There are significant methodological and philosophical differences between ethnography and laboratory-based processes in the product development cycle. These differences set users of these data collection methods at odds with one another. Frequently, these debates occur less within the user research community and more among the people using or responding to the findings and solutions presented. Whenever these arguments come up, the naysayers endlessly debate methodological purity, ownership and expertise. One side fears a lack of scientific rigor, and the other worries about a contextually detached environment yielding irrelevant results. Both sides make valid points, but the debate draws attention away from the fundamental question of product design: Does the product work in the broadest sense of the term? Can the people for whom the product is designed use it in the correct contexts? To defuse the debate and get back to this primary question requires an approach that blends the rigor of laboratory-based processes with the contextual richness of ethnography.
In the iterative product design process, what typically shapes the design are findings from in-lab usability testing. However, while the data are reliable in a controlled situation, they may not be valid in a real-world context. It is possible to obtain perfect reliability with no validity when testing. But perfect validity would assure perfect reliability because every test observation would yield the complete truth. Unfortunately, perfection does not exist in the real world, so the reliable data recorded during laboratory testing must be supported with valid data that is best found through field research..
Consider RCA’s release of the eBook in 2000. The product tested very well, but no one asked where, when and how people read. Consequently, the UI did not match user real-world needs. Had it been tested in context, the company might have avoided millions of dollars in losses. Fast forward eleven years and you find product, such as the Nook, that make sense and can be used in the “right” conditions. They aren’t going to replace the book or the book store, they will compliment them.
To ensure validity, an anthropologist or ethnographer can spend time with potential users to understand how environment and culture shape what they do. When these observations inform the design process, the result is product innovation and improved design.
At this point, however, the field expert is frequently removed, and the product moves forward with little cross-functional interaction. The UI designers and usability researchers take responsibility of ensuring that the product meets predetermined standards of usability. While scientific rigor is a noble goal, the history of science includes countless examples of hypothesis testing and discovery that would fail to satisfy modern rules of scientific method, including James Lind’s discovery of the cure for scurvy and Henri Becquerel’s discovery of radioactivity. Arguably, both scientists conducted bad science from the standpoint of sample size and environmental control, but that doesn’t negate the value to the millions of people that have benefited from these discoveries. Similarly, by allowing more testing in the field, we can learn insights about a product’s usability that might go undiscovered in a strictly controlled environment.
If we fail to account for the context in which the product will be used, we may overlook the real problem. A product may conform to every aspect of anthropometrics, ergonomics, and established principles of interface design. It may meet every requirement and have every feature potential users asked for. It may have also improved participants’ response time by a second or two in a lab study. But what if someone using the product is chest deep in mud while bullets fly overhead? Suddenly, something that was well designed and tested becomes useless because no one accounted for shaking hands, awkward positions, and decrease in computational skills under physical and psychological stress. Admittedly, some conditions can be simulated in a lab. However, it would not be cost effective or ethical to create the heat, dirt, fear and general discomfort described in the example above. Furthermore, users in their natural environment have a reduced need to provide answers that would placate the researcher. Context, and how it impacts performance is of supreme importance, and knowing the right question to ask and the right action to measure become central to accurately assessing usability.