In some cases, you may also want to loop back and interview this particularly knowledgeable and articulate subject again at the end of the interview cycle to address any topics that you weren’t aware of during your initial interview.
08_084113 ch04.qxp 4/3/07 6:02 PM Page 65
Chapter 4: Understanding Users: Qualitative Research
65
Basic methods
The basic methods of ethnographic interviewing are simple, straightforward, and very low tech. Although the nuances of interviewing subjects takes some time to master, any practitioner should, if they follow the suggestions below, be rewarded with a wealth of useful qualitative data:
Interview where the interaction happens
Avoid a fixed set of questions
Focus on goals first, tasks second
Avoid making the user a designer
Avoid discussions of technology
Encourage storytelling
Ask for a show and tell
Avoid leading questions
We describe each of these methods in more detail in the following sections.
Interview where the interaction happens
Following the first principle of contextual inquiry, it is of critical importance that subjects be interviewed in the places where they actually use the products. Not only does this give the interviewers the opportunity to witness the product being used, but it also gives the interview team access to the environment in which the interaction occurs. This can give tremendous insight into product constraints and user needs and goals.
Observe the environment closely: It is likely to be crawling with clues about tasks the interviewee might not have mentioned. Notice, for example, the kind of information they need (papers on desks or adhesive notes on screen borders), inadequate systems (cheat sheets and user manuals), the frequency and priority of tasks (inbox and outbox); and the kind of workflows they follow (memos, charts, calendars). Don’t snoop without permission, but if you see something that looks interesting, ask your interviewee to discuss it.
Avoid a fixed set of questions
If you approach ethnographic interviews with a fixed questionnaire, you not only run the risk of alienating the interview subject but can also cause the interviewers to miss out on a wealth of valuable user data. The entire premise of ethnographic interviews (and contextual inquiry) is that we as interviewers don’t know enough about the domain to presuppose the questions that need asking: We must learn what is important from the people we talk to. This said, it’s certainly useful to have
08_084113 ch04.qxp 4/3/07 6:02 PM Page 66
66
Part I: Understanding Goal-Directed Design
types of questions in mind. Depending on the domain, it may also be useful to have a standardized set of topics that you want to make sure you cover in the course of your interview. This list of topics may evolve over the course of your interviews, but this will help you make sure that you get enough detail from each interview so that you are able to recognize the significant behavior patterns.
Here are some goal-oriented questions to consider:
Goals — What makes a good day? A bad day?
Opportunity — What activities currently waste your time?
Priorities — What is most important to you?
Information — What helps you make decisions?
Another useful type of question is the system-oriented question:
Function — What are the most common things you do with the product?
Frequency — What parts of the product do you use most?
Preference — What are your favorite aspects of the product? What drives you crazy?
Failure — How do you work around problems?
Expertise — What shortcuts do you employ?
For business products, workflow-oriented questions can be helpful:
Process — What did you do when you first came in today? And after that?
Occurrence and recurrence — How often do you do this? What things do you do weekly or monthly, but not every day?
Exception — What constitutes a typical day? What would be an unusual event?
To better understand user motivations, you can employ attitude-oriented questions:
Aspiration — What do you see yourself doing five years from now?
Avoidance — What would you prefer not to do? What do you procrastinate on?
Motivation — What do you enjoy most about your job (or lifestyle)? What do you always tackle first?
Focus on goals first, tasks second
Unlike contextual inquiry and the majority of other qualitative research methods, the first priority of ethnographic interviewing is understanding the why of users — what
08_084113 ch04.qxp 4/3/07 6:02 PM Page 67
Chapter 4: Understanding Users: Qualitative Research
67
motivates the behaviors of individuals in different roles, and how they hope to ultimately accomplish this goal — not the what of the tasks they perform. Understanding the tasks is important, and the tasks must be diligently recorded. But these tasks will ultimately be restructured to better match user goals in the final design.
Avoid making the user a designer
Guide the interviewee towards examining problems and away from expressing solutions. Most of the time, those solutions reflect the interview subject’s personal priorities, and while they sound good to him, they tend to be shortsighted, idiosyncratic, and lack the balance and refinement that an interaction designer can bring to a solution based upon adequate research and years of experience. That said, a proposed design solution can be a useful jumping off point to discuss a user’s goals and the problems they encounter with current systems. If a user blurts out an interesting idea, ask “What problem would that solve for you?” or “Why would that be a good solution?”
Avoid discussions of technology
Just as you don’t want to treat the user as a designer, you also don’t want to treat him as a programmer or engineer. Discussion of technology is meaningless without first understanding the purpose underlying any technical decisions. In the case of technical or scientific products, where technology is always an issue, distinguish between domain-related technology and product-related technology, and steer away from the latter. If an interview subject is particularly insistent on talking about how the product should be implemented, bring the subject back to his goals and motivations by asking “How would that help you?”
Encourage storytelling
Far more useful than asking users for design advice is encouraging them to tell specific stories about their experiences with a product (whether an old version of the one you’re redesigning, or an analogous product or process): how they use it, what they think of it, who else they interact with when using it, where they go with it, and so forth. Detailed stories of this kind are usually the best way to understand how users relate to and interact with products. Encourage stories that deal with typical cases and also more exceptional ones.
Ask for a show and tell
After you have a good idea of the flow and structure of a user’s activities and interactions and you have exhausted other questions, it is often useful to ask the interviewee for a show and tell or grand tour of artifacts related to the design problem.
These can be domain-related artifacts, software interfaces, paper systems, tours of the work environment, or ideally all the above. Be careful to not only record the
08_084113 ch04.qxp 4/3/07 6:02 PM Page 68
68
Part I: Understanding Goal-Directed Design
artifacts themselves (digital or video cameras are very handy at this stage) but also pay attention to how the interviewee describes them. Be sure to ask plenty of clarifying questions as well.
Avoid leading questions
One important thing to avoid in interviews is the use of leading questions. Just as in a courtroom, where lawyers can, by virtue of their authority, bias witnesses by suggesting answers to them,
designers can inadvertently bias interview subjects by implicitly (or explicitly) suggesting solutions or opinions about behaviors.
Examples of leading questions include:
Would feature X help you?
You like X, don’t you?
Do you think you’d use X if it were available?
After the interviews
After each interview, teams compare notes and discuss any particularly interesting trends observed or specific points brought up in the most recent interview. If they have the time, they should also look back at old notes to see whether unanswered questions from other interviews and research have been properly answered. This information should be used to strategize about the approach to take in subsequent interviews.
After the interview process is finished, it is useful to once again make a pass through all the notes, marking or highlighting trends and patterns in the data. This is very useful for the next step of creating personas from the cumulative research. If it is helpful, the team can create a binder of the notes, review any videotapes, and print out artifact images to place in the binder or on a public surface, such as a wall, where they are all visible simultaneously. This will be useful in later design phases.
Other Types of Research
This chapter has focused on qualitative research aimed at gathering user data that will later be used to construct robust user and domain models that form the key tools in the Goal-Directed Design methodology described in the next chapter. A wide variety of other forms of research are used by design and usability professionals, ranging from detailed task analysis activities to focus groups and usability tests.
While many of these activities have the potential to contribute to the creation of useful and desirable products, we have found the qualitative approach described in this chapter to provide the most value to digital product design. Put simply, the
08_084113 ch04.qxp 4/3/07 6:02 PM Page 69
Chapter 4: Understanding Users: Qualitative Research
69
qualitative approach helps answer questions about the product at both the big-picture and functional-detail level with a relatively small amount of effort and expense. No other research technique can claim this.
Mike Kuniavsky’s book Observing the User Experience is an excellent resource that describes a wide range of user research methods for use at many points in the design and development process. In the remainder of this chapter, we discuss just a few of the more prominent research methods and how they fit into the overall development effort.
Focus groups
Marketing organizations are particularly fond of gathering user data via focus groups, in which representative users, usually chosen to match previously identified demographic segments of the target market, are gathered together in a room and asked a structured set of questions and provided a structured set of choices.
Often, the meeting is recorded on audio or video media for later reference. Focus groups are a standard technique in traditional product marketing. They are useful for gauging initial reactions to the form of a product, its visual appearance, or industrial design. Focus groups can also gather reactions to a product that the respondents have been using for some time.
Although focus groups may appear to provide the requisite user contact, the method is in many ways not appropriate as a design tool. Focus groups excel at eliciting information about products that people own or are willing (or unwilling) to purchase but are weak at gathering data about what people actually do with those products, or how and why they do it. Also, because they are a group activity, focus groups tend to drive to consensus. The majority or loudest opinion often becomes the group opinion. This is anathema to the design process, where designers must understand all the different patterns of behavior a product must address. Focus groups tend to stifle exactly the diversity of behavior and opinion that designers must accommodate.
Market demographics and market segments
The marketing profession has taken much of the guesswork out of determining what motivates people to buy. One of the most powerful tools for doing so is market segmentation, which typically uses data from focus groups and market surveys to group potential customers by demographic criteria (such as age, gender, educational level, and home zip code) to determine what types of consumers will be most receptive to a particular product or marketing message. More sophisticated
08_084113 ch04.qxp 4/3/07 6:02 PM Page 70
70
Part I: Understanding Goal-Directed Design
consumer data also include psychographics and behavioral variables, including attitudes, lifestyle, values, ideology, risk aversion, and decision-making patterns.
Classification systems such as SRI’s VALS segmentation and Jonathan Robbin’s geodemographic PRIZM clusters can add greater clarity to the data by predicting consumers’ purchasing power, motivation, self-orientation, and resources.
These market-modeling techniques are able to accurately forecast marketplace acceptance of products and services. They are an invaluable tool in assessing the viability of a product. They can also be powerful tools for convincing executives to build a product. After all, if you know X people might buy a product or service for Y dollars, it is easy to evaluate the potential return on investment.
However, understanding whether somebody wants to buy something is not the same thing as actually defining the product. Market segmentation is a great tool for identifying and quantifying a market opportunity, but an ineffective tool for defining a product that will capitalize on that opportunity.
It turns out, however, that data gathered via market research and that gathered via qualitative user research complement each other quite well. Because market research can help identify an opportunity, it is often the necessary starting point for a design initiative. Without assessing the opportunity, you will be hard pressed to convince a businessperson to fund the design. Also, as already discussed, ethnographic interviewers should use market research to help them select interview targets, and finally, as the video-editing story earlier in this chapter illustrates, qualitative research can shed critical light on the results of quantitative studies. We will discuss the differences between segmentation models and user models in more detail in Chapter 5.
Usability and user testing
Usability testing (also known, somewhat unfortunately, as “user testing”) is a collection of techniques used to measure characteristics of a user’s interaction with a product, usually with the goal of assessing the usability of that product. Typically, usability testing is focused on measuring how well users can complete specific, standardized tasks, as well as what problems they encounter in doing so. Results often reveal areas where users have problems understanding and utilizing the product, as well as places where users are more likely to be successful.
Usability testing requires a fairly complete and coherent design artifact to test against. Whether you are testing production software, a clickable prototype, or even a paper prototype, the point of the test is to validate a product design. This means that the appropriate place for usability testing is quite late in the design cycle, after
08_084113 ch04.qxp 4/3/07 6:02 PM Page 71
Chapter 4: Understanding Users: Qualitative Research
71
there is a coherent concept and sufficient detail to generate such prototypes. We discuss evaluative usability testing as part of design refinement in Chapter 7.
A case could certainly be made for the appropriateness of usability testing at the beginning of a redesign effort, and the technique is certainly capable of finding opportunities for improvement in such a project. However, we find that we are able to assess major inadequacies of a product through our qualitative studies, and if the budget is limited so as to allow usability testing only once in a product design initiative, we find much more value in performing the tests after we have a candidate solution, as a means of testing the specific elements of the new design.
Because the findings of us
er testing are generally measurable and quantitative, usability research is especially useful in comparing specific design variants to choose the most effective solution. Customer feedback gathered from usability testing is most useful when you need to validate or refine particular interaction mechanisms or the form and expression of specific design elements.
Usability testing is especially effective at determining:
Naming — Do section/button labels make sense? Do certain words resonate better than others do?
Organization — Is information grouped into meaningful categories? Are items located in the places customers might look for them?
First-time use and discoverability — Are common items easy for new users to find? Are instructions clear? Are instructions necessary?
Effectiveness — Can customers efficiently complete specific tasks? Are they making missteps? Where? How often?
As suggested previously, it is also worth noting that usability testing is predominantly focused on assessing the first-time use of a product. It is often quite difficult (and always laborious) to measure how effective a solution is on its 50th use — in other words, for the most common target: the perpetual intermediate user. This is quite a conundrum when one is optimizing a design for intermediate or expert users. One technique for accomplishing this is the use of a diary study, in which subjects keep diaries detailing their interactions with the product. Again, Mike Kuniavsky provides a good explanation of this technique in Observing the User Experience.
Alan Cooper, Robert Reinmann, David Cronin - About Face 3- The Essentials of Interaction Design (pdf) Page 13
