by Steve Krug
5 A copy of the script is available on my Web site (www.sensible.com) so you can download it anyway. And sometimes I have to research
and edit it for your own use.
OK, I guess.
something at work.
6 If you didn’t work on the part that’s being tested, you can also say, “Don’t worry about hurting my feelings. I didn’t create the pages you’re going to look at.”
7 You’ll find a sample recording consent form on my Web site.
own testing based on the small amount of
[ 145 ]
[ 146 ]
[ 147 ]
[ 148 ]
[ 149 ]
[ 150 ]
c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay Review the results right away
(a) the categories you’ve used to organize your content aren’t the ones they There’s a lot going on here. But I have no
I guess I’d click on that 3D graphics thing.
TESTING A TASK
Well, now I’m not sure what to do. I can’t
As it turns out, she’s
would use, or (b) the categories are what they expect, but you’re just not using idea what any of it is.
I’m interested in 3D graphics.
click on Home Improvement, so it looks like
mistaken. Fixed-price (in
After each round of tests, you should make time as soon as possible for the the names they expect.
OK, now we’re going to try something else.
Now I give her a task to
this case) means services
I have to click on either “RFPs” or “Fixed-
development team to review everyone’s observations and decide what to do next.
perform so we can see
available for a fixed hourly
> There’s too much going on. Sometimes what they’re looking for is right Before you click on it, I have one more
I ask this question because
Can you think of something you might want to
whether she can use the
Price.” But I don’t know what the
rate, while an RFP (or
I strongly recommend that you do three or four tests in a morning and then Request for Proposal) is
there on the page, but they’re just not seeing it. In this case, you need to either If you had to take a guess, what do you think it
This user is doing a good
the site’s designers think
question. What about these pictures near the
post as a project if you were using this site?
site for its intended
difference is.
debrief over lunch.
actually the choice that
(a) reduce the overall noise on the page, or (b) turn up the volume on the things job of thinking out loud on
most users are going to
purpose.
might be?
top of the page—the ones with the numbers?
will elicit quotes. This is the
information that was there. This book, on
her own. If she wasn’t, this
start by clicking on the
Hmm. Let me think. I think I saw “Home
Fixed price I sort of understand; they’ll
You’re doing two things at this meeting:
they need to see so they “pop” out of the visual hierarchy more.
What did you make of them?
kind of misunderstanding
is where I’d start asking
pictures of the five steps,
Improvement” there somewhere. We’re
give me a quote, and then they have to
that often surprises the
Well, it seems to have something to do with
her, “What are you
and that everyone will at
Whenever possible, it’s
> Triage—reviewing the problems people saw and deciding which ones need to thinking of building a deck. Maybe I would
stick to it. But I’m not sure what RFPs is.
people who built the site.
thinking?”
least look at them.
good to let the user have
buying and selling...something.
be fixed.
Some triage guidelines
post that.
some say in choosing the
task.
> Problem solving—figuring out how to fix them.
Here’s the best advice I can give you about deciding what to fix—and what not to.
[Looks around the page again.] Now that I
So if you were going to post the deck as a
Well, which one do you think you’d click on?
From here on, I just watch
It might seem that this would be a difficult process. After all, these are the same look at the list down here [the Yahoo-style
project, what would you do first?
> Ignore “kayak” problems. In any test, you’re likely to see several cases while she tries to post a
team members who’ve been arguing about the right way to do things all along. So category list halfway down the page], I guess
Fixed price, I guess.
project, letting her
where users will go astray momentarily but manage to get back on track I guess I’d click on one of the categories
what’s going to make this session any different?
maybe it must be services. Legal, financial,
continue until either (a)
almost immediately without any help. It’s kind of like rolling over in a kayak; down here. I think I saw home
Why don’t you go ahead and do it?
she finishes the task,(b)
creative...they all sound like services.
I noticed them, but I really didn’t try to figure
Just this:
as long as the kayak rights itself quickly enough, it’s all part of the so-called improvement.[Looks.] There it is, under
she gets really frustrated,
them out. I guess I thought they were telling
or (c) we’re not learning
fun. In basketball terms, no harm, no foul.
“Family and Household.”
The important things that you learn from usability
me what the steps in the process would be.
anything new by watching
her try to muddle through.
testing usually just make sense. They tend to be
As long as (a) everyone who has the problem notices that they’re no longer So what would you do?
obvious to anyone who watches the sessions.
headed in the right direction quickly, and (b) they manage to recover without Any reason why you didn’t pay much
I’d give her three or four
Well, I’d click....[Hesitates, looking at the
help, and (c) it doesn’t seem to faze them, you can ignore the problem. In attention to them?
more tasks to do, which
two links under “Family and Household.”]
should take not more than
Also, the experience of seeing your handiwork through someone else’s eyes will general, if the user’s second guess about where to find things is always right, No. I guess I just wasn’t ready to start
45 minutes altogether.
often suggest entirely new solutions for problems, or let you see an old idea in a that’s good enough.
the process yet. I didn’t know if I wanted
new light.
Of course, if there’s an easy and obvious fix that won’t break anything else, to use it yet. I just wanted to look
And remember, this is a cyclic process, so the team doesn’t have to agree on the
then by all means fix it. But kayak problems usually don’t come as a surprise to around first.
perfect solution. You just need to figure out what to try next.
the development team. They’re usually there because of some ambiguity for OK. Great.
which there is no simple resolution. For example, there are usually at least one So I guess that’s what it is. Buying and
Typical problems
or two oddball items that don’t fit perfectly into any of the top-level categories selling services. Maybe like some kind of
of a site. So half the users may look for movie listings in Lifestyles first, and the online Yellow Pages.
Here are the types of problems you’re going to see most often when you test: other half will look for them in Arts first. Whatever you do, half of them are going to be wrong on their first guess, but everyone will get it on their second
> Users are unclear on the concept. They just don’t get it. They look at the site OK. Now, if you were at home, what would you
guess, which is fine.8
or a page and they either don’t know what to make of it, or they think they do click on first?
but they’re wrong.
8 You may be thinking “Well, why not just put it in both categories?” In general, I think it’s best the other hand, is intended to be a complete
> The words they’re looking for aren’t there. This usually means that either for things to “live” in only one place in a hierarchy, with a prominent “see also” crosslink in any other places where people are likely to look for them.
[ 152 ]
[ 153 ]
[ 154 ]
[ 155 ]
[ 156 ]
[ 157 ]
Chapter 9 from Don’t Make Me Think
teach-yourself-how-to-do-it guide.
And besides, all of the headings in the first book were red.
8 At one point, I was a little concerned about the possibility of unwittingly quoting large passages of the fi rst book without attribution and then facing the unpleasant prospect of having to sue myself for plagiarism. I think I’ve managed to avoid it. If not, I hope I can at least convince myself to settle out of court.
[ 10 ]
call me ishmael
What if I don’t intend to do any testing? Should I still read this book?
Yes. Even if you’re sure you’re never going to do the kind of testing I’m recommending here, I think you’ll find reading about the process—
particularly the chapters about fixing problems—worthwhile.
I also highly recommend that even if you’re not going to be doing full-scale testing, you force yourself to spend half an hour doing a very simple usability test of something that you’re working on. If you give it a try, you may find that quick, informal usability testing is a great tool to have at your disposal.
Aren’t you oversimplifying this?
Yes. That’s the whole point. Doing this kind of testing is enormously valuable if you do it, and people don’t do it because they have the impression that it’s more complicated than it needs to be. So I’m trying very hard to keep it as simple as possible.
Does this work only for Web sites?
The focus in this book is on testing Web sites, because that’s what most people are working on nowadays, and to keep the book short and uncomplicated. But the same method and principles can be used to test and improve almost anything that people use. Web applications and desktop software are obvious candidates, but I think it applies equally well to ballots, cell phones, PowerPoint presentations, instructions for digital cameras, and the forms you fill out in your doctor’s office. I’d like to think that you could substitute “your product” wherever I refer to “your Web site.”
How can you have “Frequently Asked Questions” in a brand new book?
Good question. They’re the questions that always come up at my
workshops. I figure it’s safe to assume that readers will have the same questions.
[ 11 ]
chapter
chapter1
1
You don’t see
any elephants
around here,
do you?
what do-it-yourself usability testing is, why it
always works, and why so little of it gets done
[ 12 ]
Why are you waving that chicken around over your head?
To keep the elephants away.
Does it work?
You don’t see any elephants around here, do you?
—VERY OLD JOKE
OK, before we get to “do-it-yourself usability testing,” first, what is
“usability testing”?
It’s pretty simple:
Watching people try to use what you’re creating/designing/building (or something you’ve already created/designed/built), with the intention of (a) making it easier for people to use or (b) proving that it is easy to use.
There are a lot of different “flavors” of usability testing, but the one thing they all have in common is that they involve watching people actually use the thing.
This element of actual use is what makes usability testing very different from things like surveys, interviews, and focus groups, where you’re asking people for their opinions about things, or their past experiences using things.
One useful way to categorize all the different flavors is by thinking of them as either quantitative or qualitative.
In a quantitative test, you’re interested in proving something (“Is this latest version better than the previous one?” “Is our site as easy to use as our competitors’ sites?”) and you do this by measuring things like success rate (how many people finish the tasks you give them to do) and time-on-task (how long it takes them).
Since the purpose is to prove something, quantitative tests are like scientific experiments: They have to be rigorous, or the results won’t be trustworthy.
This means you have to define a test protocol and follow it consistently for 1
all of the participants. You have to collect data carefully. You have to have a 1 In usability testing, we call the people we’re observing “test participants,” not “test subjects,” to remind ourselves that we really aren’t testing them—we’re testing the thing they’re using.
[ 13 ]
chapter 1
large enough sample of participants to make your conclusions statistically significant, and they need to be representative of your actual users so you can extrapolate the results to a larger population. All of this means you have to know what you’re doing, and you have to be careful doing it.
In quantitative testing, you usually try to minimize the amount of interaction with the participant to avoid influencing the results. In an extreme form (sometimes called “Voice of God” testing), the participant sits in a room by himself with a facilitator giving him instructions over an intercom, while an observer watches through a one-way mirror and records the data.
So, what’s “Do-It-Yourself
Usability Testing” ?
As you might have guessed by now, the kind of testing I’m recommending you do is at the opposite end of the qualitative–quantitative spectrum.
“Do-it-yourself” usability tests are definitely qualitative. The purpose isn’t to prove anything; it’s to get insights that enable you to im prove what you’re building.
As a result, do-it-yourself tests can be much more informal and, well, unscientific. This means you can test fewer users (as long as you get the insights you need), and you can even change the protocol mid-test. For instance, if the first participant can’t complete a particular task and the reason why is obvious, you can alter the task—or even skip it—for the remaining participants. You can’t do that in a quantitative test because it would invalidate the results.
Basically, a facilitator sits in a room with the participant, gives him some tasks to do, and asks him to think out loud while he does them.
There’s no d
ata gathering involved. Instead, members of the development team, stakeholders, and any other interested parties observe the session from another room, using screen sharing software. After the tests are finished, the observers have a debriefing session where they compare notes and decide what problems should be fixed and how to fix them.
That’s really about all there is to it.
[ 14 ]
you don’t see any elephants around here, do you?
The funny thing is, it just works
When I teach my usability testing workshops, I always begin by doing a live demo test—“live” in the sense that it’s completely unrehearsed. The only preparation I do is to choose a site that belongs to one of the attendees and use it just long enough to come up with a task that I think people are likely to want to do on that site. (For example, if it’s a health care site, I might make up a task about booking an appointment.)
Then I ask for a volunteer to be the test participant
and spend 15 minutes doing an abbreviated version
of a test. (Real tests typically last about an hour,
although they can be as short as five minutes and
as long as an entire day.)
The result is almost always exactly the same:
The participant has a good time and gets a round of applause at the end for being brave enough to volunteer.
The site’s “owner” spends the entire 15 minutes furiously scribbling notes about things to fix and asks if she can have the recording to show to her 2
team and her boss.
Everyone else ends up thinking, “Gee. Is that all there is to it? I could do that.”
When it’s over, I ask, “Does that seem like a worthwhile way to spend 15
minutes?” and everyone nods their head in agreement.
The point of doing the live demo is to show people that (a) there’s nothing to it, and (b) it always works. I can tell that some of them suspect that I’m able to make it look easy because I’ve done it a lot. But by the end of the day, after everyone has tried conducting a test themselves, they all seem to understand that there’s no magic involved and that it really is as easy as it looks.
2 One “owner” wrote me a few months later to tell me that after viewing the demo test of his site, his team had immediately made one simple change that they calculated—based on the data from the fi rst few months—was going to save their company $100,000 a year. (It had to do with getting customers to sign up for online billing.)
