by Steve Krug
The one right way to do things. There are many ways to do most of this.
Where there are options, I’ve usually chosen the one I think works best for most people, or the one that’s easiest for a beginner to do. But that doesn’t mean I think it’s the only way that works.
[ 8 ]
call me ishmael
The obligatory companion Web site
Yes, there is a companion Web site (www.rocketsurgerymadeeasy.com), with files you can download, like the demo test video and all the scripts, forms, and handouts in the book.
These files are available to everyone, because I really do want as many people as possible to do their own testing. They may be updated at some point, although, knowing myself as well as I do, I have to admit that’s pretty unlikely.
Maxims? Really? You’re sure you
want to call them “maxims”?
One thing you will find in this book is a series of what I’m calling—for lack of a better word—maxims. They’re easy to spot, because they look like this: Recruit loosely and
grade on a curve.
What are they? I suppose they’re what some people would call critical success factors. In teaching people to do their own testing, I’ve found that there are really only a few things you need to keep in mind to succeed. But for some reason, people seem to have a hard time remembering all of them. So over time, I’ve reduced them to hopefully-more-memorable maxims.
If you forget everything else in this book, try to remember these; they’re my most important pieces of advice. You’ll find a list of all of them—suitable for framing and hanging on a cubicle wall—in Chapter 16.
A few words of encouragement
Four words, to be exact: You can do this.
For years, my corporate motto has been “It’s not rocket surgery™” because I believe that at its heart most usability work is really not very hard to do. I have yet to come across someone who can’t do a pretty good usability test—
certainly good enough that doing it is much better than not doing it.
[ 9 ]
opening remarks
Since you’re reading this, it’s very likely that you’re the de facto user advocate in your organization or department: the person most interested in making sure that your “product” (whether it’s a Web site, a Web or desktop application, or whatever) is user-friendly.
You may not have much (or any) support for this interest. Or you may have moral support, but no resources. As a result, you’re probably going to be pursuing it in what we laughingly refer to as your copious spare time.
But take heart, and be of good cheer: it’s easy, pretty much foolproof, and you can start doing it next week. And one more thing people always forget to mention: it’s fun. All the people I know who have been doing usability tests for years still get a kick out of it and find them fascinating.
So get started as soon as you can, keep it as simple as you can, and have fun with it.
FAQ
Isn’t this just a rehash of your other book?
Who
let
you in?
No, it really isn’t. The first book was about how to think about usability; this one is about how to do usability.
Why didn’t we do this sooner?
c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
—w h at ev e ryo n e says at s o m e p o i n t du r i n g t h e In some ways, this book is an expanded
f i r st usa b i l i t y t e st o f t h e i r w e b s i t e And while usability testing will sometimes settle these arguments, the main As the launch date approaches, the Marketing people may feel that their only hope Testing reminds you that not everyone thinks the way you do, knows what you
> The importance of recruiting representative users is overrated. It’s good THE TOP FIVE PLAUSIBLE EXCUSES FOR NOT TESTING WEB SITES
thing it usually ends up doing is revealing that the things they were arguing of sanity prevailing is to appeal to a higher authority: research. And the kind of know, uses the Web the way you do.
to do your testing with people who are like the people who will use your site, about aren’t all that important. People often test to decide which color drapes are research they know is focus groups.
but it’s much more important to test early and often. My motto—as you’ll see—
It’s true that most Web development schedules seem to be based
I used to say that the best way to think about testing was that it was like travel: We don’t have
on the punchline from a Dilbert cartoon. If testing is going to add A
best, only to learn that they forgot to put windows in the room. For instance, they is “Recruit loosely, and grade on a curve.”
I often have to work very hard to make clients understand that what they need is a broadening experience. It reminds you how different—and the same—people the time.
to everybody’s to-do list, if you have to adjust development
might discover that it doesn’t make much difference whether you go with the usability testing, not focus groups. Here’s the difference in a nutshell: are, and gives you a fresh perspective on things.
> The point of testing is not to prove or disprove something. It’s to schedules around tests and involve key people in preparing for
horizontal navigation bar or the vertical menus if nobody understands the value them, then it won’t get done. That’s why you have to make testing bout once a month, I get one of these phone calls:
inform your judgment. People like to think, for instance, that they can use proposition of your site.
> In a focus group, a small group of people (usually 5 to 8) sit around a table and But I finally realized that testing is really more like having friends visiting from as small a deal as possible. Done right, it will save time, because you testing to prove whether navigation system “a” is better than navigation system won’t have to (a) argue endlessly, and (b) redo things at the end.
react to ideas and designs that are shown to them. It’s a group process, and much out of town. Inevitably, as you make the tourist rounds with them, you see Sadly, this is how most usability testing gets done: too little, too late, and for all
“b”, but you can’t. No one has the resources to set up the kind of controlled Ed Grimley at XYZ Corp
of its value comes from participants reacting to each other’s opinions. Focus things about your home town that you usually don’t notice because you’re so Forget $5,000 to 15,000. If you can convince someone to bring in
the wrong reasons.
experiment you’d need. What testing can do is provide you with invaluable input gave me your name.
…two weeks?
groups are good for quickly getting a sampling of users’ opinions and feelings used to them. And at the same time, you realize that a lot of things that you We don’t have
a camcorder from home, you’ll only need to spend about $300 for
which, taken together with your experience, professional judgment, and the money.
each round of tests.
We’re launching our site
about things.
take for granted aren’t obvious to everybody.
common sense, will make it easier for you to choose wisely—and with greater in two weeks and we want to do
Repeat after me:
some usability testing.
> In a usability test, one user at a time is shown something (whether it’s a Web
> Testing one user is 100 percent better than testing none. Testing always confidence—between “a” and “b.”
Focus groups are not usability tests.
site, a prototype of a site, or some sketches of individual pages) and asked to works, and even the worst test with the wrong user will show you important
> Testing is an iterative process. Testing
isn’t something you do once.
The least-known fact about usability testing is that it’s incredibly either (a) figure out what it is, or (b) try to use it to do a typical task.
things you can do to improve your site. I make a point of always doing a live We don’t have
easy to do. Yes, some people will be better at it than others, but Sometimes that initial phone call is even scarier:
You make something, test it, fix it, and test it again.
user test at my workshops so that people can see that it’s very easy to do and it the expertise.
I’ve never seen a usability test fail to produce useful results, no Focus groups can be great for determining what your audience wants, needs, and matter how poorly it was conducted.
always produces an abundance of valuable insights. I ask for a volunteer and
> Nothing beats a live audience reaction. One reason why the Marx likes—in the abstract. They’re good for testing whether the idea behind the site
…we’re launching our site in
have him try to perform a task on a site belonging to one of the other attendees.
Brothers’ movies are so wonderful is that before they started filming two weeks and we want to do
Focus group
makes sense and your value proposition is attractive. And they can be a good way These tests last less than ten minutes, but the person whose site is being tested they would go on tour on the vaudeville circuit and perform scenes some focus group testing.
testing?
to test the names you’re using for features of your site, and to find out how people usually scribbbles several pages of notes. And they always ask if they can have from the movie, doing five shows a day, improvising constantly and feel about your competitors.
You don’t need one. All you really need is a room with a desk, a
We don’t have a
the recording of the test to show to their team back home. (One person told me noting which lines got the best laughs. Even
computer, and two chairs where you won’t be interrupted.
usability lab.
As soon as I hear “launching in two weeks” (or even “two months”) and “usability But they’re not good for learning about whether your site works and how to improve it.
that after his team saw the recording, they made one change to their site which after they’d settled on a line, Groucho
Mrs. Teasdale (Margaret
Dumont) and Rufus T. Firefly
testing” in the same sentence, I start to get that old fireman-headed-into-the-they later calculated had resulted in $100,000 in savings.)
would insist on trying slight variations to
The kinds of things you can learn from focus groups are the things you need to eavesdrop in Duck Soup.
burning-chemical-factory feeling, because I have a pretty good idea of what’s see if it could be improved.
learn early on, before you begin designing the site. Focus groups are for EARLY in
> Testing one user early in the project is better than testing 50 near the going on.
One of the nicest things about usability testing is that the
We wouldn’t know
the process. You can even run them late in the process if you want to do a reality end. Most people assume that testing needs to be a big deal. But if you make it important lessons tend to be obvious to everyone who’s watching.
how to interpret
If it’s two weeks, then it’s almost certainly a request for a disaster check. The check and fine-tune your message, but don’t mistake them for usability testing.
into a big deal, you won’t do it early enough or often enough to get the most out Lost our lease, going-out-of-business-The serious problems are hard to miss.
the results.
version of the chapter in Don’t Make Me Think launch is fast approaching and everyone’s getting nervous, and someone finally They won’t tell you whether people can actually use your site.
of it. A simple test early—while you still have time to use what you learn from sale usability testing
says, “Maybe we better do some usability testing.”
it—is almost always more valuable than a sophisticated test later.
When the last-minute request is for a focus group, it’s usually a sign that the Usability testing has been around for a long time, and the basic idea is pretty If it’s two months, then odds are that what they want is to settle some ongoing Several true things about testing
Part of the conventional wisdom about Web development is that it’s very easy that were statistically significant. It was Science. It cost $20,000 to $50,000 a shot.
request originated in Marketing. When Web sites are being designed, the folks in simple: If you want to know whether your software or your Web site or your internal debates—usually about something very specific like color schemes.
to go in and make changes. The truth is, it turns out that it’s not that easy to It didn’t happen very often.
Marketing often feel like they don’t have much clout. Even though they’re the Here are the main things I know about testing:
VCR remote control is easy enough to use, watch some people while they try to Opinion around the office is split between two different designs; some people make changes to a site once it’s in use. Some percentage of users will resist ones who spend the most time trying to figure out who the site’s audience is and use it and note where they run into trouble. Then fix it, and test it again.
But in 1989 Jakob Nielsen wrote a paper titled “Usability Engineering at a like the sexy one, some like the elegant one. Finally someone with enough clout
> If you want a great site, you’ve got to test. After you’ve worked on a site for almost any kind of change, and even apparently simple changes often turn out what they want, the designers and developers are the ones with most of the Discount”1 and pointed out that it didn’t have to be that way. You didn’t need a to authorize the expense gets tired of the arguing and says, “All right, let’s get even a few weeks, you can’t see it freshly anymore. You know too much. The to have far-reaching effects, so anything you can keep from building wrong in In the beginning, though, usability testing was a very expensive proposition. You hands-on control over how the site actually gets put together.
some testing done to settle this.”
only way to find out if it really works is to test it.
the first place is gravy.
had to have a usability lab with an observation room behind a one-way mirror, 1 Proceedings of the Third International Conference on Human-Computer Interaction, Boston, and at least two video cameras so you could record the users’ reactions and the MA, Sept. 1989.
thing they were using. You had to recruit a lot of people so you could get results
[ 131 ]
[ 132 ]
[ 133 ]
[ 134 ]
[ 135 ]
[ 136 ]
that explained how to do a usability test.8
c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay How many users should you test?
ONE TEST WITH 8 USERS
TOTAL PROBLEMS
If you can afford to hire someone to recruit the participants for you and it won’t
> If using your site requires specific domain knowledge (e.g., a currency Where do you test?
I don’t recommend using the camcorder to videotape the sessions. In fact, I used to reduce the number of rounds of testing that you do, then by all means be as exchange site for money management professionals), then you need to recruit recommend not doing any video recording at all, because the tapes were almost In most cases, I tend to think the ideal number of users for each round of testing is FOUND: 5
8 users
&n
bsp; All you really need is an office or conference room with two chairs, a PC or Mac specific as you want. But if finding the ideal user means you’re going to do fewer people with that domain knowledge for at least one round of tests. But don’t do never used and it made the whole process more complicated and expensive.
three, or at most four.
Eight users may
(with an Internet connection, if you’re testing a live site), a camcorder, a long find more problems
tests, I recommend a different approach:
it for every round if it will reduce the number of tests you do.
video cable, and a tripod.
In the past few years though, three things have changed: PCs have gotten much The first three users are very likely to encounter nearly all of the most significant in a single test.
Take anyone you can get (within limits) and grade on a curve.
When you’re recruiting:
faster, disk drives have gotten much larger, and screen recording software has problems,2 and it’s much more important to do more rounds of testing than to But the worst prob-LOST-OUR-LEASE USABILITY “LAB”
improved dramatically. Screen recorders like Camtasia4 run in the background on wring everything you can out of each round. Testing only three users helps lems will usually
In other words, try to find users who reflect your audience, but don’t get hung up
> Offer a reasonable incentive. Typical stipends for a one-hour test session keep them from
the test PC and record everything that happens on the screen and everything the ensure that you will do another round soon.3
about it. Instead, try to make allowances for the differences between the people range from $50 for “average” Web users to several hundred dollars for getting far enough
Well, I’ll be
user and the facilitator say in a video file you can play on the PC. It turns out that to encounter
