by Steve Krug
appointment for your 11-year-old son.
You are getting very drowsy. Find
Then you expand these tasks into
It needs to be after school, and he gets
yourself a cup of coffee
out at 2 pm.
scenarios—the little scripts that
Book the appointment online.
add any details of context they’ll
need to know to do the tasks.
First, come up with a list of tasks
The first step is to jot down a list of the most important tasks that people need to be able to do on your site.
Try it right now.
1. Get a sheet of paper.
2. Make a list of five to ten of the most important things people need to be able to do when using your site.
For example, here’s my list for my site:
Get info about my workshops.
Sign up for my workshops.
Read a sample chapter of my book.
Buy my book.
Find out about my consulting services.
[ 51 ]
chapter 6
Go ahead, do it now. I’ll wait.
Still waiting.
There. That wasn’t very hard, was it? (You did actually do it, didn’t you?
You aren’t just saying you did, right? Because if you are, you really should try it. It’ll only take a minute or two. I’ll wait.)
It’s almost always easy for people to come up with a list of the key tasks for their site. In fact, when I ask an entire Web team to make this list, I’m always surprised at how much overlap there is across their lists. This is one thing they tend to agree on.
The trick is to make sure the tasks you test reflect your users’ actual goals, not just your idea of what they want to do.
Decide which ones to test
Once you have your list, you need to decide which ones you want to do in this month’s round of testing.
In a normal “50-minute hour” session, you have about 35 minutes for the participant to spend doing tasks. You might just have one long task or as many as ten.
People will work at different speeds, so you always need to have extra tasks for people who finish early. One good “filler” task is to have them do one of the tasks on a competitor’s site.
Your choice of which tasks to test is based on a number of factors: What are your most critical tasks? These are the things that people must be able to do. If they can’t do them, your site will be a failure. For instance, if you’re selling books online, people need to be able to find books they’re interested in and they have to be able to pay for them.
[ 52 ]
find some things for them to do
What’s keeping you awake at night? Things that you suspect people are going to have trouble with. That may confuse people. That aren’t as clear as they need to be.
What does your other user research suggest may not be easy to use?
Have you asked customer support what kinds of problems they hear about frequently? What red flags do your Web analytics raise about possible problems people have using your site?
Make the tasks into scenarios
Once you’ve decided which tasks people are going to do, you have a writing job ahead of you: converting the simple description of the task into a script that the user can read, understand, and follow.
The scenario is like a card you might be handed for an improvisation exercise in an acting class: it gives you your character, your motivation, what you need to do, and a few details.
Task:
“Apply for a doctoral program at Harvard Business School”
Scenario: “You’ve got an MBA, and after a lot of research you’ve decided to enter the doctoral program at Harvard Business School in Science, Technology & Management.
Apply for admission to the program.”
A scenario provides some context (“You are...,” “You need to...”) and supplies information the user needs to know, but doesn’t (e.g., username and password for a test account). Don’t go overboard: trim any detail that doesn’t contribute.
There’s really only one thing that’s hard about this: Not giving clues in the scenario.
[ 53 ]
chapter 6
You have to phrase it so that it’s clear, unambiguous, and easy to understand, and you have to do it without using uncommon or unique words that appear on the screen. If you do, you turn the task into a simple game of word-finding.
For instance:
Bad: “Customize your LAUNCHcast station.”
Better: “Choose the kind of music you want to listen to.”
Don’t fence me in
There are two restrictions you may want to place on how people do the tasks:
Don’t use search. (Unless you’re testing search, of course.) You will usually want to instruct the participant not to use the search feature when doing the tasks. If they use search, all you’re really testing is whether the site’s search is returning good results. If they forget and try to use search, you can remind them that you don’t want them to use it.
Stay on this site. In most cases, you’ll want people to spend all of the limited test time on the site that you’re testing. In most cases, they’ll do this naturally, so I wouldn’t bother stating it up front. Instead, I’d just mention it when they do stray, saying something like, “For the purposes of this test, I’d like you to stay on this site for now.”
Pilot test the scenarios
Once you’ve got your scenarios written, you want to pre-test them in what’s called a pilot test. The pilot test doesn’t take as long as a full test: you can usually do it in about fifteen minutes. The purpose is to ensure that the scenarios are clear, complete, and unambiguous.
All you have to do is sit someone down in front of what you’re testing, read them the scenarios and have them try to start doing each task. Anything that wasn’t clear in the scenario will probably be obvious immediately. This is a case where you can definitely use almost anybody as a participant—in fact, this is a perfect time to use friends and family.
[ 54 ]
find some things for them to do
Usually you will do this a day or two before the testing. By that time, your scenarios are written and the developers and designers are [almost] finished producing what you’re going to test.
Print them
Once you’ve made any final changes to your scenarios, you need to print them in two formats:
One per sheet, for participants. At the start of
each task, you’re going to hand them the scenario
Plan a trip,
involving 30 stops.
so they can refer back to it while they do the task.
Each one should be on a separate piece of paper,
in fairly large type. I find that the easiest way to
Register to receive
information about
do this is to print two on a page, then cut the pages
your IRA.
in half.
Don’t number the tasks, because you may want to change the sequence or skip a task.
All on one page, for you and the observers.
[ 55 ]
chapter
ter 3
7
Some boring
checklists
and why you should use them even if, like me,
you don’t really like checklists
[ 56 ]
a morning a month, that’s all we ask
Ready when you are, C.B.! 1
—ANOTHER VERY OLD JOKE
I’m not really a fan of checklists. To me, checklists always imply a rigid process, and frankly, I’m much more of an improvisational kind of guy.
But I am a fan of things that work, and in some contexts checklists work really well.
When you’re running an event like a usability test, there are a lot of things that have to happen at specific times and a lot of small details to keep track of.
Odds are you’ll remember to do most of them, but checklists can keep things from falling through the cracks—particularly the one item everybody forgets: I can guarantee that at some point you’ll forget to turn on the screen recorder, and you won’t realize it until the session is halfway over. For some reason, this happens to everybody, so I’ve included reminders in the test script, too.
On the day of testing in particular, having a checklist gets these mundane details out of your head so you can be more relaxed and give your full attention to the participant.
You’ll find downloadable versions of these checklists on the book Web site which you can edit to fit your own circumstances. (For instance, you may have to requisition the money for incentives a month in advance or order lunch for the debriefing from your in-house catering service several days ahead of time.) 1 Hence the old joke: Legendary director of biblical epics Cecil B. DeMille was fi lming a hugely expensive scene involving chariots, collapsing fi ery towers, and thousands of extras, all shot in one take with a dozen cameras. When the action was over and DeMille had called “Cut!” he shouted through his megaphone to the camera supervisor on a nearby hill, “Did you get all that?” From the top of the hill, the supervisor waved back and shouted enthusiastically, “Ready when you are, C.B.!”
[ 57 ]
chapter 7
Three weeks before
�Figure out what you’re going to be testing (site, wireframes,
prototype, etc.)
�Create your list of tasks to test
�Decide what kind(s) of users you want to test with
�“Advertise” for participants
�Book a test room for the entire morning with Internet access, table or desk and two chairs, and speakerphone
�Find a place near the test room for participants to sit and wait when they arrive
�Book an observation room for the entire morning with Internet access, table and enough chairs for observers, speakerphone, and projector and screen (or plan to bring a projector or large monitor)
�Book the observation room or a similar-size room for the
debriefing lunch
Two weeks before
�Get feedback on your list of tasks from the project team and stakeholders
�Arrange incentives for participants (e.g., order gift certificates, requisition cash)
�Start screening participants and scheduling them into time slots
�Send “save the date” email inviting team members and stakeholders to attend
One week before
�Send email to the participants with directions, parking instructions, location of the test room, name and phone number of someone to call on the test day if they’re late or lost, and the nondisclosure agreement if you’re using one
[ 58 ]
some boring checklists
�Line up a standby participant in case of a no-show
�If this is your first round of testing, install and test the screen recording and screen sharing software
One or two days before
�Call participants to reconfirm and ask if they have any questions
�Email reminder to observers
�Finish writing the scenarios
�Do a pilot test of the scenarios
�Get any user names/passwords and sample data needed for the test (e.g., account and network log-ins, dummy credit card numbers, or test accounts)
�Make copies of handouts for participants
� Recording consent form (page 153)
� Sets of the scenarios on individual pieces of paper
� Extra copies of the nondisclosure agreement (if using one)
�Make copies of handouts for observers
� Instructions for Usability Test Observers (page 94)
� List of scenarios
� Copy of the test script (pages 147–152)
�Recruit someone to greet participants as they arrive and make them comfortable
�Recruit someone to manage the observation room for you, and give him/
her a copy of the Hall Monitor’s Guide (page 99)
�Make sure incentives for participants are ready
�Make sure you have your USB microphone, external speakers, extension cords, and thumb drive or CDs for screen recording files
�Order snacks and beverages for the observation room
[ 59 ]
chapter 7
�Verify that no one has double-booked your test and observation rooms
�Find someone (your Designated Greeter) who can welcome the
participants when they arrive, give them a comfortable place to sit while they’re waiting, and then escort them to the test room when you’re ready to start
Test day (before the first test)
�Order lunch for the debriefing
�Put observer handouts in the observation room
�Make sure whatever you’re testing is installed on the test computer or accessible via the Internet and is working
�Test the screen recorder: Do a short recording (including audio) and play it back
�Test screen sharing (video and audio) with the observation room
�Turn off or disable anything on the test computer that might interrupt the test (e.g., email or instant messaging, calendar event reminders, scheduled virus scans)
�Create bookmarks for any pages you’ll need to open during the test
�Make sure you have any phone numbers you might need:
Observation room: ___________________________
Test room: ___________________________
Greeter: ___________________________
Developer: ___________________________ (for problems with prototype) IT contact: ___________________________ (network or server problems)
�Make sure the speakerphones in the observation room and test room are working
[ 60 ]
some boring checklists
Before each test
�Clear the browser history
�Open a “neutral” page (e.g., Google) in the Web browser
While the participant signs the consent form
�Start the screen recorder!
At the end of each test
�Stop the screen recorder!
�Save the recording!
�End the screen sharing session, if necessary
�Take time before the next session to jot down a few notes about things you observe
�If it’s the last test of the day and you’ve been using a desktop computer, copy the screen recording file to a CD or thumb drive
[ 61 ]
chapter
chapter3
8
Mind reading
made easy
conducting the test session
[ 62 ]
a morning a month, that’s all we ask
I was thrown out of NYU my freshman year
for cheating on my metaphysics final.
I looked into the soul of the boy sitting next to me.
—WOODY ALLEN, IN ANNIE HALL
Now for the main event: the test itself.
I’m assuming that you’re going to be the facilitator: the person sitting in the room with the participants, giving them their instructions, and asking them questions. Eventually you may want to train someone else to do it too, but at least in the beginning you’re probably going to be it.
In this chapter, I’m going to describe the facilitator’s job, explain how to set up the test room, lay out the timeline for the test, and then discuss how you deal with participants.
What the facilitator does
As facilitator, you have two roles to play:
As tour guide you’re responsible for telling the participants what to do, keeping them moving, and keeping them happy.
Unlike an actual tour guide, though, you’re not going to be answering their questions about the sights (or in this case, the sites). The participants have to figure out how to use th
e thing on their own.
As therapist your main job is to get the participants to verbalize their thoughts as they use what you’re testing.
This means that you’re going to encourage the participants to think out loud as much as possible. You want them to do a running narration of what’s going through their head as they use the site: what they’re trying to do, what they’re looking at, what they’re reading or scanning, and what questions they have in mind. In other words, what they’re thinking.
This process, called the think aloud protocol, is the “secret sauce” that makes usability testing so effective.
[ 63 ]
chapter 8
I like to say that you and the observers want to be able to see the thought balloons forming over the participant’s head. You’re particularly interested in the moments when the balloons contain question marks or exclamation points, indicating that the user is confused, puzzled, or frustrated.
Whenever the participant’s thought balloon isn’t visible, it’s your job as facilitator to ask, “What are you thinking?”
What are
you thinking?
???
The combination of watching them use the thing and hearing what they’re thinking while they do it allows you to see your site through someone else’s eyes (and mind)—someone who doesn’t know as much about it as you do. This is what produces design insights you can’t get any other way.
[ 64 ]
mind reading made easy
The test room
For the test, you need a quiet space with a table or desk and two chairs—
usually either an office or a conference room.
So, what would
you do next?
I think I’d
click here…
b
a
d
To observation room
e
(page 96)
c
a Computer b External monitor and keyboard c Mouse d Microphone e Speakerphone Here’s what you need to have in the room:
a) A computer with Internet access, screen recording software, and screen sharing software.
The computer can be almost any laptop or desktop PC or Mac. If possible, I recommend using your own laptop instead of a desktop computer that happens to be in the room where you’ll be testing. That way you can have the screen sharing and recording software fully installed, configured, and tested and know that no one is going to uninstall it or change your settings.
