by Steve Krug
and all nearby points of interest. People spend years acquiring “The Knowledge.”
[ 40 ]
recruit loosely and grade on a curve
Take domain knowledge, for instance.
Obviously, there are cases where
domain knowledge and experience
matter. For instance, if you’re testing
the form people use to order an
industrial crane and they have to
fill in fields like Span (feet), Height
Under Boom (feet), and Capacity
(lbs), then you probably want people who know something about cranes.
But even where domain knowledge matters, it can be a tricky thing.
Your audience is probably more diverse than you think. For instance, beginners often don’t have domain knowledge, but they probably need to use your site anyway. If you’re selling car insurance online, you probably want to focus on people who have cars and know something about the car insurance domain. But you also want first-time buyers to be able to use your site.
People who presumably have domain knowledge don’t always know
what you think they know. For instance, years ago I was doing a usability review of a product designed for real estate agents. There was a term used prominently in the interface that I didn’t recognize, so I asked the designers about it. They told me that every agent knew this term and used it often. Later in the project I paid the agent who had sold us our house to do a quick usability test for me. As soon as he started looking around the product, he pointed to the term and said, “What’s this?”
And many of the most serious usability problems have nothing to do with domain knowledge anyway; they’re related to things like navigation, page layout, visual hierarchy, and so on—problems that almost anybody will encounter.
I’m not saying that you shouldn’t try to recruit people who are like your actual users. When you do need “actual users,” by all means get them. I’m just saying don’t obsess about it. For some sites you’ll have no problem finding actual users, but for others it can make the process much more time-consuming and costly—and it’s not always necessary.
[ 41 ]
chapter 5
Yes, there are things you can learn only by watching a target audience use the site. But there are many things you can learn by watching almost anyone use it. When you begin doing usability testing, your site will probably contain a lot of serious problems that “almost anybody” will encounter, so you can recruit much more loosely in the beginning. As time goes on, you’ll want to lean more in the direction of actual users. But even then I would try to recruit one
“ringer” in each round.
I also find that people who aren’t from your target audience will sometimes reveal things about your site that you won’t learn from watching “real” users, just because they have an outsider’s perspective—the emperor’s new clothes effect. And I’d rather have one articulate outsider with reasonable common sense who’s comfortable talking than ten “real” users who are tense, quirky, etc.
I’ve had a motto about recruiting for years:
Recruit loosely and
grade on a curve.
What this means is try to find users who reflect your audience, but don’t get hung up about it. Instead, try to make allowances for the differences between the people you test with and your real users.
When a participant has a problem, just ask yourself: Would our users have that problem? Or was it only a problem because the participant wasn’t familiar with the jargon or didn’t know the subject matter—a problem we’re sure our actual users wouldn’t have?
[ 42 ]
recruit loosely and grade on a curve
Three is enough
The debate over how many test participants you need has raged for a long time in the usability community, like one of those coal mine fires that burn underground for decades.
Almost everyone agrees that there are diminishing returns from having more users do the same tasks: the more users you watch, the fewer new problems you see. Most of the research that’s been done—and the arguing—is about how many users will uncover most of the usability problems in what you’re testing.
For instance, “Testing with five users will find 85 percent of the problems.”
But that’s the wrong argument for you, the do-it-yourselfer. You’re not interested in what it takes to uncover most of the problems; you only care about what it takes to uncover as many problems as you can fix.
After many years, I’ve settled on three users in each round of testing for a number of reasons:
The first three users are very likely to encounter many of the most significant problems related to the tasks you’re testing.
Finding three participants is less work than finding more.
It’s much more important to do more rounds of testing than to wring everything you can out of each round. Testing with just a few users makes it easier to do more rounds.
Testing with three users makes it possible to test and debrief in the same day.
With only a few users, it’s easier to encourage people to come and observe.
In addition to diminishing returns, there’s the tedium factor—for you as facilitator and for the observers. Starting with the fourth user of the day, there’s usually a lot more snack eating, checking of voice mail, and side conversations.
When you test with more than three at a time, you often end up with more notes than anyone has time to process—many of them about things that are really “nits.” This can make it harder to see the most serious problems—the “can’t see the forest for the trees” effect.
[ 43 ]
chapter 5
Testing with a lot of users can uncover an overwhelming and dispiriting amount of problems. Prioritizing and triaging them becomes a problem in itself, another process to manage.
The easy way out:
Throw money at the problem
If you happen to have some money lying around and you don’t have the time or inclination to do recruiting, you can make it somebody else’s problem by hiring a recruiter.
These are the same people who recruit participants for focus groups, and the process is exactly the same. To find one, just search online for “focus group rental” or “market research” and your city. The people who rent focus group facilities will usually do recruiting for you even if you’re not renting their space, or they can recommend someone who does.
The recruiter will work with you to define what kinds of people you’re looking for. Then they’ll locate possible candidates (either from their database or by some sort of advertising), screen them, schedule them, and even send them reminders to make sure they show up.
All of this is not as expensive as you might think— perhaps $100 or less per participant for the recruiter’s time, more if the people you need are hard to find.
Recruiting is the only part of the testing process that I’d recommend outsourcing. But since this is a do-it-yourself book, let’s assume that you’re going to do the recruiting yourself. Here’s how.
Where do you find them?
The first thing you need to do is think about where to look for the kind of participants you want. I think Willie Sutton’s answer when asked why he robbed banks (“Because that’s where the money is”) says it all: look in the places where the kinds of people you’re looking for tend to congregate.
For instance, if you want to test with senior citizens, consider senior centers,
[ 44 ]
recruit loosely and grade on a curve
libraries, and church groups. If you want users of your product, try user groups, SIGs, and trade shows. (You may even want to do testing right at a show.)
If you want people who use your Web site, put a link on your Home page or create a pop-up invitation that appears when they enter or leave.
If you need “just anyone,” consider friends, family, and neighbors. You don’t have to feel like yo
u’re imposing on them, because most people enjoy the experience. It’s fun to have someone take your opinion seriously and get paid for it, and participants often learn something useful that they didn’t know about the Web or computers in general.
Testing with people who work for your own organization is tempting. They’re right there, they’re comparatively easy to find, and they’re probably willing to help out. In a large enough organization, you may even be able to find people who more or less match the profile of your actual users.
But chances are they know too much. You certainly can’t use people who work on what you’re testing or people who support it, sell it, train it, or document it. But there may be people internally who know very little about it—people who work on completely different products or divisions, administrative staff, receptionists, people in finance or HR.
On the other hand, if you’re testing your company’s intranet, new employees are perfect. They’re usually eager to create a good impression, they probably have domain knowledge, and not only are they like your target audience—they are your target audience.
There is one source of participants that I can almost guarantee won’t work, although it may seem very promising: your marketing department’s offer to get you a list of users. They mean well, but in my experience there always turns out to be some wrinkle: either someone in management decides they don’t want anyone “contaminating” their customers, or there’s a privacy issue, or something. All I know is I’ve never seen it work out.
If you’re having a particularly hard time finding a certain type of user, consider doing some remote testing (Chapter 14). This usually makes recruiting much easier because it instantly broadens your potential pool from
“people who live nearby” to “everyone who has broadband Internet access.”
[ 45 ]
chapter 5
Put out an invitation
Once you’ve decided where they’re lurking, you need to put out some kind of notice announcing that you need people. For example:
We’re going to be doing a usability test of a Web site on the morning of Thursday, June 25th and we need to find a few participants. It will take about an hour of your time at our offices in the Belmont area. We’re specifically looking for people who have used online check paying to pay their bills.
If you’re interested and available on the 25th, send email to Larry Smith at [email protected]. Include your name, phone number, and a good time to reach you.
Don’t give your phone number or you’ll be swamped. Scanning dozens of emails is much more efficient than listening to dozens of voicemail messages, and the people you want to recruit will all have email.
Where do you put it? Wherever you think people will see it.
Tack it up on bulletin boards.
Post it on message boards.
Email it to your professional or personal network and ask them to pass it on to anyone they think might be interested.
Put a link on your Home page or create a pop-up invitation that appears when they enter or leave.
In recent years, people seem to have had a great deal of luck using Craigslist to find participants.
[ 46 ]
recruit loosely and grade on a curve
Screen the most promising
Once you’ve got a bunch of people to choose from, you need to screen them.
This means getting on the phone with them and having a brief conversation.
In this call you’re trying to accomplish several things:
See if they’re available on test day.
See if they meet any qualifications you’ve decided they need. (Believe it or not, some people will stretch the truth to make a few dollars. You don’t want to discover this on test day.)
Tell them what to expect: the session will last for about an hour, they’ll be using a Web site, you’ll be recording the session (but not their face), etc.
Explain how they’ll be compensated for their time.
Decide if they sound like a good participant. Do they seem like they’ll be comfortable thinking aloud? Are they articulate?
Make an appointment for one of your three test slots.
Follow up
As soon as you get off the phone, send your recruit email that confirms the appointment and gives the details: when, where, and what. Include things like:
Directions (driving and mass transit) to your site
Instructions about where to park
The location of the room where you’ll be testing
A phone number where they can reach you (or someone else) on the day of the test or the night before in the event of an emergency
Your nondisclosure agreement (if you use one) so they can read it before test day
Call them a few days before the test to confirm that you’re expecting them and to answer any last-minute questions.
[ 47 ]
chapter 5
A hearty handclasp
Years ago I had a wonderful boss who gave me a bonus with a card that read,
“Ever since the Phoenicians invented money, gratitude is no longer enough.”
Sometimes it actually is enough. Some people, like government employees, aren’t allowed to accept anything for being a test participant. And some people will be happy just to help you out, such as users of your product who are flattered that you’ve asked for their help or hope they can have some input into your future development plans. In cases like these, a gracious letter (not an email) of thanks will do.
Some people will be very happy with some kind of tangible memento—a mug or a T-shirt or one of your products.
But most of the time you need to offer people some reasonable compensation for their time, which includes the time it takes them to get there and home again.
Typical incentives for a one-hour test session range from $50 for “average”
Web users to several hundred dollars for professionals from a specific domain, like cardiologists, for instance. It depends largely on what value the people you’re recruiting place on their time. I like to offer people a little more than the going rate, since it makes it clear that I value their opinion, and people are more likely to show up on time, eager to participate.
Each method of payment has its own problems. If you give people cash, you have to get the cash, keep track of the cash, get receipts for the cash, and so on.
Checks will usually require getting the participant’s social security number and getting your accounting department to cut checks beforehand.
Probably the easiest solution all around is gift certificates. Amazon and AMEX seem to be the most trouble-free and popular.
Heir and a spare
If you’ve established a personal connection with your participants, most people will show up on test day. But at some point you’re going to find yourself staring at your watch, wondering where the next participant is. Someone’s car won’t start, or they’ll get lost en route, or…something.
And when someone doesn’t show up, your observers will wander back to their offices and may not return for the remaining tests.
[ 48 ]
recruit loosely and grade on a curve
To avoid this, you should always have a substitute available. Depending on how loosely you’re recruiting for this particular round, you have two options:
Just about anybody—someone who’s going to be nearby anyway.
This might be someone you know who works for another company in the same building as yours or someone who works in another department.
An “actual user” who you can test with remotely. If you need someone who fits a specific profile, it’s usually unlikely that there’s one sitting nearby and available. This is where remote testing (Chapter 14) can be a lifesaver, kind of like the “phone-a-friend” wildcard on Who Wants to Be a Millionaire.
In either case, the person needs to be “on call” for the entire morning: able to interrupt what they’re doing on short notice and come to your office
—or get on the phone—for an hour.
FAQ
With only three users, isn’t there a possibility we’ll miss some serious problems?
It’s not just a possibility. It’s pretty much a certainty that you won’t uncover some of the serious problems in a given round of testing. That’s why you’ll be doing more than one round.
Can you use the same participants again in later rounds of testing?
For the most part, no. Once you’ve used them, they know too much, so you can’t use them for later tests of the same site or application.
But you can use them on another site or another application. In fact, you probably want to, since you already know they’re interested and are good participants. Pass them along to other teams in your organization.
I get the general idea, but I feel like I need more advice.
All of the general books on usability testing in my recommended reading list (page 141–142) have very good sections on recruiting. But if you really want to dig in on the topic, Jared Spool and Jakob Nielsen have both published excellent reports about how to do recruiting (page 143).
[ 49 ]
chapter
chapter3
6
Find some
things for them
to do
picking tasks to test and writing scenarios
for them
[ 50 ]
Open the pod bay doors, HAL.
—DR. DAVID BOWMAN (KEIR DULLEA)
IN 2001: A SPACE ODYSSEY
If you’re going to watch people try to use what you’re building you’ve got to give them something (or some things) to do. It’s a two-step process:
First you choose the tasks to test—
1. Book an appointment online.
the things you want them to try
2. Find a gynecologist.
3. Cancel an appointment.
to do.
You need to book a physical therapy
