Recruiting

Recruiting is the most crucial piece to start early. It needs to be carefully timed and precise, especially when outsourced to a specialist recruiting company. Finding the right people and matching their schedules to yours takes time and effort. The more time you can devote to the recruiting process, the better (although more than two weeks in advance is generally too early, since people often don’t know their schedules that far in advance). You also need to choose your screening criteria carefully. The initial impulse is simply to recruit people who fall into the product’s imagined target audience, but that’s usually too broad. You need to hone in on the representatives of the target audience who are going to give you the most useful feedback. (We’re presenting a condensed explanation of the recruiting process here. See Chapter 6 for a more in-depth discussion.)

Say you’re designing a website that sells forks. Your imagined audience consists of people who want to buy forks.

In recruiting for a usability test, that’s a pretty broad range of people. Narrowing your focus helps preserve clarity, since different groups can exhibit different reactions to the same features. Age, expertise, and motivation, just to name a few common differences, can result in very different user experiences. Choosing the “most representative” group can reduce the amount of research you have to do in the end and focus your results.

The best people to invite are those who are going to need the service you are providing in the near future or who have used a competing service in the recent past. These people will have the highest level of interest and knowledge in the subject matter, so they can concentrate on how well the interface works rather than on the minutia of the information. People who have no interest in the content can still point out interaction flaws, but they are not nearly as good at pointing out problems with the information architecture or any kind of content-specific features since they have little motivation to concentrate and make it work.

Say your research of the fork market shows that there are two strong subgroups within that broad range: people who are replacing their old silverware and people who are buying wedding presents. The first group, according to your research, is mostly women in their 40s, whereas the second group is split evenly between men and women, mostly in their mid-20s and 30s.

You decide that the people who are buying sets of forks to replace those they already own represent the heart of your user community. They are likely to know about the subject matter and may have done some research already. They’re motivated to use the service, which makes them more likely to use it as they would in a regular situation. So you decide to recruit women in their 40s who want to buy replacement forks in the near future or who have recently bought some. In addition, you want to get people who enjoy shopping online, so that you do not confuse problems with this specific website with dislike of online shopping in general. Including all these conditions, your final set of recruiting criteria looks as follows:

Notice that there is some flexibility in the age and gender criteria. This is to make the recruiter’s life a little easier. You may insist that the participants be all female and that they must be between 40 and 50 years old, but if a candidate comes up who matches the rest of the criteria and happens to be 33 and a man, you probably don’t want to disqualify him immediately. Purchasing experience, on the other hand, requires precise requirements, since getting people who aren’t going to be puzzled or surprised by the details of purchasing online is key to making the test successful. Testing an e-commerce system with someone who’s never bought anything online tests the concept of online shopping as much as the specific product. You rarely want that level of detail, so it’s best to avoid situations that inspire it in the first place.

How many participants do you need? Unfortunately, there’s no simple answer. More evaluators means catching more potential problems—but at a certain point the cost of recruiting and the extra effort needed to run the tests and analyze the results lead to rapidly diminishing returns. For many years, the conventional choice, following a study by Jakob Nielsen, has been at least five evaluators per simple test. Larger groups still produce useful results, especially for complex tests with many different elements, but after eight or nine users, you may find that the majority of problems appear several times.

However, careful recruiting and task coverage matters. Research from scientists Gitte Lindgaard and Jarinee Chattratichart suggests that the number of tasks and type of evaluators are as important as the number of evaluators. A large number of evaluators who are unrepresentative of your target users will likely be less effective than a smaller number of representative evaluators. Similarly, a large number of representative evaluators performing a very limited number of tasks—or tasks that are irrelevant to actual use—will likely catch fewer severe problems than the same number performing more tasks.

So, to check your understanding of your primary audience, you can recruit one or two people from secondary target audiences—in the fork case, for example, a younger buyer or someone who’s not a recent online shopper—to see whether there’s a hint of a radically different perspective in those groups. This won’t give you conclusive results, but if you get someone who seems to be reasonable and consistently says something contrary to the main group, it’s an indicator that you should probably rethink your recruiting criteria. If the secondary audience is particularly important, it should have its own set of tests, regardless.

To offset no-shows, it’s a good idea to schedule a couple of extra people beyond the basic five. And to make absolutely sure you have enough people, you could double-book every time slot. This doubles your recruiting and incentive costs, but it ensures that there’s minimal downtime in testing. So, for focused task-based usability testing, you should recruit from six to ten people to make sure you have at least five evaluators for each round of testing.

Having decided whom—and how many—to recruit, it’s time to write a screener and send it to the recruiter. (Chapter 6 of this book describes screeners and recruiting.) Make sure to discuss the screener with your recruiter and to walk through it with at least two people in-house to get a reality check.

Then pick a couple of test dates and send out invitations to the people who match your criteria. Schedule interviews at times that are convenient to both you and the participant and leave at least half an hour between them. That gives the moderator enough buffer time to have people come in late, for the test to run long, and for the moderator to get a glass of water and to discuss the test with the observers. With 60-minute interviews, this means that you can do four or five in a single day, and sometimes as many as six. With 90-minute interviews, you can do three or four evaluators, and maybe five if you push it and skip lunch.

At this point, you will have your evaluators picked and scheduled. To help keep everything straight, we suggest making a master spreadsheet (see Table 11.2) that contains all the pertinent information—time of test, name, responses to the screener questions, and which test conditions each person saw (if you are testing multiple versions of the products). This will help you later, when you need to analyze the data and want some context for your analysis. It will also help speed up writing evaluator profiles for the report.

Table 11.2. A Sample Spreadsheet Layout

Choosing Features

The second step is to determine which features to test. These, in turn, determine the tasks you create and the order in which you present them. Give yourself enough lead time to fine-tune the procedure. A 60- to 90-minute interview can cover five features (or feature clusters). Typical tests range from one to two hours. Initial or broad-based testing requires two-hour tests, while in-depth research into specific features or ideas can take less time (though it’s perfectly acceptable to do a 90-minute broad-based test).

Test individual functions in the context of feature clusters. It’s rarely useful to test elements of a set without looking at least a little at the whole set. Here’s a good rule of thumb: something is testable if you would draw it while making a 30-second sketch of the interface. If you would sketch a blob labeled “nav bar,” then think of testing the entire navigation bar, not just the new link to the home page.

The best way to start the process is by meeting with the development staff (at least the product manager, the interaction designers, and the information architects) and making a list of the five most important features to test. To start discussing which features to include, look at features that are:

Once you have your list of the features that most need testing, you’re ready to create the tasks that will exercise those features.

In addition, you can include comparative (or competitive) usability testing. Although comparing two interfaces is more time consuming than testing a single interface, it can reveal strengths and weaknesses between products. Performing the same tasks with an existing interface and a new prototype, for example, can reveal whether the new design is more functional (or, as every designer fears, less functional). Likewise, performing the same tasks or conducting similar interface tours with two competing products can reveal relative strengths between the two products. In both situations, however, it’s very important to not bias the evaluator toward one interface over the other. (Competitive research is covered extensively in Chapter 5.)

Creating Tasks

Tasks need to be representative of typical user activities and sufficiently isolated to focus attention on a single feature (or feature cluster) of the product. Good tasks should be:

One or more tasks should test each feature on your list. Usually, it’s useful to have two or three alternative tasks for the most important features in case there is time to try more than one or the first task proves to be too difficult or uninformative.

People can also construct their own tasks, within reason. In fact, many usability researchers prefer to have evaluators come up with their own tasks. At the beginning of a usability test, you can ask the participants to describe a recent situation they may have found themselves in that your product could address. Then, when the times comes for a task, ask them to try to use the product as if they were trying to resolve the situation they described at the beginning of the interview.

Another way to make a task feel authentic is to use real money. For example, one e-commerce site gave each of its usability testing participants a $50 account and told them that whatever they bought with that account, they got to keep (in addition to the cash incentive they were paid to participate). This presented a much better incentive for them to find something they actually wanted than they would have had if they just had to find something in the abstract.

Although usability testing is generally a qualitative procedure, you can also add some basic quantitative metrics (sometimes called performance metrics) to each task in order to investigate the relative efficiency of different designs or to compare competing products. Some common quantitative measurements include:

Because such data collection cannot give you results that are statistically usable or generalizable beyond the testing procedure, such metrics are useful only for order-of-magnitude ideas about how long a task should take. Thus, it’s often a good idea to use a relative number scale rather than specific times.

For the fork example, you could have the set of tasks in Table 11.4, as matched to the features listed earlier.

Table 11.4. Sample Fork Tasks

When you’ve compiled the list, you should time and check the tasks. Do them yourself and get someone who isn’t close to the project to try them. This can be part of the pretest dry run, but it’s always a good idea to run through the tasks separately if you can.

In addition, you should continually evaluate the quality of the tasks as the testing goes on. Use the same guidelines that you used to create the tasks and see if the tasks actually fulfill them. Between sessions, think about the tasks’ effectiveness and discuss them with the moderator and observers. Although it’s a bad idea to drastically change tasks as you go, it’s okay to make small tweaks that improve the tasks’ accuracy in between tests, keeping track of exactly what changed in each session.

Writing a Script

With tasks in hand, it’s time to write the script. Sometimes called a “protocol,” sometimes a “discussion guide,” the script is really just that—a list of instructions for the moderator to follow so that the interviews are consistent and everything gets done.

A script generally has three parts: the introduction and preliminary interview, the tasks, and the wrap-up. The one that follows is a sample from a typical 90-minute e-commerce website usability testing session for people who have never used the site under review. About a third of the script explores the participants’ interests and habits. Although those topics are typically part of an interview or a focus group, it’s often useful to investigate them as well in usability testing. Another third focuses on task performance, where the most important features get exercised. A final third is administration.

The introduction is a way to break the ice and give the evaluator some context. This establishes a comfort level about the process and their role in it.

Video equipment is often blatantly obvious and can be intimidating. Talking about it helps relieve a lot of tension about it. Likewise, if there’s a two-way mirror, mentioning it—and the fact that there are people behind it—also serves to alleviate anxiety. Once mentioned, don’t bring it up again. The equipment fades quickly into the background, and discussing it again is a distraction.

Also, note that the script sounds conversational. It’s unnecessary to read the script verbatim, but the style of the script should remind the moderator to keep the tone of the interview casual. In addition, every section has a duration associated with it so that the moderator knows about how much time to spend on each one.

No matter what stage the product team says the product is in, consider telling evaluators that it’s in an early stage. Telling evaluators that the product is a work in progress helps them relax and gives them more license to make comments about the product as a whole.

Explicitly laying out the whole procedure helps the evaluator predict what’s going to come next and gives them some amount of context to understand the process.

The informed consent statement tells the evaluators that their input is valuable, that they have some control over the process, and that there is nothing fishy going on.

The preliminary interview establishes the context for the participant’s later comments. It also limits the interview’s focus to the space of the evaluator’s experience by beginning with general questions and then narrowing the conversation to the context of the product. For people who have never participated in a usability test, it increases their comfort level by asking some “easy” questions that build confidence and gives them an idea of the process.

In this case, the preliminary interview also features a relatively extensive investigation into people’s backgrounds and habits. It’s not unusual to halve the number of questions and to allot the initial context-setting interview 5 minutes, rather than 10–15.

When applicable, it’s useful to ask about people’s offline habits before moving the discussion to the online sphere. Comparing what they say they do offline and what you observe them doing online provides insight into how people perceive the interface.

It’s important that evaluators don’t feel belittled during the test. Badly designed tools and arrogant companies have conditioned people to blame themselves for unsuccessful experiences. Although it’s difficult to undo a lifetime of software insecurity, the evaluation instructions help get evaluators comfortable with narrating their experience, including positive and negative commentary, in its entirety.

A major component to effective usability tests is to get people to say what they’re thinking as they’re thinking it. Introduce the technique upfront, but also emphasize it during the actual interview.

First impressions of a product are incredibly important, so testing them explicitly is always a good thing and quick to do. Asking people where they’re looking and what they see points out the things in an interface that pop and provides insight into how page loading and rendering affects focus and attention.

If you are evaluating a website, the interview begins with the browser up, but set to a blank page. Loading order affects the order people see the elements on the page and tends to affect the emphasis they place on those elements. Knowing the focus of their attention during the loading of the page helps explain why evaluators view certain elements as more or less important.

At this point, the script can go in two directions. It can be a task-based interview, where the user immediately begins working on tasks. Alternatively, it can be a hybrid interview—half task-based and half observational interview.

The task-based interview focuses on a handful of specific tasks or features. The hybrid interview is useful for first-time tests and tests that are early in the development cycle. In hybrid interviews, the evaluator goes through an interface tour, looking at each element of the main part of the interface and quickly commenting on it before working on tasks.

A task-based interview would look like the following.

When there is a way to observe participants through a one-way mirror or streaming video, it is sometimes useful to ask them to try a couple of the listed tasks on their own, without the moderator in the room. This can yield valuable information about how people solve problems without an available knowledge source. In addition, it gives the moderator time to discuss the test with the observers. When leaving the room, the moderator should reemphasize the need for the evaluator to narrate all of his or her thoughts.

Including a specific list of issues to probe helps ensure that you answer all the important questions. The moderator should feel free to ask the probe questions whenever it is appropriate in the interview.

A hybrid interview could look as follows. It begins with a quick general task to see how people experience the product before they’ve had a chance to examine the interface in detail.

With the tasks completed, the heart of the information collection and the interview are over. However, a wrap-up is useful for the observers and analysts to get a perspective on the high points of the discussion. In addition, a blue-sky discussion of the product can provide good closure for the evaluator and can produce some good ideas. You can also use the time to ask people to draw what they remember of the interface as the moderator leaves the room and asks the observers if they have any final questions for the participant.

Finally, it’s useful to get some feedback about the testing and scheduling process.

As with every phase of user research, the product stakeholders should have input into the testing script content. The stakeholders should still vet the complete script draft to assure that the priorities and technical presentation are accurate. Give the first draft to the development team at least a week before testing is to begin. Show them a second version incorporating their comments at least a couple of days beforehand.

The Physical Layout

The test site should resemble the kind of space where people will use your product. If you’re doing remote usability testing, or visiting participants in their own homes or workplaces, then you have nothing more to do. If you are having participants come to your own facilities, and it’s a business product, then conduct the test in an environment that resembles a nice office, preferably with a window. If for home use, say, in a living room, then find some furniture for the test that resembles what your users would have in their homes. The illusion doesn’t have to be exact; you can achieve an appropriate impression with just a few carefully chosen props. Soft indirect lighting, a comfortable chair, and a tablecloth over an office desk instantly make an office feel a little more home-like.

Often, however, the usability test must take place in a scheduled conference room or a rented lab, where extensive alteration isn’t possible. In those situations, make sure that the space is quiet, uncluttered, and as unintimidating as possible.

Video record interviews whenever possible. Ideally, you’ll want to have a “picture in picture” recording of the evaluator’s face and hands and a recording of their on-screen activity (see Figure 11.1). The moderator does not have to appear in the shot. You can either use the computer’s built-in camera or position an external camera on the monitor or to one side. As of 2012, Morae (www.techsmith.com/morae.asp) is a standard, if expensive, commercial usability testing software. Lower-price alternatives, such as Silverback (Mac-only, www.silverbackapp.com) are useful if you can accept a more limited set of features.

Accurate, clear audio is extremely important, so the video camera should have a good built-in microphone that filters out external noise. Alternately, you can invest in an external microphone. The evaluator can clip it onto his or her clothing, or the moderator can tape it to the monitor.

How you accommodate observers depends on whether you have access to a traditional usability lab facility. Lab facilities enable in-person observation with a space divided by a two-way mirror—one room for the test, and a soundproofed separate observation room.

Outside of a traditional usability lab, the most efficient way to enable remote or next-door observation of usability test is to use commercial usability software (such as Morae) that supports easy sharing of picture-in-picture video and screen activity. In that case, your computer setup would resemble Figure 11.2. Many commercial software makers offer generous free trials—but over the long term, investing in specialized usability testing software can simplify your life.

Without using expensive commercial software, it gets complicated to simultaneously record and broadcast screen activity, video from camera, and audio. Bandwidth is limited; computers crash; no one program will do everything you want. As of 2012, you will have to be prepared to do some work and make some trade-offs.

Still, if you don’t have a lot of money and your needs exceed the capabilities of a trial software version, you still have some options. The cheapest (though incomplete) solution is to use free software (such as Skype) that allows remote observers to view screen activity and hear audio from the participant’s computer (setup resembles that in Figure 11.3). However, you won’t be able to share video, and you may have compatibility problems in trying to get a separate program to record your audio and screen sharing. If you have some money to spend, you can subscribe to a web-based conferencing software, such as GoToMeeting, that will broadcast and record screen activity and audio. However, as of 2012 you will still not be able to record video.

In both cases, if you really need video and you need to support live observers, the best choice may be to run an external video camera, record that video to another computer, then later recombine the video and screen activity into a picture-in-picture video using any one of a number of lower-cost commercial video editing choices.

If your observers are on site, it may be more practical to use a combination of external video camera and shared monitors. That is, you would record screen activity as usual on the participant’s computer. Then you would run another cable to the separate room and share the participant’s screen with another monitor. Finally, you would set up an external video camera to capture audio and facial expressions, then show the live video on a separate computer and monitor in the observation room (e.g., Figure 11.4).

Moderation

The moderator needs to make the user feel comfortable and elicit useful responses at appropriate times without drastically interrupting the flow of the user’s own narration or altering his or her perspective. Use the nondirected interviewing style, as described in-depth in Chapter 6.

Apart from the general interviewing style outlined in Chapter 6, there are several things that moderators should do in all interviews.

Managing Observers

Getting as many members of the development team to observe the tests is one of the fastest ways to relate the findings of the test and win them over.

Make the appropriate staff watch the usability tests in real time, if possible. There’s nothing more enlightening to a developer (or even a vice president of product development) than watching their interfaces misused and their assumptions misunderstood and not being able to do anything about it.

Bring in plenty of food (sandwiches usually work). The team can then lounge in comfort and discuss the tests as they proceed (while not forgetting to watch how the participants are actually behaving). Since they know the product inside and out, they will see behaviors and attitudes that neither the moderator nor the analyst will, which is invaluable as source material for the analyst and for the team’s understanding of their customers.

If neither a two-way mirror nor live video is available, it’s possible to have members of the team observe the tests directly. However, there should never be more than one observer per test. It’s intimidating enough for evaluators to be in a lab situation. To have several people behind them, sometimes scribbling, sometimes whispering, can be too creepy for even the most even-keeled. If an observer is in the room, introduce him or her by name. An introduction allows the participant to treat the observer as a legitimate, nonthreatening presence, not as “that guy watching me silently from the corner.”

Give observers some instructions on acceptable behavior. This will also set their expectations for what’s going to happen.

For in-room observers, add the following instructions:

If the observers are members of the development team, encourage them to wait until they’ve watched all the participants before generalizing and designing solutions. People naturally want to start fixing problems as soon as possible, but teams need to determine the context, magnitude, and prevalence of a problem before expending energy to fix it.

Tips and Tricks

Remote Usability Basics

The initial tasks of remote usability, such as choosing tasks and writing a protocol, generally resemble those of in-person testing. There are some small differences, though. A remote usability protocol will have more up front explanations of the study, any incentive payments, how to use the screen sharing tool, and the necessity for clear verbal communication at all times.

From there, recruiting and interviewing require different tools. Both moderated and automated tests usually involve a custom pop-up window that “intercepts” potential participants as they visit a website. This means that you will need access to the website you are testing in order to install the pop-up code. The pop-up usually has a screening questionnaire (see Chapter 6 for more on how to write one) that asks a few simple demographic questions, ascertains willingness to participate in a usability test, and gets voice contact information. The researcher then telephones any eligible, willing candidates immediately or at an agreed-upon later time and date. If there are observers, they can listen through conference call functionality. Of course, you will need to prepare some initial explanations of what you’re doing, since most people don’t anticipate being called by strangers who want to look at their computer screen!

In order to see what the evaluator is doing, the researcher asks the evaluator to share a view of his or her computer screen—typically with an online tool, such as Adobe Connect. Software on the evaluator’s computer then records this screen activity. Some screen-sharing tools have built-in recording functionality. Otherwise, you will need to install a program to capture the screen activity on your own computer. As well, some programs may allow you to access the evaluators’ webcams in order to record their faces as they speak. However, taking over someone’s webcam to record his or her face may be seen as too invasive—you’ve only just introduced yourself. Of course, the evaluator needs to give explicit permission before you make and store any kind of recording.

Then the evaluator follows the protocol, much as in an in-person test (see Chapter 6 for a discussion of telephone interviewing techniques). At the end of the test, the evaluator terminates the phone call and the screen sharing, assuring the evaluator that his or her screen is no longer visible and confirming the necessary contact information for sending the evaluator an incentive payment.