Recruiting on Your Own from Existing Contacts

Existing contacts can include current customers; people who have created accounts on your website; people who have signed up for your newsletter or given you their business cards at trade shows or events; your company’s friends/fans/followers on Facebook, Twitter, or other social networking services; your employees’ own families and friends; and even brand-new employees.

Many of the channels you already use to interact with customers can also be used to recruit research participants. For example, you could post a call for participants on your company website (Figure 6.1), email newsletter, or Facebook fan page, or tweet it out on your Twitter account. Obviously, make sure to choose the channels where the people who fit your research profile can be found. If you’re looking for expert users, don’t announce it in the new user help forum.

For some research needs, you may have a very specific set of current users you want to contact; for example, customers who recently upgraded from the free version of your service to the pay version. In these cases you may want to email users and invite them to participate in your research. The advantage of this method is that you may already have information that you can use to prefilter the people you ask. For example, if you want to learn about new users’ experiences, you can invite only people who have signed up for your service in the past month. On the downside, some users may find it intrusive or annoying to be contacted in this way. To avoid alienating invitees, state the purpose of your contact in the subject line of your message, offer them an incentive for participating, and give them a way to opt out of being contacted for research in the future.

Recruiting on Your Own from a Wider Population

Maybe you don’t yet have a product, or even a web presence. Perhaps you are a new company or are trying to enter a new market where you don’t have many contacts yet. If you’ve done the work to define your audience precisely, it should not be too hard to find other ways to reach out to them. But it does take some effort! This is why people hire commercial recruiting companies—locating people on your own often costs less, but it can require some creativity and extra effort.

Say you want to do research with urban dog owners. You could post messages in online discussion forums for dog owners, buy ads on pet care websites, or post flyers in pet supply stores, vets’ offices, and city dog parks. In other cases, it may be better to do your outreach more personally. If you are looking for parents of children with learning disabilities, you probably need to get to know a few of those people and earn their trust. Once they understand your objective, they may be able to introduce you to others.

If you’re at a loss for how to target a certain group of people, first find one person who matches your research participant profile. Then sit down with him or her and brainstorm about other ways that similar people can be reached. Talk about places he or she goes—online and offline—and habits and interests. Suggest some of the methods in the preceding list and ask how to tailor them specifically to his or her lifestyle.

Using a Commercial Recruiting Service

Commercial recruiters have databases with thousands of people from all walks of life. For them, the task of finding people is fairly straightforward: they search through their databases for basic demographic matches, and then they screen those people for the criteria that you’ve determined. We describe how to work with commercial recruiters later in the chapter.

The decision about whether to work with a commercial recruiting firm or do it yourself largely depends on available resources. Recruiting is a time-consuming process, especially for relatively rare groups or for people with complex schedules. However, basic recruiting can be a fairly straightforward process if organized well and given sufficient time.

Keeping Track of People Available for Recruiting

However you connect with potential research participants, you should keep a database of people who are willing to have you contact them directly for user research. Be sure to record how you got in touch with each person. You should also keep a list of people who have asked not to be contacted, so you can make sure not to bother them. Ideally, this should be integrated with whatever customer contact management system you use.

You should try to have at least ten times as many people in your contact list as you expect to invite to your next round of research.

Additional Recruiting Tips

Rather than including lengthy explanations in email, you can make a web page that explains the research program and then send email that points people to it. This can be the same page that collects their demo or web info, but it can be a purely informative page, too. The page should explain the purpose of the recruiting, the incentive, the occasional nature of the research (so people don’t think it’s a job), and how they can get more information. Add frequently asked questions to it as the recruiting process matures.

Clearly state the location of the research and the geographic area from which you’d like to recruit in all invitations. You can’t predict who will read a request, and it’s little use to have people offer to participate from halfway around the world (or even 100 miles away) unless you’re willing to pay them for their transportation costs or you’re doing research remotely.

Keep track of when and how people found out about your program. This will tell you which techniques work best and, moreover, how your recruiting methods may affect the perceptions of the participants. People who have purchased something from your company will know more about your product and have stronger opinions than people recruited through your neighbors, who will in turn have stronger opinions than people who saw an ad.

Don’t get to the point where everyone you’re recruiting has identical qualities or has been recruited in the same way. Every attribute in a profile has a range of values that could satisfy it, and recruiting should strive to get a diverse group of people within the range. For example, when participant ages are defined to be between 25 and 35, it would be better if the recruits ages encompassed the whole range rather than all being 35.

Try to keep the number of repeat recruits down. In most cases, it’s generally acceptable to recruit someone who hasn’t participated in any kind of usability or marketing research in six months. In some cases, however, this requirement will have to be waived—for example, when time is short or the person has a rare profile or he or she is in a popular market that gets recruited a lot (IT managers are few, and are the targets of much market research, for example).

When someone gives you especially good feedback, make sure to note it in the database (you can have a check box for “great response”). If you ever need a couple of people on short notice, you can just pull people who you know are articulate and insightful.

If time allows, run a small recruiting test. Recruit one or two people and run them through the research. If they provide good feedback and they’re in the right audience, proceed with recruiting everyone else the same way.

The Screener

Your next task is to create an appropriate script for the recruiter to filter the general population for the criteria specific to your research. This is called a screener.

The screener is probably the most important part of the recruiting process. It’s a script that filters out the people who will give you good responses from the ones who merely match your basic criteria. The screener can be a form posted on the web, a script read by the recruiter over the phone, or a questionnaire sent in an email. If you are doing an unmoderated study, such as an online survey, you can have people complete the screener at the beginning, with only screened participants proceeding to the actual research.

Getting the screener right can get you people who are likely to be interested in the product and can speak about their experiences intelligently and eloquently. Getting the screener wrong means getting people who are, at best, only marginally interested in what you have to offer and, at worst, uninterested and inarticulate.

Screeners vary from project to project and from recruiter to recruiter, but there are some general rules that apply to most.

Sample Phone Screener

This is a telephone screener for a site that offers an online calendar. The target audience is mostly existing users, but the screener has been expanded to include a couple of potential users and a power user (to get a taste for several different kinds of user experiences).

This screener consists of three sections: an introduction for the recruiter that specifies the ideal target audience, an introduction for the participants, and the main set of questions and selection criteria. It is a relatively complex screener that would probably be too involved for a first usability test.

The introduction simultaneously sets people’s expectations and serves as the first round of questioning since people will immediately say if they’re unavailable on the given date or if they’re unwilling to participate. It’s also careful to describe the general nature of the research without going into specifics that may skew people’s responses.

Alternatively, you can use a more generic introduction (“We are conducting a study, and I would like to ask you a few questions”), but you run the risk of going through the whole process just to find that the person is not available on a given day or is uninterested in participating in this type of research.

Eliminate people who work in industries that can present a conflict of interest. People who are in advertising, usability, user interface design, and market research should almost always be eliminated since they’re too aware of the kinds of issues that research is aiming to uncover and are unlikely to give an unbiased perspective (even if they want to).

“Terminate” is an instruction to the recruiter that tells him or her to stop the recruiting process and wrap up the interview. There is some termination text provided at the end of the screener.

Although many of these questions will have already been filtered for if these names were pulled out of a database, it’s still useful to make sure that the information is accurate, so it’s a good idea to verify the information again. Additionally, the age and income questions may be considered to be invasive by some people. In many cases, that information doesn’t affect people’s behavior and the questions can be eliminated; in cases where the information is of secondary importance, it’s possible to move the questions to the end.

When people have participated in user research recently, they’re more likely to give unconsciously biased responses since they’ll be familiar with the format of the research and may try to anticipate the “appropriate” answers. Since you’re almost always looking for unguarded, unbiased responses, it’s generally a good idea to eliminate these people from the research unless you have no other choice. Moreover, some people see the incentive payments as a good way to supplement their income and may try to get into any kind of marketing or usability research project. Inviting such people should be avoided entirely since they’re unlikely to provide natural or truthful responses.

When you are short on time or when you can interview only a couple of people, it’s sometimes useful to filter for people you’ve interviewed before rather than filtering them out. People who are known to give good, honest, articulate feedback provide a shortcut to useful information. In such situations, it’s important to inform the analyst about the participant’s background since it’ll probably affect the way he or she interprets the results.

Open-ended questions like this serve two purposes. They give the recruiter an idea of how articulate a potential participant is, and they can collect information that’s not easily formatted as a multiple-choice question. Save them for the end, and don’t put more than one in any given screener since they’re time-consuming and don’t filter many people out. (That said, some recruiters prefer to ask the open-ended question at the beginning since it’s less intimidating and can catch terminally inarticulate participants early in the process.)

If you’re using the more generic introduction, you can also replace the termination statement with something less specific, such as “That’s all the questions I have. Thank you very much for participating in our study.”

Online Screeners

Online screeners are similar to telephone screeners in terms of the kinds of questions that are asked, except that online screeners are designed to be answered directly by the participant. This makes them similar to a survey questionnaire in format. In fact, you can use the same writing style and online tools to create them that you would use to run a survey. See Chapter 12 for how to write survey questions and select an online survey tool.

Ideally, you should use a tool that lets you create screeners with branching logic, so that you can terminate people right away if their answers disqualify them, just as you would with a phone screener. Include a question at the beginning to determine whether they are available at the times and locations of the study and a form at the end for qualified respondents to provide their contact information.

Once your screener is online, you need to invite potential participants to fill it out; email is generally the easiest way of doing this. Your first task is to convince recipients not to ignore the invitation, so if you are paying an incentive, make that clear in the subject line. However, don’t put the subject in all caps or use exclamation points, or your invitation will look like spam. Make it clear to readers why you are contacting them, describe the incentive they can earn if they qualify, and state how long the screener will take to complete. Since they haven’t qualified yet, there is no need to describe the research in detail.

The Invitation

The invitation should reiterate what the research is about, why participation is important, how much the participant will be paid, where the research will take place, and when they should show up (or, in case of site visits, when they can expect the visitors to arrive). Here’s an example of an invitation email:

Confirmation and Reconfirmation

Since a lot of dates and times fly around during the scheduling process, it’s a good idea to confirm the final date and time. Send a short, clear email or other message, like this:

All confirmations should also contain a line to the effect of “Because we have tight schedules, it’s critical that you show up exactly at the time you’re scheduled or a few minutes early,” as well as specific transportation and parking instructions.

In addition, we found it helpful to standardize the format of all correspondence with a one- to two-sentence summary of the message at the beginning; a standard layout for all messages; and a contact person’s name, email address, and phone number at the end. Encourage the participants to contact that person as soon as they know they won’t be able to make an appointment or if they have any questions. Keeping the text tight, but with humor and personality, helps make the process feel less clinical, and people are more likely to read it.

Finally, the single most effective way of preventing no-shows is a combination email and telephone reminder the day before the scheduled appointment. It reminds people, subtly reinforces the importance of their input, and gives you time to schedule an alternate in case they can’t make it.

The Wrong People

Sometimes you recruit the wrong group. Maybe you misworded a key screener question, and everyone answered it opposite to what you had intended. Maybe you forgot a key element when determining the target audience. Maybe the recruiter always picked the minimal criteria when you only wanted a couple of people with that description. Regardless, sometimes you will mistakenly invite the wrong people. Every researcher has stories about coming into a room and realizing that it’s full of people who shouldn’t be there at all.

Preparation, a good screener, a carefully chosen target audience, and asking some key questions at the beginning of the research will minimize the chances of this situation, but it still happens. If you find yourself in such a situation, there are two things you can do: either cancel the session and re-recruit, or try to get as much information out of the group, hoping that some of their feedback will be usable. Sometimes the wrong group can be enlightening, since the comparison of answers with a group of the right people can lead to a better understanding of the whole population. Maybe your target audience isn’t so unique after all. However, when time is short and accurate results are critical, cancellation is a perfectly acceptable option. Then you can channel your efforts into getting a group of the right people.

Before you decide to re-recruit (or you start blaming the recruiter), you have to determine why the recruiting failed. In some cases, it will be really obvious—you wanted job seekers, but not people who have never had a job—but sometimes it may not be clear why a certain group didn’t fit your vision. Identify the elements that resulted in the wrong group: is it because of demographic factors? Is it because of their experience (or lack thereof) with the product? Perhaps you did not specify a key criterion (“factory workers” versus “factory workers who work on the shop floor,” for example). Or maybe your expectation of how people behave may not match how your target audience thinks. In rare cases, there may be no way to recruit the perfect group because the factors you want simply don’t exist together.

No-shows

It seems to be human nature that about a quarter of any invited group won’t show up, for one reason or another. You can compensate for this by scheduling extra people and adding unexcused no-shows to your list of people not to ask again.

If you would like to guarantee a full roster, you can double-schedule every time slot. This makes for twice the recruiting and scheduling work and twice the incentive expense (since you have to pay everyone who shows up, whether or not you need them to participate in research), so it’s rarely worth the trouble. But it’s useful in certain situations, such as when a senior stakeholder plans to observe the research. You can also use the extra people for other studies if you have the resources and a secondary research project. You could have a usability test as a primary research project and a questionnaire or card-sorting task as a secondary project, since these don’t need as much interaction with a researcher as the usability test. You could even have members of the development team who aren’t observing the usability test interview the participants (provided those team members have some user research experience).

Likewise, you can schedule a “floater” for a block of time, say, three hours, to hang out in case someone doesn’t show up. However, qualified people who have two to three hours in the middle of their day are difficult to find, so this isn’t as attractive an option as it may seem at first. Floaters should be paid the proportional amount of incentive for the amount of time they spend waiting (two to three times what someone scheduled for a single slot would get), and they, too, can participate in secondary research if they’re not needed as substitutes.

If the schedule and facility rental budget permit, you can also create “makeup” slots a day or two after the initial batch of research. People who can’t make their initially scheduled time may be amenable to rescheduling.

Note down the names of no-shows. Unless the people warned you or the recruiter ahead of time, do not pay them and remove them from your contact list, or tell the recruiter to do likewise.

Bias

Your recruiting methods will alter the kinds of people you find. If you rely on paper flyers in your local businesses, for example, you are limiting potential participation to people who visit those businesses. That’s great if you’re doing a study on people who probably spend a lot of time in your neighborhood. But those people might not have anything relevant to say about your research question.

A truly random sample is impossible, so there’s no getting away from some amount of bias. The question is whether the bias matters for your project. Often it does not. Nevertheless, you should mind how bias in recruiting might affect the research results. Keep track of how each person was recruited and evaluate frequent sources for his or her potential to skew the results. If half of your participants come from the same place, you should consider whether that recruitment method is affecting your results.

Anonymity

Preconceptions about your company can also bias the attitudes of people coming to test with you. A famous name can be intimidating, and an unknown company or product can seem more trivial than it really is.

If you don’t want people to know your company’s name as they are recruited and scheduled, it may be useful to maintain an alternative identity. Don’t deceive people, but working through an alternative email address (say, one from a free service) and describing the company and research without giving any specifics can be useful. In general, people don’t mind if you tell them that you’d rather not reveal the name of the company or its products.

Building and Space Preparation

Although not technically a recruiting pitfall, a common problem with scheduling participants for in-person research is that the venue isn’t prepared to greet them. If they’re arriving at an office, everyone they meet should be expecting them. Security staff should know where to send them; the receptionist should know they’re coming. You can even make signs to direct them. If you are meeting in a public place (like a shopping mall or a park), make sure they have some way to coordinate on the go.

The Neutral Interviewer

As the person writing and asking the questions in a nondirected interview, your job is to step outside everything you know and feel about your product. Forget all the hard work and creativity. Put away all hopes for success and all fears of failure. See it in a neutral light, as if it’s not yours at all.

This seems harsh, but it’s necessary in order to be able to understand the feedback people give you, both positive and negative, and be able to relate that to the process of making the product into what they want and need, not what you think they want and need. Otherwise, you’ll always be seeing either the silver lining or the cloud at a time when you need to be seeing both.

Zen aside, asking questions so as to not bias the respondent’s answer involves a lot of self-imposed distance and a rigorously critical examination of your assumptions. This can be especially difficult when you are intimately familiar with or emotionally invested in the product under examination. At first, it’s going to feel like your questions take too much energy to formulate and sound stilted. Experience will clarify which questions lead people and how to phrase questions neutrally. Eventually—when you’ve achieved nondirected question enlightenment—your questions will sound natural, analysis will be easier, and the unbiased answers you get will give you greater confidence in your results.

Composing Nondirected Questions

Most important, every question should be focused on the person answering it. It should focus on experience, not extrapolation. Our understanding of our own behavior rarely corresponds to how we really behave. When we try to put ourselves into others’ shoes, we idealize and simplify. That’s useful in trying to understand people’s ideals, but it’s rarely useful in understanding their behavior. A question such as, “Is this a useful feature?” can be easily misinterpreted as, “In the universe of all things, do you think that someone somewhere could find some use for this feature?” Even taken at face value, the potential for misunderstanding makes all replies questionable. “Is this feature valuable to the work you do right now?” clarifies the perspective.

Similarly, questions should concentrate on immediate experience. People’s current behavior better predicts their future behavior than do their predictions. If you ask people, “Is this interesting to you?” they may imagine that at some point they could find it interesting and say yes. But some things that seem interesting in theory are quite different from things that people will remember and return to. If they find something compelling right now, they’re likely to continue to find it compelling. Thus, the responses to “If it were available today, would you use it? Why?” will be more useful.

Questions should avoid judgmental language. The person answering the question should not think that you’re expecting a specific answer or that any answer is wrong. You can (and should) state this explicitly, but it works better if the question reinforces that view. “Don’t you think that this would be better if it was also available on smart phones?” implies that the person asking the question thinks will disapprove if they hear otherwise. “If this feature were available tomorrow on smart phones, would you use it?” doesn’t imply an expected answer (though it’s a binary question, which we’ll describe later). An even better question would be, “Is there any other way you might use a feature like this?” and then ask about smart phones after they’ve stated their initial thoughts.

Focus questions on a single topic. An “and” or an “or” linking two ideas is ambiguous. It’s hard to tell which concept the response addresses. Divide a question like “How would this product be useful to you in school or at work?” in two.

Keep questions open-ended. If forced to choose, people will pick something, even if none of the options match what they believe. “Which feature from the following list is most important to you?” assumes that there are features that are important, and it assumes that there is one that’s more important than any other. A better way would be to say “Rate from 1 to 5 how important each of the following features is to you, where 1 is least important and 5 is most important. Put 0 if a feature is completely unimportant. Write down any features we may have missed.” Or don’t rate the features. Instead ask, “Does the product do anything that’s particularly useful to you? If so, what is it? What makes it useful?”

Avoid binary questions. Binary questions are of the form “yes/no” or “true/false” or “this/that,” and they force people to make a black-and-white choice when their attitude may not lie near either extreme. “Is this a good product?” misses a lot of subtlety. Although it may be nice to quickly check people’s off-the-cuff opinions, it’s more valuable to know what they find good and bad about the idea, rather than just if the whole thing seems good or bad. “What, if anything, do you like about this product?”

Running a Nondirected Interview

There are a number of ways to increase the quality of the responses in a nondirected interview.

Define terms. “That thing” can refer to a button, a feature, or the whole site. Personal definitions of words differ from the dictionary definition and the development team’s definition. Someone may speak of a simple function as a “module,” whereas the development team may call complex clusters of functions “modules.” When using a technical term, make sure that you clearly define it first. Whenever possible, use the respondent’s definition of a word (even if it’s not how you use it), but make sure that you understand what that definition is first (which may mean asking the respondent to define it). This is especially important in group interactions, where everyone can come in with different definitions.

Don’t force opinions. There are topics about which we just don’t have an opinion. We may have never thought about them, or we may not have enough information. When asked for an opinion, most people will express one. But it’s not going to be carefully considered or deeply held. Before framing a question in terms like “Would this be better or worse if…,” try to gauge whether the participant has any knowledge of “this.”

Restate answers. Bouncing the respondent’s answer back at him or her using different words can cut through problems with questions. It clarifies terminology and verifies that you and the participant understand each other. Immediately after someone has finished a thought, you can say something like “So I hear you saying that…” and state what you just heard using different words. Restating answers requires a little extra care because restatements themselves can lead the discussion. If you have a bias, it’s easy to subtly restate in a way that better fits your assumptions.

Follow up with examples, but always wait for an undirected answer first. Sometimes people understand a question, but may not know how to start answering it. If you are precise with your wording, it shouldn’t be an issue. Occasionally, though, you may want to ask a question that’s intentionally broad, to see how people understand a concept or what their most general thoughts are. Prepare an example (or two) for the questions you feel may need examples. After the participants have given their initial answers, you can refocus their thoughts with an example. Say you’re running a focus group that’s brainstorming new features. If they’re defining features too narrowly and seem to have reached an impasse, you can say, “Now what if it were to email you whenever items you liked were on sale?” and see if the participants can come up with other ideas along the same lines. Don’t give more than a couple of examples since that tends to frame people’s perceptions too strongly.

Use artifacts to keep people focused on the present and to trigger ideas. Artifacts are the material products of people’s work: the notes, the papers, the tools, and so on. Bring participants back to their immediate environment by asking questions about the everyday objects that they deal with on a regular basis. When someone mentions “shopping carts” in the abstract, ask about “this shopping cart.” When you’re in the field and they’re talking about how a certain procedure is done, ask them to show it to you with the actual objects. The idealized situation people imagine and discuss in the abstract is often different from the lived experience. Focusing on specific objects helps recall the grungy details missing from the ideal.

Be aware of your own expectations. Watch for situations that surprise you. Despite the exhortations at the beginning of this section, it’s impossible to be a blank slate. There are going to be things you assume or expect from the interaction, and these are going to affect how you run the interview. If you’re aware of these assumptions, it makes avoiding them easier.

Never say the participant is wrong. Even if someone’s understanding of how a product works or what it’s for is completely different from what was intended, never tell anyone that his or her perspective is wrong. Try to understand where it comes from and why he or she has it. Even the most minor, superficial contradictions can make participants feel clueless or unprepared and thus reluctant to volunteer their opinions. Steve Portigal, founder of the design research firm Portigal Consulting, tells the story of an interview he conducted about home entertainment technology with a participant who clearly considered himself very knowledgeable. At one point the interviewee called one brand-name digital video recorder a “Tie-vo,” instead of pronouncing it “Tee-vo,” as its makers do. Steve switched to saying “Tie-vo” himself rather than imply that the participant was incorrect.

Listen carefully to the questions people ask you. Questions reveal a lot about how people understand a product or a situation, and they’re important for understanding experience and expectations. Probe why people are asking the question. For example, if someone asks, “Is that how it’s supposed to work?” answer with a question: “Is that how you think it works?” or “Is that how you expected it to work?”

Keep questions simple, both in language and in intent. Use questions to uncover assumptions and perceptions, not prove points or justify actions. A good question does the minimum necessary to elicit an undirected response. Analyzing the answers will provide any proof or justification. Questions should focus on getting the clearest raw information.

Review your recordings when possible. It’s easy to miss a key statement or a subtle distinction when relying on your memory and notes. Try to Spend some time with your tapes—whether audio or video—verifying that your views of the discussion accurately represent what happened. It’s also the way to improve your interviewing skills, even though it can be painful to watch yourself.

Often, two people conduct interviews—one asking questions, and one taking notes. Cross-cultural research may also involve an interpreter (see Chapter 13 for more). In general, try to avoid having more than two interviewers (and, if necessary, an interpreter). Determine when and how the notetaker should ask questions in advance and introduce everyone involved to the interview participant at the beginning of the interview.