This chapter covers situations where the participant doesn't fit your recruiting criteria or doesn't meet your expectations in some other way. These include situations such as a participant who does not meet your recruit criteria, a participant who either refuses to or can't do a key task that was a condition of his recruit, and a participant who has a physical feature that does not align with your setup. For each situation you'll learn what to do, what to say, what not to do or say, and how you might be able to avoid it in the future.
usability recruitment; usability misrecruiting; user research participant qualifications; user research participant low literacy
4.1 Participant does not seem to meet a key recruit criteria
4.2 Participant either refuses to or can’t do a key task
4.3 Participant has an unexpected physical feature
4.4 Participant is unfamiliar with the equipment
Recruiting mishaps can happen despite your best efforts. A participant who doesn’t meet your criteria or who has some other unexpected quality means that your session may immediately start with a go or no-go decision. These situations require you to make quick decisions about whether and how to proceed with the session, and to make any necessary accommodations for any unexpected participant characteristics.
First, verify that the participant does not actually meet the desired criteria. One way to do this is to let the participant know that the information you have about him indicates that he has a specific qualification, and ask him directly if that is correct. If you’re looking for participants who perform a certain type of task, ask the participant to describe his role and, in particular, what he does regarding the task that you’re interested in. Another way is to be subtle with your line of questioning, so that it comes off as normal background questions. That way if you immediately take a break and/or end the session, the participant won’t feel like it’s his qualifications that caused the problem.
Keep in mind that once the participant arrives, you’re obligated to provide him with compensation (unless there was another agreement in place; see section 15.2 for more about your compensation decisions), so you might see if there is a way to adapt the session to include the participant’s feedback, or shift the focus of the session entirely to another method. To decide this, see if the participant’s actual qualifications would still be useful for your research. For example, based on your research goals, you may know that talking to a novice manager will provide useful feedback even though your primary target for recruiting was experienced managers. If your stakeholders are observing in person, you should take a moment to discuss this with them privately. Especially if your project has time or budget restrictions, you may not have a chance to replace the participant with someone who’s a better fit. Be prepared to argue the case for your observers as to why the participant still may yield useful and valid data.
If you need to let the participant go and not proceed with the session, consider being candid about why you’re unable to proceed. This approach may be appropriate if the criteria was role-based—for example, “I apologize for the misunderstanding. For the purpose of this study, we need to talk with managers who have direct reports.”—or any other reason where the participant won’t feel like he “failed.” However, if the reason is potentially sensitive, use a pretext such as having “technical difficulties.” Be appropriately apologetic to the participant, especially if he was excited about giving his feedback, and provide him with his full compensation.
If the participant is a very important person (VIP) or customer, or a colleague hooked in with your project team, you probably can’t get away with the technical failure pretext. Check with your observers/stakeholders on the best way to handle this particular participant. They may encourage you to proceed anyway so the participant feels like he has had a chance to provide feedback that will be taken seriously, even if it isn’t in line with your original research goals. One way to get this feedback is to adjust your study plan to learn more about what the participant does or is interested in doing.
“Before we continue, I just want to verify a few things about your <experience/expertise/role>. According to the information provided by our recruiter.…”
“I’m sorry, <participant name>, but for the purpose of this research, we need to talk with people who <criteria>. Thank you so much for your willingness to come in today, and here is your agreed-on compensation.”
Ending early:
Don’t be accusatory about the participant’s qualifications (or lack thereof). He may take it personally. When verifying the participation criteria, do so in a subtle and nonjudgmental way.
Don’t assume that the participant lied about his qualifications. The screener questions could have been poorly worded, or your recruiter may have been more lenient than he should have been. After the session, go back and review the requirements with the recruiter.
Likewise, don’t assume that it was the recruiter who messed up. Until you’re able to look into what happened, you won’t know the truth of the situation. When interacting with the participant, always take the responsibility for the misunderstanding yourself, even if you’re pretty sure the fault lies with your recruiter.
Ensure that the recruiter has detailed information about each of your desired criteria and how to evaluate whether or not the participant matches the criteria. Avoid using “give-away” questions in a recruiting screener where a respondent can guess what the correct answer is. For example, instead of asking a yes/no questions such as, “Do you use an iPad?” ask the more open-ended, “Do you use any mobile devices or tablets? If so, which ones?”
Verify the participant’s qualifications at the beginning of the session, especially if any of your criteria are deal-breakers that would require you to end the session.
Consider having backup or floater participants available. This way, if a participant doesn’t meet your requirements, you can still get feedback from someone during the session time. For more on backups and floaters, see section 15.1.
Talk to your team ahead of time to understand their tolerance for misrecruits, and create an action plan for how to address them. It will be helpful to set expectations upfront, so that you can have a quick and efficient discussion during an actual session.
If you were explicit about participants needing to do something specific during the session as a condition of the recruit, and the participant is unable or unwilling to do this, end the session. The exceptions are if there is a way for you to reschedule the participant, shift the focus of the session, or you have some other backup plan in place. You’ll need to provide the participant with his compensation unless you were crystal clear throughout the process that the compensation was contingent on performing this task. However, even in this circumstance, err on the side of the participant, especially if you have any sense that he misunderstood what he signed up for.
If the participant is willing to do the key task but is unable to (e.g., because he forgot his password), see if there is a way for you to help him get whatever information he needs to proceed. For example, if he forgot his password, ask him to go through the Forgot Password or Password Reset process. If this process involves him using the phone or answering sensitive questions on the screen to reset account information, leave the room to avoid overhearing personal identifying information and/or be sure that nothing is being screen-shared or recorded during that time. If the participant seems reluctant to make this attempt, do not force him, and instead end the session.
Offer to reschedule if the participant is genuinely disappointed about not being able to participate. Be sure to confirm and reconfirm your expectations for what he’ll need to do during the rescheduled time!
If you’re making the participant’s compensation contingent on his ability to attempt the key task, confirm this task with him several times before the session. Be as explicit as possible about what he may need to have prepared for the session, especially if it will involve accessing potentially sensitive information. As soon as the session begins, verify that the participant will be able to attempt the key task before he signs the consent form (and thus before being recorded or observed by others).
Alternatively, you can avoid conflict altogether and compensate any participant who shows up for testing, regardless of whether he is willing or able to move forward.
Despite your best efforts, accept that you may have a participant arrive who still falls into this category. Have a backup plan in place for these participants in case any misunderstandings occur. Your backup plan may include having backup or floater participants available, shifting the focus to a different type of study, or setting up a test account that a participant can use to attempt tasks on his own. For more discussion on backup participants, floater participants, and backup plans, see sections 15.1 and 15.2.
If the feature is something you notice on your own (e.g., very tall, left-handed), make any adjustments to your facility as soon as possible. For example, show the participant how to lower the chair, or move the mouse from the right to left side). When you make the adjustments, place any blame on you or your facility for not being set up correctly.
If a participant is very large and has trouble fitting into his chair, offer to get him a different one without making a big deal about it. A chair without any arms can work very well. While most participants won’t take you up on your offer, those who do will be very grateful.
If the participant apologizes for inconveniencing you, reassure him that it’s no inconvenience and that you’re looking forward to his feedback.
“I apologize that we’re not set up as well as we could be. Let me show you how we can adjust the chair to a more comfortable height.”
“Is there anything I can do to make you more comfortable?”
“Sorry, but before we continue, I’m going to help you lower/raise the chair so our recording equipment can capture you.”
Don’t call unnecessary attention to the participant’s physical attributes.
Occasionally, a participant will call attention to his own attribute to acknowledge it. If he does so, try not to follow up with personal questions. Focus any questions on understanding how you can better accommodate him. If the participant makes fun of himself for some sensitive attribute, do not laugh along. Smile politely, and then move on to your study plan.
Don’t complain about any extra work created by making the accommodations for the participant.
Familiarize yourself with the adjustments that your facility and equipment can support. For example, most chairs and desks can be lowered or raised, other types of chairs are usually available from somewhere, and a mouse (and its control panel settings) can be moved from the right side to the left side.
First, figure out if the participant is just unfamiliar with the platform that is being used. For example, participants who are used to older Apple computers and mice may not understand how to use a Windows two-button mouse, but have an understanding of how a mouse is supposed to work. If this is the case, provide a short explanation of how the device works on this platform or, if possible, switch to what he is used to (e.g., if you’re testing an application on an Android operating system and he only knows how to use an iPhone with iOS, you may want to switch devices if you’re able to do so).
If the participant is truly unfamiliar with the equipment, experiment with a short five-minute explanation and demonstration of what to do. Depending on what your research is focused on, that short time period may be enough to get him to a level where he can proceed. Make it clear to the participant that you’re training him just for this first five minutes, after which you’ll no longer be training but rather observing.
Let the participant attempt the first task, with you providing minimal technical support as necessary. Ask him to think aloud so you can track whether any problems he runs into are based on the interface he is interacting with or his unfamiliarity with the equipment.
If the participant continues to struggle, shift the focus of the session to limit the use of this equipment and end early. Handle this as gracefully as possible. See section 10.8 for additional tips for dealing with struggling participants.
If at any point the participant seems like he is feeling bad or inadequate, reassure him that he is not being tested and that the feedback he is providing is helpful.
Consider adding some basic equipment experience into your recruit criteria. We’ve found that recruiting participants who spend at least five hours a week using the Internet doing varied activities (e.g., not just checking email but reading online, purchasing from e-commerce websites, etc.) typically ensures that they can use a computer and a mouse.
Depending on your study goals, this situation may not be something that you want to avoid. The perspective that a very new or inexperienced user provides can be extremely valuable. However, for a typical usability study, the extra time spent helping an inexperienced participant results in less time getting the feedback that you need, so you’ll have to weigh this trade-off on a study-by-study basis.
If the participant is having difficulty or seems nervous reading tasks aloud, read the task to him or ask him to read it quietly to himself and then verbally reiterate the task. You may find that despite having issues reading the tasks, he may approach the tasks with the product without issue.
If the participant continues to struggle with the tasks and is getting increasingly nervous or frustrated, shift the focus of the session to reduce the amount of reading required. For example, if he communicates well verbally, turn the session into an interview. If necessary, use a pretext to explain the shift (e.g., tell the participant that the prototype isn’t working as intended so you need to change the format).
Whether or not you shift the focus of the session to another method, end early. If possible, try to do it in such a way that the participant doesn’t realize that you’re ending things early (e.g., by saying that you got through everything faster than anticipated).
Switch over to reading tasks to participants:
“Let’s say that your next task is to.…”
“I apologize—I didn’t word this task very well. Let me try to rephrase it.…”
Change the focus to an interview:
“It looks like our prototype is not working as we planned it to, so if you don’t mind I’m going to just ask you some questions.”
End the session early:
At beginning of the session: “I’m sorry, but we’re going to have to cancel the session due to some technical difficulties. Thank you so much for coming in, and here’s your compensation.”
If a little later in session: “That is actually all I had for you today, so you’ll get some time back in your day! Thank you so much for your feedback, and here’s your compensation.”
During recruiting, let participants know that they will be interacting with a product rather than just answering questions.
If low-literacy participants are part of your target recruit, be sure to set up your study plan appropriately to accommodate their abilities. For example, you should not require them to read tasks.
Clarify your purpose for being onsite and what you want to get out of the contextual inquiry. Emphasize that you wouldn’t typically be there to help, so you would rather see how the participant handles his work on his own.
The participant and your other contacts may perceive your session to be something informal, especially if you have a preexisting work relationship with them. If this is the case, disengage slightly and adapt to a more formal moderating style. For example, you may want to take a less social tone, and let them know that you need to start/resume your planned session goals.
If you have the resources available, offer to help after the session is over (if there is time) and/or to find them more help.
“I’m sorry, I would like to help—it actually looks fun/interesting. But my role is really to be a neutral outside observer here. If I’m helping, I won’t be able to focus on all that I need to.”
“If I helped you, you’d be dividing up labor that you usually do yourself, so I wouldn’t get an accurate depiction of your day-to-day work. I’d rather see what you’d do if I wasn’t here.”
“I wish I could be of help, but I can’t really leave to do <favor asked of you>. My role is to sit here and watch <the participant’s> end-to-end workflow. I’m happy to help after, though.”
If asked to train:
Don’t be offensive or condescending about not helping. Remain professional and just explain your viewpoint.
Avoid helping participants do their work, especially in a business environment. Doing so may create liability issues if you give them bad advice and they lose or corrupt data. If you’re an expert in the systems used by the participant, you may offer some basic tips like shortcuts or ways to be more efficient at the end of the session. However, keep in mind that you don’t know all the details about their systems, so it may be better to direct the participant to someone who is in a better position to help.