If the participant is overly diplomatic in his responses, you may be able to see right through his sugar-coating. But less tactful participants may seem to be lying or avoiding a question, and it’s usually because they know they’re being observed or out of fear that their feedback may not remain confidential.

 If you suspect that the participant is lying or avoiding a question to spare someone’s feelings, try to gently coax the truth out of him. Remind him that his feedback will benefit future users and help improve the product. For example, you could say, “I just want to assure you that any feedback you provide will be used to make improvements to the product that will help other users. Your name will not be associated with any of our results.”

 If you know a question will be particularly controversial or sensitive for a participant, consider prefacing it with something that will make him feel better about being candid. For example, if you’re interviewing a call center representative about his call procedures and sense that he is worried about getting in trouble for not following them, consider saying something like, “I know that there is a certain way you’re trained to do things, however, I’d like to understand more about when you have to do something that is different from your training. Please know that any feedback you share will be kept confidential.”

 Talk to your observers after the session and remind them to respect the participant’s feedback and anonymity, even (and especially) if his feedback was controversial.

 “I know that this is a product you’re required to use in your job, but tell me candidly.…”

 “Please know that your honesty is appreciated. Nothing you say will hurt anyone’s feelings. You’re here to help us make the product better for yourself and future users.”

 “I got the sense that this participant was worried about offending someone with his feedback. So I just want to remind all of you, for him and our other participants, please respect the participant’s confidentiality and feedback. Don’t approach the participant to discuss his participation or talk to one another about his participation since he is an employee here.”

 Avoid making the participants feel like you don’t believe them. If you suspect a participant is not being entirely honest, don’t say, “It’s okay, you can tell the truth,” or “Could you tell me how you really feel?”

 Set expectations with the participant that his data will be kept confidential, and that his superiors or team members are not observing the session and will not have access to the raw identifiable data from the study.

 6.7 Participant does not have any negative feedback

 8.3 Participant looks or sounds uncomfortable and/or nervous

 10.5 Participant gives vague responses to questions

 If you have observers who are subject-matter experts or if the session is being recorded, you can capture the feedback and continue with your study plan. After the session, verify with observers what the participant was talking about. This will move the session along smoothly, but you may lose opportunities for immediate follow-up.

 If you know enough about the domain that what the participant says is within reach of your understanding, ask him to clarify what he means. Be honest—tell him that you’re not a subject-matter expert and would appreciate hearing his thoughts in more common terms.

 In some cases it helps to have an in-room observer (or on-phone observer for remote sessions) who is a domain expert but is also trained in talking to users and knows the ground rules of moderating user research. Allow that observer to ask follow-up questions at times where you might have missed the opportunity due to lack of understanding. Be sure to coordinate with the observer ahead of time how this will work.

 Alternatively, you might have another way to communicate with observers (e.g., through an instant message program) where domain experts can answer your questions in real time or can help you pose follow-up questions. Keep in mind that this type of multitasking can be challenging and may take some practice, but it can be incredibly valuable.

 If there is time for interruption during the participant’s workflow, ask him to explain what he’s doing in laymen’s terms and why he’s doing it.

 If it would be too disruptive to his workflow to ask for clarification, make sure to capture the exact process/steps with your notes and/or recording equipment. If possible, have domain experts observing or share your notes with them later. If there is time after he’s completed the workflow, ask the participant about what he was doing.

 “I’m not a subject-matter expert here, so bear with me for a second—can you explain in plain terms what you mean by that?”

 “I’m not sure I understand. Could you rephrase that for me?”

 “<Trained observer>, do you have any follow-up questions about that for <participant>?”

 “I noticed you doing something that I couldn’t quite follow. Could you help me understand better what you were doing and why? It was when you <did thing>.”

 Don’t make the participant feel like he was being inarticulate or supercilious with his feedback. Be humble yet professional when asking for clarification.

 Don’t ignore the feedback altogether just because you don’t understand it. Try to seek clarification either during the session with the participant or afterwards with a domain expert. You may actually learn a few things!

 A good practice when performing research in an unfamiliar domain is to gain a basic level of familiarity with the domain and product before you start your sessions. Having even a shallow foundation will help you filter relevant information from the participant and ask more targeted questions. You don’t have to become an expert—in fact, some user researchers argue that it is actually helpful if you’re not too immersed in the subject matter so that you can maintain a neutral and objective perspective. However, if your research is with expert users, you’ll find that a bit of additional knowledge will help you come up with meaningful follow-up questions. Some evidence of understanding on your part will help establish your credibility with the participant and may even encourage his feedback.

 Tell the participant at the beginning of the session that you’re not a domain expert. Stress that your role is to get his feedback and report it back to the team. Tell him that you may not be able to answer his questions, and that you may ask him for clarification at times when you don’t understand something.

 10.3 Participant does something very unexpected

 Give the participant another moment or two before prompting him. He may need to process what he is seeing on the screen before verbalizing his reaction to it. As discussed in section 2.1, taking this moment may be enough for this situation to resolve itself.

 After a few moments ask the participant what he is thinking. This cue is usually enough to get him to think aloud for the rest of the session.

 If the participant is persistently silent, remind him to try to think out loud as he continues.

 If the participant still doesn’t give much feedback, try to build engagement; draw him out with more general questions about what he is doing or what his experience is like. For example, you may ask him to compare his experience to other products he has used.

 If the participant is remote, encourage him to think aloud by describing what you see him doing. Use the excuse of not being beside him as motivation to keep him talking, For example, “Since I’m not able to see you as you use this product, it’s helpful if you tell me what you’re trying to do and whether what you’re seeing is working for you.” This will get him used to being verbal on the call, and will lead to more feedback.

 “What’s going through your mind right now?”

 “What are your thoughts on this?”

 “I’d like to remind you to try to think out loud as you go through these tasks. I know it may feel uncomfortable at first, but it helps us understand more about how you’re approaching the tasks.”

 “<Participant>, since I’m not there beside you to see when you’re doing certain things like clicking, I’ll ask you to just explain to me what you’re doing as you’re doing it. If there is anything along the way that you particularly like or don’t like about what you’re seeing, let me know that too.”

 “I can’t see you, so it will be hard to know what’s going on when you fall silent. So please remember to think out loud for me as you’re going through the tasks. If you’re too quiet, I’ll remind you to think aloud!”

 “It looks like you tried to <click or do something>. What were you expecting to happen?”

 Try not to be machine-like when asking participants to think aloud. Paralanguage (how you say something) matters. “What are you thinking?” may not roll off your tongue comfortably and you may feel (and sound) awkward saying it. Try to use a friendly tone and play with wording to find something that sounds more natural for you. For example, “What is going through your mind right now?”

 Explain the value of thinking aloud in the introduction to the test, specifying what you’re interested in hearing. For example, “What I mean by thinking aloud is, tell me what steps you’re taking, what you’re looking for, what you’re trying to do, what you expect to happen, and whether or not what happens meets your expectations.”

 Partway through the study, thank the participant for thinking aloud to confirm that he is doing what you asked and encourage him to continue.

 6.5 Participant ignores or pretends to understand your question

 8.6 Participant is unwilling or unsure

 Decide how and when to give the participant an assist. Time permitting, try to allow the participant to keep working on his own for a bit to see if he self-corrects. Then, if the participant has not self-corrected, give him a broad or vague assist, followed by more specific ones if needed. Think of it as a funnel slowly leading the participant to the desired location if he can’t get there on his own. For example, say the participant is using a mobile device to look for a place to change a setting for his email account. He keeps overlooking the Settings screen and navigating elsewhere to do it. When he is ready to give up, you might say, “What if I were to tell you that there is a way to do that from the Settings screen?” It gives him broad direction but isn’t pointing out anything specific. If he still can’t find it, you might then point him toward a specific area of the Settings screen. Ultimately, you could point out the particular link that you need him to select.

 When you need to point out a specific area to the participant, find a way to do it that doesn’t make the participant feel stupid or embarrassed. In the previous example, if you finally have to point out the “Email Settings” button, you might say something like, “This has been helpful seeing where you were expecting to find that option. I’ll just point out this button here. What do you think about where it’s located?” By providing this direct assist:

 You acknowledge that what he did up until this point was not wasteful.

 You allow the session to proceed.

 You reassure the participant that he is there to help evaluate the design.

 Be careful with the feedback you get immediately following a direct assist—participants will often say something snarky like, “Well, obviously I didn’t find it so it’s not intuitive,” or become overly accommodating and say, “Now that I know it’s there, that is fine.” Asking for the participant’s feedback here more about his emotional well-being than your data collection.

 We also like to use a stealthier tactic in situations like this. For a detailed description, see the Chapter 3 sidebar “The Diversionary Assist.”

 Another option is to set the product to the necessary state yourself while the participant’s not looking. For example, you can pause the screen-sharing with a remote participant, or have an in-person participant look away briefly so that you can do whatever needs to be done. While this avoids the complications of a direct assist, it can be awkward to do, especially if you need to ask the participant to look away. Some gentle humor and a smile can make this less awkward, for example, “I know this sounds strange, but I need you to look away for a few seconds while I reset the prototype for our next task”

 “What if I were to tell you that.…”

 “I’ll just let you know that there is a way from here to do that.…”

 “Let me draw your attention to this area. What are your thoughts on it being here?”

 “I’ll point out that you can find that functionality here. Talk to me about this versus your expectations.”

 “It’s good to see your thought process in doing that. I will just let you know that there is actually a different way of doing that task. I’m curious to know what you think of it” <point him to correct way>.

 “Let’s move on for now. I wanted to ask you about what you did earlier.”

 “Why don’t we stop this task here, because I wanted to ask you about.…”

 “Let’s stop this here and we can come back to it later. Read the next scenario.…”

 Try to avoid asking, “Did you notice this link?” when the participant obviously hadn’t. It can work in a pinch with nothing else to say, but it often makes the participant feel a little uncomfortable.

 Avoid saying anything implying there is a correct answer. For example, “What you’re looking for is here” or “This is the right answer.”

 The nature of usability studies is to discover usability problems so, unfortunately, there is no way to avoid this. Just be prepared for it!

 4.2 Participant either refuses to or can’t do a key task

 6.8 Participant believes he has successfully completed a task

 Chapter 3 sidebar: “The Diversionary Assist”

 Blame yourself as the moderator for not being clear with a question/task. For example, “I’m sorry, I didn’t explain this very well. What I’d like to hear from you is.…” This technique reassures the participant that he is not being evaluated and that he is not stupid for failing to understand your question.

 Rephrase the question/task to make it easier for the participant to understand, or break down your question into a set of smaller questions that will lead to your goal.

 If the participant is watching your lips closely or tilting his head to be closer to yours, he may be having trouble hearing you. Move your chair a bit closer to his, speak a little louder, and use short, direct sentences.

 If the participant still doesn’t understand you or is still being vague, move on to your next task or question.

 “If anything I ask doesn’t seem to make sense, please let me know.”

 “I know this may not be what you typically work on, so please let me know if I’m not being clear about what I’m asking you to <do/talk about>.”

 “I’m sorry, I don’t think I’m being clear with my question. Let me rephrase.…”

 “Would it help if I rephrased the question?”

 “Let me back up a little, and first ask you this.…”

 Don’t ask the same question over and over in hopes of getting different responses. This will just exacerbate the problem and further frustrate the participant.

 Try not to say anything that may be perceived as blaming the participant. For example, “Did you not understand my question?”

 At the beginning of the session, reassure the participant that he is an important part in your research. For example, “I will be asking you some questions during this session, but if anything I ask isn’t clear, please don’t hesitate to tell me.” Another example might be, “I’m not a subject-matter expert here, so if I’m not asking a question that makes sense, please let me know that.”

 If the participant is an internal employee, provide reassurance about how his feedback will be used to help alleviate any concerns about looking bad in front of his fellow employees. For more information, see section 15.1.

 6.3 Participant is not thinking aloud

 8.6 Participant is unwilling or unsure

 10.5 Participant gives vague responses to questions

 Survival Story: “I would have trusted my gut” in Chapter 11

 Since it can be difficult to tell if a participant is approaching something naturally or not, look for these signs:

 He seems to struggle a lot with a task that he supposedly does on a regular basis.

 He uses descriptions like, “we do …” or “we then have to do …” rather than “I do. …”

 He seems nervous, looks at you frequently, and/or otherwise acts like he is being evaluated. For example, he asks, “Are you going to grade me on this?”

 Instead of showing you the workflow, he just describes the process, even after you prompt him otherwise.

 When you notice these things happening, clarify with the participant that you want to see what he actually does, even if it’s not official procedure or the most efficient way to do things.

 If the participant seems afraid of getting into trouble because of what he shows or tells you, talk to him about how the collected data will be used (e.g., the data are kept confidential and aggregated before they’re shared with the project team). Reassure him that he is not being evaluated in any way.

 “In a normal work week, how often would you do this?”

 “Is this how you typically do this task day-to-day?”

 “About what percentage of the time would you say that you do it this way?”

 “What are some other ways you’ve tried approaching this task before?”

 “I am here to watch how you work so that I understand what does and doesn’t work well for you. We then take this information and use it to design a better experience. Just to be clear, I am not here to evaluate your skills or how you do your job.”

 “Please know that anything you say or do will not be associated with your name.”

 Don’t joke about evaluating the participant.

 Don’t tell the participant that he is doing a good or bad job.

 Don’t act entitled to watch the participant work, especially if someone else signed him up for it. Instead be appreciative that you’re able to watch him and behave as if he is doing you a favor.

 In the recruiting communication to the participant, try to make it clear what your role is, why you’re there, and how the information will be shared/used.

 If you’re doing a contextual inquiry onsite, try to provide a short general briefing to all the participants you’ll be talking with when you arrive. This briefing can help set context for what you’ll be doing, how you’ll collect data, and what you’ll do with the results.

 5.5 Participant treats a contextual inquiry like an interview

 If a usability study participant completes his tasks without experiencing any difficulties, don’t panic! Receiving all or mostly positive feedback is unusual but not unprecedented. Do not pressure the participant to come up with negative feedback. Remember that it is useful to know what works well in a design so you can make sure it doesn’t get changed.

 If the participant seems to have contradictory reactions (e.g., struggles excessively with tasks or usage of the product but loves everything about the design), think about what may be in play:

 Did you forget to mention at the start of the session that participants should be candid and provide both positive and negative feedback? If so, find a way to sneak that into discussion, even if you need to take responsibility and admit that you forgot to say it.

 Does the participant think or know that you worked on the design, or that the team who designed the product is observing? If so, make it clear that everyone wants to hear both positive and negative feedback so they can keep what is working and fix what isn’t.

 Is the participant a die-hard fan or early adopter of the product and feels that it can do no wrong? If this seems likely, try to specifically draw out things that he does not like about the product since he may not mention those things on his own. For example, “If you could wave a magic wand and change one thing about this product, what would it be?”

 “If you could wave a magic wand and improve any one thing about this product, what would it be? Why?”

 “<Participant>, I’m sorry! I forgot one point to tell you when I was going through the introductory information. I didn’t introduce myself properly—I’m a user researcher, which means I’m a neutral observer here—and anything you have to say, positive or negative, will not praise me or hurt my feelings.”

 “Thank you for this feedback—it is all very helpful to hear. The project team is really excited to get all feedback—both positive and negative. So just continue to let me know anything you want to tell me, good or bad. Let’s move on to.…”

 “This has been very useful feedback, so thank you. Let’s think about this <task/topic/area> for a moment. What did you like most about it? What did you like least? What would you want to improve first if you were using this at work or home?”

 Try not to call out the participant for giving only positive feedback, or pressure him to come up with negative feedback. Doing so may unnaturally bias the participant’s approach to the product during the session, and might ostracize your stakeholders.

 Make sure that participants know upfront that you want them to provide both positive and negative feedback.

 6.1 Participant is reluctant to say anything negative

 Does the participant actually need to know that he did not complete the task? As a moderator, you shouldn’t tell him when the task was “successful” or not. Instead, let the participant decide for himself when he is done. If he happens to do something wrong, you usually don’t need to point anything out and can simply move on. Just remember that if he gives subjective feedback or ratings, his performance may not be correlated with his rated experiences. Because he doesn’t know that he wasn’t successful, the participant may think everything is great and rate his experience very highly.

 If the participant does need to know that he wasn’t successful (e.g., if tasks are interdependent in such a way that requires a task to be completed before attempting the next one, or the participant never got to a key area of focus for your research), follow the recommendations in section 6.4 to provide an assist.

 “Okay, why don’t we move onto the next task?”

 Don’t make the participant feel like he did anything wrong or is being tested. Avoid saying things like “This task isn’t yet complete” or “Let me show you the correct way.”

 Participants will do this from time to time, so there is no way to avoid it! Remember that this type of feedback is just as valuable as seeing participants struggle to complete a task.

 6.4 Participant is not able to complete a necessary task

 If the observers are in the same room as you and the participant, you may need to take a short break so that you can ask them to pay more attention to the session. Remind them that the participant’s feedback is valuable and he needs to feel comfortable to provide it. A disengaged observer in the room distracts from this.

 Provide ways for observers to take ownership over the session results. Ask them to track the usability issues that they see during the sessions, and review their findings with them after each session. Post the list as they create it so that it is constantly visible for all the observers. For more ideas on how to involve your stakeholders, refer to It’s Our Research (Sharon, 2012).

 Encouraging engagement from remote observers can be more challenging. However, you can do something similar to the previous recommendation by asking each observer to keep track of the issues that they observe during sessions, and then running quick debriefs after each session so the observers can share their findings. If some of the remote observers are in the same location, recommend that they get a conference room so they can watch the sessions together and create a unified list of their findings.

 “How do you think the sessions are going so far? Have you seen or heard anything that was surprising?”

 “During this next session, I’d like you to keep track of any issues that you see, both positive and negative. We can then compare that list with mine and discuss further after the session.”

 Don’t lash out at the observers. While it can be very frustrating to know that your hard work is not being observed (or actively appreciated), recognize that most people are very busy and that it can be difficult to get enough time to commit to a whole session. Instead, try to make it as easy as possible for observers to become, and remain, engaged.

 If the observers will be in the same room with you and the participant, provide them with guidelines for their expected behavior prior to the sessions. See section 15.4 for more on this and providing ground rules to observers in general. For example, you may want to ask the observers to track issues that they see during sessions to encourage them to pay attention, and then you can review and compile results with them.

 Reach out to individual stakeholders before the sessions and personally invite them to attend. This can help get them excited and engaged in the research.

 9.2 Observers are loud and distracting