Chapter 4: Writing a Discussion Guide

It sounds obvious, but your research sessions will be better if you think them through beforehand. You do this by imaginatively playing through what might happen, plotting out how to achieve your aims, and weighing up different questions and activities you might use. Most of this planning takes place in your head, but it also has a physical product, which goes by the name of the discussion guide.

The research cycle: setup

What’s the Discussion Guide For?

A discussion guide has three jobs:

1. As mentioned above, the main purpose is to think through your approach. The process of writing a discussion guide forces you to do this.
2. Getting agreement from colleagues. A discussion guide is a great way to prompt debate among the project team and stakeholders. It’ll also help to make them feel involved and invested in the outcome. You’ll probably find that your colleagues have different views on priorities, and how best to tackle things. Allow for an iterative process, and when you’re finished, the agreed discussion guide can be signed off by your stakeholders.
3. A reminder of what to cover in your session. When you’ve finished your discussion guide, you probably won’t need to refer to it much in the session: it’s in your head. But it can still be useful to have a guide to refer to, if you think you might have forgotten something, or if you’ve lost your place.

What’s a Discussion Guide Not For?

A discussion guide isn’t a script. Think of it as a set of questions to answer, not questions to ask. Nor is it a rigid sequence that you have to follow, because you should allow for some improvisation depending on what the participant says or does. Think of the guide as the skeleton around which the real discussion is shaped.

Elements of a Discussion Guide

Whatever your research is about, the discussion guide will always include certain standard elements. In a depth interview, these elements will be topics: the aspects of the user and their world you’d like to understand better. In a usability test, they will be tasks: the product features or journeys you need to evaluate. The sequence of tasks or topics, plus the intro and outro sections that you shape around them, are the core structure of your discussion guide. So whether you’re in the discovery stage or the validation stage of a product lifecycle, the structure will be roughly the same:

1. Introducing the session (5 minutes of a 60-minute session)
2. Participant introduction (5-10 minutes)
3. Setup for the main task or topic (3-5 minutes)
4. Main task or topic (5-20 minutes)
5. Exploring the detail (10-20 minutes)
6. New versions or comparisons with competitors (0-10 minutes)
7. Summing up (5 minutes)

What goes into each of these sections? We’ve described them in more detail below, and later on in this chapter you can see how this varies for different kinds of project.

1: Introducing the Session

In this first section, you’re going to lay the ground rules for the participant, explaining what you need from them in the session, and answering any questions they may have. It’s also important to tell them about any recording or observation you’re planning to do. In the session, this is also the moment at which you’d give the participant a consent form to read and sign. It’s also a good idea to offer them a hot or cold drink at the same time.

Here’s an example from one of our projects:

• Meet and greet the participant in reception, then transfer to interview room once they’re comfortable.
  • Thanks for coming to our research session
  • It’ll take about 60 minutes
  • I’d like to record it, so we can remember what happened properly. Is that OK?
  • Some of my colleagues would like to watch, but we’ve put them in a separate room so they don’t distract us. I might go to ask them if they’ve got any questions at the end
  • Views shared and recording are confidential. They’ll only be shared anonymously, and only within the project team
  • Just to be clear: I don’t work for the company that’s made this website. That means that I’m neutral: I can’t take the credit for anything you see today. And I won’t be upset if you don’t like it, either
  • There are no right or wrong answers. I’m just interested in the way you see things
  • At the end of the interview, we’ll give you $60 as a thank you
  • Do you have any questions about what we’re going to be doing to?
• Don’t reveal the client identity at this stage

Try to make this section conversational and relaxed, rather than formal. Use the guide as a reminder, rather than reading directly from it, otherwise your introduction will feel robotic and awkward. Treat this as a checklist rather than a prepared speech, and try to get the participant talking using easy, closed, questions such as “Have you done this before?”, “What was it about?”.

Time: Around five minutes, but may be longer if the participant takes a while to relax.

2: Participant Introduction

The purpose of this section is to get the participant talking and feeling comfortable with the interview setting. It also provides you with some background information about their life and habits, which you can refer to later on. If you’re testing a fully-functional prototype or product, you can use this information to create scenarios.

For example, a participant has said they’re going to a friend’s wedding in a few weeks’ time, and they want to get some shoes to go with their outfit. We could use this scenario as the basis for one of our tasks later in the session, because it will be more realistic than a made-up scenario, and the participant will therefore engage with it more truthfully.

Here’s an example of a participant introduction section:

• Participant to introduce themselves in general terms:
  • What have you been doing today, before you came along to the interview?
  • Do you live locally, or have you had to travel a distance to get to the interview?
  • What’s your occupation?
  • Do you have any children?
• Once they’re warmed up, participant asked to tell the story of the last time they bought a pair of shoes:
  • What was the trigger?
  • What did you buy?
  • Where did you buy them?
  • Who did you buy them for?
  • What other options did you consider?
  • What stopped you from buying those other options?

In the first section, you’ll notice that these questions are the kind of small talk you expect if you were chatting to a stranger in any situation. There’s a reason we start with these kind of questions: people find them unthreatening and are very used to giving answers to them. Just as you would in a social situation, though, you should choose your questions to fit the person, and avoid reading from a script. For example, if you’re talking to a student, they will probably be used talking about their course, but a question about children might be inappropriate. It’s good practice to avoid questions that put participants under pressure at this stage. For that reason, it’s best to avoid asking about favourite websites or hobbies: these questions can make participants feel under pressure to invent an answer.

In the second section, we’re using storytelling, another familiar format, to put participants at ease and get them used to talking, while at the same time giving us useful information about past behaviour.

Time: 5-10 minutes. This is often the hardest section to judge. On the one hand, it’s best not to hurry if you feel the participant is nervous. If that’s the case, try to establish a rapport with them before you move on to the next section by focusing on easy questions and uncontroversial topics, otherwise you may find they freeze up in the most important part of the session. On the other hand, if they seem comfortable don’t linger too long: it’s easy for this section to turn into general, unfocused chit-chat, which means you’ll run out of time later on!

3: Setup for the Main Task

Before you go into the main task, you need to lay the groundwork correctly.

A Repeat Briefing

This is especially important if you’re testing a prototype. In this, you should explain:

• Not every link will work. Don’t worry about breaking it!
• It’s not a test of your internet skills, it’s a test of the thing in front of you. You’re here to get feedback for the designers, so don’t hold back
• There are no right or wrong answers.

It might feel like you’ve already said these things at the beginning of the session. Don’t worry. The participant may not have taken them in properly at the time, and it’s good to recap.

Setup Questions

You may want to establish a benchmark against which to compare the actual experience. For example:

• What are you expecting this thing to be like?
• Have you used this thing before? What do you remember about it?
• Tell me about the last time you needed to achieve this objective. This information will be very useful to refer back to later on, once your participant has tried out the thing you’re getting them to explore.

Time: 2-5 minutes. This is a short section, but it’s important.

4: Main Task

This is the heart of your session. Although it may only be a fraction of the total running time of your session, it’s crucial because it’s the point at which your participant will be responding most naturally to the topic or product. From here on in, it’s likely that they’ll become more and more familiar with it (and increasingly aware of what you want to know about it, based on your line of questioning). So make the most of it.

Because you need to make the most of this key moment, you’d be wise to focus on the task that’s most important to you. If your research is about testing an e-commerce journey, then you would get your participant to complete the journey now. If your research is about getting reactions to a new concept, then this is when you would do the big reveal. If it’s about understanding participants and their needs, then you would focus on the main area you’re interested in.

You’re aiming for as natural a response as possible, so it’s best to dial back on your questioning at this point. If you’re testing a product, get the participant to think aloud, and try not to interrupt them. If you’re getting a reaction to a new concept, keep your questions extremely open-ended, such as “What do you think?”, or “What do you make of it?”. Once they’ve reviewed the thing you’re interested in, you can ask them to summarise, as in the example below:

• Having completed the task, participant to talk through:
• How did you get on?
• How did you feel the site performed?
• What was easy / difficult?
• How clear was it?
• What (if any) were the points where you had to work harder?
• How easy was it to make a decision about which product to go for?
• Were there any points where you were uncertain what to do next?
• Were there any points where you were worried about what would happen?
• What feedback would you give to the website’s designers?
• How ‘Shoestore-like’ did it feel?
• What impression did it give of Shoestore?

Time: It depends on the length of the activity you’re focusing on. If it’s a concept test, it might only be five minutes. If it’s a full e-commerce journey, it could be half an hour.

5: Exploring the Detail

Once your participant has been exposed to the prototype / concept / main topic and given their initial reaction, it’s time to explore it in more detail. This is where the session can be most unstructured, because the order in which you tackle things may depend on what exactly happened in stage 4. So rather than have a rigid order of questions, we’d recommend creating two lists: tasks or activities to use, and areas to cover.

• Tasks or activities to use. These are different ways to get your participant to explore the topic or product, especially the parts that they might not otherwise have discovered. For example, you might set them a task to cancel an order in their account section. They might not otherwise have chosen to do this, so you need a task that will nudge them in that direction if you want to understand how the account section works.
• Areas to cover. These are questions or rolling hypotheses that you’re interested in. By listing them here, you’re reminding yourself to get answers, whether it’s through spontaneous actions or comments of the participant, through a task that you set them, or by probing with questions.

Another reason why it’s a good idea to keep an open mind about the order in which you tackle the tasks and areas in this section is that you can then tackle them in a different order with each participant. This means you’re reducing the bias from one question preceding another (AKA the ‘order effect’). Also, you can tailor your approach to each person, making the session feel more realistic.

You can see how this approach works in the example below:

Tasks & activities

• You’re going to visit your cousin in Canada, and you need to buy a pair of snow boots. What would you expect to spend in a situation like this? Now, try to find a pair for that price.
• Your friend has recommended a brand of snow boots called Aggi. Try to find them on the Shoestore website.
• Your trip to Canada has been called off. Try to cancel your order for snow boots.
• You want to leave a Google review of the Shoestore website. What would you write?

Areas to cover

• Homepage
  • What draws your attention?
  • Who does it appear to be aimed at?
  • Is it clear what Shoestore’s offering is, from this page?
  • Is it clear what you can do here?
  • How can you move on from this page?

• Category page
  • Reactions to the product range shown on the page
  • What sense of Shoestore’s strengths / offering / style / range do you get from this page?
  • To what extent do you engage with ‘Shoe Discovery’? How does this compare to ‘Shoe Rack’?
  • How easy is it to move on from this page?

• Search / product listing page
  • How easy is it to find the search function?
  • How well does this work?
  • How do you feel about the presentation, sort order, abundance of results?
  • If you wanted to change any of this, how would you go about it?
  • How easily can you filter the results by size or any other important facet?
  • How do you feel about the amount of information about each product?
  • How well supported does this feel on mobile, eg, predictive, filtering, subcategorization, null results?

• Account section
  • How easy is it to find the Account section?
  • How clear is the status and delivery details?
  • How easy is it to cancel an order? How confident are you that the cancellation is confirmed?
  • How easy is it to change a password?

Time: 10-20 minutes. However it’s a good idea to include more tasks or activities than you need, so you have a selection to choose from depending on how the session unfolds. So we’d recommend writing enough for 20-30 minutes, and cherry-picking the ones that are most useful to you.

6a: New Versions

If you’ve spent most of the session looking at an existing product or a relatively complete version of a new product, then towards the end you’re in a position to show some new variations. The reason for doing it here is that participants will have enough knowledge from their earlier activities to be able to put isolated pages or lo-fi sketches in context. If you do it earlier, it’s harder for them to envisage how they’d work.

Time: 5-10 minutes.

6b: Comparison

If there are no new designs or concepts to look at, we can use this part of the session for other purposes. One of the most common ways to do this is to look at one or two competitors’ versions of the same journey and see how they compare. This is useful because a) it’s easier for participants to evaluate your product if they’ve got other examples to refer to, and b) you may see how alternative approaches to the same problem work, before you adopt them yourself.

Time: 0-10 minutes. This section is generally the most expendable of all, which is one of the reasons why we do it virtually at the end of the session: if you’re running out of time, it’s ok to ditch it.

7: Summing Up

In this section, you’re asking the participant to summarise their experiences and reactions. It’s unlikely you’ll learn anything new, but it’s an important section for a number of reasons:

• You get a sense of priorities and proportion. For example, your participant may have encountered several issues over the course of the session, and now you can ask them to say which felt like the most important.
• You can ensure your observing colleagues feel involved in the session, by asking them for any additional questions.
• You will often get your best quotes here. In fact, you can ask participants to give a quote ‘direct to camera’, which is useful for your outputs.
• It’s also important to give the participant a sense of closure, so they understand that the session is over and their contribution is complete.
• You can ask if the participant has any questions for you.

Different Kinds of Session

Every research project is different. While the core structure we’ve outlined above works for any kind of one-to-one session, there are variations in the detail depending on whether you’re running discovery research (eg, understanding someone’s life and needs) or validation research (eg, testing a product’s usability). You can see how these differences play out in the table below:

Discovery	Evaluation
Introduction	Introduction
Warmup subject	Context of behaviour
Focus on main journey or question	Big reveal or main task
Explore context (e.g. look round home)	Follow-up tasks
Wrap up	Wrap up

Questions, Tasks, and Activities

There are a few problems with interviewing users. Firstly, they’re not always great at telling you what they need. On top of that, they struggle to remember what they’ve done, they aren’t always honest about what they think, and they don’t always do what they say they’ll do. These problems are the main reason why we try to rely on observable behaviour, not people’s opinions. In fact, given all of these problems you might wonder why we bother asking users at all. The reason is that it’s often our only way of finding out (if, for example, we’re talking about something that happened in the past). And it’s also our only insight into why something happened: observation alone can’t tell us about user’s perceptions or motivations. So we do need to ask questions, but we shouldn’t just take the answers totally at face value.

The best research uses questions and observed behaviour in combination to try and get around these problems (an approach we referred to as ‘triangulation’ in Chapter 1). We compare and contrast data from these two complementary perspectives, to get around the blind spots in each. So when it comes to writing your discussion guide, you should plan how you’ll use questions and activities together. For your most important questions, it’s a good idea to approach them from several different angles, using a mix of more direct and more playful questions, tasks and creative activities. Each of these will give you a different perspective, and in combination they’ll be much more insightful (and less misleading) than any single approach.

For our Shoestore research, we’re interested in how parents buy school shoes for children. As it’s an important question of our project, we’ll approach it in several different ways, at different points in the session:

• As a question: “How do you buy school shoes for your children?” The weakness of this question is that participants will tend to simplify and generalise, giving explanations that are overly-rational or that paint them in a good light. It’s useful to ask, but we need additional perspectives.
• As a question: “Tell me about the last time you bought school shoes for your eldest child.” This question is more specific, and therefore less prone to bias. However, the example we’re asking about may have been atypical, and it doesn’t leave room for the participant to give a broader sense of how they approach this task.
• As a task: “You’ve got $40 to spend on school shoes for your eldest child. Go and shop for a pair online.” This is the behaviour we’re interested in, however it lacks context. For example, their child may not currently need shoes, or they may not be using their normal device.
• As an activity: “Imagine you had to write a blog post advising parents how to shop for school shoes online. What would you write?” This gives an insight into the participant’s perceptions and knowledge. However, it risks being too general, and ignoring real-world constraints like budgets.

None of these approaches on their own gives the entire truth. Taken together though, they add up to a much richer picture of what participants are doing, and why. Additionally, by exposing contradictions, they enable you to raise probing questions that the participant might otherwise dismiss. For example, you could observe: “When you were talking about your approach to buying school shoes, you said that you always went for the cheapest pair, but when you were using the website you spent quite a lot of time looking for more stylish pairs that weren’t the cheapest. It feels like cost isn’t the only thing that’s influencing your approach. What do you think?”

Types of Content

As you have seen, a discussion guide is constructed of several types of content, arranged within the skeleton structure we described earlier. Let’s look at each of these in more detail:

Tasks

Tasks are the tool we use to be able to observe users’ behaviour. By asking users to perform a task, we’re directing them down a broad path of action, so we can sit back and watch the options they choose, what works and what doesn’t.

Tasks can range from the very basic:

“Buy a pair of shoes on the Shoestore website.”

To the very elaborate:

“Imagine you’re going to start a new job next Friday. You’ve just found out that the dress code at your new workplace requires all staff to wear unbranded black shoes. Because your job involves standing on your feet for most of the day, you need your shoes to be as comfortable as possible. You have $60 to spend, and you’re only going to be at home on Tuesday. Buy a pair of shoes that are suitable for your new job.”

The difference between the two is that the first makes no assumptions about how or why the user should be buying the shoes, whereas the second includes a lot of extra detail. We call this extra detail the scenario. Knowing how to write a good scenario is the key to creating effective tasks. You need to strike a balance between giving enough information, but not too much.

While the second version above looks more realistic, there are actually several problems with it:

• It’s overly-prescriptive. It’s leading them towards a particular course of action or solution. For example, by mentioning that they’re only going to be home on Tuesday, we’re suggesting that they would prefer to get the shoes delivered rather than collect them.
• It may be unrealistic. The participant may be used to working at a desk, and so might find it hard to identify with the role in the scenario where they have to stand at on their feet all day.
• It’ll be hard for the participant to remember all this information while they complete the task.

On the other hand, the first example can feel too broad. It’s very unlikely that someone would go to shop on a shoe website with absolutely no context: no trigger, no sense of budget or urgency, and no reason for choosing to buy shoes.

The solution lies in three principles:

• Wherever you can, use scenarios that are realistic to the participant. If they’ve got a new job coming up, ask them to buy shoes for the job. If they’ve got a trip to the ski slopes coming up, ask them to buy snowshoes. By taking this approach, the participant brings realistic context with them, rather than you having to impose made-up context on them.
• Provide just enough context to set the objective, but don’t specify the route to get there. “You’re going to a job interview next Friday, and you need shoes to go with your suit.”
• Have the rest of the context ready, but don’t share it unless you need to. This involves creating a more detailed backstory around the task, but only providing it if the participant is unable to come up with their own details. For example, if the participant asks, you might invent the colour of the suit they’re going to wear, the nature of the job, or their address.

Protocols

When you’re asking participants to complete a task, you need to give them guidance on how to verbalise their thoughts. There are a number of different ways of doing this, which we refer to as protocols.

The Thinking aloud protocol asks the participant to complete the task while giving a running commentary on their thoughts.

• Pros: The most generally useful protocol, which gives a good balance of observable behaviour and insight into that behaviour.
• Cons: Greater cognitive load for the participant, which can make it harder for them to complete the task itself. Participants can also drift into silence.

The Silent completion protocol asks the participant to complete the task in silence, unless they want to ask a question.

• Pros: More realistic and less unnecessary cognitive load for the participant.
• Cons: Little insight into how participants perceive the product, their decisions or behaviour.

The Reconstructed thinking aloud protocol aims to bring together the best features of the above two protocols. In this version, participants complete the task in silence, but are then shown the recording of their journey and asked to narrate their thoughts and decisions.

• Pros: Realistic cognitive load, more thoughtful reflection on the process.
• Cons: Complex and time-consuming to set up. Also, if they have to explain their behaviour retrospectively, participants might want to choose explanations that flatter their choices, rather than the real reasons.

The Deconstruction protocol involves the participant as a critic, rather than a user. This is particularly relevant when the person you’re talking to is an expert on the subject or the product you’re using, for example if you’re testing enterprise software.

• Pros: Can be more engaging for expert participants, leading to more useful feedback.
• Cons: Misses out on some of the detailed usability feedback you’d get from other protocols.

It’s a good idea to change protocols through the session, as the participant becomes less natural in their responses and more used to the interview setup. So you might start with silent completion, before moving on to thinking aloud protocol and perhaps even deconstruction protocol at the end.

You also don’t have to stick to the same approach for each interview. Over the course of a day, we sometimes use silent completion protocol for the main task in the first couple of sessions, then use thinking aloud protocol thereafter.

Questions

We’ll cover how to ask questions in much more detail in chapter 7. At this point, it’s just worth mentioning that your bigger research questions and hypotheses should definitely be included in your discussion guide. You may also want to add some possible probes and follow-up questions, to help you think through how to approach each topic. Also, after each session you will be updating your rolling hypotheses, and you may want to update the guide to cover these.

Activities

Activities are another way to get an insight into participants’ decision-making and behaviour. Unlike tasks, we’re not asking them to use the product we’re interested in; instead, they tend to be more lateral, often using playful or creative approaches.

A word of warning. While activities can much more energising than asking questions, and often more insightful too, they have drawbacks:

• They can be time-consuming, both in terms of setup and within the session itself.
• Not everyone is willing to engage with them. In some settings (in research with business people, for example), creative activities may be inappropriate. In other cases, openness to activities is down to the individual, and you will need to use your judgment in the session.
• The outputs are harder to interpret than questions or tasks

Don’t let these problems put you off, though. Often activities are the highlight of a research session, and generate the most engaging outputs. Here are some examples, but there are many more:

• Card sort adjectives to describe the website you’ve just used.
• Take a print advert / bill / screengrab from the product website, and rewrite it in red pen.
• Using a highlighter, mark up any parts of the content that don’t make sense.
• Using Post-It notes, create a timeline of your most recent process of buying new shoes.
• Imagine the Shoestore website as a person. What job would they do? What kind of house would they live in? What would they dress like? What would they be like if you met them?
• Create a collage from magazines to illustrate how buying shoes makes you feel.
• Write a pretend Amazon review of the website you’ve just used.

How to Write a Discussion Guide

When you start writing a discussion guide, it’s tempting to piece together sections from previous guides to save time and effort. Resist the temptation, and always start from scratch. The reason it’s important is that if you don’t write the guide yourself, you’ll miss the vital process of imagining and internalising your approach. You’ll have created a long document, but you won’t have done the actual work of thinking through how you’re going to approach the research.

Before you starting writing the detail, plan out your approach at a high level:

1. Write out your main research questions or rolling hypotheses. Remember, you don’t need to worry at this stage about how you’re going to ask these questions, you just want to be clear about what your priorities are.
2. Check how long you’ve got, and divide up your time into rough sections using the approach described earlier in this chapter.
3. Identify your main task, activity or topic. This will make up section 4 of your guide.
4. Think about your secondary activities and areas to cover. These may be different angles on your main questions, or other aspects of the experience that you’re interested in. These will go into section 5.
5. If you’ve got any new designs to include, these will go into section 6. Otherwise, create a list of competitors or other products you’d like to compare.
6. Think about the context that will be useful to understand about the participant and their relationship to the product. This will form the basis of section 2.

Once you’ve sketched out your approach at a high level, you can go through and add in more detailed questions, activities and tasks to do the rest. Remember to approach your key research questions from several different angles, as described above.

A moment ago, we said that you should always start your discussion guide from scratch. The exception to this is section 1. Because the items on this list should almost always be the same, you can copy this from a previous guide, while remembering to check if there are any changes for the project you’re currently working on.

Use Your Imagination

When athletes are training for a competition, they spend time envisaging it beforehand, mentally pre-living each stage of the race and preparing for how they’ll react when the time comes. You can use a similar process for writing a discussion guide. Go somewhere quiet, and think through each part of the session. How do you expect participants will respond to your questions, tasks and activities? How long will it take? What kind of guidance will they need? By anticipating what’ll happen, you can avoid being caught out, and plan for different eventualities.

Summary

• Discussion guides have several jobs, but the main one is to help you thinking through the session beforehand.
• Structure the outline of your session first, focusing on the main questions, tasks and activities you want to use.
• If you include more material than you can use, you’ll have the option of tailoring some aspects of your session to each participant.
• Don’t treat it as a script, but as a flexible framework you can depart from when you need to.
• Use your imagination to think through how the session might unfold, so that you can plan for different outcomes.