User research can significantly mitigate an organization’s risk—but only when the studies are run correctly and insights are accurate. Unfortunately, user research is prone to error. Organizations need an effective system of governance to ensure research quality. This play lays out six attributes that will help you gain reliable insights and drive real value for your users and business.

Who Are the Key players in the Research Quality Play?

ROLE	WHO’S INVOLVED	RESPONSIBILITIES
DRIVER	User researcher	Design experiments; Produce relevant, reliable, and unbiased insights.
CONTRIBUTOR	Experience practitioners	Provide research questions; Create research stimuli; Share accountability for relevant, reliable, and unbiased insights.

THE HOW

To run an effective research quality play, you need to be mindful of:

1. Talking to the Right USER

Insights must always come directly from the right users. These users can never be replaced by a proxy (an SME, support rep, sales rep, PM, or CEO), regardless of how well they may know them.

Recruit users (Chapter 26: “Research Recruitment Play”) who most accurately represent the group of people for whom you are solving a problem. More specifically:

• They have the specific needs that your product or service solves;
• They are in the specific role/industry for which your product or service was made;
• They have deep familiarity with the various contexts in which your product or service is most applicable.

2. Focusing on the Right QUESTIONS

While there’s no such thing as “too much data” in user research, there are such things as relevant and irrelevant data. Think about the desired outcomes of your research, and define what insights you need in order to make decisions that will lead you to this outcome. In other words, identify which research questions (RQs) are most relevant. Be mindful of what phase of product development you are in when you identify your RQs.

If your desired outcome is to understand how to improve the next iteration of your network security product, some relevant research questions might be:

• What parts of the design are helpful/hindering to the user?
• What parts of the design are more/less intuitive to the user?
• What parts of the design are most/least valuable to the user?

While some irrelevant research questions (in context of network security) might be:

• What kind of social media platforms does the user use?
• What are the user’s retirement plans?
• What is the user’s marital status?

3. Choosing the Right METHOD

While there are many right methods you can use to get to the specific insight, you need to mitigate the risk of picking the wrong methods. Different research methods are best suited for answering certain types of questions or gathering certain types of data, so using the wrong research method can result in misleading insights and increased churn. For example if you are looking to understand what journey and problems a physician faces while using a surgical product, the right method could be leveraging the contextual inquiry or fly‐on‐the‐wall methods, but a survey would not give you similar quality of insights. Choose the method that best fits the needs of your study by evaluating key variables such as the current phase of the project, what you want to do with the insights, and the types of questions you want to ask (Chapter 25: “Picking a Research Method Play”). This will help achieve your research goals in the best way possible.

4. Carrying Out Your Method with the Right EXECUTION

Research is a rigorous craft. Any variance in the execution of your study can potentially skew your data and lead to false conclusions.

When you conduct your studies, maintain consistency and avoid making any last‐minute changes.

• Ask the same sets of question to all participants;
• Present all participants with the same prompts;
• Show the same stimulus to all participants;
• Give all participants the same amount of time to complete tasks.

Conduct a pilot study at least two days prior to your first participant session to iron out any issues in the script or stimuli, and build in at least one hour between each session so you can have enough time to debrief and prepare for the next session.

Be mindful of how you frame your questions to your participants. How you ask your questions can affect what your participant says in response. In order to generate the most real and accurate responses, avoid asking:

Leading questions, which may influence users to answer in a way that confirms a preexisting assumption, opinion, or desired response.

• Leading question: “Would you say this screen feels cluttered?”
• Non‐leading question: “Can you describe anything you may like or dislike about what you see on this screen?”

Dead‐end questions, which expose the what, but not the so what or why. Try asking questions that start with “What,” “Why,” “How,” or “Can you show me/tell me about a time…”

• Dead‐end question: “Is this feature valuable?” (Yes/No, or dead end)
• Non‐dead‐end question: “How would you describe your experience with this feature?”

Dead‐end questions that are not grounded in historical or behavioral data.

• Hypothetical question: “What kind of features would you want to see listed when you evaluate a car?”
• Non‐hypothetical question: “Can you evaluate these two cars and walk me through your decision‐making process?”

5. Finding Accurate Insights with the Right SYNTHESIS

Data interpreted correctly will lead to valuable, actionable insights. Misinterpreted data will lead to incorrect insights, which can lead to significant risk for your business.

Research findings can be misinterpreted in many ways:

• A single or statistically insignificant data point may be over‐amplified as an insight. Avoid this by objectively evaluating the statistical significance of the data.
• Hypotheses are stated as facts when they are not backed by reliable sources. Avoid this by documenting and citing supporting data for each insight and by maintaining a running list of hypotheses and facts.
• Observations, or what a user did or said, may be used to make knee‐jerk product decisions. Avoid this by asking yourself, “Why did they say/do this, and why is this important to know?”
• Biases may affect the direction of the synthesis, so the product designers should not be the ones running the tests. Avoid biases by looking at the data objectively and seeing where patterns naturally emerge, instead of searching for data points that specifically support predicted insights.

6. Socializing Your Insights with the Right PRESENTATION

The purpose of conducting research is to empower everyone in the organization to act on the findings. To do this effectively, insights must be presented in a way that is registered with different audiences; otherwise, all prior steps in the quality checklist will be in vain.

Most researchers will compile all the findings from a study in a comprehensive report, but the reality is that most stakeholders will not read the report. And if no one ever reads the report, does the research even drive any value?

What can you do to make sure your research insights are heard?

• Make the report relatable and consumable: When most people hear “research report,” they think “lots of pages of text, numbers, and graphs.” There’s no rule against using playful language, fun graphics, or even enticing animations in your artifacts! The takeaways of research should be memorable and attention‐grabbing, and a big part of that is the actual presentation—the “skin,” if you will—of the finding.
• Tailor your artifacts to the audience: Different stakeholders have different perspectives, goals, and areas of interest, which is why the research insights you present need to be tailored to match these varying needs. Think of these stakeholders as your different “users,” and design your communications and research artifacts so they are valuable and easy to comprehend for each of these users.

• Include user researchers in the design process: The most important part of user research is making sure the insights are acted on in the right way. Ask your user researchers to contribute to ideation sessions and design reviews. This way, insights can be reiterated throughout the product development process, ensuring a higher likelihood of delivering a product that meets the user’s needs.