“Supposing is good, but finding out is better.”
—Samuel Clemens
Selecting Techniques for Your Usability Test
Analyzing and Presenting Usability Test Results
After you design your software, hardware, or Web user interface, it’s time to put your interface to the test by letting users preview it and provide feedback so you can make changes as needed before you release it to the public.
You need to start by selecting techniques for your usability test. You will run into resistance, and this chapter discusses the different questions you may be asked and gives you some answers that you can give in response.
When you know what technique you want for your usability test, you need to define your test to determine what information you need from the users and how you will gather that information.
After you design the initial test, you need to conduct a pilot test that will allow you to hone your observational and interviewing skills. When you’ve worked out the kinks, you’re ready to conduct the real test.
When the test is over, it’s time to crunch the data and prepare a report and presentation for your stakeholders that tells them what you found and gives them your recommendations for improving the interface and the overall product.
You can use several techniques for conducting your usability test (Hackos and Redish, 1998). Depending on your situation, you can use one or all usability techniques to get the information you need to make your product or documentation better.
One of these techniques is to go onsite to visit the users in their natural habitat. You may run into resistance when you propose a site visit, especially if the trip to the customer costs money. If you do encounter the following questions (Hackos and Redish, 1998), which are fairly common, you can counter with the following arguments:
As I said in Chapter 3, “Making the Business Case,” be sure to perform a return on investment (ROI) analysis, and include any site travel information in your cost estimate.
When you observe users in their natural environment, you should adhere to the following rules as you plan for any type of site visit (Hackos and Redish, 1998):
If you also bring this information—particularly a plan for the site visit—to your decision maker, it will help make your case even stronger.
There are other methods of user interaction that you can employ either in place of or in tandem with site visits (Hackos and Redish, 1998). These methods, which will be familiar from related concepts in earlier chapters, include usability evaluations away from the customer site as well as more traditional marketing techniques:
Each of these methods has drawbacks, generally falling into three categories: bias for lack of adequate feedback, lack of information provided by users, and misunderstandings. These misunderstandings are caused by confusion, miscommunication, or not being able to see how the users actually use the product.
There is no one best way of conducting a usability test. However, observing and engaging users onsite have been shown to be the most effective ways of gathering usability information from users (Hackos and Redish, 1998).
The first step in the testing process is to define and plan your usability test. If you don’t know how you are going to test and what you are testing for, then you will be wasting the time of a lot of people—not to mention the company’s money. You should keep good written records of what you are testing, the responsibilities of project and usability team members, and the decisions the team makes.
—Dumas and Redish, 1999
Dumas and Redish (1999) identify five tasks that you must complete as you define your usability test:
The following sections describe the tasks to complete (Dumas and Redish, 1999).
After you have determined who your users are, you have to make choices when you create your test—for example, whether you want your usability test to be geared toward advanced users or the majority of users classified as intermediate. You create your goals by starting with general goals, and from there you build specific goals. These goals can come from several sources:
You must be choosy when you determine who you want to participate in your usability test. When you create a persona, as you learned about in Chapter 6, “Analyzing Your Users,” you’re determining the characteristics that you want each user in your test to fit into. You need to think about two types of characteristics: those that all users share, and those that may cause differences between the users. Following are the decisions you need to make when determining characteristics:
You should think broadly about your users when creating profiles. Following are some examples of thinking broadly:
From here, you can create groups and even subgroups of users who share the same characteristics so you can, for example, learn if there are differences between subgroups toward a new feature in your program.
Unfortunately, you can’t test every possible task that the user could do with the product. So how do you narrow it down? Use tasks that
As you select tasks, you must also keep in mind how long the task will take for the user to do and what hardware, software, procedures, and other information are needed for the user to do the task. You should write down your tasks by giving each task a number and description (just as you did with paper prototyping). Each task should show the time it will take, the hardware and software needed, and the high-level instructions and procedures required to complete the task.
You can use a scenario to tell participants what you want them to do during the test. A scenario describes the task in a way that helps bridge the task (which is artificial) with what the user would be doing in the world. For example, “You have three new hires. Add accounts for them.”
A good scenario has the following characteristics:
Your tasks and scenarios don’t have to be written. You can have human actors playing different roles, such as customers, support staff, or supervisors. You can also have the participants stop between each task, such as after a longer task or if you want to distribute a printed questionnaire to all participants after each task. However, you must provide audio cues to tell the participants to stop and start again because participants may become focused on the task and won’t remember when to stop and start.
Usability measures two dimensions:
In the case of performance measures, you can easily log each time a user exhibits a certain behavior during the test, like expressing frustration. Subjective measures are harder to quantify unless more than one participant tells you the same thing, such as that the email button is hard to find on the page. There are commercially available programs for logging usability data, or you may want you or a programmer on your product team to create a program that meets your specific needs. If you can’t use a computer-based program, you can create a printed form to use.
As you create your logging form, you need to set criteria for performance measures. A typical criterion for performance measures is a four-point scale, which forces a choice toward positive or negative because there is no strictly neutral response. This four-point scale, in fact, has three passing grades and only one failing one. You must also set performance measures that are directly tied to your general and specific concerns. For example, if you’re concerned about how easy it is for a user to read a message, some of the measures you may want to add include the time it takes for the user to perform the task and the time it takes for the user to recover from errors.
You’ll want to follow the same performance measures for most tasks in the same test whenever possible to get a good idea of how users perform. However, different tasks within a test may require different performance measures. For example, a function that is available in one Web page may not be available in a sublevel Web page, so you wouldn’t log errors for that function in that sublevel page.
You may also have to take into account the test situation, such as whether the participants have to read the instructions for each task. If you’re testing the time it takes users to complete a task, you have to add to the total time it takes the users to complete the task to account for the test situation. For example, you should add 30 seconds to the beginning of the test so the testers have enough time to read and absorb the task instructions.
Before you test, you must prepare legal forms for the treatment of human participants. As the tester, you are responsible for the following:
You should consult with your company’s legal department or attorney (if possible) to produce these forms and possibly present them to your participants. If you are required to explain and present these forms, do so in a neutral but friendly tone.
You should also have a testing script so that you test all users in all groups the same way. If you remember standardized testing from high school, you’ll remember that all the teachers followed the same script to ensure that everyone was tested the same way so as not to skew the results and to make sure that all tasks were completed at the same time. The script should also include a checklist so you know that everything has been completed. If you have other team members with their own checklists, you must ensure that they have completed their checklists.
You may also want to distribute written questionnaires before the test, after each task, or at the conclusion of the test to get the following information from your users:
Written questionnaires are useful and efficient because you ask all participants the same questions, and you don’t forget to answer the questions. However, you must ensure that all the questionnaires ask the right questions so that you get the most effective answers. For example, if you want to ask a question about the difficulty of completing a task, it would be more effective if the question asked participants to rate the difficulty on a scale from 1 to 5 (5 being very difficult) rather than being close-ended.
It’s time to assess your preparations by first conducting a pilot test to see how well it works. After you have conducted the pilot test, you need to learn how to take proper care of your test participants before you start your actual usability test.
You should conduct a pilot test before you conduct the real usability test (Dumas and Redish, 1999). A pilot test allows you to “debug” your test and find out if there are any initial problems with the product or Web site you’re testing, its documentation, its test methods, and its test materials. Following are bugs you can encounter during the pilot test:
Always conduct the pilot test exactly as you will conduct the full usability test, and use one test participant who represents the users you want to test. By mimicking the same conditions in the full usability test, your pilot test will give you the most accurate results. The pilot test will also let you test the way you approach your users.
To give yourself enough time to make any necessary changes, schedule the pilot test two days before the live usability test. That will give you a full day (and perhaps longer if you schedule the pilot test in the morning) to make any changes without feeling the pressure of an immediate deadline. If the pilot test exposes problems that require more substantive changes, you can also determine whether to escalate the issues.
From your pilot test, you will get clues that will help you hone your skills, especially if you’re going onsite at a customer’s location to view how users work and use your product. Many factors go into a successful site visit (Hackos and Redish, 1998). Before you go to the user site, keep the following in mind:
When you arrive, do the following:
Do the following while you are onsite:
When you leave, do the following:
After you leave, be sure to send a thank-you note to the users and the managers.
When you take notes as an observer, write them on a form that ensures that you capture the important information about what the users are doing and that you answer the question you have. Although the form should be specific to your usability needs (and perhaps customized further to meet the needs of the users you’re testing), it should include the following:
You should also write down inferences and questions about the users and the task during the observation. Ask questions during the observation so you can get as much information as possible.
You can interview the users as they are performing the task, but you can also determine both from the users you’re interviewing and from the pilot usability test what interview methods and skills are best for your site visit. In addition to obtaining information while the users are performing the task, which is called a concurrent, contextual interview, you can also perform one or a combination of the following types of interviews (Hackos and Redish, 1998):
No matter which interview process you decide to use, you should always keep three things in mind when you interview:
Within this overall three-point philosophy about interviewing, there is a set of fundamental skills you should adhere to so you can get the most out of your interviews:
Sometimes your interviews may require you to create an ongoing relationship to track the progress of a product or document. As users progress from beginner to expert, they go through several stages (Kuniavsky, 2003):
There are a number of methods for obtaining usability information over a longer period of time (Dumas and Redish, 1999):
You need to ensure that you take care of your test participants. When you start the test, you need to ensure that your testers are comfortable and that you’re calm and focused on them. A checklist can help keep you and your testing team on track and ensure that you create a rapport with your testers from the beginning (Dumas and Redish, 1999).
Some of that rapport can include small talk and having the testing staff and testers introduce themselves by providing information about their jobs, their organization, and what they want to get out of the test. You should also talk with participants about the environment, and if you have a videotape, show it. If you have a testing room either at the user site or your own, you should show users the room and introduce them to any monitors who will be watching them and working with them throughout the test.
As you go through the test, remind your testers to think out loud whenever possible (Dumas and Redish, 1999). Thinking aloud helps focus the testers’ thoughts and helps them understand what they’re thinking. The success rate for thinking aloud can vary because some testers are more willing to share their thoughts than others.
Sometimes testing can go awry. Following are some common situations and what you can do in response (Dumas and Redish, 1999):
Note that if you’re compensating testers, you’ll have to determine how to compensate any who leave on a case-by-case basis.
During the test, you should always observe problems and create a problem list. Those on your testing team should also create a problem list because everyone has a different perspective on what’s happening. Write down your observations, your hypotheses about the actions you observe, and your interpretations. Keep your observations as neutral as possible, and record all user problems. That way, you’ll get a complete list without discounting anything. After you write down all your problems, you may want to discuss some of them with your testers to get more information. After the test, you should talk with your testing and project teams about what you found.
Some of the information in this chapter repeats what has been covered in earlier sections and chapters, but now you should see how all the information fits together so you can conduct your usability test. After you complete your usability test, you must analyze and present your data and then recommend a plan of action, as we’ll discuss in the next section.
A usability test generates a lot of data that you need to go through (Dumas and Redish, 1999). After the test, your data can include one or more of the following:
The first step in analyzing the data is to tabulate and summarize quantitative data. This is something you can do with any spreadsheet program. You can also compile all the comments into a word processing program. More powerful software programs such as the ones found in Microsoft Office let you link your spreadsheet in Excel to the document in Word. When the spreadsheet is updated in Excel, it is automatically updated in Word.
After you have entered all the data, you can analyze it for trends and surprises. Spreadsheet programs are also useful in calculating statistical information about the data, such as the mean score for a question in a questionnaire. However, as Mark Twain said, there’s always the problem of “lies, damned lies, and statistics,” especially when it comes to inferential statistics. Inferential statistics take a sample from a larger set of data and make inferences about the larger data from the sample. This approach contrasts with descriptive statistics, which describe a set of data, like the average time it takes to complete a task.
A useful technique for processing data is triangulating it (Dumas and Redish, 1999). This involves looking at all the data together to see how each set of data supports the other sets. Each apex contains a different set of data:
You measure the data against your usability goals and the quantitative criteria you set before the test to determine what the problems are inside the triangle.
You may find some surprises that warrant further research. For example, you may find that one user had different reactions to several questions. Perhaps that user felt that performing a task was a lot harder than the other respondents thought. Because the number of users in a usability test is small, you should always treat this outlying data seriously. If the outlier may represent a large group of potential users, the data may suggest that you need to schedule another usability test with more users like the outlier to see if the problem is with that set of users or is confined to that one user for some reason.
Dumas and Redish (1999) recommend that you adhere to the following guidelines to make statistical analysis as relevant as possible:
Both your quantitative data analysis as well as the qualitative data from feedback and notes will help you organize the information into two areas (Dumas and Redish, 1999):
Research is not an exact science, and there is bias in all facets of research. When you present your report, be sure to start with two questions in mind (Kuniavsky, 2003):
By addressing these questions first, you can minimize any issues and not only help the data become clearer, but also lend weight to your arguments.
The type of report you create depends greatly on the audience. You may be presenting to one or more groups within your company, so your report language needs to be tailored for one or more of these audiences (Dumas and Redish, 1999):
In addition, you should determine what format these audiences expect. You may need only a Word file attached to an email message and send the message to the appropriate people, or you may give your presentation at an executive board meeting and therefore need a printed paper report to present to board members. If you want to ensure that your format is acceptable, send a draft to the intended recipient(s) and get feedback.
After you have established the formats, you need to categorize the report’s information in the most effective manner for your audience. You may also want to produce several versions of the report depending on your audience (Dumas and Redish, 1999). For example:
Write your report in newspaper style—that is, structured like a newspaper story where the first sentence contains the most important fact, and the least important information is saved for last (Dumas and Redish, 1999). Break out the report into several sections or chapters, such as these:
Begin the report with an executive summary that summarizes the information in the report on one page so people who don’t want to read the report can get a broad idea of what’s in the report and what your recommendations are.
When you get ready to give your presentation, adhere to the following guidelines for making that presentation successful (Dumas and Redish, 1999):
After you set up your presentation, practice. It’s best if you can practice in front of someone else, especially someone who is similar to your audience members.
You may not be able to give a formal presentation. What’s more, some people may never read the report because they’re too busy. The development of multimedia technologies has made it easy not only to create a video presentation, but also to publish that information in a streaming video file that you can attach to an email message or post on the company intranet. For example, when I worked as a contractor at Hewlett-Packard (HP), I created streaming audio and video files using software that was available for $50, which was within the manager’s discretionary budget for his department. The manager liked the production so much that we placed the file on our group page within the HP intranet.
There are advantages and drawback to making a highlight tape or streaming video file (Dumas and Redish, 1999):
When you create your video, write down your plans using the following criteria:
Note that you may also need to buy hardware and software to produce your video, which could be another impediment. If you already took videos of your usability test and you have the video equipment to produce video and audio recordings of yourself or another person as the narrator, chances are that your company has the hardware and software you need to produce the video.
After you’ve imparted your information to the project team, how do you turn that information into action so you can improve the usability of your product and the process? You can be most helpful to managers, developers, and other stakeholders—especially those who are resistant to change—by keeping three things in mind (Dumas and Redish, 1999):
These three guidelines also hold true for changing processes, because usability testing can well expose process defects that are leading to usability problems in your test(s). If you find that you need to change processes, your role goes beyond just testing the product—you’re now a change agent for the entire organization. And when you change the processes for the company, you’re helping not only to improve usability for one product, but for all future products that the company produces.
You’ve interviewed your testers and reviewed the existing applications. Now it’s time to create the draft of the paper prototype test, place the materials in the binder, and then have Evan conduct the pilot test.
In this pilot test, Evan will give the test to you, one observer, and one note taker as he will give it to the testers in the actual test. The remaining observers and note takers you hired will be observing the pilot test and will provide feedback about that pilot test in the debriefing session immediately following the test.
The pilot test is originally scheduled to run for 65 minutes with a 10-minute introduction at the beginning of the test, which includes an introduction of the primary persona and a 10-minute question and answer session afterward. Based on the information you have gathered, you have come up with nine tasks for the project team to test:
Each task will take no more than 5 minutes, so this will leave 45 minutes for the remainder of the test. Evan has the test binder ready (including the task sheets) as you and the other two pilot testers sit around a round table to take the test. The observers are Ann and Sam, and the note takers are Debbie, Jim, and Robyn. Ann and Jim took part in the pilot test as the testers.
The note takers and observers will be in corners of the room and out of the way of Evan and the other testers (see Figure 9.1).
Figure 9.1. Positioning of the “computer” and observers.
When the test starts, Evan shows the primary persona based on user feedback so the testers know where the paper prototype options are coming from. He then gives some brief instructions for the test:
After the pilot test, you, Evan, and the observers and note takers share your information and thoughts about the test.
You: “I want to go last. I want to hear what you thought of the test.”
Ann: “I thought the test went well, but Evan was a bit rushed, so we didn’t finish the last test until 2 minutes before the end of the testing session.”
Jim: “I agree, and I think we need to add about 30 minutes to the test so Evan will have enough time to complete the test.”
Evan: “You don’t think the time should be extended to 15 minutes?”
Ann: “I think 30 minutes is about right, especially because our discussion about using the product availability page went too long. If you make changes to the product availability page, especially putting more information on the page, I think you’ll alleviate that problem.”
Robyn: “Especially if Evan needs to answer questions. I think he did a great job of that during the test.”
Debbie: “Evan, you did a great job with the audio cues that were the same as the current application, which Ann noted during the test. That helped the testers follow along and know when the users did something right or wrong.”
Evan: “This was a great test. I think we can tweak the interface a little bit per Ann’s suggestion and then run the real test.
Sam: “Giving an extra 30 minutes should cover any problems the testers find about the test, especially if they want to use the Q and A time to make suggestions.”
You: “I agree. I think we need to include a group of sticky notes for people to write down their suggestions, and then we can add more sticky notes from the observers and testers to make a to-do list for a follow-up session.”
Evan: “I’ll talk to Mike and get permission to extend the session to 95 minutes and have a second paper prototype test.”
You: “Be sure to tell him that we may need a third paper prototype session depending on what happens with the second paper prototype test. And then we’ll need to conduct a usability test when the draft of the application is ready for testing. Please work with Mike on the schedule, but if he has any problems, have him contact me.”
The interview with your pilot test team happened 6 weeks ago. You and Evan went through the paper prototype test and a follow-up prototype test with the entire project team. Happily, those tests were finished within one work-week, and the developers implemented the team’s recommendations into the draft upgrade of the application in only 12 working days.
You and Evan interviewed all members of the project team to learn how they used the draft upgrade of the application on the test site. They provided valuable information that you summarized in a report to Mike. That report included recommended changes that Mike approved. The coders made the changes quickly, and Mike approved the final changes to the application quickly.
Two days ago, the changes went online. And you visited Mike’s Bikes earlier today to follow up with him about how much users like (or dislike) the changes to the system.
As you opened the door, you noticed Mike and Traci sitting around Mike’s desk with big smiles on their faces. “The response has been incredible!” Mike exclaimed.
“We’re already on track to exceed our ROI estimates,” Traci added.
You smiled back, inwardly relieved that Mike was happy. “I’m glad you like it. Is there anything else you need?”
Mike folded his hands on the desk. “I’m glad you asked,” he said. “It’s time to have our customers test our new Web site that ties into our database application. When can you start usability testing?”
This chapter began with a discussion about selecting techniques for usability tests and the various types of tests available to you. There is no one best usability test; you will have to determine the best usability test for your situation. The primary usability test discussed in this chapter is observing, listening to, and engaging users. With this type of usability test, you learned that you may run into resistance and a number of questions. This chapter discussed arguments that you can use to answer these questions.
Defining your usability test was covered next. You learned about defining goals and measurements for the test, picking test participants, creating and selecting test scenarios, determining how to measure usability, and preparing test materials. When you create test scenarios, a scenario is short, unambiguous, and gives participants enough information to do the task. Although a scenario is in the user’s words, it is also directly linked to your tasks and concerns.
The section on conducting the usability test followed. You learned about conducting a pilot test before the real test, honing your testing skills, and caring for the test participants. You learned that you have to conduct a pilot test to work out any problems with the test before you conduct the test with real participants. When you conduct the test with real participants, you learned that you should encourage your testers to think out loud whenever possible and what to do when the testing goes awry.
This chapter ended with a discussion of analyzing the usability test results, ways you should present the results, and ways you can present the results to your intended audience. As with a usability test, there is no one best way to present the information to your viewers—that depends on your specific situation. This chapter discussed what you need to do to create a formal presentation as well as producing a highlight presentation if you can’t give a formal presentation. After you provide the information, you need to turn the information into action so you can improve not only the usability of the product, but also apply what you learned from your users to other company products.
Now it’s time to review what you’ve learned in this chapter. Ask yourself the following questions, and refer to Appendix A to double-check your answers.
1. What are the three general goals that a user looks for when he uses something?
2. What are the three phases of the UEL?
3. What rules should you adhere to as you plan for a usability test?
4. What types of scenarios should you test?
5. Why should you conduct a pilot test?
6. Why is it useful to conduct a worksite visit?
7. How many users are required for a useful and valid pilot test?
8. What are the possible sources of bias in your test results?
9. How should you address the question of bias in your report?
10. How do you ensure that your project team implements your recommended changes?