A usability study, or user testing, is the measured observation of users’ behavior as they engage in the use of your software application. It’s scientific in practice and favors metrics, measurements, and data to prove assumptions. There are two ways you can approach usability studies.

You can choose to complete the study in the user’s environment. For instance, if you’re building a call management application, you may elect to observe users in a live call center. This is referred to as a natural setting or an in-the-wild study. Much like the contextual inquiries we discussed in the previous chapter, usability studies are even more focused, systematic, and consistent.

The overall goal of a usability study is to measure the effectiveness of a feature or set of features within your application. To do this, you establish metrics such as time to complete or number of errors or a combination of these measurements. The study can also be combined with a survey to measure things that are difficult to observe like satisfaction or perception of value.

To complete a usability study successfully, you’ll want to have a plan.

It may seem obvious, but before you can conduct testing, you’ll need to know what you’re looking for. If you’re building a mobile phone application, don’t just hand the phone to someone and see if they like it. You’d be better served by a measured and organized approach. You’ll want to have the right subjects, a prepared script, and a set of guidelines. These ensure that the data you collect is consistent and viable.

Just like creating surveys, it’s important that you have the proper selection of users. If you’re building a line-of-business application for a specific group of users, you’ll obviously want to include them in your research. If you’re building a mobile phone application for novice and experienced users, then you’ll want to make sure you’re including people from all levels of experience and familiarity. As we discussed with questionnaires, make sure to avoid selection bias, selecting users who are misrepresentative of the group.

A script can be a useful way to achieve consistency in your testing. It’s easy to veer off course or become distracted when conducting a study. Preparing a script will allow you to deliver the test exactly the same way to each subject.

It may feel odd to read from a script, but it truly is the best way to ensure consistency. I find that a script sets the tone for a good quality study. Users will know that you’re serious about getting valid results and you’ve come prepared for the study. If you need an example of what a script might look like, be sure to check out Appendix A at the end of this book.

Here are some things you should include in your script:

You may be wondering how many people should participate to have an effective study. One option is to apply Jakob Nielsen’s no-more-than-five rule described in the previous chapter. Remember, the only guarantee is that if you don’t do any studies, you can expect to discover zero usability problems. I would rather conduct a study with three people than not have any study at all. On the other side, having too many subjects can be a logistical nightmare that complicates and delays your efforts.

Don’t think that testing a handful of people is a waste of time. If you only have access to a small group of users, I still encourage you to try usability testing. I’ve conducted very small studies and still gained valuable insights.

Other than the number of participants, here are some other things you should have while preparing for a study:

After you put a plan in place, it will be time to conduct your study. I recommend printing each task you plan to study on its own piece of paper. If you place a packet of instructions in front of subjects, there’s a good chance they’ll begin to leaf through it and get ahead of themselves. Having the tasks printed individually minimizes those distractions and prevents your subjects from skipping ahead.

Additionally, you’ll want to have a printed copy of your script for you to read. Prepare the script to coincide with each task. Ideally, you’ll have an overall script for the study and mini-scripts for each task. Make sure the language is consistent from task to task, even if it feels like you’re repeating yourself.

One thing that you want to continually remind subjects to do is to think aloud. This is the process of the subject telling you what she’s thinking while she completes the task. It’s all too easy for users to quietly focus on completing the task. The problem with that is that you’re missing out on their thought process. You can’t really tell what’s going through their head as they use your application unless they tell you.

Your script should include the instruction to talk aloud several times. You may consider giving this instruction before the start of every task. Most subjects will resist because it feels silly—they’ll feel like they’re rambling—but encourage them to anyway. Their stream of consciousness will be full of insight, even if they don’t realize.

Imagine that the user is looking for the Search button in your application. If she doesn’t say, “OK, I’m looking for the search button now,” there’s no way for you to connect with what she’s trying to do. By having subjects think aloud, you can better document their actions and hone in on what might be tripping them up. Let’s say you’re testing an email and calendar application. A great usability study should have your subjects sounding like this:

In this case we would document that the user was having trouble decoding the difference between creating a new message and a new appointment. This will remind us to review how we’re handling the creation of new items in our application. Perhaps it will drive us to consider a new design decision. If the user hadn’t mentioned that she was trying to create a new message, we might’ve missed that she was having trouble.

If a user has questions about how to complete a task, I strongly suggest that you only clarify what the task requires. Limit your instruction and do your best to eliminate any training while the user is trying to complete the task. You want the subject to try and figure it out for herself and think aloud as she does.

Too often, usability studies end up turning into training sessions. There’s nothing wrong with showing the user how to complete tasks, just make sure this training doesn’t occur during the study. If your application requires training (e.g., a complex line-of-business application that users have never seen before), then consider doing training before the study. If possible, have enough time between training and the study so that users are able to recall how the application works.

If subjects ask for help, rather than telling them what to do, consider flipping the question around:

By responding to the subject with a question, I was tipped off to the fact that I should consider putting the Print function where other applications place it (under the File menu). Avoid the temptation to provide training to increase your chances of catching things you might’ve missed.

If it’s possible, position yourself behind subjects. I find that it’s much easier to take notes and make observations without distracting them.

When each task is complete, take a moment to capture your measurements (elapsed time, number of errors, etc.). Consider giving subjects a small questionnaire while you jot down these last minute notes. I’ve found this to be a smooth rhythm that doesn’t make subjects wait while you’re furiously writing in your notebook. Also, while they’re completing the questionnaire, you can prepare for the next task (reset the stopwatch, get the next printed instructions, and so on).

The questionnaire could be satisfaction-based, with questions about users’ experience while completing the task. See Chapter 7 for more information about conducting surveys.

Things happen quickly during a usability test. You’ll be surprised by how many observations and off-handed comments you’ll want to write down for review. Having a clear script, printed materials, and any other tools ready to go will keep your study moving efficiently.

If you’re eager to get out there and begin, that’s great! However, I encourage you to fight the temptation to prematurely study your users. A great way to test the script and structure of your study is to practice. Consider asking a friend or colleague to run through the entire study with you. This will give you an opportunity to find any typos or confusing language in your script.

Ideally, you’ll find someone who is a good candidate for your application. If you practice with a fellow project member, he may be too familiar with the application to point out confusing areas.

Regardless of whom you find, spend time practicing to be prepared to conduct the official study.

Your usability studies should provide you with a trove of insight and feedback. It may be tempting to take one comment and start making design modifications, but you’ll be better served by taking the time to compile all your findings. By calculating your measurements and organizing your comments, you’ll convert the data into meaningful conclusions. The data from your usability studies can help you justify your design decisions, especially when there’s conflict among team members about how the application should function.

Here’s an example. While studying a redesign of our corporate intranet, I discovered that it was significantly less efficient for users to navigate a list of items rather than searching for a specific item. During this study, we examined our employee directory, which gave users the ability to search for an employee and see a list of employees by selecting a letter of the last name, as shown in Figure 9-1.

Figure 9-1. Our original employee directory provided users with a list of employees and the ability to search for someone

When asked to locate a specific employee, more than 90% of employees clicked to view a list of all employees with the shared letter of the last name. This would require them to scan a lengthy list. Some users took more than four minutes before locating the employee they were looking for.

When I asked why they chose to view a list of all the employees rather than using the available Search box, I received a common response: they were concerned they would spell the employee’s name wrong. They believed that it would be better to scan the entire list of all employees rather than take a risk of entering the name incorrectly.

The study helped me realize that if given the choice between searching and viewing a list of results, our staff did not feel confident in using Search, even though it would’ve been a far more efficient option. Primarily, their concern was spelling an employee’s name wrong.

Misspelling a name was a legitimate issue. Employees have unique spellings and long last names. It could take several queries to get the name you were looking for.

Therefore, I decided to build suggestions into our search, as depicted in Figure 9-2. I realized that we already had the list of employee names. Why not create a way to suggest all the possible employees as the user was typing his query? This would reduce errors and give users confidence to rely on a search engine rather than scanning lengthy lists.

Figure 9-2. Revised employee search using suggestions

As with any change, I experienced pushback from some staff and leaders. They were adamant that reviewing the lists was easier than typing an employee’s name. However, their disagreement was merely anecdotal compared to my data.

I was able to prove that the average user spends 21 minutes a year looking for employees. At over 3,500 employees being paid an average rate of $20.50 an hour, that was an estimated expense of $25,000 a year! Admittedly, this isn’t very sturdy math (in fact this is about as fuzzy as it gets), but the point is that having measurements allowed me to prove to the naysayers that their assumptions were incorrect.

This is why usability studies are the crux of the user-centered design methodology—they correct assumptions through the systematic observation of users and collection of their feedback. There’s no going with your gut in the user-centered design world.

I’m confident that when you begin usability studies, not only will you strengthen your relationship with users, but you’ll also be a better programmer. You’ll learn what works, what doesn’t, and most importantly, you’ll have the data to back it up.