Independent Analysis

The business model for most research companies is pretty simple: They do independent in-depth analysis on a specific topic or a specific industry, and then they sell reports to companies from the industry they just analyzed. These reports are not cheap, but they often represent a lot of comprehensive thinking by a group of industry experts (or, at least, research experts).

As an outgrowth of the marketing research industry, these companies tend to focus on the financial health of specific markets, industries, or companies. As part of this, however, they often research into the needs and desires of those markets’ target audiences. Much of the knowledge necessary to sell to a target audience is the same as that which is necessary to give that audience a good experience, and such marketing-focused information can be immediately valuable when creating user personas or setting expectations for contextual inquiry.

Some companies specializing in this kind of research include:

Additionally, some other firms create research focused on evaluating the user experience presented by companies or industries rather than just their business metrics. These companies include:

Although these reports present a ready-made trove of potentially useful knowledge, it’s important to read them closely. The researchers who write them will often not know the industry as well as insiders; they may misjudge the behaviors of users and the motivations of companies. A careful reading of the research methods is important.

Traffic/Demographics

By knowing who is using your competitors’ products, how much they use them, and what they use them for, you can avoid your competitors’ mistakes and capitalize on their strengths. Unfortunately, as great as it would be to have access to your competition’s log files and survey data, this is rarely a legal option. Fortunately, services exist that collect some of these data independently, reporting the conglomerate information and selling access to specific slices of the data. By using these services’ data and tools, it’s possible to gain insight into the makeup and behavior of your competition’s users.

Some companies that provide this kind of research include:

Interpreting these results and how they apply to your product is more difficult than reading an analyst’s report, and the sheer amount of data received from one of these services can be daunting. There are typically two kinds of data in one of these reports: the participants’ behavior, as it was tracked by the company, and the participants’ profile, as it was reported to the company. Linking these produces a powerful set of measurements. For example, you can (hypothetically—not all services allow you to do this directly) get data about the most popular sites in a given market and then get a profile of the people who use those sites.

Of all the data that’s possible to extract, often the most immediately interesting information is the demographic makeup of your competition (or of companies in a parallel industry) and their technological and Internet usage profile. You can immediately see how old, how tech-savvy, and how affluent their audience is (among many other variables). These are, of course, aspects that you probably considered when creating your own audience profile, but a set of independent data can confirm your assumptions or cast them in doubt.

Like all research, the process by which the data were collected needs to be carefully examined since that aspect is least under your control. Sometimes the data collection methods can introduce subtle biases, which need to be taken into account. For example, Comscore Media Metrix requires its participants to install a piece of software on their computer; the software tracks what sites they visit and when they visit them. Although they can get a mostly representative sample of users this way, this approach misses a key group of people: those who are unable to install this software on their work computers because it violates their company’s software installation rules. This means that the data collected by the service skew toward home-based computers and companies with lax internal security standards. For many situations, this bias does not affect the applicability of the final data. But the data would be insufficiently representative for a business-to-business (B2B) sales site targeted toward Fortune 100 companies. It would be important to track B2B users at work, but it would be impossible under the management information system security rules of most of the target audience.

Marketing Research

As is obvious from many of these descriptions, the tools of marketing research can be used for user experience research. The marketing department of your company is interested in what will make people want to go to your site and use your product. The reasons that they will want to go to your site and will be able to use your product are directly related to the user experience.

Research marketing often can help in understanding your user population. Ask for any audience profiles or market segmentations that have been created. These can offer a high-level view of your users, often supported by quantitative data, that will help you define your users’ likely interests and concerns.

Publications and Forums

Familiarizing yourself with the publications that cover and influence your field is probably a good first step in any kind of user research. Time spent at the library (or with a search engine) is rarely wasted and often reveals sources of information that would otherwise take a lot of work to replicate. Books and trade magazines are obvious sources, as well as blogs, podcasts, and especially RSS feeds. Subscribing to a handful of industry newsfeeds provides the media monitoring that the old newsclipping services used to, only for free. While industry-focused publications can be expensive, they are often accessible at your library.

White papers are essentially analyst reports, but from a biased source. They’re often written to justify a particular company’s perspective and explain their technology, but that doesn’t mean that they’re not useful. In defending their perspective, they often contain valuable information, though it should always be examined with attention to the bias inherent in its source.

A number of user experience pundits, consultants, and groups publish blogs and online magazines, which offer advice not just in print form, but through podcasts and videos as well.

Finally, you can track breaking news, issues, and debates in your industry of interest by subscribing to email lists and joining industry-focused discussions on LinkedIn, Twitter, Quora, and other social networking sites.

Timing

A key to using specialists well is calling them in at the right moment. Often, consultants get the call to produce a perfect solution late in the game, after all in-house methods have failed and a deadline is approaching. More often than not, this is asking for a miracle. Unfortunately, despite the way some advertise themselves, consultants are not saints.

The work that hired specialists do is not all that different from what your in-house staff can do, and it needs to be scheduled just like in-house work. Actually, it needs even a little more time than what you give your in-house projects since the specialists will need to learn about your product and the task that’s involved.

As consultants, we have been asked a number of times to “do a little user testing a couple weeks before launch.” In response, we must tell the caller that this is not unlike looking up to see where the moon is after you’ve already launched the rocket. The kinds of results that testing a completed product will reveal may help with tiny course corrections, but no amount of testing and adjustments will help if the rocket was pointed in the wrong direction. This holds for any other kind of specialty.

In addition, consultants, as opposed to other kinds of specialists, need to be called in especially early. Technical specialists don’t teach you; they do it for you. They don’t know your business before they come in and, most likely, won’t after they leave. Good consultants, on the other hand, absorb enough of your business to recommend solutions and strive to transfer some of their expertise to you. Good consultants will leave your company in a state where you won’t have to go to them with the same problem again. Good technical specialists will do their job quickly and accurately, but if it has to be done again, you’ll probably have to call them back in.

Fortunately, it’s hard to call user researchers in too early, but it’s still important to do the right research at the right time. As discussed in Part 1 of this book, a good iterative development process involves user input at nearly every iteration. The responsibility for picking what research is needed is as much the project developers’ as it is the researchers’. For example, if a product’s interaction is usability tested before the feature set has been defined, much of the information may go to waste since people’s use of it will likely change based on the options available to them. Likewise, testing a product that’s been built without first researching its audience’s needs will result in a lot of unnecessary effort: If the product’s audience has no interest in it, then they have little motivation to understand how it works or to use it in a realistic way.

Find a Specialist

For tasks with relatively straightforward needs, such as a single round of usability testing or some interviews to set feature priority, the procedure is similar to finding a carpenter for your house.

Write a description of your research needs and goals. What kind of research do you want to do? Why? How are you going to use the results? This is similar to how you would prepare for your own research, as described in Chapter 4.

Make a list of specialists to contact. Ask colleagues for recommendations or contact one of the usability professional organizations, all of which maintain lists of consultants and contractors. Some prominent organizations are the following:

Check qualifications. The specific experience of the research companies should be investigated before you hire them. You probably don’t want a carpenter who specializes in houses to make furniture or a furniture builder making a house. A user experience specialist may not have any experience doing marketing research even though the techniques are quite similar (and vice versa).

Get quotes and an explanation of philosophy and techniques. If possible, get a sample results document from all the consultants under consideration. Read the results for an explanation of techniques and look for a sensitivity to the needs of the product and the client.

Ask for references to several recent clients and follow up with the references. Inquire into the quality and timeliness of work, but also the quality of service. Did the consultant follow through on what was promised? Did they listen? Were they responsive?

Set Expectations

Going into a relationship with a specialist, especially a consultant or a consulting company, requires setting appropriate expectations on both sides.

As a client, you know your business better than they do. If they knew what you know, they’d be your competitors. Even if you tell them everything that you know, you will still know your business better than they will. You have more experience and should frame what you expect specialists to do from that perspective. The role that they play is not as replacements, but as information sources and tools.

Some consultants are qualified to help you uncover problems and to create solutions to those problems. It’s tempting (and may be advantageous) to get both services from a single source, but they require different capabilities and are best managed as separate pieces of consulting work. Scope the user research and the design as different projects and get separate proposals for each. Even if you end up working with one firm from start to finish, it will have been worth making sure they could meet your expectations throughout. And remember, consultants will happily give you what they believe is the best solution for a given problem, but it will still be coming from the perspective of someone who has limited experience with your business. The responsibility for taking their advice, understanding what it means in your situation, and applying it correctly is still yours.

Specialists provide perspective based on experience. Because of their experience, they know some general solutions that work better than others, and they can tailor those solutions to your problem. What they tell you may not match your perception of the world, but it’s important to listen to them. Inside your development process, you come to conclusions based on a certain set of assumptions. No matter how honest and forthright you and your staff are, eventually you’re all going to see the problems from much the same perspective, based on the same information. Outsiders come in with an entirely different set of assumptions and information, so their conclusions can be much different from yours. This does not mean that they are right and you are wrong, or vice versa, but the perspective they bring enriches yours. Allow them to ask fundamental questions and reintroduce topics that you may have already decided on.

One way to think about consultants is that they are people who know now what you will find out in two to three months if you don’t hire them. That’s it. Think of them as people who are a couple steps ahead of you, and by hiring them you are shaving a couple of months off your development schedule. A couple months of development time is really expensive, so bringing in specialists is generally worth the money. But what you’re buying is time, not magic.

The relationship between a specialist and client can be a valuable one. When it works, both parties benefit and learn from each other, while making the development process a bit more efficient. When it doesn’t, it can be broken off easily, and neither party is much the worse than before. In the long term, the most valuable aspect of reusing knowledge and experience is it makes the whole industry more efficient and gives everyone a reason to think about the things that really matter. Don’t reinvent the wheel if you don’t have to.