Chapter 10
Diary Studies
Dear Diary…
Imagine you want to learn more about how people commute by car. Following all those people in person from their homes to workplaces over days, weeks, or even months would take up all your time—not to mention how much it would irritate them. This isn’t an unusual problem. Most, if not all, processes or activities don’t happen only once. Nor do they unfold in convenient locations.
The solution is to use a diary study. Diary studies, as the name implies, ask a group of people to report their activities over time. Diarists track which mistakes they make, what they learn, how often they use a product, and anything else that is of interest to the project. Diarists can also give you a more general window into their lives, documenting the places and people they encounter.
Letting people track their own progress can give you an unobtrusive view into their experience without actually having to stare over their shoulders—literally or metaphorically—for months at a time. Even with minimal analysis, diary studies can provide a source of feedback that reveals patterns that would be difficult to identify otherwise. They can help you observe infrequent or brief events. Moreover, they reduce the time between an event and its documentation. Memory is reliably unreliable; diary studies help avoid asking people to remember events of interest.
Diaries are also one of the few geographically distributed qualitative research methods. You can have people complete diary entries around the country (or the world, for that matter) without leaving your office. This allows you to perform research into internationalization and into how cultural and geographic differences affect people’s experiences of your product. And unlike such methods as log file analysis (Chapter 16) and remote usability testing (Chapter 11), a diary study will help you observe activity on- and offline.
For example, in 2011, Beverly Freeman of the PayPal user research team conducted a “Managing Expenses” online diary study to better understand how nannies, caregivers, and small business employees handle business expenses with their employers. Participants across the United States completed online activities that involved sharing photos and commenting on various actions such as communicating with others, paying for expenses, and getting paid back. Freeman told us, “Because these actions occurred across multiple roles, times, and locations (both online and offline), the diary study proved to be a practical data-collection tool for revealing the good, bad, and ugly of today’s situation. In addition, projective exercises such as ‘Please take a picture of something that represents how you feel about your role in managing expenses’ helped crystallize the emotional aspects of a topic that extends far beyond the mere exchange of money” (Figures 10.1 and 10.2).
Figure 10.1 Paper diary study booklet tracing media use.
Image courtesy of Adaptive Path.
Figure 10.2 “Managing Expenses” diary study using Revelation online research tool.
Image courtesy of PayPal.
When to Do a Diary Study
Diary studies can track usage of a product, document particular activities, or follow a specific type of experience. As such, they typically occur at two main places in a product development cycle. Early in design, diary studies can help you follow behaviors and activities. They can help you scope a survey or serve as the basis for more in-depth later interviews. Later in the process, a diary study can serve as a kind of extended remote usability test for a working prototype.
Steve August, maker of the qualitative research tool Revelation (www.revelationglobal.com), proposes four main types of diary studies. His taxonomy is a useful starting point for deciding where and how a diary study might help you.
• Usage diaries document specific moments of interaction with a product or service. The subject can be a website, a restaurant, an ATM, or a railway. The interactions can be organic—activities the participant would perform outside the diary study—or activities that you specify. If the latter, the usage diary can serve as a type of remote usability test.
• Spotter diaries identify where and how the presence of companies, products, or services matter in people’s lives. Unlike usage diaries, spotter diaries map the place of these objects in the broader context of people’s lives.
• Process/purchase diaries follow activities that unfold over time and space. They involve longer narratives, such as those of buying a car or planning a wedding.
• Behavior diaries are more exploratory in scope. They examine a range of activities or objects that comprise a specific topic. For example, a behavior diary could examine how participants deal with money in their lives—the tools and activities of earning, spending, and saving.
Usage diaries and spotter diaries are useful when you already have a product, company, or service that is stable enough to support long-term engagement. Process and behavior diaries, however, are often helpful earlier in the design process to explore a potential design space.
How to Do a Diary Study
Preparation
Because you have to design and manufacture a customized digital or paper workbook for participants, diary studies require more advance preparation than a simple interview or even a field visit. Even experienced researchers don’t try to put together exercises for a diary study in a hurry—they allot plenty of time for recruiting and pretesting (as in Table 10.1).
Table 10.1. A Typical Diary Study Schedule
Pretesting for paper-based diary studies is particularly important because once you send the activities out to your participants, you won’t have much opportunity to clarify vague instructions or modify an activity that isn’t delivering interesting results. Once it’s in the field, you just have to live with what you get back. Online diary platforms offer slightly more flexibility. They make it easier to run a study in which participants begin and end at different times, which lets you pilot your activities with early participants.
However, do not underestimate how much time it will take to monitor an online diary study while it is in progress. A typical study has 10 participants and seven days of activity. If there are three diary exercises per day, you will have 210 entries. You will need to review entries as they come in. Participants will often have questions, and you may need to coach them if their participation wanes in the middle of the study.
Recruitment
Diary study recruiting resembles any other user research activity. However, diary studies require self-directed action from participants, sometimes over weeks. Given the demands of diary studies, recruiters should filter potential participants not just by eligibility and availability over the study, but also for likely reliability and articulateness. Ask yourself: do potential participants respond promptly to messages? Do they seem willing to perform the activities involved? Are they good at expressing their thoughts and emotions in words or pictures? Since even careful screening cannot prevent all mid-study dropouts, save yourself some stress by securing backup participants early on. A typical diary study has about 10 participants, so a careful researcher will recruit two extras if possible.
Recruiting people who are dedicated enough to fill out diaries over an extended period of time will likely bias your sample. Their diligence may come with other qualities that are not representative of your general user population. It’s still worth doing diary studies, but be aware of the potential bias when analyzing your results.
See Chapter 6 for more information about recruiting.
Designing the Study
Whatever the purpose of your study, there are two main types of activities: feedback activities and elicitation activities. Human-computer interaction researchers Scott Carter and Jennifer Mankoff explain the difference this way: In feedback activities, participants complete a questionnaire. For example, a participant might record date, time, location, purpose, and purchases at every visit to a grocery store. In elicitation activities, participants capture media, which later prompts interview questions. For example, a participant might be asked to take a photo or record some sound during that shopping trip.
Because they are so prolonged, diary studies tend to have their own rhythms. They often have multiple phases, such as an initial phone interview, the study itself, and then a follow-up interview. The study itself has its own internal tempo created by the frequency and duration of the activities. For that reason, designing diary studies requires some attention not just to the activities that will best answer your questions, but also to how you and your participants will experience them over time.
The type of study you are planning and the kinds of activities you think will best capture your interests will influence the final three decisions: duration, schedule, and sampling rate.
• What is the duration of the study? There should probably be at least a half dozen diary entries for trends to be observable (though the exact number, of course, depends on many factors). If your topic of interest happens once a week, the study will need to go on for about two months to be able to measure a change. However, if there will be a relevant event every day, then a week may be sufficient to see changes in people’s use.
• What kind of schedule will entries follow? You can ask people to make entries at fixed times (such as every hour) or variably. Variable entries can take place at random intervals (usually prompted by a telephone call or text message from the researcher) or whenever the participant starts or finishes a specific activity. The Arbitron radio diary, for example, is a long-running American market research study with hourly data on participants’ radio listening activities. However, radio listening is a fairly easy phenomenon to track. What about less clear phenomena, like “being happy”? People aren’t always able to recognize and respond to their moods in the moment. If you wanted to track mood swings, it might make sense to employ a variable schedule and prompt participants to report their moods at random intervals during the course of the study. If you were just interested in how people cook dinner, obviously you would ask for neither hourly or random entries. Instead, people would complete the diaries as they begin to prepare food.
• How many entries do you predict? The study’s sampling rate determines the level of detail that you can observe. The more frequently people respond, the more subtle the changes you can notice in their experience. However, changes happen at all levels of detail, and there are important monthly and yearly trends even in products that get used many times a day.
Since people aren’t diary-filling machines, picking a sampling rate, schedule, and duration that won’t bore them or take too much effort is likely to get you better quality information. For example, a study of how search engine users learn a new search engine recruited people who searched roughly once per day (with a different search engine than the one being researched). Picking daily searchers defined the maximum sampling rate since asking them to fill out more than one diary entry per day would not have produced any additional information.
The balance of feedback and elicitation activities will also affect the rhythm and duration of the study. The need to fill out a questionnaire tends to mean that feedback activities take more time in the moment of action. If you are following a variable schedule, this could cause inconvenience, as you prompt participants at random intervals to make entries. However, feedback activities do not necessarily require any conversation after the diary is completed. Elicitation activities, on the other hand, tend to take less time for participants in the moment (it only takes a few seconds to snap a photo, after all). But follow-up interviews require scheduling and time.
Inventing Good Exercises
The first step in creating good diary exercises is to write down everything you might possibly want to know about participants’ activities. Then come up with diary tasks that can help you track them.
Like survey questions (see Chapter 12 for more on question writing), diary exercises produce both structured and unstructured data. Structured data are the result of choosing responses from a limited set of possibilities. The Arbitron diary study (Figure 10.4) produces mostly structured data: hours of the day and radio stations. Unstructured data are, essentially, unconstrained. Unstructured data in diaries can result from notes or sketches on paper, text messages or voice calls, or photography and video. Cultural probes (Figure 10.3) produce mostly unstructured data. Structured data are well suited to the quantitative data analysis methods described in Chapter 12, unstructured data to the qualitative methods in Chapter 15.
Figure 10.3 A cultural probe investigating the lives of 10- to 13-year-olds in Lebanon (a). Designed to appeal to both boys and girls, it includes postcards (b), a disposable camera, an audio recorder (c), a food and eating diary (a) and a set of “like/dislike” stickers to place on a collection of images (a). The project’s client is a nonprofit organization working to promote health awareness in kids. Images courtesy of MENA Design Research Center.
Diary studies are extremely flexible as a method. You can ask people to do anything that you think will help you understand their lives. Typical diary activities include:
• Taking photos of places (such as the kitchen) and activities (such as cooking)—and writing captions for them
• Using stickers or icons to indicate one’s mood
• Drawing maps of the home or workplace
• Noting the time, location, and purpose of each occurrence of an activity of interest (such as checking email)
• Tracking emotions or energy level at key events (such as at the beginning or end of a daily commute)
• Writing or recording longer reflections about an activity or event at the end of each day
Diary studies also commonly include one-time activities, such as photographing one’s workplace or home or answering a set of initial attitudinal or behavioral questions.
Cultural Probes
Traditional diary studies are descriptive—they ask participants to record activities accurately, as they happen. By and large, they do not encourage speculation, whimsy, personal idiosyncrasy, or flamboyance from participants. More recently, many researchers have begun to use a type of diary activity—the cultural probe—that encourages imaginative personal reflection through structured, but playful, exercises.
As explained by their original creators at Britain’s Royal College of Art in 1999, cultural probes deliver “inspiration, not information.” They are not intended to produce an “objective picture” of the conditions of people’s lives. Instead, cultural probes are used to stimulate creative discussion between designers and potential users—to provoke the imagination and generate empathy.
That’s why MENA Design Research Center chose to use a cultural probe to learn about Lebanese children’s lives for a project on promoting health awareness. Doreen Toutikian, MENA’s director, said, “All these various multi-sensual and interactive methods allow the child to be more responsive and intuitive. This form of design research has proven to be far more effective than concrete interviews with researchers in sterile observation rooms with stalking cameras. It allows the child to feel free in his/her environment and explore his/her world with creativity and insight.”
Common probe materials and activities include:
• Pre-addressed, stamped postcards with an evocative image on the front and an open-ended question on the back. Postcards in MENA’s cultural probe (see Figure 10.3b), called the KonfiKit, asked questions such as, “If you ever met an alien from outer space, what would you ask it?” As an informal way to communicate with friends, postcards can feel more casual and intimate than a survey form on a website. For people who mostly use digital mail, they can also simultaneously feel exciting and special.
• Maps of familiar places, such as neighborhoods and cities. A probe might ask participants to mark up maps with sketches and stickers to answer questions like, “Where would you go to be bored? Where are you afraid?” A probe might also ask the participant to sketch a map in lieu of a pre-printed one.
• Cameras (disposable or digital). MENA’s KonfiKits, like the original cultural probes, use disposable cameras whose cardboard cases had been recovered with new paper to match the other materials. Instead of documenting specific activities, probe “shot lists,” like those in the KonfiKits, might include requests like, “Take a photo of your favorite clothes” or “Take a photo of your favorite game.”
• Space for sketches and collage. Paper probe packets often include space for participants to make their own compositions—to draw representations of their relationships to people and places, or maybe to select and collage images that represent parts of their lives.
Probes exist to stimulate multiple interpretations. As evocative objects, they often serve to elicit discussion in interviews. See Chapter 8 for more instructions on using probe materials in interviews.
A note on terms: Some people use “cultural probe” as a synonym for “diary study.” Others classify a diary study as a type of cultural probe. However, diary studies were frequently used in social science research long before the publication of the original cultural probe study in 1999. Given the history of the diary study, we think it makes most sense to call the cultural probe a diary study variant—one that focuses on reflection and interpretation rather than factual documentation. Given this varying terminology, make sure that all your stakeholders understand what you mean when you propose doing a cultural probe.
Because diary studies are so flexible, there are no firm guidelines for their design. However, here are some characteristics of good diary activities that you can use to evaluate your own.
• Relevant. Just as good survey questions do not ask people to guess at information they do not possess, good diary activities don’t waste participants’ time—and your own—by demanding actions that are not natural to them. Partially, this is a matter of recruiting. When studying kitchen appliances, don’t recruit people who mostly eat out. But the larger question is: How can people effectively document a phenomenon of interest? To design kitchen appliances, you probably should ask people to document not only cooking, but cleaning and storing appliances as well. However, adding extra activities creates new problems, such as disruption.
• Nondisruptive. The other side of relevance is disruption. The struggle to follow directions can distort or obstruct diarists’ normal activities. As the study’s demands escalate, participants might give more attention to the study than to the activities it documents. At that point, your study can self-destruct. Sometimes, the resulting irritation can lead participants to skip exercises, falsify data, or even drop out altogether. Obviously, all diary studies disrupt people’s everyday activities to some extent. The question is, how disruptive is too disruptive? The best way to tell if your study is too disruptive is to pretest it yourself or have a sympathetic friend give it a shot.
• Nonbiasing. During a project on physical fitness, Elizabeth realized that participants’ embarrassment about “not exercising enough” might keep them from accurately reporting their own gym-going. Moreover, they did not classify some physically demanding work, such as moving furniture, as “exercise” and thus might not report it. So direct questions about physical exercise were likely to produce unhelpful responses (see sidebar on self-reporting). Instead, Elizabeth avoided the words “fitness” and “physical exercise” in diary instructions. She simply asked participants to wear a pedometer all day while listing their activities. The pedometer helped establish a ground truth about participants’ level of physical activity movements, while the written accounts gave more detail. Between the two, she could estimate participants’ level of physical exercise without asking directly.
Self-reporting
When you ask people to report their own experiences, you ask them to step outside their normal perception of themselves and comment on their behavior. Some people will provide accurate answers. Others will have more difficulty. Even if they want to help you and think that they’re telling you everything, they may not feel comfortable admitting failure or revealing that they don’t know something. This problem applies to any self-reported information, such as support comments, survey questions, or interview responses. But it’s especially problematic in diaries. When the diary entry is your only contact with the person, there’s no way to ascertain what’s called the “ground truth” of their behavior.
One way to mitigate the effects of self-reporting bias is to gather multiple perspectives on the same event. Andrea once conducted a diary study to learn more about how close friends and family members share information. She recruited diary study participants in pairs (mother and daughter, boyfriend and girlfriend, and so on) who communicated daily. Every day of the study, each participant wrote about an interaction they had had that day with the other member of the pair. This not only helped Andrea to understand what had actually happened, but also highlighted the differences between what each person remembered and considered important.
Another way to mitigate self-reporting bias is to follow-up with additional interviews. Use the interviews to ask for clarification of important statements. And always keep a grain of salt handy.
Choosing Your Platform
Paper booklets, voice messaging, and online tools can all help people document their experiences. You can use them singly or in combination. For example, you could use a paper booklet to direct people to upload digital photos to a website (see Figure 10.5).
Paper Booklet
Traditionally, diary studies have relied upon giving participants a paper booklet with entry forms (see Figure 10.1 and Figure 10.3) to return at the end of the study. In fact, they are still widely used. Participants can fill out timesheets (as in Figure 10.4), sketch maps, or even collect and collage paper documents.
Figure 10.4 Instructions for the Arbitron radio diary form, which collects information about U.S. radio-listening habits.
Image courtesy of Arbitron, Inc.
It pays to put some thought into the aesthetics of your booklet. The design of the booklet affects readability and ease of completion, especially if you are providing forms (as in Figure 10.4). Furthermore, the appearance of the booklet is a demonstration of your attitude and commitment. Participants are more likely to spend time on attractive booklets that look professionally prepared. Figure 10.1 is a good example of a well-designed paper booklet that is easy to produce. It uses a standard-sized piece of paper folded in half.
The advantages of paper booklets are many: they are cheap to produce, require no technical know-how to complete, and support a variety of writing, collage, and sketching activities. However, during the course of the study there is no practical way to monitor completion. Researchers must wait until the diaries are completed and mailed back, which means that there is no way to correct any misunderstandings or mistakes.
Voice Messaging
With the prevalence of mobile phones, voice messaging can be very useful—especially when engaging with people who are not comfortable writing. Voice message diaries are simple to set up; they only require a phone number and voice mailbox. You can even send instructions and reminders by text message (although you’ll probably want to give people an initial email or paper letter with full directions).
Save yourself some time by using a voice-to-text service to have the voice entries transcribed as they come in.
Voice diaries are expressive. People often talk more freely than they write. They may choose to update their diaries more often, in more concrete detail, and with more emotional richness. Because people carry mobile telephones with them, voice diaries remove the need for them to remember a paper booklet. As well, participants can call in while they are on the go, which keeps the time between an event and its documentation to a minimum. Text message reminders can help bridge that gap even more quickly than email to a computer.
Voice messaging, however, will not always be the most appropriate solution. You may want to make sure you get precise data, such as the call signs for radio stations, from your participants. Maybe each entry requires complex steps. Or maybe you know that in some locations and activities, speaking aloud is not an option.
Online Applications
It’s becoming more and more common to have participants fill out their entries using email, a website, or mobile application. Online entries allow researchers to monitor participation and ask for any adjustments in the moment. Online entries also streamline the incorporation of digital media, especially video. If your participants have a computer with a camera, you can instruct them on how to record video messages—thus adding expressiveness and conversational spontaneity to any text entries.
Visit the book’s website at www.mkp.com/observing-the-user-experience for more information on tools for online diary studies.
An email form is perhaps the simplest digital tool. You simply email your participants a form and have them respond to the email with their answers. Not only is email free, but it’s a familiar way of communicating for many people. It’s a neat trick: the diary entry email reminds people to reply promptly. You can also have participants attach photographs and video to the email. If you then set up a blog (at time of writing, Tumblr.com and Posterous.com are popular choices) to accept email posts, you can automatically have those emails posted to the web for easy aggregate viewing by you and potentially your clients. Using the blog commenting functionality, you can even start up a conversation about specific entries during the run of the study.
If you do use an email-to-web solution, make absolutely sure you password protect access to the resulting website. The same thing goes for every other online tool you use. Don’t allow participants to see each other’s data, and don’t make data publicly viewable. Your participants didn’t sign up to have the details of their lives potentially exposed.
If most of the data you’d like to capture is relatively structured, you can also use a spreadsheet (either an online document or an individual file on a desktop). You name the columns, and respondents fill in the rows with their own data. The Arbitron radio diary essentially follows a spreadsheet format.
You can also create your own diary entry forms using one of the many free online survey tools (Google Forms is popular). Diarists complete entries by “responding” to the survey. All you have to do is make sure that the survey includes a free text entry area for participants to identify themselves. Then you can use any built-in survey analytics to analyze the resulting data.
There are also custom online tools for diary studies. At the time of writing this book, a popular one is Revelation (www.revelationglobal.com). They simplify the task of designing and deploying online diaries by offering a standard library of exercises to choose from. Participants can report the details of an event, upload photographs, or capture video. They also simplify analysis by providing built-in tools for aggregating the data and extracting trends. Mobile versions of such applications can sometimes use the phone’s capabilities to add location data to each entry as well. However, such tools are generally not free. Also, you may want the greater flexibility that assembling your own diary may give you.
A final online tool is not a tool so much as an overall strategy. You can study people’s lives through the social tools they already use, such as Facebook, text messaging, and Twitter. Just ask your respondents to give you temporary access to their daily communications. They can use whatever built-in access controls are already available to keep certain messages private—but you will see the vast majority of the everyday activity they would share with friends.
This type of diary study, called “lifestreaming,” by Chris Khalil, Director of User Experience at News Digital Media, does not demand artificial action from participants. When participants are asked to step back and comment on their own activities, they can unwittingly bias their responses in order to fit them into the diary study “frame.” That frame also means that diary studies can fail to capture some of the most important minutiae, simply because participants don’t think it’s important. With lifestreaming, that frame is absent. Participants are simply permitting the researchers to watch them going about their daily business. There is no need for them to judge whether an event is “important enough.” If a feeling or event is important enough to post to Facebook, then the lifestreaming diary study will catch it.
As with an email diary, you can set up a blog to automatically bring together these sources into one “stream.” At the time of writing, most blogging services include plug-ins that automatically integrate feeds of new content from a variety of websites. Then you can ask participants to add their own commentary later.
Incorporating Images and Video
In the days before cheap digital cameras, diary studies relied on disposable film cameras. Researchers would distribute the cameras along with detailed written instructions on what to photograph. Then they had to wait until the end of the study to discover what participants had done. As you can imagine, the developed rolls of film often brought unwelcome surprises: out-of-focus or poorly exposed shots, unhelpful or irrelevant images, or even an entire blank roll.
Luckily, there are many relatively low-cost digital cameras these days. We recommend digital so you don’t have to wait until the end of the study to see how your participants are doing. And with a digital camera, there is no such thing as “running out of film.” If you have the funds, you can even allow participants to keep their cameras after the study as an incentive for participation.
Nevertheless, don’t fret if your budget does not run to buying 10 or 15 digital cameras. You can ask participants to use their own cameras, if you believe they own their own and are likely to find this acceptable. For some participants, using their own cameras may actually be more comfortable than learning an unfamiliar interface. Alternately, disposable cameras are still better than nothing. You can customize them by making your own cardboard cover, as in Figure 10.3a.
Assembling the Diary Components
Since your participants will be completing diary entries outside of your presence, you will need to provide whatever materials you think will help them consistently give you the information you want.
A typical diary (or probe) has four components:
Throughout this section, we will use as an example text from an email diary designed to track use of HotBot, a search engine.
Introductory Letter
After being recruited, screened, and invited to participate, participants will still need a full introduction to the study. It’s likely that they’ve gotten this information in bits and pieces already, but it’s always good to condense all the key facts into one place. An introductory letter addresses:
• Basic study information: its goals, motivation, and sponsorship
• Why participants were recruited (optional)
• Compensation and warning of any potential harms
• Permission/consent letter (if not already completed during recruitment)
If you are using an online tool, the introductory message should also include a link to the website you are using.
The introductory letter is also a good place to remind diarists of why—besides any money—they agreed to do all this work for you. However, no one wants to read a lengthy exposition; this letter should be short and sweet.
Thank you for participating in our evaluation of the HotBot search engine. We are in the process of evaluating it for redesign, and we would like your input so that we can base the design changes on people’s daily experiences with it. We are asking for your help as someone who uses our search engine frequently and who might be interested in helping us improve your experience with it.
If you have any questions or comments, please feel free to contact Mike Kuniavsky at [email protected].
Once again, thanks for your time and effort!
Explicit instructions to diarists make it more likely that they will comment on what’s important to you. However, detailed instructions can bias the responses. The participants may look at parts of the product they have not looked at before, describe it in ways that they normally wouldn’t, or use it in novel ways. Stay conscious of how your wording might bias your results.
Instructions
After the introductory letter, diarists will need some instructions. The instructions should be specific, brief, and complete. They should give the diarists a set of guiding principles about what kind of behavior to record while encouraging active participation (Figures 10.5 and 10.6).
Figure 10.5 Instructions for diary activities in a paper booklet.
Image courtesy of Adaptive Path.
Figure 10.6 Participants’ view of instructions for intro activity using Revelation.
Image courtesy of PayPal.
The most important thing is to keep the directions simple and short. Remember that your participants are completing your questions in breaks between other activities.
As with surveys, question wording really matters, because you will not be present with participants to clarify any instructions they find confusing. Depending on your participants’ projected education, you may want to aim your instructional text at a sixth-grade (11-year-old) to high school (16-year old) reading level.
Fry’s Readability Graph is a standard method for measuring the reading level of American English. Directions for using Fry’s Readability Graph are available at www.idph.state.ia.us/health_literacy/common/pdf/tools/fry.pdf.
The forms for diary study entries resemble survey questionnaires. The following sample set of instructions comes from an email study, but you can have people answer your questions on paper, on a website, or using a custom application.
Here’s a sample set of instructions:
Please use HotBot for as many Internet information research needs as you can. Your thoughts and experiences are very valuable to us.
For the next month, you will receive this instruction sheet along with a diary form twice a week (on Mondays and Thursdays) in email.
It’s a good idea to remind people of their obligations and to set some expectations. Attaching the instructions to every email reminds the participants of how they should think about filling out the diaries.
We would like you to fill out the diary form and email it to [email protected] before the next one arrives (mailing it on the same day is fine). We estimate that it will take 20–30 minutes to fill out completely.
If you can’t email the diary to us by the time the next form arrives contact us as soon as possible. Contact information is at the bottom of this sheet.
Of course, you already gave diarists your contact information in your introduction. Just keep on reminding them. Including contact information prominently with every communication will help remind people that you are there to answer questions and lower any barriers to contacting you.
The form is designed to help you describe your experiences while using HotBot to look for information. You don’t have to fill it out at any specific time and you can write as much as you want, but we would like you to fill out as much as you can.
Here are some things to keep in mind while you’re filling out the form.
• Relate as much as you can about your experiences, positive and negative, big and small. We are interested in all of it, no matter how minor it may seem at the time. We are especially interested in experiences that were surprising or unexpected.
• Our goal is to make HotBot work better for you. If you can’t get some feature to work, it is not your fault. Please describe any such situations in detail.
As with all situations where people report their experiences, they should be assured that they’re not the ones being tested, it’s the product, but that it’s their responsibility to accurately report such failings.
People may not know when they’re using something new, but it doesn’t hurt to encourage them to look for and record novel situations.
Encourage specifics when possible. Search terms and products are relatively easy to note down, so it’s feasible for people to record them, but avoid more abstract or labor-intensive tasks. If people are asked to write down everything they clicked on and every checkbox they selected, they’re likely to spend all of their time writing about the minute elements of their experience instead of the more important issues. And they’ll get sick of doing it.
If you’re not sure about whether to put something in the diary or not, please put it in.
If you have not performed any searches by the due date of the form, please mark it as such and return it.
If you have any questions, please don’t hesitate to contact Mike Kuniavsky at [email protected] or (415) 235-3468.
Questionnaire Forms
The previous example collected relatively unstructured data on the use of a search engine. However, it’s possible to treat the diary entry more as a structured questionnaire. Here’s a sample questionnaire-structured diary email form.
HotBot diary
Please return this diary entry on or before Thursday, June 22, 2013
Today’s date:
The current time:
1. Approximate number of searches since your last diary entry:
2. Of those, the approximate number of searches using HotBot:
Please describe what you searched for most recently, providing the search topic and the exact search terms you used, along with any + or - modifiers.
As described earlier, asking people for the specifics of their actions is worthwhile as long as it’s not burdensome. Note that this questionnaire directs users’ attention to a specific part of the interface, which turns this diary study into a kind of remote usability study.
(Please rate the search from 1 to 5, where 1 means that it was unsuccessful, 3 means that the information you found was adequate, and 5 means that you found exactly what you were looking for.)
(Please rate your experience from 1 to 5, where 1 means that it’s not working at all, and 5 means that it’s working very well.)
5. In your recent searches, did you use any of the search options in the left-hand margin of the main search page (the first page you see if you go to www.hotbot.com)? If so, which ones?
(Please rate their effectiveness from 1 to 5, where 1 means that they did not help your search at all, and 5 means that they were critical to its success.)
Open-ended questions give participants the opportunity to explain their experiences in depth. Asking people to present a narrative of the changes in their experience, although prone to the bias of selective memory, can compress the analysis process, providing direct insight into how people’s views and thoughts are changing.
8. Have any of your views about HotBot changed since the last diary entry? If so, how have they changed, and was there a specific experience that caused the change?
9. Other comments. Are there any other issues you’d like to tell us about or questions you’d like us to answer?
10. When you’ve completed this form, please email it to [email protected]. Thank you very much for helping us make HotBot a better product.
If you have any questions or comments about this form, please contact Mike Kuniavsky at [email protected] or (415) 235-3468.
This form adds several questions about a feature cluster on the front door interface. This allows the researchers to get feedback on a specific part of the product.
Of course, the specific content and layout of your diary forms will depend on the product and your goals for the research. Feel free to experiment.
Pretesting
Before you finalize your diary design, test whether the instructions are clear and the activities are doable. The question that you think is clear may confuse your audience; the activity that you think will take five minutes may actually take fifteen. You should at least try them yourself. If you have the time, get someone else to run through a day of the study.
Conducting a Diary Study
Diary studies may allow the researcher to do other things while they run, but they do require some continuing attention. After the diaries are distributed, a successful diary study rarely runs on autopilot. You will need to stay in touch with participants—not just to remind them of their commitment to the study, but also to troubleshoot any technical problems they might have with forms or devices. Moreover, diary studies, especially lifestreaming studies, can produce a surprisingly large amount of data very quickly. If you are not using an online tool for diary management, you will give yourself extra work at the end of the study if you don’t organize diary entries as they come in.
Managing Participation over the Study
One of the keys to a successful diary study is the management of responses and respondents. During the course of a diary study, day-to-day contacts with participants will require less sustained attention than an interview or site visit. But because they are so drawn out, diary studies challenge researchers’ skills in creating and sustaining participant engagement.
After sending out the instructions, you might review them with the diarists either in person or through some other real-time method (the phone, instant messaging, etc.) to clarify any subtleties.
Review the early responses carefully. If they do not meet the goals of the research, adjust the question phrasing or activity design. This is easy to do with online tools. If the changes are drastic, you may have to mail out a new paper booklet.
Don’t be surprised by dropouts. Often, a small percentage of people who sign up for your diary study will quit midway through. Some participants didn’t understand how much work the study entails. Some participants feel that the incentives don’t sufficiently compensate them for the disruption to their everyday life. And some participants are just flaky. Don’t be offended or upset if some participants just quit without any warning. You probably didn’t do anything wrong—it happens to everyone.
However, a very high dropout rate should worry you. First, it indicates a problem with the study design. Second, those who finish the study are probably unusual in some way. They are likely not representative of your target audience.
Incentives
Align incentives to the amount of time required to complete the work. Six entries may well take a total of three hours to complete, with an extra hour of overhead spent managing them. Diarists should be paid at a rate somewhere between 60% and 100% of what you would pay participants who came to your lab. Assuming a standard rate of $60 per hour, this means that each participant in a two-month study that requires two diaries per week should receive roughly $200 as an honorarium at the end of the study. It is possible to do this less expensively, but you risk high dropout rates. Mike remembers a very long diary study that paid $50 and a t-shirt—but only one person out of eight finished it. You can encourage completion, with smaller “surprise” incentives (t-shirts, restaurant gift certificates, etc.) placed throughout the process. Staged incentives—paying out a little at a time—can also help. It’s better to pay too much than too little in order to ensure good results after you invest all that time and effort.
When working with an organization where incentives may be inappropriate, consider asking the participants’ managers to allocate time to fill out the diaries.
Reminders
Reminders can be important tools in bridging the gap that happens when you’re not doing research face to face. Since much of the study work is self-initiated, it’s easy for people to treat the diaries as optional. Reminding participants of their promise to complete the diary and of the importance of their effort to the product can inspire them to provide better feedback—and to put the diary on their to-do list. Text messages or email follow-up to missed entries tells the participants that you’re still there and that their feedback is important to you. Thanking them for their responses and promptly answering their questions lets participants know that you are committed to the study as well. To maximize response, you have to give people incentives and remind them of their obligations. But please, don’t spam them with a dozen “keep up the good work!” messages a day.
Schedule some time every day for checking for uploads, monitoring compliance with instructions, and organizing the collected data. Keeping up with these project management tasks will help you move more smoothly into data analysis.
Distributing and Collecting Diaries
If you are using paper booklets, you will need to include a pre-addressed, stamped envelope for any diary materials you need returned. You will also—since people have busy lives and will probably forget—need to send a reminder to mail the diary back at the end. If you have loaned any media devices, such as video cameras, to participants, make sure they know you’re expecting to get them back as well.
Follow-up Activities
Coupling your diary study with follow-up focus groups or interviews allows you to clarify their responses, ask follow-up questions, and check any frameworks you are beginning to develop. This is where elicitation activities come in handy as the generators for “show and tell” discussions. Here are some sample elicitation questions:
You took this photo while you were making dinner? Can you tell me more about what’s happening in this picture?
What prompted you to take this photo?
See Chapter 8 for pointers on using media prompts in interviews.
Make sure to schedule these follow-up activities relatively soon after the events described in the diary.
Conclusion
Diary studies assist in studying everyday activities, long-term processes, and rare events. They minimize recollection bias and maximize geographic coverage. Paper diaries need no batteries; mobile phone diaries are always in one’s pocket. They can even go places—like a crowded train commute and the locker room—where a researcher might be unwelcome. They encourage participants to share deep thoughts and casual observations. All in all, diary studies solve some thorny problems in research. But they do not run on autopilot.
We’ve given you a number of tips for designing and managing a successful diary study. In closing, remember these basic suggestions:
1. Recruit for responsible and articulate participants.
2. Select a few extra participants.
3. Make your diary materials look attractive and professional.
5. Regularly contact participants to catch problems that they don’t think to report.
6. Encourage the kind of participation you want. Praise and thank participants, remind them of deadlines, and let them know if you want more or different responses.
7. Remind participants that they are getting paid and helping improve the world.
8. Schedule your follow-up interviews promptly, while the events in the diary are still fresh.