3.1 Designing Effective Activities

Effectiveness is a tricky concept. It is tempting to equate effectiveness with efficiency—in other words, with productivity, the cost of achieving something with respect to time or space. But, in fact, effectiveness is a more general concept. Simply put, a design is effective to the degree that it satisfies the needs it is intended to meet. If efficiency is important in the problem situation, then efficiency will be one aspect of effectiveness. But the real question is how to make sure your solution has the “right” services.

Designers begin with their own beliefs—explicit and implicit—about technology. Coming up with novel ideas is intrinsically rewarding; this is often what leads people to be designers in the first place. Innovation can lead to radical transformations in how we view tasks, and even in what goals are reasonable.

Unfortunately, technology for its own sake often gets in the way of effective design (Tradeoff 3.1). For example, technological determinism is a school of thought wherein technology is seen as the most important factor influencing success of an organization (Woodward 1965). Under this view, tracking and applying leading-edge computer technology become driving organizational goals. But clearly not all innovations are useful in all situations. The World Wide Web is full of useless sites where interactive hypermedia systems have been created simply because being on the Web is the “in” thing; the infamous paper clip in Microsoft Office is another generally recognized example of technology overstepping its usefulness.

Sometimes the most effective designs introduce only minor changes to a situation. Small changes are easier for people to understand and appreciate, as well as easier to implement. But there is clearly a dilemma here: If designers focus too much on users’ current practices, they may miss important opportunities for innovation. Most users will not be familiar with novel technology options. A user-centered development team wants to be solicitous of and responsive to users’ needs and preferences, but in general these users will have a limited ability to apply new technology to their tasks. Of course, this is a major motivation for iterative development, where users are exposed to technology at many different phases of the development process.

In cooperative design (sometimes also called participatory design), an interactive system emerges through a direct collaboration between designers and prospective users. The overall process is one of mutual learning: Software developers interact with user representatives to learn more about the work setting and vice versa (Kyng 1995; Carroll, Chin, et al. 2000). Changes are made and incorporated into work practices in an incremental and iterative fashion. Through direct contributions to design, the user participants come to identify with the new technology, which also aids in acceptance and adoption of the system once it is complete (Greenbaum & Kyng 1991). Cooperative design is a whole lifecycle approach—user participation starts during requirements analysis and continues throughout design, prototyping, and implementation.

When envisioning new applications of software technology, it is important to remember that a person’s experience depends on much more than just a piece of software. It is easy to become consumed with software production. After all, this is what software developers are trained to do—analyze requirements, produce design specifications, and build software. Design documents and prototypes are concrete indications of progress; they can be tested to find out what to do next. However, as discussed in Chapter 2, people’s activities are very much influenced by the social or organizational context of work (Eason & Harker 1989). The scope of design must encompass the entire situation.

The implicit goal of design is to provide computer support for people’s tasks, but putting one task or subtask “online” may make related activities difficult or impossible (Tradeoff 3.2). For instance, shoppers often bring lists along on their expeditions. A designer might note this and plan to include computerized support for shopping lists: If lists are created and maintained online, it becomes easy to keep records, request auto-fulfillment, repeat orders, and so on. But would online shopping lists contribute to an effective design in general? Most shopping lists are created at many different times in the midst of other activities that evoke shopping needs. Often multiple people contribute to a list over a period of time. Unless these interrelated activities can be integrated within a single system, moving the lists online may disrupt the social context of shopping.

Analyzing the role of shopping lists in shopping activities is an example of distributed cognition (Hutchins 1995). Distributed cognition is concerned with how task-relevant information is distributed throughout a situation in many different forms—the knowledge and skills of the people involved, the state of the tools or other artifacts in use, and the recent or long-term memories of specific events. This framework has been applied to the detailed analysis of complex tasks such as flying an airplane (Hutchins & Klausen 1996). However, the general concepts apply just as well to more mundane activities such as shopping, where the contents of a cupboard or a closet represent information that may play a critical role in one or more shopping tasks.

Scenarios help to reason about which aspects of an activity are best supported by computer software. By thinking through a hypothetical situation, designers “try out” new ideas about technology in a realistic setting. The possibilities already present in the real-world are highlighted, providing a balance against the tendency to identify and computerize as many task elements as possible. Scenarios emphasize the context in which activities take place, so they help to recognize how the many different aspects of a situation (including other people and tools) can be combined into an effective activity design.

A third tradeoff in designing for effectiveness is choosing an appropriate level of generality. Designers normally seek solutions that cover as many cases as possible. For example, given the job of designing an online product database, most designers would develop a scheme for classifying, organizing, and retrieving all products in the same way. Such generality has obvious benefits—the database is simple to build and maintain, and browsing and selection work the same for every product. This makes the design quite effective from the perspectives of ease of learning and software maintenance.

The problem is that designing a general solution can diminish effectiveness for specific tasks (Tradeoff 3.3). For instance, suppose specific products are tightly linked in people’s buying activities—when shoppers buy one, they virtually always want the other. A common example is computer hardware and peripherals; a new printer cannot be used until the connecting cable is also purchased. The general-purpose solution treats each product the same, so shoppers would be forced to go through two seemingly independent selection procedures, first browsing and selecting from the printer category, then doing the same for cables. The result is likely to be irritation and uncertainty (how to be certain you have the right cable?). For scenarios like these, a database that supports special-case “bundling” is a better solution, even though it complicates creation and maintenance of the database. Scenario-based reasoning of this sort helps designers find an effective combination of general and task-specific functionality.

3.2 Designing Comprehensible Activities

Designing for effectiveness is not enough—the new activity design must also be understandable. As people work with a computer system, they must be able to tell what goals are possible and whether they are making progress on these goals. They will use whatever knowledge they possess to figure out what is happening; this lets them choose and carry out appropriate actions. Someone who cannot make sense of events will have difficulty planning further action, and is likely to give up in frustration.

In the end, it is a system’s user interface—the information presentation and interaction techniques that are the topics of Chapters 4 and 5—that makes the system more or less comprehensible to users. But during the early phases of activity design, the designer’s model of the system begins to form—his or her own understanding of the information and tasks that will constitute the system. This model will be elaborated and refined through the development process; it guides the development of concrete design artifacts such as scenarios, user actions, and screen displays.

Figure 3.2 contrasts a designer’s model of an online shopping system with how a user might understand the same system. The designer’s model is depicted as an abstract network, implying that it is a systematic and relatively complete body of knowledge. Although a designer’s model is also a mental representation, designers often use technical representations such as feature lists, task analyses, data-flow diagrams, or object-action tables to express these models (van Harmelen 2001). In SBD, designers write scenarios and claims to document and share their design models.

Figure 3.2 A user’s mental model of a software system may be quite different from the designer’s model.

A user’s understanding of a system is likely to be less well formed. We use the term mental model to mean a loosely organized body of concepts and procedures used to select relevant goals, choose and execute appropriate actions, and understand what happens (Carroll & Olson 1988). The diagram depicts the case of a new user who has not yet interacted with the online shopping system—at this point the model consists largely of knowledge about shopping in general (a cashier, a shopping cart) and about other Web systems (e.g., how Web browsers work).

Early in use, a mental model is very much influenced by knowledge of related systems and activities (e.g., shopping in the real world). As experience increases, the user’s mental model becomes more detailed and specific to the system in use. For example, an experienced online shopper may possess very well-defined mental scripts for frequent tasks such as retrieving and browsing sale items. However, mental models are always incomplete and are constantly updated through use.

Designers naturally want to anticipate and facilitate construction of useful mental models. Unfortunately, the psychological processes that create users’ mental states are not directly observable; designers can only guess at mental models based on users’ behavior or comments (Tradeoff 3.4). Different users begin with differing backgrounds, or will have goals that lead them to interpret the same system behaviors in different ways. Thus, despite the strong requirement to design concepts and services that are intuitive and easy to comprehend, designers are simply unable to observe many of the factors that will influence users’ understanding.

One thing we do know is that users find it easier to understand new ideas when they are similar to familiar concepts (Holyoak & Thagard 1995). Users often learn to interact with systems by applying metaphors—a metaphor is a known concept used to understand a new concept by analogy (Carroll & Thomas 1982). In Figure 3.2, the user is relying on pieces of a store metaphor (cashier, shopping cart) to understand online shopping.

Designers often use metaphors deliberately as an aid to comprehension and planning (Erickson 1990; Madsen 1994). For example, most online catalogs have incorporated some version of the shopping cart metaphor, and many advertise sales or provide coupons that can be applied to obtain special prices. The hope is that if a metaphor is followed faithfully, even users with little computing expertise will be able to understand how that part of the system works.

Building a new system by analogy to an existing system simplifies initial comprehension and use. Unfortunately, metaphors can also have a narrowing influence on mental models, because they may limit the kinds of activities users will expect or seek out (Tradeoff 3.5; see also Halasz & Moran 1982). For example, researchers noted that the early users of word processors made extensive reference to a typewriter metaphor (Carroll & Mack 1985; Mack, Lewis, & Carroll 1983). But while this metaphor was helpful in thinking about text input and printing, it led to an overly “physical” view of digital documents. Learners had difficulty understanding editing features such as insert mode, or special characters that turn formatting controls on and off.

Although metaphor breakdowns may lead to errors initially, they also encourage revision and updating of mental models (Carroll, Mack, & Kellogg 1988). For example, Amazon.com provides a shopper review service; people who browse or purchase a product are invited to write a review for others to use in the future. Note, though, that this concept breaks the store metaphor; in the real world, shoppers do not (at least not by convention!) leave personal notes among the shelved goods. But digital views of products are flexible: They can exist many places at once, and may include different views for different tasks. Recognizing why it is okay to “write on” digital versus physical products enriches the user’s mental model of online shopping.

Metaphors are also useful in design brainstorming, when people seek to generate completely novel ideas (see “Of Warehouses and Meeting Places” sidebar). Often there are one or two obvious metaphors, or real-world domains, that share many features with the new functionality. A physical store is an obvious metaphor for online shopping, a town library for a digital library. But in addition to these very similar domains, a designer might explore more unusual analogies to see if new ideas are provoked. Suppose you imagined that an online catalog is like a storybook. This might lead to novel ideas about how to present the company’s products (e.g., via narratives that have actors, motivations, and a plot line). In SBD, metaphorical brainstorming is a key technique for expanding the space of ideas.

3.3 Designing Satisfying Activities

In business settings, computer use is often mandatory. Once an employee is hired, he or she is expected to use the information technology already in place. But even when an organization’s tools appear to be useful and comprehensible, work will suffer if the employee finds them irritating, depressing, or otherwise unpleasant. Such cases may “simply” lead to poor employee morale. But when use is discretionary, tools that fail to satisfy their intended users will simply be discarded.

One way to satisfy users is by automating tedious or error-prone tasks (e.g., retrieving addresses from an alphabetized list of client records). Indeed, many software development projects take task automation as a prime objective, with the general intention of saving time or money. But lurking behind this objective is an important tradeoff: Automating unrewarding work makes people happy, but it is possible to remove so much responsibility from the human that he or she feels unimportant (Tradeoff 3.6). An extreme case is a worker who ends up in front of a computer console, pushing buttons at the appropriate time so that a computer can carry out what was formerly his or her job.

Knowing what to automate requires careful study of the employee’s responsibilities and sources of reward. Most jobs involve some degree of knowledge work—workers apply personal expertise and knowledge to collect the right information and make the right decisions (Muller et al. 1995). Contrast the knowledge work of a reference librarian who works with clients to find useful materials to that of a catalog librarian who indexes and organizes new materials. Giving the reference librarian an online database that automates reference retrieval saves time that can be spent instead on the knowledge-intensive tasks of resource analysis and recommendation. But giving the catalog librarian an automatic classification system (e.g., via keywords) might remove job responsibilities that are crucial to his or her feelings of personal accomplishment.

To find the right balance between automation and personal control, designers must carefully consider all organizational implications of redesigning a business procedure. It is not enough to determine whether a particular job will be faster or easier to complete. In sociotechnical systems theory, technology and the surrounding organization are analyzed, designed, and iterated as a single co-evolving system (Eason 1988; Mumford 1987). This paradigm assumes from the start that successful systems will be those where technology and the social system evolve together over time, each adapting to the other.

Another important question is “satisfying for whom?” Most computer software is used by a single person at a workstation. Individuals create documents in a word processor or debug financial models in a spreadsheet. Ease of use and satisfaction are defined in terms of an individual user’s reactions. But as networks and collaborative systems have become more pervasive, this view has become too simplistic (Tradeoff 3.7). Virtually all work now has collaborative elements. Even the most private and creative activities (e.g., writing a research paper) are shared, displayed, or evaluated by others at some point.

When designing for satisfaction, the increasingly collaborative nature of computer use creates a dilemma: Activities that benefit a group may be experienced as tedious or frustrating for individual group members (Grudin 1994). The analysis of shared calendars at the beginning of this chapter is illustrative—a shared database succeeds to the extent that an entire workgroup cooperates. But if some individuals are expected to do extra work so that others benefit, the perceived inequity will lead to dissatisfaction.

There is no easy answer to this dilemma. It always takes extra effort to share and coordinate activities (Bannon & Schmidt 1991). Information that otherwise might have been left informal and tacit must be made explicit and archived. Policies and decisions about shared resources, access, control, reporting, and so on, must now be developed, communicated, and implemented. The challenge is to design activities that support these group processes, but that are pleasurable or otherwise satisfying for individuals as well.¹

Activity scenarios help to reason about shared work. The real-world usage context contains clues about other stakeholders in an activity. Even if a scenario focuses on the goals and experiences of one individual, it contains information about the setting and about the actor’s motivations and responsibilities. These details lead designers to question and analyze related impacts on coworkers and on the group.

Science Fair Case Study: Activity Design

Moving from problem scenarios to activity design scenarios represents a crucial move in SBD: This is where the design team first introduces concrete ideas about new functionality, and new ways of thinking about users’ needs and how to meet them. Although the overall development process is iterative and highly responsive to feedback, the ideas introduced now will have an especially significant impact, because they will scope and organize much of the detailed design work that follows.

During activity design, scenarios and claims serve as a flexible and inexpensive medium for imagination and discussion. The focus on realistic situations helps designers to generate specific ideas, as well as to think through the implications that these ideas have for the scenario actors. The combination of problem scenarios and claims allows designers to conduct a deliberate and systematic investigation of design possibilities.

Figure 3.3 summarizes the major components of activity design. Problem scenarios are transformed into activity design scenarios through a combined process of brainstorming about design ideas, reasoning from previous claims, and working through the general concerns of activity design (e.g., as discussed in the first half of this chapter). Throughout, the scenarios are evaluated and refined by identifying key features and their consequences for use (via claims analysis). The general design goal is to maintain or enhance positive consequences while removing or minimizing negative consequences (Carroll & Rosson 1992).

Figure 3.3 Designing activities in scenario-based design.

Exploration of ideas (divergent thinking) is a key element of any design process. Although many effective designs make only small changes to current practice (Tradeoff 3.1), it is always important to push on the boundaries of how things are currently done. Detailed problem analysis should be complemented with “out of the box” thinking aimed at opening up the space of possibilities. In SBD we do this by brainstorming about metaphors and technology options.

3.4.2 Activity Design Scenarios and Claims

After exploring metaphors and technology, the design team begins to transform current practice, envisioning new ways to meet the needs of the stakeholders in the problem scenarios. Just as we did when generating problem scenarios, we use a tightly interleaved process of scenario writing and claims analysis.

It is hard to say how to “do design” because so many things contribute at once. Begin by considering a problem scenario, using the claims to identify features of the situation that might be changed by introducing technology. Use the ideas from the metaphors and technology to come up with possible changes. Try out the changes in the scenario to see how it would now transpire, thinking through how the actors would react, what they would want to do next, and so on. At the same time, think about other situations that the feature might impact, or other consequences that it might have. In other words, try to think through the many possible side effects of introducing a design feature into a scenario.

Let’s take as an example one central transformation—putting the science exhibits online. Looking back at Sally’s exhibit-planning scenario, we can think concretely about how such a change might impact her goals and activities (Figure 3.4): Sally thinks the virtual exhibits might give her more options for presenting her work, as well as making the exhibit more interactive. She also understands that the physical constraints of a physical board are no longer in place. At the same time, Sally is concerned about the physical models she has created, wondering how she can share them in an online system. Overall, the positives outweigh the negatives, so we incorporate this feature. We document our analysis of the pros and cons as a claim (the first entry in Table 3.3 on page 100).

Figure 3.4 A problem scenario is transformed into an activity scenario.

Table 3.3 Claims analyzing the key features of the activity design.

Proposed Activity Design Feature	Hypothesized Pros (+) or Cons (-) of the Feature
Putting exhibits online	+ remove many constraints regarding space and diversity in layout + facilitates an iterative process of design, construction, and editing + simplifies access to the exhibits by people separated in space and time
	- but may lead to a decreased emphasis or interest in physical components
	- but exhibitors may try to include too much, making exhibits complex
An exhibit template with traditional science project components	+ simplifies and guides the exhibit planning process + builds on prior exhibiting experience of fair participants + enhances consistency and comparability of exhibits for viewers and judges
Integrating the products of common tools into the online exhibits	- but may discourage more inventive and creative exhibit structures
	+ builds on exhibitors’ existing skills and preferences
	+ extends the apparent diversity of the fair and its services
	- but visitors may be confused about what is and is not “part” of the fair
	- but students may wish that flashy new tools had been provided
Email notices of the virtual science fair	+ can be directed specifically to individuals expected to be interested + may include a direct link to the online activity, simplifying access
	– but people without email accounts may feel excluded or slighted
Exhibiting projects that are not yet completed	+ emphasizes the extended and ongoing nature of science projects + encourages future visits for purposes of checking progress
	- but students may be embarrassed about showing a project that is not yet done
Archiving discussions at an exhibit	+ enables less redundancy in question answering by exhibitors + offers visitors more options, for a richer browsing experience + emphasizes the ongoing and community-oriented nature of the fair
	- but visitors may feel obliged to read all archives before asking anything
Editable judging forms	+ acknowledges that judging is never completely objective or predictable
	+ increases judges’ feeling of control and contribution to the rating process
	- but may lead to evaluations that are difficult to interpret or compare
Authentication of the judging forms	+ reminds judges that their evaluations are valuable and confidential
	- but may be annoying if the number of forms is large
Preserving exhibits after the fair is over	+ simplifies access and review of example projects + emphasizes a view of the fair as an ongoing event, extended in time
	- but isolated exhibits (e.g., without students or other visitors)
provide only a partial and perhaps misleading picture of the overall fair activity

This activity scenario incorporates less obvious ideas as well. The claims analysis of the original scenario documented how the lab table supported Sally’s planning activities, and we wanted to provide similar support in the new design. The solution proposed is a template, offered as a guide to the layout process. Again, we used claims analysis to think through the pros and cons of a template (second claim in Table 3.3): It might be seen as too heavy handed or restrictive by experienced students. We conveyed this as a concern of Sally’s, also noting that a template must have sufficient flexibility for her creative ideas.

We applied similar reasoning to the other scenarios, looking for opportunities to apply new metaphors or insert new technology. The results appear in Figure 3.5. To simplify the case study content, the figure presents abbreviated scenarios that describe just the new design features. Readers may want to refer to the actors described in Figure 2.12 and the problem scenarios in Figure 2.13 to recreate a more complete setting and context for the envisioned activities.

Figure 3.5 Activity scenarios for the virtual science fair (in summary form).

Some of the activity design ideas can be traced to specific metaphors or technology exploration. For instance, the idea to make the judging forms editable came from our thinking about judging as a discussion, which is a more open-ended activity than filling out a form. As we tried this out in the judging scenario, we recognized that providing Ralph with an editable form would give him a greater sense of control, and might encourage him to take action on special cases. This led us to envision a situation in which he has difficulty comparing Sally’s and Jeff’s exhibits, and how he edits the form to deal with this problem. At the same time, we noted a negative consequence—a set of forms may become inconsistent and difficult to merge.

Other design ideas can be traced to general tradeoffs in designing interactive systems. The idea to build exhibits out of existing desktop documents (e.g., Word, Excel; see the first three scenarios) was motivated by the general belief that building from existing practice is a good idea. But the claim documenting this reasoning in Table 3.3 also points to a negative impact—students may be disappointed when they are not offered special-purpose tools with flashy graphics or other user interaction novelties (Tradeoff 3.1).

It is important to remember that the claims in Table 3.3 were generated in parallel with the activity scenarios. Each claim documents a design feature that was considered and debated with respect to its pros and cons. Other features were considered but rejected because the cons outweighed the pros. For example, we considered that form editing could be a collaborative activity involving all judges, but decided that this would require too much inter-judge coordination and was not realistic for volunteers at a science fair.

In addition to documenting key design decisions, claims bring out issues for future design work. For instance, one claim raises a flag about the potential complexity of an unlimited presentation space. In response, we have proposed the general idea of exhibit layers, but this needs further design work. Similarly, we have suggested that conversations at an exhibit can be stored, but future design work should consider how to convey that the implied task of reviewing these archives is optional.

Later on, we will build design prototypes to test our claims, so that we can see if the analyzed pros and cons match users’ actual experience (Chapter 6). But right now, the scenarios serve as a flexible and inexpensive medium for trying out the new ideas. Each proposal is elaborated and given substance as the actors’ actions and experiences are specified. The scenario context helps us to consider whether and how a feature is meeting the actors’ needs, what the actors might be thinking as a feature is encountered, and whether they will understand what has happened when the task has been completed.

These scenarios and claims address system functionality only; they explore people’s general goals and activities, not the details of information presentation or physical interaction. The exhibit planning scenario describes Sally’s concerns and plans, but says nothing about how she interacts with the science fair or its tools. The judging scenario assumes that the form can be edited, but offers no details about form appearance or editing procedures. These user interface details will be the topic of Chapters 4 and 5.

3.4.4 Participatory Design

Activity design scenarios and claims can be elaborated and refined in cooperation with stakeholders. Because the design at this point describes only the system’s functionality, such interactions will be quite informal. For example, a design team might revisit the individuals or groups observed during the field study. They can present the activity design scenarios and gather reactions and suggestions for change. Some stakeholders may even enjoy using claims analysis or the point-of-view technique to come up with new ideas about their task objects.

The use of participatory development methods was an important starting assumption for the virtual science fair (as documented in our root concept; see Table 2.2). Thus very early in the project, we conducted participatory sessions with stakeholders (community members, students, and teachers). We asked them to imagine system tools that could be active and helpful agents in the task. We gave them index cards labeled “Object name” and “Object responsibilities” as prompts for these ideas. A sample result appears in Figure 3.6, summarizing how a calendar tool could help out in an exhibit.

Figure 3.6 Card describing possible contributions of a “helpful” calendar in a science fair scenario.

Of course, inviting users to work with a computational metaphor is just one technique for expanding their views of current activities with computing support. Any form of metaphorical thinking can help users think “outside the box.” For example, we could have asked users to discuss and expand the metaphors in Table 3.1. In participatory design work, we have collaborated with users to develop claims analysis of their current practices, and then together generate design ideas that addressed the issues raised by the shared analysis (Chin, Rosson, & Carroll 1997).

Plate 3 A user’s experience with a system is a combination of the underlying system functionality and the user interface provided for interacting with these functions.

Plate 4 An information display designed using a grid. The image below shows the grid used to size and position the elements in a regular and pleasing fashion.