1. The Nature of the Beast

In 1962, The first modern video game was invented by some hackers at MIT. It was called Spacewar!, and it ran on a DEC PDP-1, the world’s first mini-computer, connected to a CRT display. One of the game’s designers explained that the game was born as a group sat around trying to figure out what sort of “interesting displays” they could create for the CRT with some pattern-generating software they had developed. “We decided that probably, you could make a two-dimensional maneuvering sort of thing, and decided that naturally the obvious thing to do was spaceships.” The MIT hackers weren’t the only ones to invent Spacewar!. As Alan Kay noted, “the game of Spacewar! blossoms spontaneously wherever there is a graphics display connected to a computer” (Brand 1974).

Why was Spacewar! the “natural” thing to build with this new technology? Why not build a pie chart, an automated kaleidoscope, or a desktop? Its designers identified action as the key ingredient, and conceived Spacewar! as a game that could provide a good balance between thinking and doing for its players. They regarded the computer as a machine naturally suited for representing things that you could see, control, and play with. Its interesting potential lay not simply in its ability to perform calculations, but in its capacity to co-create and represent actions with human participants.

1. A way for a person to communicate with a computer.

2. A way for a computer to communicate with a person.

3. A surface through which humans and computers can communicate.

4. A way for humans and computers to construct actions together.

Number three comes close, but it implies a membrane or separation between the human and the computer. But the object is not the membrane; rather it is the action co-created by the human and technical forces at play. The difference in emphasis may be the impetus of the trend toward replacing the term “human-computer interface” with “human-computer interaction” in recent years.

There are two major reasons for belaboring such a seemingly obvious point. First, it wasn’t always true—and the design disciplines for applications and interfaces still bear the marks of that former time. Second, reconceptualizing what computers do as enabling and representing actions that involve both human and technological participants suggests a design philosophy that diverges significantly from much of the received wisdom about interface design.

“Interface” became a trendy (and lucrative) concept in the 1980s and 1990s—a phenomenon that is largely attributable to the introduction of the Apple Macintosh. Interface design was concerned with making computer systems and applications easy to use (or at least usable) by humans. When we thought of human-computer interfaces in those days, we were likely to visualize icons and menu bars or perhaps command lines and blinking cursors. But, of course, many conceptions came before as well as after.

John Walker (founder and president of Autodesk, Inc.) provides an illuminating account of the “generations” of user interface design (Walker 1990). In the beginning, says Walker, there was a one-on-one relationship between a person and a computer through the knobs and dials on the front of massive early machines like the ENIAC. The advent of punched cards and batch processing replaced this direct human-computer interaction with a transaction mediated by a computer operator. Time-sharing and the use of “glass teletypes” reintroduced direct human-computer interaction and led to the command-line and menu-oriented interfaces with which the senior citizens of computing (people over forty) are probably familiar. Walker attributes the notion of “conversationality” in human-computer interfaces to this kind of interaction, in which a person does something and a computer responds—a tit-for-tat interaction.

This simplistic notion of conversation led many early interface specialists to develop a model of interaction that treats human and computer as two distinct parties whose “conversation” is mediated by the screen. But as advances in linguistics demonstrated, there is more to conversation than tit for tat. Dialogue is not just linearized turn-taking in which I say something, you go think about it, then you say something, I go think about it, and so on. An alternative model of conversation employs the notion of common ground, as described by Herbert H. Clark and Susan E. Brennan (1990):

It takes two people working together to play a duet, shake hands, play chess, waltz, teach, or make love. To succeed, the two of them have to coordinate both the content and process of what they are doing. Alan and Barbara, on the piano, must come to play the same Mozart duet. This is coordination of content. They must also synchronize their entrances and exits, coordinate how loud to play forte and pianissimo, and otherwise adjust to each other’s tempo and dynamics. This is coordination of process. They cannot even begin to coordinate on content without assuming a vast amount of shared information or common ground—that is, mutual knowledge, mutual beliefs, and mutual assumptions (see Clark and Carlson 1982, Clark and Marshall 1981, Lewis 1969, Schelling 1960). And to coordinate on process, they need to update, or revise, their common ground moment by moment. All collective actions are built on common ground and its accumulation.

In her work in applying the notion of common ground to human-computer interfaces, Brennan (1990a) suggests that common ground is a jointly inhabited “space” in which meaning takes shape through the collaboration and successive approximations of the participants. Brennan’s work was aimed at designing human-computer interfaces so that they offer means for establishing common ground (“grounding”) that are similar to those that people use in human-to-human conversation, such as interruptions, questions, and utterances and gestures that indicate whether something is being understood (Brennan 1990b).

Successful graphical interfaces, exemplified early on by the Macintosh, explicitly represented part of what Clark called the “perceptual common ground” of interaction through the appearance and behavior of objects on the screen (Clark 1996). Some of what goes on in the representation is exclusively attributable to either the person or the computer, and some of what happens is a virtuous artifact of a collaboration in which the traits, goals, and behaviors of both are inseparably intertwined.

The concept of common ground not only provides a superior model of the conversational process, but it also supports the idea that an interface is not simply the means whereby a person and a computer represent themselves to one another; rather, it forms a shared context for action in which both are agents.¹ When the old tit-for-tat paradigm intrudes, the “conversation” is likely to break down, once again relegating person and computer to opposite sides of a “mystic gulf”² filled with hidden processes, arbitrary understandings and misunderstandings, and power relationships that are competitive rather than cooperative. Mistakes, unanticipated outcomes, and error messages are typical evidence of such a breakdown in communication, in which the common ground becomes a sea of misunderstanding.

1. This book employs the noun “agent” to mean one who initiates action. This definition is consistent with Aristotle’s use of the concept in the Poetics.

2. The term “mystic gulf” is attributed to composer Richard Wagner to refer to the gap between audience and actors created by the orchestra pit.

But even “good” metaphors don’t always work as intended. Several years after the introduction of the Mac, in an informal survey of Macintosh-literate university students, for instance, many people failed to employ the word “desktop” anywhere in their description of the Finder.³ Where an interface metaphor diverges significantly from its real-world referent, people proceed by accounting for the behaviors of particular “objects” on the screen with ad hoc explanations of system operation that are often incorrect: a “naïve physics” of computing (see Owen 1986). In such cases, metaphors do not serve as “stakes in the common ground,” but rather as cognitive mediators whose labels may be somewhat less arcane (but possibly more ambiguous) than a computer scientist’s jargon.

3. The Macintosh Finder is an application for managing people’s file systems and for launching other applications. It comes with the system and is automatically launched when the machine is turned on.

Since the introduction of the Mac, we have seen a variety of interface metaphors, both local and global. With the advent of the World Wide Web, we began to speak of a Web page as if it were a page in an enormous book. When things turned out to be a little more complicated, with hyperlinks both within and without, the terrain of the Web was re-visualized in terms of geography with Web sites. The notion of the Web portal was based on the idea that a wise provider would open a view of the Web that would protect us from chaos and provide uniform representations of information, but which might well lob us into sites and pages with diverse characteristics and an unexpected entrance into the Wild Wild Web, or else, if we wanted to stay safe, we could choose to remain penned up in walled gardens.

Names also change as technology and design advance. In the 1940s, for instance, some people had “car phones” that worked with radio technology. With the development of a reliable cellular system, we had handheld mobile devices that we called “cell phones.” When mobile phones began to have something like a browser, some “apps,” and messaging capabilities, we began to call them “smart phones.” These phone names are not so much driven by metaphor as by somewhat naïve understandings of technology. But metaphors are still with us; for example, we see terms like “notebook” and “tablet” used to describe computers with certain dimensions and capabilities, even though one cannot typically sketch on, scribble on, or tear out and wad up a page.

A behavioral metaphor that has been rather more successful and pervasive is the notion of direct manipulation (discussed in more depth ahead), in which users can move objects about the screen in much the same way as they might in the physical world. Although the operations and conventions implicit in direct manipulation interfaces require more procedural learning than actually picking something up, the value of the metaphor is strong enough to boost most people up the learning curve.

Although interface metaphors can fail in many ways (as discussed later in this book), their prevalence has expanded the domain of interface design to admit contributions from specialists in graphic and industrial design, linguistics, psychology, education, and other disciplines. The metaphorical approach contributed to making interface design an interdisciplinary concern. They became lightning rods for people from many disciplines, either in service of or in reaction against them.

Engelbart believed that there were fantastic new horizons for human potential with computers. The notion of augmentation, while not a metaphor, was a vision that drove all of his work and solidified his team. It was only much later that the world rewarded him for it. He was ahead of his time, and the fledgling industry predictably pulled away key members of his team for shorter-term profit-making ventures.

Engelbart’s legendary demo in 1968 was an incredible theatrical triumph as well as a technological one. Later nicknamed “the Mother of All Demos,” Englebart sat on stage in San Francisco while his team was in Menlo Park. Engelbart recalls:

Our computer was down at SRI in Menlo Park. In order to demo it, we beamed two channels of video along two microwave links up to San Francisco, bouncing them off dishes above the airport. There was only one video projector on the West Coast powerful enough for the conference hall, a Swedish Eidophor that I had to borrow from NASA. It was huge, maybe 6 feet tall. Then we rigged up a homemade modem—2,400 baud—to get signals from my console in San Francisco back to SRI over a leased line.

On stage right was a big screen, 22 feet high. At the side of my display monitor, a camera pointed right at my face. Another camera was pointing down to capture my hands at the keyboard. It was pretty elaborate. My face would be on one side of the screen, with text on the other—or on a split screen with people in Menlo Park showing something as I talked about it. I’m told that this is the original videoconferencing demo (Jordan and Englebart 2004).

The theatre of the live performance and the skin-of-your-teeth presentation technology may sound familiar to theatre folk, but they were so far away from the culture of computing at the time that they made an indelible impact on the audience. Engelbart’s demo lived on in the culture of SRI as well as the culture of PARC. The MIT Architecture Machine Group (later to become the MIT Media Lab) was still relying on demos that were often mock-ups, deconstructed as soon as they had been shown. At the same time, the content of the demo marked a major turning point in the practice, technology, and purpose of interface design.

4. The literature in human factors and other psychological perspectives on human-computer interaction is huge. It is beyond the scope and purpose of this book to provide even a cursory survey of the entire domain. The work mentioned in this chapter is selected in terms of its relevance to the thesis of this particular book. Interested readers may wish to review The Human Factor by Richard Rubinstein and Harry Hersh, which includes an excellent bibliography; Readings in Human-Computer Interaction, by Ronald M. Baecker and Willam A.S. Buxton; or the various proceedings of ACM SIGCHI and the Human Factors Society.

Norman’s emphasis on action as the stuff that interfaces both enable and represent bores a tunnel out of the labyrinth of metaphor and brings us back out into the light, where what is going on is larger, more complex, and more fundamental than the way that the human and the computer “talk” to each other about it.

Norman’s insights dovetail nicely with those of the “common ground” linguists, suggesting a notion of the interface that’s more than screen-deep. The interface becomes the arena for the performance of some intentional activity in which both human and computer have a role. What is represented in the interface is not only the task’s environment and tools, but also the process of interaction—the contributions made by both parties and the evidence of the task’s evolution. I believe that Norman’s analysis supports the view that interface design should concern itself with representing whole actions with multiple agents. This is, by the way, precisely the definition of theatre.

Norman was also a key figure in the development of another pivotal interface concept, the idea of direct manipulation. Direct manipulation interfaces employ a psychologist’s knowledge of how people relate to objects in the real world in the belief that people can carry that knowledge across to the manipulation of virtual⁵ objects that represent computational entities and processes. The term “direct manipulation” was coined by Ben Shneiderman of the University of Maryland, who listed these key criteria:

5. The adjective “virtual” describes things—worlds, phenomena, etc.—that look and feel like reality, but which lack the traditional physical substance. A virtual object, for instance, may be one that has no real-world equivalent, but the persuasiveness of its representation allows us to respond to it as if it were real.

1. Continuous representation of the object of interest.

2. Physical actions or labeled button presses instead of complex syntax.

3. Rapid incremental reversible operations whose impact on the object of interest is immediately visible (Shneiderman 1982).

Shneiderman (1982) reported that direct-manipulation interfaces can “generate a glowing enthusiasm among users that is in marked contrast with the more common reaction of grudging acceptance or outright hostility.” In a cognitive analysis of how direct manipulation works, Edwin Hutchins, James Hollan, and Don Norman suggest that direct manipulation as defined may provide only a partial explanation of such positive feelings. They posit a companion effect, labeled direct engagement: A feeling that occurs “when a user experiences direct interaction with the objects in a domain” (Hutchins et al. 1986). They add the requirements that input expressions must be able to make use of previous output expressions, that the system must create the illusion of instantaneous response (except where inappropriate to the domain), and that the interface must be unobtrusive.

It seems likely that direct manipulation and direct engagement are head and tail of the same coin (or two handfuls of the same elephant): one focusing on the qualities of action and the other focusing on subjective response. The basic issue is what is required to produce the feeling of taking action within a representational world, stripped of the “meta-context” of the interface as a discrete concern. Hutchins et al. sum it up this way: “Although we believe this feeling of direct engagement to be of critical importance, in fact, we know little about the actual requirements for producing it” (Hutchins et al. 1986).

Nearly 20 years later, in his book Emotional Design, Norman (2004) says:

We cognitive scientists now understand that emotion is a necessary part of life, affecting how you feel, how you behave, and how you think. Indeed, emotion makes you smart.

In more recent years, Norman might say that direct engagement arises from the emotional pleasure of a well-designed affordance; the characteristics of immediacy and lack of fussy procedural steps simply make direct manipulation feel good to us.

Here, I think, is an important articulation between psychology, interface design, and theatre. Direct engagement in the theatre arises first of all from real-time enactment and the enhanced attention it evokes. Audiences (and actors) have immediate emotional responses to the action on stage. Over the course of a play, emotions take on greater resonance, ideally producing empathy (literally, “feeling with” the characters). The interface (the venue, stage machinery, etc.) is not a matter of direct concern; when an audience is directly engaged with the action of the play, these elements literally disappear from conscious awareness. Further, theatrical audiences have an expectation of emotional pleasure. We will examine the nature of that pleasure in the next chapter.

Psychology is a familiar domain to dramatists, actors, and other theatre artists because of its focus on the human mind, behavior, and emotions. Understanding how psychology and theatre are alike and different may illuminate the distinct contributions that each can make in the field of human-computer interaction. The two domains have several elements in common. Both concern themselves with how agents relate to one another in the process of communicating, fighting, solving problems, building things, having fun—the whole range of human activity. Both interpret human behavior in terms of emotions, goals, conflicts, discoveries, changes of mind, successes, and failures. Both observe and analyze human behavior, but each employs those means to different ends: In general, psychology attempts to understand what goes on with humans in the real world with all their fuzziness and loose ends, while theatre means to represent a kind of thing that might go on, simplified for the purposes of logical and affective clarity. Psychology explicates human behavior, while theatre represents it in a form that provides intellectual and emotional closure. Theatre is informed by psychology, but it turns a trick that is outside of psychology’s province through the direct representation of action.

In many ways, the role of the graphic designer in interface design is parallel to the role of a theatrical scene designer. Both create representations of objects and environments that provide a context for action. In the theatre, the scene designer provides objects like teacups and chairs (props), canvas-covered wooden frames that are painted to look like walls (flats), and decorative things like draperies and rugs (set dressing). The behaviors of these elements are also designed: doors open, make-believe bombs explode, trick chairs break in barroom brawls. The lighting designer uses elements of color, intensity, and direction to illuminate the action and its environment and to focus our attention on key areas and events. In interface design, animation has been used increasingly as processing power has grown.

Scene and light designers use such elements as line, shadow, color, texture, and style to suggest such contextual information as place, historical period, time of day, season, mood, and atmosphere. Theatrical designers also employ metaphor (and amplify the metaphors provided by the playwright) in the design of both realistic and non-realistic pieces: The looming cityscape around Willy Loman’s house in Death of a Salesman metaphorically represents his isolation and the death of his dreams; abstract webs of gauzy fabric suggest the multiple layers of illusion in the personality of Peer Gynt. At Ohio State University, the Advanced Computing Center for the Arts and Design (ACCAD) has collaborated with the Departments of Theatre and Dance to produce real-time visual effects, including characters projected from the motion-capture studio onto the stage where they interact with “live” actors.

In interface design, graphic designers and animators make the same sorts of contributions. They render the objects and environments in which the action of the application or system will occur, imparting behaviors to some objects and representing both concrete and ephemeral aspects of context through the use of such elements as line, shadow, color, intensity, texture, and style. Such familiar metaphors as desktops and windows provide behavioral and contextual cues about the nature of the activity that they support.

Sound and music design in interactive media—especially games—has become increasingly important and sophisticated. The introduction of spatialized sound into computer games in the late 1990s accelerated the development of sound-design tools, technology, expertise, and curricula. Simpler sounds give us cues as well as a sense that something “real” is going on, from the minimalist “whoosh” when you send a message to the “crackle” of “paper” when you drag something to the “trash” on the Mac.

Both theatrical design and interface design are aimed at creating representations of worlds that are like reality, only different. But a scene design is not a whole play; for that we also need representations of character and action. Likewise, the elements of interface design are only part of the whole representation that we call human-computer interaction.

How should an interface come to be? In effective interaction design, the interface does not come last; it develops throughout with the entire design process. It is deeply entwined with functionality. It shows sensitivity to the interactor and sometimes even constrains functionality that cannot or need not be touched effectively by the interactor. If we think of an application as an organic whole, the process by which it is created should be organic as well.

I became a software designer and programmer, working primarily on interactive fairy tales and educational programs for kids. The company was called CyberVision, and the machine was a lowly 1802 processor with a four-color, low-resolution display and an alphanumeric keypad. The CyberVision computer was cassette-loaded with 2K of RAM, and it had the capacity to synchronize taped audio with animation on the screen. My first “feature” was an interactive, animated version of “Goldilocks.” Later, I created the first lip-synching on a microcomputer for a game of “Hangman” in which the evil executioner delivered menacing lines in a Transylvanian accent (all this with only sixteen lip positions). I immediately became immersed in mapping my knowledge of drama and theatre to the task at hand because the two media were so obviously alike. There were characters, emotions, and actions. I could imagine other worlds through the looking glass, and I could imagine reaching into them.

Working at CyberVision was a time of wonder. It was also a time of wrestling with unfamiliar technologies with for novel purposes, hamstrung by the tiny RAM, the slow BAUD rate, fat pixels and “cyberwind” that smeared colors across the screen, and an alphanumeric keypad as an input device. The ability to synchronize audio with video was a great aspect of cassette tape; oxide dropouts were emphatically not. Most troubling was the need to create converging nodes in our branching tree architectures because consequential choices could not be held in tiny memory past the boundary of a 2K data load. A vision of possibility was emerging for me—meaningful interactions, responsive worlds—but I didn’t know how to pursue it.

When CyberVision folded to its competition (an upstart company called Atari), I asked my boss to help me think about what kind of job to look for next. He said, “Why don’t you go work for a bank? They need people to help design automated teller machines.” “I don’t know anything about that,” I cried. “Of course you do,” he replied. “That’s human factors.” In response to my blank look, he elaborated, “That’s making computer things easy for people to use.”

What a concept!

I ended up going to work for Atari, not a bank, but the notion of ease of use as a design criterion fit neatly and permanently into my developing intuitions about how theatrical expertise could inform the art of designing software. There’s nothing between the audience and the stage but some good illusion. Clearly, I was on the right track. But I hadn’t run into the other “i” word yet.

I got off to a rocky start in the software branch of the newly-minted Atari Home Computer Division, first as a software specialist for educational applications, then as a producer, and finally as director of software product management. That job included thinking up what would make good applications, getting buy-ins from marketing and engineering, and doing much of the basic design work. Various producers managed budding application areas as we struggled to understand what would differentiate the Atari 400/800 from the Atari Video Computer System (VCS, later 2600) as well as from competitors like the Apple 2.

Product differentiation (or imaginings of what personal computers could do) was the sticking point between me and the president of the company. He wanted me to devote 80% of our budget to porting VCS games to the 400/800 computers. I consistently resisted. Finally, I went to the head of the Home Computer Division and drew him a map of all the areas in which we could be creating applications that had no equivalents in game systems—personal finance, education, personal development, and useful tools like word processing and spreadsheets. He gave me a raise and a promotion.

Warner, the new corporate owner of Atari, did not share my views on diversifying the product offering. The Warner folks believed that, while videogames may be permanent fixtures of our culture, personal computers were likely a fad (in fact, I participated in a research gathering where some significant luminaries agreed, including Stewart Brand, who compared personal computers to jogging—no offense to Stewart; these were jogging’s early days, and it was hard to tell whether it was a fad). Further, Warner’s idea of great videogames went no further than great movie or comic licenses. The great game designers recruited by Atari founder Nolan Bushnell got little support for developing high-quality games from the new regime, including a lack of personal credit, but those who remained after the Warner acquisition continued to do mostly great work, at least until the E.T. disaster. Most important, perhaps, was the fact that the Atari VCS actually provided both a better processor and a better interface for most Atari games.

As the corporate axe began to swing my way, I literally ran over to the Atari System Research Lab to ask Alan Kay to let me work there. I wanted to devote time to thinking through what I had come to believe about computers and theatre (I also needed to begin my dissertation, which I had decided would be on that subject). Alan gave me the opportunity. “Interface” was every other word in the conversations of the bright young MIT wizards that populated the lab. I dimly perceived that there must be more to it than ease of use, and so I signed up for a weekly interface seminar that one of the psychologists on staff was conducting.

Compelling as its simplicity might make it, this model was immediately dismissed by everyone in the group. In order for an interface to work, the person has to have some idea about what the computer expects and can handle, and the computer has to incorporate some information about what the person’s goals and behaviors are likely to be. These two phenomena—a person’s mental model of the computer and the computer’s “understanding” of the person—are just as much a part of the interface as its physical and sensory manifestations (see Figure 1.2).

But in order to use an interface correctly, a person must also have an idea of what the computer is “expecting” her to do. If you are going to admit that what the two parties “think” about each other is part of what’s going on, you will have to agree that what the two parties think about what the other is thinking about them must perforce be included in the model (see Figure 1.3). This elaboration has dizzying ramifications.

Faced with this nightmare, the group at the Atari Lab abandoned the topic and turned their attention to more manageable concepts, such as the value of multisensory representations.

Over the years, I have frequently observed interface workers backing away from such gnarly theoretical discussions in favor of the investigation of more tractable issues of technique and technology—such subjects as direct manipulation, “user” testing, online help functions, animation, and sound and speech, gesture, body tracking, and facial recognition. These areas contain hard problems and add greatly to the potential for interface design, but they do not necessarily advance the theoretical conversation. The working definition of the interface has settled down to a relatively simple one—how humans and computers interact (see Figure 1.4)—but it avoids the central issue of what this all means in terms of reality and representation.

It occurs to me that when we have such trouble defining a concept, it usually means that we are barking up the wrong tree.

6. In his book The Elements of Friendly Software Design (1982), Paul Heckel remarks, “When I design a product, I think of my program as giving a performance for its user.”

Part of the technical magic that supports the performance is embodied in the scenery and objects on the stage (windows that open and close, teacups that break); the rest happens in the backstage and wing areas (where scenery is supported, curtains are opened and closed, and sound effects are produced), the loft area above the stage, which accommodates lighting instruments and backdrops or set pieces that can be raised and lowered, and the lighting booth, which is usually above the audience at the back of the auditorium. The magic is created both by people and machines, but who, what, and where they are do not matter to the audience.

It’s not just that the technical underpinnings of theatrical performance are unimportant to audience members; when a play is working, audience members are simply not aware of the technical aspects at all. For the audience member who is engaged by and involved in the play, the action on the stage is all there is. In this sense, plays are like movies: When you are engrossed in one, you forget about the projector, and you may even lose awareness of your own body. For the actor on stage, the experience is similar in that everything extraneous to the ongoing action is tuned out, with the exception of the audience’s audible and visible responses, which are often used by the actors to tweak their performance in real time (this, by the way, reminds us that theatrical audiences are not strictly passive and may be said to influence the action). For actor and audience alike, the ultimate reality is what is happening in the imaginary world on the stage—the representation.

As people grapple with the notion of interaction in the world of computing, they sometimes compare computer users to theatrical audiences (see Figure 1.6). “Users,” the argument goes, are like audience members who are able to have a greater influence on the unfolding action than simply the fine-tuning provided by conventional audience response. In fact, I used this analogy in my dissertation in an attempt to create a model for interactive fantasy. The user of such a system, I argued, is like an audience member who can march up onto the stage and become a character, shoving the action around by what he says and does in that role.

But let’s reconsider for a minute. What would it be like if the audience marched up on the stage? (See Figure 1.7.) They wouldn’t know the script, for starters, and there would be a lot of awkward fumbling for context. Their clothes and skin would look funny under the lights. A state of panic would seize the actors as they attempted to improvise action that could incorporate the interlopers and still yield something that had any dramatic integrity. Or perhaps it would degenerate into a free-for-all, as performances of avant-garde interactive plays in the 1960s often did.

The problem with the audience-as-active-participant idea is that it adds to the clutter, both psychological and physical. The transformation needs to be subtractive rather than additive. People who are participating in the representation aren’t audience members any more. It’s not that the audience joins the actors on the stage; it’s that they become actors—the notion of observers goes away.

In this view, the “stage” is a virtual world. It is populated by agents, both human and computer-generated, and other elements of the representational context (windows, teacups, desktops, or what-have-you). The technical magic that supports the representation, as in the theatre, is behind the scenes. Whether the magic is created by hardware, software, or wetware is of no consequence; its only value is in what it produces on the “stage.” In other words, the representation is all there is (see Figure 1.8). Think of it as existential WYSIWYG.⁷

7. WYSIWYG stands for the rubric “what you see is what you get,” coined by Warren Teitelman at Xerox PARC. It has been held up as a paradigm for direct-manipulation interfaces, but some theorists have contested its value (see, for instance, Ted Nelson’s (1990) article “The Right Way to Think about Software Design” in The Art of Human-Computer Interface Design).

Focusing on human agency allows us to simplify another perpetually problematic concept, the notion of interactivity. People in the computer game business have been arguing about it for decades. In 1988, the first conference aimed at bringing together people from all sectors of the interactive entertainment business took place in New York.⁸ People came from such diverse industries as personal computers, videogames, broadcast and cable television, optical media, museums, and amusement parks. Over the course of the two days, a debate about the meaning of the word “interactive” raged through every session, disrupting carefully planned panels and presentations. People seemed to regard “interactivity” as the unique cultural discovery of the electronic age, and they demanded a coherent definition. Several speakers tried to oblige, but no one succeeded in presenting a definition that achieved general acceptance. Many participants departed angry and dissatisfied. Such conversations persist today, at ACM SIGCHI, South by Southwest Interactive, the Game Developers’ Conference, and many others. It has also become the topic of dozens—if not hundreds—of books. The conversation has become much more diversified and nuanced, but the nature of interactivity continues to generate new theories and controversies (see, for example, Dubberly et al. 2009).

8. INtertainment was an annual conference sponsored by Alexander Associates.

In the past, I posited that interactivity exists on a continuum that could be characterized by three variables: frequency (how often one could interact), range (how many choices were available), and significance (how much the choices really affected matters) (Laurel 1986a and b). In his book Expressive Processing (2009), Noah Wardrip-Fruin gives us a good test for significance: “What changes to the state of the system and influence on future operations can be produced by this interaction” (p. 75). A not-so-interactive computer game judged by these standards would only let you do something once in a while, only give you a few things to choose from, and the things you could choose wouldn’t make much difference to the whole action (or produce significant changes to the state of the underlying system). A very interactive computer game (or desktop or flight simulator) would let you do something that really mattered at any time, and it could be anything you could think of.

But these variables provide only part of the picture. There is another, more rudimentary measure of interactivity: You either feel yourself to be participating in the ongoing action of the representation or you don’t. Successful orchestration of the variables of frequency, range, and significance can help to create this feeling, but it can also arise from other sources—for instance, sensory immersion and the tight coupling of kinsethetic input and visual response. If a representation of the surface of the moon lets you walk around and look at things, then it probably feels pretty damned interactive, whether your virtual excursion has any consequences or not. It’s enabling a person to act within a representation that’s important. Optimizing frequency, range, and significance in human choice-making will remain inadequate as long as we conceive of the human as sitting on the other side of some barrier, poking at the representation with a joystick or a mouse or a virtual hand. You can demonstrate Zeno’s paradox⁹ on the “user” side of the barrier until you’re blue in the face, but it’s only when you traverse it that things get “real.”

9. Zeno’s paradox (called the theory of limits in mathematics) says that you can never get from here to there because you can only get halfway, then halfway of halfway, etc. Mathematics offers a solution; so does common sense. But the paradox is compelling enough to have interested logicians and mathematicians for centuries.

Wardrip-Fruin (2009) suggests an alternative to sensory immersion as a way to intensify the experience of interactivity. He argues for “systems that more clearly communicate their structures to audiences.” In what he calls “the SimCity effect,” the experience of interaction is enhanced, paradoxically, when players incrementally build “a model of the system’s internal processes based on experimentation.” This model brings players’ initial expectations into line with the capabilities of the game, dissolving an important barrier to successful (pleasurable) interaction.

The experience of interactivity is a “thresholdly” phenomenon, and it is also highly context-dependent. The search for a definition of interactivity diverts our attention from the real issue: How can humans participate as agents within representational contexts? Actors know a lot about that, and so do children playing make-believe. Buried within us in our deepest playful instincts, and surrounding us in the cultural conventions of theatre, film, and narrative, are the most profound and intimate sources of knowledge about interactive representations. A central task is to bring those resources to the fore and to use them in the design of interactive systems.

So now we have at least two reasons to consider theatre as a promising foundation for thinking about and designing human-computer experiences. First, there is significant overlap in the fundamental objective of the two domains—that is, representing action with multiple agents. Second, theatre suggests the basis for a model of human-computer activity that is familiar, comprehensible, and evocative. The rest of this book will explore some of the theoretical and practical aspects of theatre that can be directly applied to the task of designing human-computer experiences. But there are a few more stones to be turned in arranging the groundwork for this discussion.

10. The same argument was used a few decades ago to ban bright colors, potted plants, and chatchkas from the workplace; but that’s another story.

The no-frills view that permeates thinking about interfaces of “serious” applications is the result of a fundamental misunderstanding of the nature of seriousness in representations. The idea that theatre is “really not real” and is therefore unsuited as an approach to serious human-computer activities is misguided, because those activities are “really not real” in precisely the same ways. Without the representation, there is nothing at all—and theatre gives good representation.

Human-computer interaction may be divided into two large categories: productive and experiential (Laurel 1986b). Experiential activities, such as computer games, are undertaken purely for the experience afforded by the activity as one engages in it, while productive activities such as word processing have outcomes in the real world that are somehow beyond the experience of the activity itself. They are often mistakenly defined in terms of their artifacts—a printed document or a spreadsheet filled with numbers. But seriousness is not equivalent to concreteness. A printed paper (such as this manuscript, for example) has “real” implications (e.g., transmitting knowledge, changing how something is done, receiving a grade, or getting paid) even though it is itself a representation. “Productivity” as a class of applications is better characterized, not by the concreteness of outcomes, but by their seriousness vis-à-vis the real world.

There is a parallel here with seriousness as an aspect of drama. In formal terms, “serious” treatments of subjects are reserved for tragedy (and in some senses, melodrama), and “non-serious” treatments are found in melodrama, comedy, farce, and satire. Here again, although the plays themselves are representations, seriousness depends largely on the consequences of the actions represented in them. In a serious work like Hamlet, for instance, falling down (as does Ophelia after her father’s death) has serious consequences both physically and symbolically, while in a farce, falling down (tripping over a piece of furniture or slipping on a banana peel, for instance) causes no permanent injury or pain to the agent.

To trace these effects through to the real world, we need to look at their impact on audiences. Ophelia’s fall and its symbolic meaning impart information about suffering, revenge, and the consequences of evil that can be contemplated, absorbed, and acted upon by an audience. The fall of a clown, on the other hand, may arouse laughter and ephemeral pleasure; it may also, as in more thoughtful flavors of comedy, communicate a philosophical view (e.g., a lighthearted attitude toward random accidents). Seriousness in both theatre and human-computer activities is a function of the subject and its treatment in both formal and stylistic terms. Drama provides means for representing the whole spectrum of activity, from the ridiculous to the sublime.

Another objection to a theatrical approach is that theatre by its very nature is “fuzzy,” while serious applications of computers require crystal clarity. The connotation of fuzziness probably derives from drama’s emphasis on emotion—subjective experience—while serious productivity is seen to require undiluted objectivity. Yet such “serious” tasks as formatting a paper for publication or designing a business plan for a new product can involve a far greater degree of subjectivity (in terms of creativity and evaluation, for instance) than “objective” skill and action (cutting and pasting, typing, and mousing around). At the farthest extreme, the notion that serious applications require objectivity, clarity, and precision is used as a rationale for rejecting natural-language interaction because the success of machine understanding, at least in leading contemporary approaches, is probabilistic, whereas the understanding of symbolic logic (in mathematical or numerical representations) is seen to be unambiguous.

Yet people often drown in precision because of the complexity and artificiality of its expression (both lexical and syntactic). From the gamer grappling with a parser to the inexperienced Linux user trying to “alias” a complicated e-mail address, people experience the requirement for precision as troublesome. This is no secret; the problem is commonly acknowledged and wrestled with by most interface designers (see, for example, Rubinstein and Hersh 1984, Chapter 6). What may stop them from making a foray into the world of dramatic representation is the view that drama is fundamentally imprecise and therefore prone to error (both in terms of interpretation and subsequent action), while people require 100% success in all of their communications with computers. My experience suggests that, in the vast majority of contexts, this simply isn’t true.

The imprecision of dramatic representation is the price people pay—often quite enthusiastically—in order to gain a kind of lifelikeness, including the possibility of surprise and delight. When “imprecision” works, it delivers a degree of success that is, in balance against the effort required to achieve it, an order of magnitude more rewarding than the precision of programming, at least for the non-programmer. When it doesn’t work (as in the case of a parser error), how it is experienced depends heavily upon how the system handles the failure. “I DON’T UNDERSTAND THAT WORD” disrupts flow and frustrates people; an in-context response based on the most probable interpretation imitates a normal conversational failure and opens the way to methods of repair that are quite natural to use (see Brennan 1990b).

Seriousness in human-computer activities is a thresholdy thing. “Serious” and “non-serious” or “playful” activities can occur within the same context and at different stages in the same basic activity. I fool around with the layout of a document, for instance, experimenting with different fonts and paragraph styles, the placement of illustrations, perhaps even the structural divisions of the paper. At the point at which I make a creative decision and wish to implement a certain aspect of the design, I experience a “mode swing” (like a “mood swing,” only different) toward greater “seriousness.” I may then swing back to a “fooling around” mode as I evaluate the effects of a choice on the evolving document.

The advent of the regular-human-friendly search engines made a different sort of mode-shifting possible. Before the World Wide Web, companies like AOL created “walled gardens” with their own content and email communities, safely encapsulated from the horrors of Gopher or File Transfer Protocols (FTPs).¹¹ The first truly robust search engines for regular folk (e.g., Lycos and Altavista) placed the human front and center as the “agent” of the action. Ask Jeeves (1997) attempted to re-characterize a search engine with a computer-based butler on top, but it was soon obvious that Jeeves was simply a cartoon intended to create the feeling of being taken care of. Jeeves’ untimely demise demonstrated that emotional comfort may be achieved in better ways. Reliability and robustness have become criteria for good search engines, although we have made little progress to date in creating search engines that can assess the truth value of their findings. But “search” does allow us to see that the experience of flow is not necessarily disturbed when such shifts occur. Further, the experience of searching and finding has its own dramatic arc.

11. And relatively unable to feed the voracious appetite of consumerism for big data that is gathered and recycled as targeted advertising today.

A dramatic approach need not be fuzzy or imprecise in its ability to produce results. It is potentially capable of supporting both serious and non-serious activities. Its evocative powers and even its ambiguities can be harnessed to enhance rather than to impede people’s serious goals, and to create the possibility of surprise and delight—things that are rarely produced by exhaustive responses to crystal-clear specifications.

For many people whose way of working can be characterized as objective or scientific, the idea of employing an artistic approach is troublesome. It’s hard to say how artists do what they do. The process seems to consist largely of imagination and inspiration, and there seems to be no forthright, dependable methodology. Yet, as we observed in the Foreword, and as we will expand upon in the next chapter, there are ways in which art is “lawful”; that is, there are formal, structural, and causal dimensions that can be identified and used both descriptively and productively. The final goal of this chapter is to justify taking an artistic approach to the problem of designing human-computer activity.

In 1967, Ted Nelson examined the evolution of film in order to understand how the new medium he envisioned—hypertext—should develop. In considering the ways in which the stage had influenced film, he noted that “stage content, when adapted, was appropriate and useful,” while stage techniques (such as the notion of a proscenium and an insistence on continuous action within scenes) were not (Nelson 1967). From the vantage point of today, we can see a migration of both techniques and content from film into the computer medium. If one takes the theatre and the film medium as subsets of a larger category, as representations of action in virtual worlds, then another key similarity between these media and computers are their fundamental elements of form and structure and their purpose.

Both Heckel and Nelson draw our attention to the centrality of “make-believe” in the conception and design of software. An engineer’s view of software design is rooted in logic, realizing an orderly set of functions in an internally elegant program. In Heckel’s view, the better approach is rooted in vision, realizing an environment for action through evocative, consistent illusions. In Nelson’s, it is the creation of “virtualities”—representations for things that may never have existed in the real world before (Nelson 1990). The role of imagination in creating interactive representations is clear and cannot be overrated. In an important sense, a piece of computer software is a collaborative exercise of the imaginations of the creator(s) of a program and the people who use it.

Imagination supports a constellation of distinctively human phenomena that includes both symbolic thinking and representation making. There is a story about a monkey and some bananas that every undergraduate psychology student has heard. A researcher places a monkey in a room with a bunch of bananas hanging from the ceiling and a box on the floor. The monkey tries various ways of getting the bananas—reaching, jumping, and so on—and eventually climbs up onto the box. A person in a similar situation would rehearse most of the possible strategies in her head and actively pursue only those which seemed promising, maybe only the successful one. For the monkey, the focus of attention is the real bananas; for the human, it’s what’s going on inside her head. Imagination is a shortcut through the process of trial and error.

But imagination is good for much more than real-world problem solving. The impulse to create interactive representations, as exemplified by human-computer activities, is only the most recent manifestation of the age-old desire to make what we imagine palpable—our insatiable need to exercise our intellect, judgment, and spirit in contexts, situations, and even personae that are different from those of our everyday lives. When a person considers how to climb a tree, imagination serves as a laboratory for virtual experiments in physics, biomechanics, and physiology. In matters of justice, art, or philosophy, imagination is the laboratory of the spirit.

What we do in our heads can be merely expedient or far-reaching, private or intended for sharing and communication. The novels of Louise Erdrich, for instance, or the plays of Bernard Shaw, create worlds where people address issues and problems, both concrete and abstract, and enact their discoveries, responses, and solutions. These representations are wholly contained in the realm of the imagination, yet they transport us to alternate possible perspectives and may influence us in ways that are more resonant and meaningful than experiences actually lived.

Art is the external representation of things that happen in the head of the artist. Art forms differ in terms of the materials they employ, the way the representations are created, what they purport to represent, and how they are manifested in the world. Different forms have different powers—to engage, to provide pleasure and information, to evoke response. But all have as their end the representation of some internal vista that the artist wishes to create beyond the bounds of his or her own skull, making it available in some form to other people.

What are such representations good for? Aristotle identified catharsis as the end cause of a play. He defined catharsis as the pleasurable release of emotion,¹² specifically those emotions evoked by the action represented in the play. In his view, catharsis occurred during the actual “run-time” of the play, but some contemporary theorists disagree. The early 20th-century German dramatist Bertolt Brecht extended the notion of catharsis beyond the temporal boundary of the performance (Brecht 1964). He posited that catharsis is not complete until the audience members take what they have assimilated from the representation and put it to work in their lives. In Brecht’s hypothesis, the representation lives between imagination and reality, serving as a conductor, amplifier, clarifier, and motivator.

12. That’s not to say that plays must arouse only pleasant emotions; the pleasure of release makes even nasty emotions enjoyable in a theatrical context. Catharsis is discussed more fully in Chapter 4.

It seems to me that computer-based representations work in fundamentally the same way: one participates in a representation that is not the same as real life, but which has real-world effects or consequences. Representation and reality stand in a particular and necessary relation to one another. In much contemporary thinking about interfaces, however, the understanding of that relationship is muddy. On the one hand, we speak of “tools” for “users” to employ in the accomplishment of various tasks with computers. We plumb psychology for information about how people go about using tools and what is the best way to design them. We arrive at notions like “cut” and “paste” and even “write” that seem to suggest that people working with computers are operating in the arena of the concrete. We often fail to see that these are representations of tools and activities and to notice how that makes them different from (and often better than) the real thing.

On the other hand, we employ graphic designers to create icons and windows, pictures of little hands and file folders and lassos and stitched leather covers for “calendars” and “address books,” to stand in for us in the computer’s world. Here the idea of representation is used, but only in a superficial sense (and Sir Ive at Apple detests it). Messy notions like “interface metaphors” are employed to gloss over the differences between representation and reality, attempting to draw little cognitive lines from the things we see on the screen to the “real” activities that psychologists tell us we are performing. Interface metaphors rumble along like Rube Goldberg machines, patched and wired together every time they break, until they are so encrusted with the artifacts of repair that we can no longer interpret them or recognize their referents.

This confusion over the nature of human-computer activity can be alleviated by thinking about it in terms of theatre, where the special relationship between representation and reality is already comfortably established, not only in theoretical terms, but also in the way that people design and experience theatrical works. Both domains employ representations as contexts for thought. Both attempt to amplify and orchestrate experience. Both have the capacity to represent actions and situations that typically do not and cannot exist in the real world, in ways that invite us to extend our minds, feelings, and senses to envelop them.

In the view of semioticist Julian Hilton (1993), theatre is “essentially the art of showing, the art of the index. . . . it involves the synthesis of symbolic and iconic systems (words and moving pictures) in a single indivisible performed event.” Hilton employs the myth of Pygmalion and Galathea (familiar to many as the basis of Bernard Shaw’s play Pygmalion, or the musical version, My Fair Lady) to express the relationship of the theatre to the domain of artificial intelligence. He describes the value of the theatre’s ability to represent things that have no real-world referents in semiotic terms:

Galathea in a literal sense imitates nothing, and as such defines a class of icon (the statue after all is a picture of itself) that can simultaneously be an index. It is this category of non-imitative index which enables the index to liberate its true power, whereby it has all the infinite valency of the symbol while retaining the immediate recognisability of the icon. (Hilton 1993)

Computers are representation machines that can emulate any known medium, as Alan Kay (1984) observed:

The protean nature of the computer is such that it can act like a machine or like a language to be shaped and exploited. It is a medium that can dynamically simulate the details of any other medium, including media that cannot exist physically. It is not a tool, although it can act like many tools. It is the first metamedium, and as such it has degrees of freedom for representation and expression never before encountered and as yet barely investigated.

Thinking about interfaces is thinking too small. Designing human-computer experience isn’t about building a better desktop. It’s about creating imaginary worlds that have a special relationship to reality: worlds in which we can extend, amplify, and enrich our own capacities to think, feel, and act. Hopefully, this chapter has persuaded you that knowledge from the theatrical domain can help us in that task. The next two chapters are designed to give you a deeper understanding of some of the most relevant aspects of dramatic theory and to apply them to interactive forms.