Chapter 22

Emerging Technologies Sound Restoration and Colorization

5.1-6.1 AND HIGHER

What Is 5.1?

Sounds like odds on a horse race—or at least it used to. Now I hear an odds quote and I think they are talking about a new speaker system.

Back in the mid-1990s, a young sound department trainee burst into my room. He was all excited as he announced that the decision had been made to mix the soundtrack we were working on in five point one! Wasn't that great?! I smiled at the news. He spun on his heels and disappeared down the hall to tell the rest of his friends.

My smile vanished as I sat there—stumped at the announcement. What the devil was five-point-one? This was the new breakthrough sound technology that everybody was talking about around town. Gee, I was going to have to learn a whole new way of doing things. I decided that I did not want to disclose my ignorance too blatantly—so I slipped down the hall and knocked on the door of our resident engineer.

“I hate to admit this, but I don't know what five-point-one is. Could you explain it to me?”

His eyebrows shot up. “Oh, you're going to love it! The sound comes out of the left speaker, the center speaker, the right speaker, as well as split surrounds—and we have a subwoofer.”

I waited for the shoe to drop. “Yeah—what else?”

He shrugged. “Isn't that great?”

I studied his face suspiciously. “That's five-point-one?”

“Yeah.”

I turned and paced back to my room. Geez-o-Pete! We've been doing that for the past 30 years! It's called theatrical format! What is so blamed new about this?! Well, of course what was so important about it was that it was an encode that would allow the home consumer to have true dedicated audio signals assigned to the proper speakers in their home entertainment system to replicate the theatrical presentation with a compression that would allow this enormous amount of audio information to take up less space on the DVD disc medium. It would mirror what we have been doing for cinema for the past three decades. Okay—that's a good thing. So let's just remove the idea that it is a new sound system—it is not. It is a new sound delivery format—a better way to bring the big theatrical sound to the consumer.

The first thing we should review is the speaker layout for theatrical presentation.

Center speaker is where all dialog (except for more recent mixing philosophy) emanates from.

Subwoofers (also known as “baby booms”) are low-frequency speakers that sit in between the left-and-center and center-and-right speakers. These speakers reproduce very low-frequency audio signals; some of them are subharmonic and are meant to be felt and not just heard.

Left speaker and right speaker is where the stereophonic presentation of sound effects and music live (with occasional hard left or hard right placement of dialog).

Figure 22.1 The theatrical speaker layout.

Left surround and right surround speakers are where a growing taste in aggressive 360° sound experience is rapidly evolving. Both music and sound effects come from the surround speakers. Since the movie Explorers (1984), the surround speakers have been “dedicated left” and “dedicated right,” allowing wonderful abilities to truly surround the audience and develop an ambience envelope that the audience sits within to enjoy the full soundtrack experience.

Dedicated speaker is placed at the rear of the theater, thanks to Dolby EX. It is meant for the hard assignment of sound effects that will perform behind the audience—for further envelopment—as well as, I am sure, pop-scare effects and dramatic swoop from rear to front aspect audio events.

Everyone knows that the home theater has literally skyrocketed both in technology as well as the ability, in many cases, to present the theatrical experience more vibrantly (in the audio sense) than the hometown cinema. This literal tidal wave of home entertainment theater development has demanded an audio encoding system that could properly place the prescribed audio channels to the correct speakers and do it with as little fuss or muss (as well as technical understanding) on the part of the consumer as possible.

Many home systems use a separate speaker on either side of their big-screen television screens instead of the built-in speakers. The dialog is phantomed from the left and right speakers, making an illusion of a center channel. More and more consumers are now buying a special center channel speaker and placing it either on top or just below the television screen, thus having a dedicated dialog signal assignment. This makes for a much fuller and more natural front dialog track experience than the phantom technique.

For many years home systems have had a surround system—where both the left and right rear speakers were fed by the same surround speaker signal—just like theatrical presentation was prior to the movie Explorers. But more and more consumers are taking advantage of the split-surround option and incorporating that into their home system as well.

A favorite of many consumers, especially those who love to irritate their neighbors, is the subwoofer speaker additive. They come in all prices and sizes, even up to a 3000 watt subwoofer that can just about move walls. This full speaker system, for consumer venues, is what the “one” in 5.1 is all about.

Now, with Dolby EX and DTS-ES, the 5.1 rises one more notch—allowing the consumer to place a dedicated rear surround channel just like they experience in the theater. Technically, this raises the encode designation to 6.1, rather than 5.1, though this term has not officially taken hold as of the writing of this book. The truth of the matter is, variations have provided us with 7.1, and only technical variations will dictate how far this will evolve. Several years ago Tom Holman (THX) was telling me that he was working on 10.1, and that was several years ago. Ten years from now I can just see two consumers standing around the off-the-shelf purchase shelf at their nearest dealer, arguing whether the 20.1 or 22.1 version is more satisfying for total immersion entertainment!

Figure 22.2 Dolby Encode software that the user can matrix encode his or her audio channels into any playback configuration listed in the submenu. The user simply presses the speaker button icon on the left and assigns the proper audio file to it.

Channel Assignments

Channel 1—Center speaker (front)

Channel 2—Left speaker (front)

Channel 3—Right speaker (front)

Channel 4—Left surround speaker(s) (rear left)

Channel 5—Right surround speaker(s) (rear right)

Channel 6—Rear (Dolby EX) speaker (rear center)

Channel .1—Subwoofer (low-frequency effect)

THE QC GAME

One would not know it by looking at the sleek disc products coming out onto the market, but there are huge issues when it comes to quality control.

All DVD authoring facilities have a QC department, of course, but only a small percentage of DVD authoring is done with proper 5.1 playback room environments. Some DVD authoring companies simply have their QC techs sit at a workstation and handle the quality control evaluation by listening to the soundtrack through headsets! As incredibly stupid as this sounds, it is no joke. You are going to have someone listen to and properly evaluate a 5.1 soundtrack through a left-right set of headsets?! I know, I know. . . the market has headsets that advertise they play proper spatial 5.1 environment with their headsets.

You have just spent umpteen million dollars producing this motion picture; you have hired the best sound team you could to record the production sound, the best team you could to handle the post-edit challenges and sound design, and the best rerecording sound team you could to mix it. They worked thousands of man-hours to produce the best sound experience that they could, and now—now, you are going to entrust a quality evaluation to an underpaid get-start trainee using headsets! Go ahead—play Russian roulette if you want to—but not me!

To add insult to injury, there is a general plus or minus four-frame sync acceptability! I'm not kidding. A top Hollywood film laboratory vice-president actually admitted this to me: “Plus or minus four frames is acceptable to us.” Ever wonder why those lips seem a little “rubbery” to you? Now you know. And I am not just talking about low-budget product. One major studio, which subcontracted its DVD authoring to an independent firm, discovered that the DVD had been mastered with the audio channel assignments directed to the wrong speakers. This was a huge-budget picture, by the way. The studio had to buy back all the DVDs and remaster—which cost them several extra million dollars. That can hurt even a mouse!

Here is a little piece of warning for those who thought it was perfectly safe to watch DVDs on your computer. Much to the chagrin of a major (and I mean major) studio executive who watched a test DVD of his $100+ million epic motion picture on his personal computer—he found out that the DVD authoring service had contracted a virus in their system, which was transplanted onto the DVD, and as he watched the movie the virus transplanted itself into his own computer hard drive, eating away and destroying his computer files.

Unfortunately the world of DVD authoring is so misunderstood, like so many technical disciplines that are misunderstood, that there are few in the business who really know their own business. The aforementioned studio put its foot down about the independent contractor that misassigned the audio channels of its major motion picture. The studio execs decided to build their own in-house state-of-the-art operation. Good idea—except for one problem. They did not hire the right person to head it up. Over time the work literally was outsourced to independent contractors again because of mismanagement and technical ineptitude from the inside. A doctor may be a doctor by virtue of the initials M.D., but you do not hire a podiatrist when you need a brain surgeon. Consequently the studio found itself back where it started. Once again, quality and craftsmanship are being destroyed and throat-cutting low-ball bidding and decision makers at the studios who think they know what is going on and how it should be done (and seldom do) make decisions based on who-knows-who relationships and low-bid greed rather than keeping the eye on the art form.

TOTAL IMMERSION OPPORTUNITIES

In addition to the obvious soundtrack mastering of linear entertainment product—such as authoring DVD media of motion pictures, rock videos, and the like—the need for sound event designers and audio authoring is rapidly expanding.

One of the most exciting emerging technologies for the future generation of entertainment venues is the total immersion experience. This is the next quantum step, with photo-realistic visuals fed to you via wraparound goggles, spherical-audio that is event linked to visual action and touch/temperature sensations phantomed throughout your nervous system, driven by complex algorithmic interdependent software that obeys the laws of physics.

Interactive programs have evolved from very crude digital color-block figures to mirroring reality. The technologies that need to drive the serious replication of such high-level audio-visual experiences require faster computers and mammoth amounts of high-speed access for compressed visual and audio files. With every passing week the computers get faster and the drive storage media get bigger, faster, and cheaper.

Several serious developers, thanks to Department of Defense dollars, are emerging with almost unbelievable audio-visual “scapes” that defy the boundaries of tomorrow's entertainment possibilities. The promises of 1980’s Brainstorm and Isaac Asimov's remote viewing from I, Robot, as well as countless science-fiction prognosticators of the past 100 years are finally coming to fruition.

Software programs have pushed the boundaries of realism, such as the seascape software that can replicate ocean waves as real as nature makes them. The operator simply fills in parameter requirements, such as water temperature, wind direction and speed, and longitude and latitude, and the sea will act and react exactly as the laws of physics dictate. Another software can replicate any kind of ship. The operator enters parameters such as historic era, wood, metal, displacement, size, cargo weight, design, and so on. The ship will replicate out and ride the ocean wave program, obeying the laws of physics as its size and displacement require.

New body scanning platforms that can scan the human body down to a hair follicle and enter the images into the computer for action manipulation in real time are bringing forth a whole new reality of opportunities. You can now create whole epic scenes, limited only by the power of your own imagination and your understanding of how to enter parameters into the multitude of programs that will work in concert to create the future vision that you have in mind. Today's war cry is, “If you can think it—you can make it!”

There will be an exploding market for the new “total immersion” sound designer who will be hired, along with his or her team of sound creators, what we may have once called the sound editor. These craftsmen and -women who will specialize on their own assignment audio events that once created and factored into the entire audio construct of the project, will work in concert with each other, all obeying the laws of physics along with the virtual visual replications to create the brave new worlds. After we come home from a long day at the office, we will want to unwind by jumping into various “total immersion” entertainment scenarios.

Various new sensor interactive technologies are currently being developed that will allow you to “feel” the visual realities. You are already able to turn your head and change the aspect of your view, turning to see a big oak tree with its leaves rustling in the warm afternoon breeze. Now with 360-degree spherical-binaural audio, you will be interlinked to visual events that will play for you the realistic sound of deciduous leaves fluttering in the breeze—and through a device such as a wristband, embedded signals in the program will activate the sensation of a warm gust of wind pressing against your chest as the breeze washes across your body. You will feel pressure as well as temperature in addition to seeing photo-realistic visuals and spherical sound.

Practical Application for Training Purposes

The armed forces believed that they could possibly save as many as 50 percent of their wounded if they could just train their combat personnel to not fear the act of being shot or seriously injured in action. They had a scenario developed for total immersion interactive whereby their soldiers train in scenarios where they are ambushed by enemy forces. Realistic spherical-audio sells the distant weapon fire and the incoming flight of deadly bullet rounds. The wristwrap phantom's bullet strikes in the body, the first in the upper shoulder, the second in the left thigh. The sharp zip of other bullets fly past the trainee, striking and ricocheting off various surfaces (exactly linked and replicating the trainee's surroundings) around the soldier.

Now the wounded trainee has experienced as realistically as possible (short of really being shot) what it is like to be hit by incoming fire. Instead of reacting with the emotional side of our psyche “Oh my God, I'm shot—therefore I am going to die,” the trainee is now trained to think, “Damn it! I've been hit! Now I have to problem-solve how to survive!”

Mental attitude makes all the difference in the world. Our self-disciplined intellect “position” will hold back the emotional side and discard the primal fears and spur the will to live.

Entertainment in Total Immersion

In the world of entertainment, one can just imagine the limitless applications to nervous system stimulation as triggered by nervous system events embedded in the total immersion program. It is hardly necessary to explain it.

New audio technologies are just now being perfected that will change everything. What if I could give you a microphone and tell you that you do not have to try to mimic or sound like Clark Gable or Cary Grant, all you have to do is perform the dialog line with that certain lilt and timing of pace and performance? Your spoken words would then be brought into a special audio software and processed through the “Clark Gable” plug-in or the “Cary Grant” plug-in and out will come, voice printable, Clark Gable or Cary Grant. Science fiction? Not at all.

This, of course, will change everything. Up to now, the visual cut-paste technique of integrating deceased celebrities into commercials and motion pictures (e.g., Forrest Gump, beer commercials with John Wayne and Fred Astaire dancing with a vacuum cleaner) have been dependent on original footage. Yes, we have been able to morph the actor's face to be absolutely accurate for some time—that is no trick at all. It is the human voice that has eluded us, and the use of voice mimics to impersonate the deceased actor has always been just detectable enough to make it useless for serious celebrity replication. With the advent of this new audio technology however, the shutters will be thrown back on all of the projects that have wanted to enjoy bringing an actor back to life to perform in a project and give a performance that he or she had never given before. We are literally on the eve of a new evolution of entertainment opportunities utilizing these new technological breakthroughs.

These new opportunities (as well as the responsibilities) are staggering. By combining the technologies, one could create a total immersion entertainment venue whereby you could come home from a stressful day and decide that you want to put your total immersion gear on and activate a Renaissance Faire program that would take you into an evening-long totally realistic experience of being at a fifteenth-century festival on a nice spring afternoon with, of course, a spherical audio experience.

It only stands to reason that the more aggressive opportunists will develop programs where you could come home and put your total immersion gear on and make passionate love to Marilyn Monroe or Tom Cruise (depending on your sex and/or preference) or whoever else is literally on the menu.

Needless to say, there will be obvious exploitation used with this new technology as well as amazing mind-blowing opportunities to do things we could not otherwise do, such as a total immersion space walk, or participate alongside Mel Gibson in the Battle of Sterling, or scuba dive amid the coral reefs. Can you imagine what it would be like to turn an otherwise boring afternoon (say that you are snowed in at home or maybe you are just grounded by your parents) to jump into an interactive educational scenario like sitting around in a toga in the Roman Senate or participating in the lively debates in the House of Lords during Cromwell's time? What a concept!

All of this is going to need a new generation of sound craftsmen, developing and pushing the boundaries of how to design, develop, and bring to the consumer the most amazing audio event experiences you can possibly imagine.

AUDIO RESTORATION

To the inexperienced, “audio restoration” probably sounds like some salvation service, with medieval monks hunkered up around their individual workstations working on would-be lost sound files. I doubt that they are monks, but this wonderful and amazing aspect of our industry is woefully misunderstood by far too many who may very well find that they need this kind of work desperately.

I must harp on the same soapbox that I have climbed upon so many times throughout this book: just because some enthusiastic would-be sound doctors go out and buy off-the-shelf software, such as no-noise, broadband, declick, or other algorithmic spectrographic resynthesizer software, does not make them audio restoration craftsmen. Only experience and veteran trained know-how can properly empower the use of such software programs. There is nothing worse than a bull swinging a sledgehammer in a china closet in place of a Dutch master painter using a sable brush!

There are a few very fine audio restoration services. One that I have personally used on several occasions to salvage nearly 85 percent of the production track of the Trimark production Frank & Jesse as well as clean out dominant fluorescent ballast hums (because the producer wanted to save money using a warehouse instead of a real sound stage) from over 50 percent of the film Dr. Jekyll and Ms. Hyde, is John Polito of Audio Mechanics (Burbank, California), one of the world's leading services in a multitude of audio restoration services.

Figure 22.3 Audio Mechanics 5.1 mastering studio.

Audio Mechanics features two mastering rooms designed for 5.1 from the ground up, a live room and vocal booth for recording and ADR, and a stereo mastering control room. A well-thought-out design with complete audio/video/data integration among all rooms and facility-wide Equi-Tech balanced power to eliminate grounding problems. The centralized equipment room allows rooms to share resources efficiently and quickly recover from equipment failures.

This state-of-the-art 5.1 mastering room features instant room tuning presets to adapt to the project. A flat setting for music mastering; a Dolby-certified film setting for print mastering; a screen compensation EQ setting for the center channel when the Stuart screen is lowered for picture projection; and a delay compensation for the digital video projector.

Up to 50 different room tunings can be stored to simulate various listening environments. The speaker system features an ITU placement of 5 matched dual 15” Pelonis Series mains, based on a Tannoy 15” dual concentric phase linear driver integrated with a super fast low-frequency driver and a super tweeter that extends the frequency response up to 80 kHz. JBL dual 18” subwoofer delivers a thundering point-one punch. The room is not only gorgeous, but the sound is amazing—the imaging is perfect. Like all great craftsmanship, it is all about the details.

Figure 22.4 shows the rear wall of the room featuring a rear center channel soffit, Pelonis Edge technology bass trapping with a first ever acoustic RPG pegboard utilizing a mathematically randomized hole pattern to permit near-perfect bass absorption along with an evenly distributed diffusion above 500 Hz.

Figure 22.4 Reverse view of Audio Mechanics’ 5.1 mastering studio.

Figure 22.5 Audio Mechanics logo.

The tools required for sound restoration and noise removal are extremely sophisticated and infinitely complex. Without the proper experience and talent, results can vary from mediocre to just plain awful. Audio Mechanics has more combined experience and talent with these tools than anyone in the world, and their track record proves it. Producers, directors, mixing engineers, and dialog editors come to Audio Mechanics because they know they will get the best results attainable.

Sound Preservation and Restoration

Audio Mechanics’ John Polito has been at the forefront of digital sound restoration since its inception. His facility has completed projects for many of the leading studios, including Paramount Studios, Sony Pictures Entertainment, 20th Century Fox, Walt Disney Studios, Warner Brothers Studios, as well as UCLA's Film & Television Archive, and the Academy Film Archive. He shares his knowledge on this important topic below:

Methodology, tools, techniques, research, and evaluation are the most important steps in preservation. The three Rs of restoration are research, research, research! Any information you can gather about the original sound production—the genealogy, quality, and condition of existing elements—will enable you to pick the best possible elements for the restoration. The goal is to find the best sounding, most original element. Oftentimes the best element will be one that is not readily available. The engineer should evaluate the elements’ physical condition and audio quality, and document the findings. Select the best element or elements for transfer.

Transfer (the second most important step in preservation): Always use the best playback electronics and digital converters available. Prepare the element by carefully inspecting and, if necessary, cleaning and repairing. Mechanical modification of playback equipment can be used to accommodate shrinkage, warping, vinegar syndrome, and so on (reduced sprockets, roller heads, etc.).

Listening environment (always important!): The quality of listening environment is as important as the talent of the engineer. Care must be taken to make the listening environment appropriate for the type of work being performed. Use high-quality speakers and amplifiers appropriate for the type of sound work being performed. Make sure that the acoustics of the room are optimized. Just as important as the room itself, the engineer must know how the room translates to the environment the sound will be heard in.

Remastering (clean up, conform to picture, re-mix): 99 percent of remastering is now performed with digital tools. Forty-eight kHz/24-bit is the current standard for digital audio restoration, but the industry is moving toward 96 kHz/24-bit. Digital tools are available to reduce hum, buzz, frame noise, perforation noise, crackle, pops, bad edits, dropouts, hiss, camera noise, some types of distortion, and various types of broadband noise including hiss, traffic noise, ambient room noise, motor noise. High-quality digital filters and equalizers can help balance the overall sonic spectrum. Digital audio tools are only as good as the engineer who is using them. The experience and aesthetics of the engineer directly relate to the quality of the end result. Knowledge of the history of the original elements guides the engineer's decision process. For example, the restoration process would be quite different for a mono soundtrack with Academy pre-emphasis than it would be with a stereo LCRS format. Knowing whether a mono show was originally a variable density track versus a variable area track will also affect key decisions early on in the process. For picture-related projects, ensure that the restored audio runs in sync with picture. It may seem obvious, but make sure you are using a frame accurate picture reference when checking sync. Oftentimes there are different versions of the picture available. The intended listening environment will determine the final audio levels.

Theatrical environments have a worldwide listening standard of 0 VU = 85 dB. This means that the sound level of a film should sound about the same in any movie theater. The music industry has not adopted a listening standard, so the level depends on the genre and delivery media. Up until recently, digital audio masters were delivered on digital audiotape: DAT, DA-88, DA-98HR (24-bit), and DigiBeta were used for film; 1630, DAT, and Exabyte were used for the music industry. The problem with these formats is that they had to be transferred in real time and the shelf life of the tapes was anywhere from 1 to 10 years. High error rates were sometimes experienced within a single year. The industry has been moving toward optical disc and file transfer over the Internet for final digital delivery, with a preference for audio data files that can be copied and verified faster than real time.

The music industry uses Red Book audio CD-R with an optional PMCD format (Pre-Master CD), and recently DDP files (Disc Description Protocol) that can be transferred and verified over the Internet, or via data CD-ROM or DVD-ROM. The film and video industries have been converging on Broadcast Wave Files audio files (BWF) on DVD-R, removable SCSI drives, FireWire drives, or transferred directly over the Internet via FTP. Higher capacity optical disc storage has recently become available with HD-DVD and Blu-ray (a.k.a. ProData for data storage).

Archiving (re-mastered and original elements): Save your original elements! Technology is continually evolving. Elements that seem hopelessly lost today may be able to be resuscitated 5 or 10 years from now. Save both the original elements and the raw digital transfer of the elements. The film industry routinely transfers the final restored audio to several formats. Analog archival formats include 35 mm variable area optical soundtracks and 35 mm Dolby SR magnetic tracks on polyester stock. Digital formats include BWF files on archival grade DVD-R, and digital Linear Tape-Open (LTO). DVD-ROM is a good choice because a high quality archive-grade DVD-R has a long expected shelf life of 50+ years. They are, however, very sensitive to mishandling and scratch easily, which can cause a loss of data. LTO tape is popular because it is an open standard with backwards compatibility written into its specifications. It is widely supported and not owned by any single manufacturer, so it is less likely to become obsolete quickly in the ever-changing digital frontier.

Even though LTO is digital tape with a fairly poor shelf life, solutions exist to allow easy migration to newer and higher capacity tapes. Software verifies the data during migration. The music industry vaults its masters in whatever format is delivered. Since the manufactured optical discs are digital clones of the original master, they can rely on a built-in automatic redundant worldwide archive! Live data storage servers are another option for digital archiving masters. RAID storage (Redundant Array of Independent Disks) is perhaps the most proactive way to digitally archive audio. RAID is a self-healing format. Since the RAID disks are always online, a computer can continually verify the data. If a drive in the array fails, a new unformatted drive can be put in its place and the RAID will automatically fill the drive with the data required to make the data redundantly complete. RAID storage is much more expensive than DVD-R and LTO. A wise digital archivist would adopt two or more digital archiving formats and create multiple copies of each using media from multiple manufacturers. Geographically separating multiple copies should also be adopted as a precaution in the unlikely event of a disaster.

For a further demonstration of the audio work that John Polito has shared with us in this chapter, go to the interactive DVD provided with this book. Choose button 10 “Restoration & Colorization.” A sub-menu will appear. Choose the “Audio Mechanics” demo button. You can then hear comparisons and examples for yourself.

Restoration and Colorization

Colorization in a book on sound? Well, yes, it might seem a bit strange at first. But this entire chapter is about the ever-evolving world of emerging technologies and the future that we, as both craftsmen and -women as well as producers and product users, are swiftly hurtling toward like a mammoth black hole in the galaxy of the entertainment arts.

Inexorabilis—the disciplines and artistic concepts come colliding together like two separate galaxies that criss-cross each other's determined path. Though you have read my touting that sound is more important than the visual image in a motion picture, I am the first to speak an opinion with regard to the colorization of film. I was a staunch resistor until in late 1989 I saw one film, just one film, that changed my thinking. John Wayne's Sands of Iwo Jima was amazingly better than any of the childish smudge coloring that I had seen up until that time.

Figure 22.6 Legend logo.

Then, out of the blue, I happened to have a fateful conversation with an old colleague that I had worked with a number of years earlier at another cutting-edge firm, no longer in existence. I asked Susan Olney where she was working. She told me she was vice president of operations at an amazing cutting-edge technology company based in San Diego, California.

In a few short conversations I found myself talking to Barry Sandrew, Ph.D., the president and COO of Legend Films. Our conversation was almost like two lost brothers finding each other after years of separation. I told him that the only colorization effort I ever saw that gave me pause was Sands of Iwo Jima. He chuckled as he responded, “That was my work.”

Well, of course it was. It had to be. As the conversation continued, it became extremely obvious to me that the evolution of our art forms is, in fact, inexorably drawing all of us into a singularity of purpose and dedication: art form and the highest standards of craftsmanship to create artificial reality.

Barry sent me several demos, some of which I am pleased to share with you on the companion website. Watch the Legend Film demo, and I am sure that, like me, you will be amazed by what you see. The interview of Ray Harryhausen is especially compelling in dispelling the standard artistic argument that we always hear against colorization—and he should know, he was there and he worked with these directors and artists. I cannot think of a professional and respected voice that can give a reason to rethink and appreciate this amazing process. So, I asked Barry if he would write a piece about his high resolution restoration and colorization processes. You can review the video demo which includes Ray Harryhausen sharing his views and commitment to colorization and the numerous comparisons that have been talked about by Barry Sandrew on this book's accompanying DVD. Choose button number 10 “Restoration and Colorization” and then the “Legend Films Video Demo.” There is also a gallery of comparison stills.

COLORIZATION: A BIASED PERSPECTIVE FROM THE INSIDE
by Barry B. Sandrew, Ph.D., president and COO of
Legend Films

The technology of colorization has changed considerably since it first arrived on the scene in August 1985 when Markle and Hunt at Colorization, Inc. released the 1937 production of Topper, the first colorized feature film. They used a system in which analog chroma signals were digitally drawn over and mixed with analog black-and-white video images. To put it more simply, colors were superimposed on an existing monochrome standard video resolution image much like placing various colored filters over the image.

Figure22.7 Barry B. Sandrew, Ph.D., president and COO of Legend Films.

Today Legend Films, the leader in all-digital colorization, is working with black-and-white and color feature film at the highest film resolutions and applying long-form and short-form nondegrading restoration and a spectrum of color special effects with a brilliance and quality never before attainable. These technological advances are largely the result of my work in digital imaging and specifically digital colorization over the past 20 years.

However, the overall workflow and processing protocol for restoration and colorization evolved over several years beginning in 2001 by a very talented team of colleagues at Legend Films who built a machine that colorizes feature films, commercials, music videos, and new feature film special effects in the most efficient and cost-effective manner in the industry. Historical accounts of colorization from its genesis in the late ’80s and early ’90s is replete with a handful of highly vocal, high-profile Hollywood celebrities who adopted a vehemently anticolorization stance. Some became so obsessed with the cause that it appeared their life's mission to rid the world of this new technology that was, to them, so evil and so vile that its riddance would somehow deliver them salvation in the afterlife.

Others much less vocal understood that the purpose of the U.S. copyright law was to eventually make vintage films freely available to the general public after the copyright holder held the title as a protected monopoly for a fixed period of time. These laws are the same as patent laws, which eventually make inventions, drugs, and other innovations free to the public for competitive exploitation and derivation. There was a significant majority who saw colorization as a legitimate and creative derivative work and who recognized that colorization was actually bringing new life to many vintage black-and-white classics that were until then collecting dust on studio shelves or viewable on television only in the early-morning hours. I believe the most balanced and accurate account of the technological evolution of colorization, as well as the political and economic perspectives underlying the colorization controversy, can be found in the IEEE Spectrum article “The Germans Wore Gray, You Wore Blue” (G.R. Edgerton, 2000).¹

For me, one of many significant revelations in the IEEE Spectrum article is that the colorization controversy might have been largely avoided had Colorization, Inc. continued the joint venture relationship they had already developed with Frank Capra to colorize three of his classics, including It's a Wonderful Life. When Markle discovered It's a Wonderful Life was in public domain he dismissed Capra's involvement in the colorization of the Christmas classic and took on the project himself, seriously alienating Capra.

Edgerton writes,

Capra did indeed sign a contract with Colorization, Inc. in 1984, which became public knowledge only after he began waging his highly visible campaign against the colorizing of It's a Wonderful Life in May 1985, when he felt obliged to maintain that the pact was “invalid because it was not countersigned by his son, Frank Capra, Jr., president of the family's film production company” (Lindsey, R., “Frank Capra's Films Lead Fresh Lives,” The New York Times, May 19, 1985; C1, C23).

Capra went to the press in the first place because Markle and Holmes discovered that the 1946 copyright to the film had lapsed into the public domain in 1974, and they responded by returning Capra's initial investment, eliminating his financial participation, and refusing outright to allow the director to exercise artistic control over the color conversion of his films.

As a consequence, Colorization, Inc. alienated a well-known and respected Hollywood insider whose public relations value to the company far outstripped any short-term financial gain with It's a Wonderful Life, especially in light of the subsequent controversy and Frank Capra's prominent role in getting it all started. Capra, for his part, followed a strategy of criticizing color conversion mostly on aesthetic grounds, while also stating that he “wanted to avoid litigation. . . at [his] age” and “instead [would] mobiliz[e] Hollywood[’s] labor unions to oppose” colorized movies “without performers and technicians being paid for them again” (Lindsey, R. “Frank Capra's Films Lead Fresh Lives,” The New York Times, May 19, 1985; C1, C23.).

Edgerton succinctly sums up the controversy:

All told, the bottom-line motivation behind the colorization controversy was chiefly a struggle for profits by virtually all parties involved, although the issue of “creative rights” was certainly a major concern for a few select DGA spokespersons such as Jimmy Stewart, John Huston, and Woody Allen, among others.

The rest of Edgerton's article is historically spot-on accurate. However, in his conclusion Edgerton writes,

In retrospect, the controversy peaked with the colorization of Casablanca and then gradually faded away. Ted Turner recognized as early as 1989 that “the colorization of movies. . . [was] pretty much a dead issue” (Dawson, G., “Ted Turner: Let Others Tinker with the Message, He Transforms the Medium Itself,” American Film, Jan./Feb. 1989, 52, pp. 36–39). Television audiences quickly lost interest in colorized product as the novelty wore off completely by the early 1990s.

While certainly a reasonable conclusion for Edgerton to make in 2000 when he wrote the IEEE Spectrum article, I'm confident it will be judged premature given significant changes in the attitude of the entertainment industry including the emerging spectrum of digital distribution channels available for entertainment of all types and the advent of Legend Film's technological advances in both restoration and colorization.

From Brains to Hollywood

I first learned that people were creating colorized versions of black-and-white classic feature films in 1986 when I was a neuroscientist at Massachusetts General Hospital engaged in basic brain research and medical imaging. My interest in colorization at the time was purely academic in nature. Indeed, a process that could create colorized black-and-white feature films where color never existed intrigued me. From the lay literature I understood the early colorization effort was accomplished using an analog video process. Consequently I did not delve much further into the early process because at the time I was not experienced in analog video but firmly entrenched in high-resolution digital imaging. When I saw the results of the analog process I was less than impressed with the quality, but certainly impressed with the technological and innovative effort. I soon conjured up an all-digital method of creating a superior all-digital method for colorizing feature films, but it was all simply an intellectual exercise with no commercial outlet. After all, my selected career track was neuroscience.

Digging further into the business rationale for colorization, it was clear why the technology had become attractive even if the aesthetic value of the process was questionable. While the black-and-white version of a vintage film remains in the public domain, the colorized version becomes a creative derivative work that (back in 1986) was eligible for a new 75-year copyright.

Coincidently, in the fall of 1986, George Jensen, an entrepreneur from Pennsylvania with the entertainment company American Film Technologies, approached me to develop an all-digital process for colorizing vintage black-and-white films that he hoped would become a major improvement over the current analog process. American Film Technologies had financed several failed development trials in an effort to create a digital colorization process that might capture the market. When approached by George to provide an alternative solution, I suggested a novel direction to the problem that was significantly different from any of the failed attempts.

Given my background in digital imaging and my credibility in academia, American Film Technologies decided to offer me a chance to develop their system. I considered the opportunity sufficiently low risk to leave academia for a period of time and take on an R & D project with company stock, considerably higher salary than one can expect at a teaching hospital, and all the potential rewards of a successful entertainment start-up company.

Part of my decision was based on the fact that George Jensen had assembled an impressive group of Hollywood talent such as Emmy-nominated Peter Engle (executive producer, Saved by the Bell) and two-time Oscar winner Al Kasha (Best Song honoring “The Morning After” from The Poseidon Adventure and “We May Never Love Like This Again” from The Towering Inferno), as well as Bernie Weitzman, former VP/business manager for Desilu Studios and VP/operations for Universal Studios. This brought me additional confidence that the company was heading toward the mainstream entertainment industry. I saw George Jensen as a visionary in that he realized one could colorize a huge library of proven classic black-and-white films in the public domain and, in the process, become a Hollywood mogul based solely on technology.

The problem that stood in his way was the development of advanced digital imaging technology and the acquisition of original 35 mm film elements that would be acceptable in overall image quality to capture digitally for colorization and subsequently for VHS release. His search for a superior, all-digital process ended soon after we met. My lack of experience in analog video was a huge advantage because I believe I would have been biased toward developing a more advanced analog solution and ultimately the system would have been less successful. Instead I started from scratch and created a digital colorization process that was technologically much different to existing systems but markedly superior in final product.

I believe my strength was the fact that I approached the problem from a totally different and perhaps naïve perspective. I was not a video engineer and had very little knowledge of the field. Consequently there were some in the entertainment industry who told me my concept wouldn't work and advised me to go back to studying brains.

The Early Digital Process

The basis of American Film Technologies’ digital colorization was a designer-created color transform function or look-up table that was applied to underlying luminance or gray-scale values of user-defined masks. Each mask identified objects and regions within each frame. In other words, instead of simply superimposing color over the black-and-white image like the old analog processes, the color was actually embedded in the image, replacing the various levels of black-and-white with color. This permitted a great deal of control over color within each frame and automated the change of color from one frame to the next, based on changes in luminance. Once a designer created a reference frame for a cut it was the colorist's job to make certain the masks were adjusted accurately from frame to frame. The colorists had no color decisions to make because the designer-created look-up tables intimately linked the color to the underlying gray scale. All the colorist was required to do was manually modify the masks to fit the underlying gray scale as it changed, due to movement every 24th of a second from frame to frame. However, because each mask in each frame had to be viewed by the colorist and possibly manually modified, the number of masks allowable within each frame had to be limited.

Of course from a production cost standpoint the more masks in a frame, the more the colorists had to address, the longer each frame took to be manually modified and consequently the greater the cost to colorize. Consequently, using this method, the number of mask regions per frame in the American Film Technologies’ system was ultimately limited to 16. Because the process was highly labor-intensive the user interface was designed to be very simple and very fast so the process could be outsourced to a cheaper labor market.

Within 5 months I had a working prototype and a demo sufficient to premier at a Hollywood press conference. The late Sid Luft, Judy Garland's ex-husband and manager, became a great personal friend and major supporter of colorization. He provided American Film Technologies with some of our early black-and-white material from Judy Garland concerts and television shows, which we incorporated into a demo. Most of the material originated on relatively poor quality 2” videotape, but the digital color was such an advancement over the analog process that it appeared stunning to us. Apparently it also appeared stunning to the press at a conference held across the street from Universal Studios in April of 1987. When Frank Sinatra appeared with baby blue eyes you could hear the amazed reaction of those assembled.

The majority of articles that resulted from that press conference were favorable and indicated that colorization had finally come of age. In reality we had a long way to go. The real proof of concept didn't happen until November 1987 when we successfully produced the first all-digital colorized film, Bells of St. Mary's for Republic Pictures. Much of the technology was refined and developed during that period of production and during the production of several feature films thereafter.

American Film Technologies took over the colorization industry within one year following the release of Bells of St. Mary's though Jensen's vision of creating a library of colorized films never really materialized. We learned that the best 35 mm prints at the time were held by the studios or by a very close-knit subculture of film collectors, so we became largely dependent on the major studios as work-for-hire clients. Those clients included Republic Pictures, Gaumont, Universal, Disney, Warner Bros., CBS, ABC, MGM, Turner Broadcasting, and many others.

In the early ’90s American Film Technologies and its competitors eventually faded away because each company overbuilt for what was essentially a limited market. Once Turner completed colorizing his A- and B- movie hitlist the colorization well essentially dried up. I believe that both the technology and the industry premiered and peaked 20 years too early. Today there is affordable computer horsepower to automate much of the process at the highest resolutions and in high-definition as well as film color space. In addition, using reliable outsourcing resources, particularly in India, virtual private networks can make it seem as though a colorization studio in India is next door to our design studio in San Diego. The affordability and, most important, quality of colorization product have advanced to a technological plateau.

The Rationale for Colorization Today

The early days of colorization were a wake-up call to studio asset management directors and archivists, many of whom saw their studio libraries neglected for over half a century. However, the very early studio asset managers and archivists (if they existed) were not at fault. Indeed it was a matter of budget, studio culture, and lack of foresight. As a result, over half of the films that were ever produced in the United States are gone today. Not because they have deteriorated over time, though this was also a contributing factor, but because the studios threw many of their film negatives in the trash!

There were actually rational excuses for this mass destruction of our film heritage. No one in the ’20s through the late ’40s saw the enormous impact of television on our lives and culture. Certainly, the idea of tape recorders in every home and later DVD players and soon high-definition DVD players was as distant as a man landing on the moon. . . probably more so because digital video entertainment in the home could hardly be envisioned. No one anticipated Blockbuster, Hollywood Video, or Netflix, where films of every genre and era can be rented. There was no sign that the cable industry and the telcos would be laying the groundwork for downloading high-definition films to home digital disc recorders and viewable on the average consumer's 50” high-definition monitor with 5.1 surround sound, as they are today.

In the early years, studio feature films had a theatrical run and those films that had a significant return on the studio investment were likely saved. Those that did not were eventually thrown out to make shelf space for newer films. The idea of films having an afterlife was foreign to only the most Nostradamus-like among studio executives. It's ironic that it took colorization, with all its controversy, to focus mass attention on the need to restore and preserve our film heritage. With so many diverse digital distribution outlets for entertainment product today and a renewed desire for proven vintage entertainment, the studios now have a great incentive to restore their classic black-and-white film libraries. Several have called on Legend Films to colorize selected classic black-and-white titles in an effort to make them more attractive to new generations of audiences.

Indeed, by now much of the furor over colorization has subsided and special effects specialists, archivists, creative producers, and iconic Hollywood talent are embracing the process. Digital distribution outlets cry for content that is economical yet proven profitable. What better time for colorization to reemerge—and now in high definition. The studios realize that new film production is typically high cost, high risk, and low margin. Colorization of proven evergreen (having timeless value) classic feature films is low cost, low risk, and high margin over time.

Legend Films, Inc.: Building a System for the Twenty-First Century

In 2000, I met Jeff Yapp, former president of international home video at 20th Century Fox Home Entertainment and later president of Hollywood Entertainment (Hollywood Video Stores). He was a major proponent of colorization for at least a decade before I met him and he always pointed to the old but valuable 20th Century Fox Home Entertainment collection of colorized Shirley Temple films that were rereleased in the late ’90s as fully remastered for broadcast and VHS sales. Those colorized classics turned into one of 20th Century Fox Home Entertainment's most successful home-video campaigns. Jeff understood the value-add of colorization if it could be done correctly, at high definition and with a color vibrancy and quality never before seen. He asked whether I could create a more automated method of colorization that would be “light years ahead of the look of the old colorization processes.” Fully equipped with the latest and most economical computer horsepower, major advances in software development, and broadband connectivity I was 100 percent certain it could be done.

Restoration of the black-and-white digitized film is the first step in the colorization process. In order to create the best-looking colorized product, it is essential to start with the best black-and-white elements with the best dynamic range. We search for the best quality 35 mm elements and then digitally capture every frame in high-definition or various film formats. While the colorization is ongoing we put each frame through the Legend Films proprietary restoration process. The software is semiautomated using sophisticated pattern recognition algorithms and the user interface is designed to be easily adaptable to unskilled outsourcing labor markets.

The colorization system had to be sophisticated enough to do the heavy lifting of correct mask placement from frame to frame with little decision making or direction by the actual colorist. Speed, efficiency, and, most of all, quality were the essential ingredients. With the cost of high-powered technology at an all-time low, and with reliable PC networking, I went to work. The result was a semiautomated colorization system, in which there were no color restrictions, and the number of masks per frame could be essentially unlimited. Legend Films currently has 10 international patents that are either published or pending along with a full-time research and development team that is constantly evolving the system to meet and overcome new production challenges.

Figure 22.8 Standing from left to right: Barry Sandrew, president, David G. Martin, CEO, and, seated, Sean Crampton, QA manager, Legend Films. Looking at QA on 20th Century Fox Home Entertainment's Heidi.

One objective was to create an intuitive user interface that would make much of the sophisticated technology underlying the color design, colorization, and restoration software transparent to the user and consequently built for unfettered creative design as well as speed and efficiency.

In the case of a feature film, the entire movie is captured digitally in high-definition or film resolution. The resulting digital frames are then dissected into cuts, scenes, characters, locations, and so on. Once the entire film is broken down into its smallest components, a series of design frames is selected to serve as a color storyboard for the film. The design process involves applying many different colors for various levels of shadows, midtones, and highlights. With 16 bits of black-and-white or 64,000 levels of gray scale available for each frame, the amount of color detail that can be applied is enormous. At American Film Technologies only 256 gray scale values were available for each frame. In the case of color film, Legend Films can apply color effects at a resolution of 48 bits of color or 16-log bits per red, green, and blue channel. Pure hues are not common in nature but in earlier digital colorization processes, single hue values were the only way to apply color to various luminance levels. In the Legend Films process, color is dithered in a manner that mixes hues and saturations semirandomly in a manner that makes them appear natural, particularly for flesh color and different lighting situations. While this sounds complicated the design process is actually unencumbered by the underlying technology allowing the designer to focus on the creative aspects of the colorization process rather than keeping track of masks or how the computer interprets the data. Once the designer has established the color design it becomes locked into a cut and scene so that no additional color decisions need to be made during the actual colorization process. The color automatically adjusts according to luminance levels.

A second key ingredient of the Legend Films color effects process is a method of applying color effects from frame to frame in a manner that requires little skill other than an eye for detail. Once the color design is established for the entire film, at least one frame per cut in the film is masked using the look-up tables appropriate for that cut. This key frame is used as a reference to move the color masks appropriately from one frame to the next. This process of moving masks from one subsequent frame to the next has been largely automated in the Legend system using many sophisticated pattern recognition algorithms that are activated with the press of a space bar. Making the process even easier, only the foreground objects (i.e., objects that move independent of the background) are addressed at this stage. The backgrounds are colorized and applied automatically in a separate step. The only labor-intensive aspect of this process is the high level of quality assurance we demand for each frame. This requires that a colorist monitor the computer automation and make slight adjustments when necessary. Any adjustments that the colorist makes updates the computer to recognize those errors in future frames so in effect it learns from prior mistakes. As a consequence of this proprietary computer intelligence and automation, the colorization software is readily adaptable to any economical labor market regardless of the skill level. In fact, a person can be trained to become a colorist in little more than a day, although it takes a few weeks to become proficient.

The third key ingredient of the Legend colorization process is the automated nature of background color effects. Each cut in the film is processed using Legend's proprietary utility software to remove the foreground objects. The background frames for the entire cut are then reduced to a single visual database that includes all information pertaining to the background, including offsets, movement, and parallax distortion. The designer uses this visual database to design and subsequently applies highly detailed background color automatically to each frame of the cut where the foreground objects have already been colorized. This sophisticated utility enhancement dramatically cuts down the time required for colorizing, essentially removing the task of colorizing the background from frame to frame. Since backgrounds are handled in design, the colorist need only focus on foreground objects and the designer can place as much detail as they wish within the background element. Complex pans and dollies are handled in the same manner.

Once the color of the foreground and background elements are combined, the entire movie is rendered into 48-bit color images and reconstructed using SMPTE timecode for eventual layoff to high-definition or 35 mm film.

Colorization—A Legitimate Art Form

In 2006, colorization matured into a legitimate art form. It provides a new way of looking at vintage films and is a powerful tool in the hands of talented color designers for realizing what many creative contributors to our film heritage intended in their work.

A color designer typically looks at an entire film and creates a color storyboard that influences the way the film will ultimately look like in color. Our designers are experienced in many aspects of production including color theory, set design, matt design, costumes, and cinematographic direction. Their study and background research of each film is exhaustive prior to the beginning of the design process. Once into the film, the designer uses visual cues in an attempt to interpret what the original director had intended in each shot.

If the shot focuses on a particular individual, object, or region, the focal length of that shot within the context of the dialog and storyline will often provide the reason it was framed in a particular way. Using color, the designer can push back all or parts of the background or bring forward all or parts of the object, person, or region of interest that is necessary to achieve the desired effect.

The designer can also enhance the mood by the use of warm colors and/or cold colors as well as enhance the intended character of particular actors with flesh tones and highlights. Lighting can be used in a spectrum of ways to increase tension or create a more passive or romantic mood in a shot. Certainly the possibilities for influencing the audience's reaction, mood, and perception of scenes are enormous when using color appropriately.

Indeed, Color has proven to be an enormously powerful tool in the hands of ad agencies in developing strong brand identification in commercials and in music video in developing a unique creative look. In working with Nirvana on their final music video back at American Film Technologies, “Heart-Shaped Box” was filmed in black-and-white. That black-and-white film served as a canvas for our designers to create what turned out to be a final and signature music video for the group.

Figure 22.9 Reefer restored (showing the black-and-white presentation only).

India Is the Future

To paraphrase Thomas L. Friedman (The World Is Flat: A Brief History of the Twenty-first Century, 2006), the world of entertainment production is getting flatter by the day. Outsourcing to Canada was yesterday; today it's China and India. I've become particularly enamored with India because we've maintained a highly efficient and professional production facility there for the past three years. I never would have considered maintaining a satellite production studio halfway around the world in Patna, India, but with broadband connectivity and a virtual private network, Legend Films is functioning 24 hours a day.

India is 12 1/2 hours ahead of us. We receive color data from India every morning that are applied to our high-definition frames here in San Diego. We review that work during the day and send QC notes for fixes, which follow the next day. The work ethic and work quality of this largely English-speaking part of the world has been a major asset to Legend Films.

Our primary studio in India is partitioned into 16 smaller studios with 10 to 15 colorist workstations in each. High-quality infrared cameras are mounted in each studio so that we can see the work that is being done and communicate with the colorists if there are any production issues.

The result is that Uno/Legend Studio in Patna and Legend Films in San Diego are one virtual studio that works in sync around the clock and around the world. We are now adding an additional studio in India at Suresh Productions in Hyderabad to increase our overall output and offer a more complete service package globally to filmmakers. The new facility acts as a third element of the Legend virtual studio, working in tandem with our Patna studio. It concentrates on film restoration but also has digital intermediates and color-grading capabilities as well as the most advanced high-definition telecine, editing, and laser film scanning equipment. The facility also includes a complete sound-stage resource as well as audio suites for voice looping and sound effect sweetening.

Hyderabad is one of the major centers of the film industry in India and as such has a large middle-class and rather affluent and highly educated population. The world is changing and professionals in all aspects of the entertainment industry, particularly in production, must be aware of this change. Don't be in denial.

The Second Coming of Colorization

In the past three years we have restored and colorized over 50 films in high-definition or film resolution and we currently have another 50 in the pipeline. Twentieth Century Fox Home Entertainment has elected to use the Legend Films process to revitalize the entire Shirley Temple catalog of films that were originally colorized in the ’80s and early ’90s.

We've also colorized The Mask of Zorro and Miracle on 34th Street for 2005 and 2006 releases, respectively, for 20th Century Fox Home Entertainment.

Many in the creative community of vintage Hollywood are looking to correct the gaps in available technology when their films were first produced. Other contemporary producers, directors, and special effects supervisors are using colorization to bring cost savings to what are already skyrocketing production costs.

Figure 22.10 Ray Harryhausen directs the color design for Merian C. Cooper's She with creative director Rosemary Horvath.

Ray Harryhausen

Even when color film was first available, the cost often outweighed the value since, unlike today, the theatre going audience back then was quite accustomed to black-and-white films. A case in point is Marian C. Cooper's sci-fi classic, She. Cooper intended that the film was to be originally shot in color, and both the sets and costumes were obviously designed for color film. However, RKO cut Cooper's budget just before production and he had to settle for black-and-white. It wasn't until 2006 that Ray Harryhausen chose Legend Films’ advanced colorization technology to rerelease She in the form that Cooper envisioned it, in vibrant color! Ray spent days in our design studio working with our creative director to select appropriate colors for the background mattes so that they retained or in most cases enhanced the realism of a shot. He knew exactly what Cooper would have been looking for had he been sitting at the designer's chair. The result is outstanding and, because of Ray's involvement and creative input, the film is one of the most important colorization projects ever produced.

Ray Harryhausen has become a staunch proponent of Legend Films’ colorization process and has asked both Warner Bros. and Sony Home Entertainment to allow him to work with Legend to colorize his black-and-white classics including Mighty Joe Young, Beast from 20,000 Fathoms, 20 Million Miles to Earth, It Came from Beneath the Sea, and Earth vs. the Flying Saucers.

When he originally produced these films, he envisioned them in color but was constrained by studio budgets and the status of film technology at the time. He told me that it was difficult to do realistic composites with the existing color film stock. Today using Legend Films’ technology he can design his iconic fantasy monster films into the color versions he originally intended.

The Aviator

Martin Scorsese used Legend Films to produce the Hells Angels opening premier scene in The Aviator. He also used Legend's technology to colorize portions of The Outlaw with Jane Russell and many of the dogfight scenes in Hells Angels. While many of the dogfight scenes were done using models, there were others that were simply colorized footage of the Hells Angels feature film. Obviously those scenes would have been very difficult to reproduce as new footage. For a tiny fraction of the cost to recreate the models and live action footage of key scenes, he received from Legend Films all the colorized footage he needed which FX supervisor Rob Legato blended seamlessly into The Aviator.

Jane Russell

I remember showing Jane Russell the seductive scene in The Outlaw that made her a star and was used in The Aviator. Of course it was the first time she had seen it in color. Although it was only about 8 to 10 seconds long and she was viewing it on my 15” laptop, she loved the way her red lips looked, the natural flesh color of her face, and the accuracy of her hair color and its highlights. She later did a running video commentary of The Outlaw for the Legend Films colorized DVD library in which she was very frank, with typically Jane Russell sass, reminiscing about her experiences with Howard Hughes.

Whenever I speak to Jane she expresses a very strong desire to colorize The Pink Fuzzy Nightgown (1957) owned by MGM. That was the last starring film role for the Hollywood icon, which she coproduced with her former husband, Bob Waterfield. Jane considers that film her favorite starring role next to Gentlemen Prefer Blondes (1953).

The film was originally intended to be a film noir, which Jane wanted directed and shot in black-and-white. She was fortunate to have that option. However, it evolved into a comedy and looking back, she has always regretted her decision to shoot it in black-and-white, and today is lobbying the studio to allow her to design the film in color using the Legend Films process. Colorization and her personal involvement in reviving the title will add considerable asset value to a film that is certainly not a revenue producer at the moment. To have Jane Russell color-design and provide commentary on one of her favorite films will fulfill a long-held wish by a beloved and significant Hollywood icon.

Terry Moore

Terry Moore is an actress who was married to Howard Hughes and starred in films such as Mighty Joe Young, Shack Out on 101, Why Must I Die, and Man on a Tightrope. She was nominated for a best supporting actress Academy Award for her role in Come Back Little Sheba. Terry is a dynamo who, at 76, is probably more active today than ever in her career. She became a believer in colorization and a very dear friend of mine when Legend Films colorized A Christmas Wish in which she costarred with Jimmy Durante.

During that process she provided running commentary on the colorized version for our DVD release in 2004. She was so engaged with the process that she commented that Howard Hughes would have loved the Legend Films technology and would have wanted to redux his black-and-white classics. Terry and Jane are happy that we were able to accommodate Hughes with our colorized version of The Outlaw. The two Hollywood stars collaborated on the video commentary for the Legend Films DVD of The Outlaw in which Moore interviews Russell and the two compare notes on Howard Hughes. They are certain he would have colorized that film if he were alive today.

Shirley Temple Black

What can you say about Shirley Temple Black? I had the awesome privilege of visiting her at her home in the Bay Area 3 years ago. I wanted to show her how the colorized Heidi looked that had been sold for the past 10 years on VHS and how it looked colorized in 2003 by Legend Films in high-definition. We lugged a high-definition workstation to her house and set it up in her living room. I'll never forget Shirley's reaction when she saw the switch from black-and-white to color. She initially said, “that looks really good in black-and-white,” but when the color popped in she said, “Wow, that looks great!”

I believe she understood at that moment that colorization, using advanced colorization technology, brings new life to those great classics.

Shirley and Legend soon entered into a partnership to restore and colorize original Shirley Temple films and TV programs that she always showed to her kids during birthday parties but which were stored in her basement after they were grown. Shirley had dupe negatives of her Baby Burlesks shorts; the very first films in which she appeared when she was 3 and 4 years old. She also had 11 hours of The Shirley Temple Storybook, which were produced when she was a young woman. She acted with Jonathan Winters, Imogene Coco, and many other famous comedians and actors of the day in those most popular children's stories.

These 11 hours were the last starring and producing project in which she was involved. The classic TV programs are rich in color and held up well as quad tapes. We restored each one-hour show and released both the Baby Burlesks and The Shirley Temple Storybook as a set that is currently being sold as The Shirley Temple Storybook Collection.

Conclusion

While there will always be detractors, the fact that we fully restore and improve the original high-resolution black-and-white digitized film elements in the process of colorization should satisfy most purists. Furthermore, we always provide the restored black-and-white version along with the colorized version on each Legend Films DVD. In a very real sense, colorization and the increased revenue it generates subsidize the restoration of the original black-and-white feature film.

The bottom line: colorization applied to public domain vintage films is a derivative work that deserves attention, as a valid interpretation of earlier studio's work-for-hire. In that regard it is separate and distinct from the original work and should be allowed to stand on its own.

Colorization has matured to a level where it is a legitimate art form. It is a new way of looking at vintage films and a powerful tool in the hands of a knowledgeable color designer for commercials, music videos, and contemporary feature film special effects.

THE SOUND DESIGN DEPARTMENT FOR INTERACTIVE GAMING

The remainder of this chapter focuses on sound for interactive video games and is contributed by sound designer, composer, audio producer, and audio director Rich Carle. With more than 15 years in the game audio industry, Rich has worked on a wide range of titles including Mortal Kombat (2011), Jon Woo Presents: Stranglehold, and Blitz: The League. He is currently director of audio at NetherRealm Studios, a Warner Brothers Games studio, in Chicago, Illinois.

The job of the production sound team is to capture the best sound possible during the shooting process, including actors’ dialog, sound effects, and locations’ ambient sound.

Although sound is considered to be half of the movie, the sound department—consisting of only three technicians on a feature film or dramatic television series, and just one or two on reality-based shows—is small considering the rest of the crew. Oftentimes 200+ crews are there, on location, to solve, on the spot, issues to keep the camera crew on schedule. Although the production mixer's work cannot be seen onscreen, his or her efforts and job quality can affect the success of a project dramatically.

Figure 22.11 Rich Carle sound designer, composer, audio producer, and audio director.

Now that we are in the new age of digital sound with theaters offering SDDS, DTS, and Dolby Digital reproduced through a carefully engineered speaker system, theaters have had to be inspected by Dolby field technicians with careful and exact measurements to strict playback standards—often leading to redesign and reconstruction so that the theater yields a faithful playback of the soundtrack. Only then can the theater secure the rights to advertise that they have a THX speaker system in that theater, boasting a split left-right surround system.

Today the movie enthusiast can even build a home entertainment system, from 5.1 to 7.1 surround and in some cases even more sophisticated and better than many theaters. Sophisticated audiences insist on better sounding DVDs and higher quality sound. Even though Blu-ray seems to be the ultimate in both high-definition picture (1080p) as well as audio reproduction, I doubt that we have seen the end of it.

With budgets shrinking, and schedules tightening, the production sound department faces more problems than ever before. Our job of recording the sound and monitoring the set to protect the soundtrack of unwanted and unnecessary background sounds is becoming more challenging. Without the support of the production department, the sound department can be made to feel unwelcome or a hindrance to the production process, even though our goal is to deliver the best sounding tracks possible for our clients’ projects. With early collaboration and strategy discussions between the client and the sound department, there is no excuse for a bad soundtrack, no matter what kinds of budget restrictions—well, within reason. A good soundtrack can always be achieved.

Having started in the film business in 1986, and I had been nominated for a Grammy before that, I have experienced many dramatic changes in both equipment and recording techniques. Going from analog to digital recording processes was pleasing and painful in some ways. Even though I am an advocate and first to support newer technologies and techniques to hone our craft, I think it is just as important to understand and appreciate past techniques that have contributed to our goal of delivering good audio tracks during the filming process.

In the past there was more courtesy and cooperation on sets between departments. There seemed to be more of a feel that everyone was working as one to achieve a common goal, to produce a quality project. It was in everyone's best interest to help out another department including sound. In the past, experienced gaffers and grips would take into consideration that the boom would be working on the set and would light the set accordingly to reduce shadows and help set flags to accommodate boom shadow problems.

Camera operators would use camera sound blimps, even packet blankets, to find a way to reduce the noise coming from a noisy camera. The electric department would keep noisy ballast farther away from the set or change out noisy equipment. It was not uncommon for the special effects department to approach the sound mixer when they suspected a “gag” (a piece of business) would be too loud, and the department heads would get together and discuss any ideas to find a quieter solution to accommodate the desired effect and protect the dialog track.

Upgrading equipment and going from analog to digital had its own set of problems and benefits. For example, recording on a Nagra 1/4” reel-to-reel magnetic tape recorder gave the mixer some comfort visually, by seeing the reels turn and monitoring the recording through his or her headsets from the playback head, confirming that you are getting the quality recording that you had planned for. The recorder also sounded very good, the limiters were great, allowing you to record very dynamic tracks safely without overmodulating. On the other hand, it was heavy and cumbersome and only could record about 30 minutes at a time on a 7 1/2” reel. But at least we could visually see how much tape was left on a reel, and we had to constantly estimate if we had enough tape in order not to run out of tape, which was a definite no-no. If the scene was dialog-heavy we were constantly reloading and putting up new reels. We constantly tried to guesstimate when this had to be done so we could do it when cameras had to be reloaded with film as well. Also if the director wanted to hear something back, we had to delicately rewind, play back the desired portion of tape, and then reset, to make sure we were not recording over any previous recorded takes. When digital DAT machines came into the picture, the problem of constantly having to reload tape was not as much of an issue anymore because the tape lengths were up to 120 minutes. Very rarely did we have to use more than one tape in a production day. On the other hand, recueing for playback and the possibility of recording too hot of a signal were new issues and made the task more difficult. Also the sound quality was not quite as good, due to the 16-bit limitations of the DAT machines. Also both of these machines limited us to only 2 tracks.

Finally with the onset of hard disk recorders, such as Zaxcom's Deva and Aaton's Cantar, the recording process became a bit easier. Worrying about recording over previous takes was no longer an issue. Recueing was a snap. The recorders would not allow you to record over previous material. With the introduction of larger hard drives, reloading was no longer much of an issue. These machines also have more tracks for us to record on.

Figure 22.12 From simple “pong” in the 1970s thru today's almost photo realistic interactive video games, the gaming industry has come a fantastic way.

Wireless microphones have also improved greatly over the years. Going from the VHF to UHF diversity was a big improvement. In the past, portable wireless were set to only one frequency. If we encountered interference all we could do was put up another system. With today's equipment we not only have the opportunity to scan for open frequencies but have a choice of hundreds of frequencies to choose. This is also a necessity due to the fact that the airwaves are becoming more crowded. So far, one manufacturer, Zaxcom, has met this challenge head-on by putting the recorder option directly in the wireless transmitter. If the wireless signal drops out from the transmitter to receiver, the backup recording of the same quality is available with matching timecode. Digital wireless has also come onto the scene, improving sound quality drastically. Hard-wired, quality digital-wireless booms with backup recorders built in give the boom operator freedom to move easily around the set and also reassures the sound mixer because he or she knows there is a backup recording available at any time in case there is a problem with wireless transmission.

SOUND FOR THE VIDEO GAME INDUSTRY

Game Audio Overview

In a world where video games generated over $11 billion in 2008 (according to the Entertainment Software Association), a basic knowledge of video game audio can be a valuable tool for a sound professional. As computer technology continues to evolve into the human experience, so do the entertainment possibilities. For the most part, video games still make use of joysticks and two-dimensional screens, but as technology innovations continue to push forward, more immersive interactivity has already arrived. For example, in 2010, Microsoft's “Kinect” hardware has turned the human body into a controller with the use of a motion-detecting camera. This hardware peripheral connects with a host Xbox 360 and allows players to use their bodies and voice to dance, gesture, and speak their way through a game experience. This full-body physical engagement is really taking the players one step closer toward experiencing a dynamic virtual reality right in their living room. Of course, with new technologies come new challenges. Having more knowledge of the fundamental issues and challenges inherent in video game audio will give you a greater understanding of what it takes to make a game experience sound alive and how you can play a role in making it happen.

First off, you cannot understand video game audio without having some understanding of video games. If you do not have any experience with modern video games, find a friend, cousin, niece, or nephew who does, and sit down and watch them play. Once you've watched them play for a few minutes, it's your turn. Pick up the controller and play—no excuses! Reading all the books in the world on video games will not give you the same information that one hour of direct experience will. There is still pervasive thought in this country that video games are only for kids and young adults. The reality, however, is that many older adults grew up with video games as part of their entertainment diet and never stopped playing them. The gaming demographic continues to widen in this country as the stigma associated with being an older gamer dissolves.

Get Back in the Game

If you're really serious about game audio you need to own at least one gaming console and play games yourself consistently. This industry is constantly changing, and when you go after a job, you'll want to make sure your knowledge is direct and current. When I interview potential sound candidates, I always ask them if they play games. If they say “no,” that is a huge sign that they may not be right for the game industry. Imagine a film mixer who doesn't watch films! If they say “yes,” I immediately ask them what game they are playing now and what they think about it. I am looking to see if this person is pursuing a career or just wants a job. I'm also interested in finding out if they are listening when they play. What did you hear that was good? What surprised you? What drove you nuts? I want details. People who really enjoy playing games, called “gamers” in the industry, have no problems coming up with these details instantly. You don't have to play hours and hours of video games every day (in fact, I highly recommend that you don't) to be part of this industry, but an hour every week will help you stay in touch, and it's just plain fun!

When looking more deeply at video games, it is critical to understand the fact that what lies at the core of every game is player to game interactivity. This all-important differentiation from film and TV transforms the more passive “audience” of TV and film shows into active “players.” Game players are consciously unfolding the game's challenges, struggling through setbacks and fulfilling goals. Progress is dictated by how well the player plays, not by how long they've watched. The feelings of accomplishment, discovery, and frustration are much more visceral with interactive entertainment and much more personal. All players are directly responsible for their successes and failures, whether they like it or not. One of the most common mistakes developers make in game sound is not thinking about how irritating sound can be after you have failed a particular task and are forced to repeat it in order to progress. This cycle of try-fail-try-again is omnipresent in games, yet many times the sound design does little to soften the blow of this necessary device. In fact, there are several games that I have played where I would rather stop playing than have to suffer through another incredibly annoying “you lose” sound yet again.

Figure 22.13 WB Games’ Mortal Kombat vs. DC Universe.

Whether it's the simplicity of reorienting the geometric shapes of Tetris into a complete row, or coordinating a complex, multitiered assault with an international group of gamers on an enemy's home base, video games provide a wide array of experiences. Much in the way that the story is the center of a film, a game's design defines all that surrounds it. What will the player be able to do in the game? The answer to this question defines the fundamental mechanics of a video game and is referred to as gameplay throughout the industry. What is the setting of the game? What, if any, is the story of the game? The better you know the answer to these questions, the better prepared you will be to create sound for your game. For big-budget, console video games such as Call of Duty, Assassins Creed, and World of War Craft, the answers to these questions are long and full of very important details.

Both linear and nonlinear entertainment share a great deal of common ground when approaching sound design techniques, many of which have already been discussed in this book. Processes such as field recording, signal flow, microphone placement, DAW usage, sound layering, sound library usage, project organization, and backup procedures all share common ground with both industries. This common background has led many sound artists who have concentrated their skills on crafting sound for the film and television world to work on a video game project at one time or another in their career. Occasionally, some film sound artists even change their specialization and choose to concentrate on games.

There are, however, many critical differences between film and TV sound and video game sound. The concept of shared resources like disk space, RAM constraints, and CPU processing are alien to TV and film at the final stage of playback. Drastically varied listening environments, controller requirements, and, of course, interactivity are more important differences. These issues have far-reaching implications on all aspects of video game sound. To ignore them is to set yourself up for failure. Attempting to produce sound, dialog, and music for a game exactly as you would for a linear production does not work in the real world. These next few pages will give you an overview of the video game industry, briefly describe how teams function, and explore some the details that make video game sound work so interesting.

Technically Speaking

Video games revolve around three interconnected systems: hardware, software, and content. A hardware “platform” refers to the physical devices the game will run on. Today this vast list encompasses handheld devices, phones, desktop gaming consoles (such as Sony's Playstation 3 and Microsoft's Xbox 360), personal computers, Internet games, and all sorts of in-betweens. Understanding exactly which hardware devices your game will support plays an important role in helping you determine the breadth and scope of the audio design. You don't have to be a computer expert or a programmer to work on a video game, but you do need to learn some basics and pay attention. For instance, a highly adaptive music system may not be the best choice on hardware that only allows for four voices of polyphony (simultaneous audio channel playback). If you need a track for gunfire, one for explosions, and another for special bonus sounds, having multiple channels of layered music will not be possible. A games hardware platform plays the most important role in shaping the entire game, and the audio design is no exception.

There are important questions that sound designers need to ask about the hardware in order to shape an appropriate audio landscape. How much random access memory is available? RAM allows for instantaneous playback of sound files, which is critical for any sounds that require picture sync. Sounds like gunfire, footsteps, bullet impacts, punches, and grunts need to be loaded into RAM in order to maintain picture lock. How much hard drive space is available? Even with today's data storage capacities, fitting within size constraints is still an issue that has to be planned for. What types of data compression are available on the hardware? Quad music and 5.1 movies are a great idea, but when coupled with translation into five different languages this can easily get up into the 1 gigabyte range, even after 7:1 data compression. Knowing which types of data compression are available and what, if any, parameters you can access within those formats, can maintain the sound quality of a game while reducing the amount of audio disk space.

It is worth noting that games don't magically work on some hardware and not on others. Game publishers very carefully target hardware systems that they feel will have a user base that will embrace the particular game they are selling. It takes both time and money to modify a game that was originally developed for one hardware platform so that it will work correctly on another. This transferred game is called a port, and if the consumers don't buy the ported version then the publisher's gamble has not paid off. The game industry is littered with ports of games. Sometimes these ports have been given a decent enough budget for the developer to do a good job recreating the powerful experience of the original, but more often than not, the budgets are too tight, the port is weak, and the sales lackluster. A well-made port, one with an extra level, a new character, or a new mode of play can have a strong allure for gamers.

Game software refers to the lines upon lines of code written by computer programmers which not only make the game work, but also create and maintain the tools that allow nonprogrammers to contribute to a game's development. The programming requirements for big-budget video games are incredibly complex and require a large (20–60 person) group of highly capable, highly organized programmers working as team. Programmers with specialties in physics, math, networking, graphics processing, tools, and gameplay all come together to form the bones of a game's structure.

If you can imagine viewing video game hardware as the giant bedrock foundation of a skyscraper, then a game's software can be represented as the steel girders, elevators, fire escapes, plumbing, electrical—the entire infrastructure that sits on top of that foundation. It is the combination of the foundation and the bones—the software—of this complex building upon which all the game art, animation, and sound are hung. Games that look, sound, and play great do so because the design, art, sound, and programming are working together to achieve one complex, engrossing experience.

You cannot talk about modern video game software without talking about game engines and tools. A game engine refers to a collection of software systems designed to facilitate the creation of a game. This is software that helps you make software. A game engine can be custom-created and maintained by the development team or, as is more commonly the case, a third-party game engine can be licensed for use. Licensable game engines, such as Unreal's immensely popular Unreal Engine and Crytek's CryEngine, offer game developers a huge leg up toward creating a video game. A “renderer” for 2D or 3D graphics, a physics engine or collision detection system, sound, scripting, animation, artificial intelligence, networking, streaming, memory management, threading, and localization support make up the core functionality provided by game engines. These out-of-the-box software solutions can be further augmented by specialty software called middleware. Middleware is a term used throughout the games industry to define ancillary software that is used to help a specific aspect of a game, such as Havok's physics engine or Firelight's FMOD sound system. One of the most powerful attractions to these middleware solutions is their ability to be easily adapted to a variety of hardware platforms. This hardware-agnostic functionality of middleware is a big attraction to video game developers who are producing their game for multiple platforms simultaneously.

The Big Picture

Every industry has hierarchical organizations that work together to produce a product, and the video game industry is no exception. Learning some basics about how the video game industry operates will help you gain some perspective on where exactly you fit in. If there is one thing the video game industry is completely aware of, it is that in today's competitive marketplace all aspects of a game—quality, pre-launch awareness, marketing muscle, launch timing, post-release support—have to be executed well in order for a game to have a chance at financial success. Computer entertainment is a fast-moving industry, and making profitable console games is no easy task. In recent years, there has been a great deal of consolidation as publishers have bought up many skilled development studios. As the cost of making AAA console games increases, it's becoming more and more important for publishers and developers to make games that have a high probability of financial success.

Publishers

The video game publisher sits on top of the food chain and is responsible for the funding, marketing, and advertising of a game. Whether the game developer is owned by the publisher (sometimes called an “internal studio”) or the developer is an independent, varies from game to game. Larger video game publishers often have several internal studios and external studios developing games for them concurrently. Publishers are also responsible for distributing the product across the globe, usually through their own internal networks. With digital distribution and social media gaining momentum in recent years, downloading games is becoming easier and more popular, and every game publisher is betting heavily on this area's growth.

Publishers struggle with figuring out which games to greenlight and which games to cancel. They keep a careful watch on the games they are currently funding by using producers who stay in constant contact with the developer. They also work with their developers to create milestones that use quantifiable goals on a set schedule. If the developer is not owned by the publisher, these milestones are embedded in their contract as a requirement for payment. These all-important milestones allow publishers to track each project's progress and make crucial decisions about a game's scope, release date, and marketing dollars.

Developers

The developer's primary concerns are all about the game. What game are we going to make and why? What are the critical features, the secondary features, the reasons for playing? Is it easy to pick up and play, but hard to master? Does it resonate with a variety of game players? Does it look and sound great? And the most challenging of all, is it fun to play? This may sound like a simple question, but there have been many games put into boxes over the years that look and sound competitive, but whose reliance on regurgitated game mechanics and cliché plot devices make them unenjoyable to play. Just like in film, quality sound and picture can enhance the core experience, not fundamentally alter it.

A typical video game developer (a.k.a. development studio) for a high-end console game requires roughly 60 to 120 internal employees (depending on the game it is making). Employees break down into disciplines such as art, design, production, programming, animation, and audio. The two largest groups of people are the programming team and the art team, comprising roughly 70 percent of the total development personnel. With this many people on the project (more counting contractor support) and a schedule that often runs over 2 years, AAA video games are very expensive endeavors. Costing anywhere from $20–$60 million to produce, and posing a relatively high risk of failure, there is great pressure on development studios to make a successful game; the ultimate prize being a franchise game that can draw in gamers with some regularity. The best developers find balance amid this chaos. They find a way to maintain deadlines, keep an eye on the bottom line, and still have fun.

Game development begins with inexpensive “clay.” Devices such as story outlines, sketch art, movie mockups, and rough gameplay mechanics can help define a direction with very little budgeting. The goal for this conceptual phase is to impart to the team at large, and potentially the publisher, the pillars, goals, and ethos of the game being pitched. Transforming a video game from a concept into a fully funded greenlit project is never an easy task, but the process is more streamlined for developers owned by publishers due to the high level of internal executive communication. Most developers pitch game ideas by building on an existing engine that they are already familiar with. Usually they have already developed a game or two with the engine. With a firm grasp on a game engine, getting basic components of the game up on screen and playable isn't a horribly time-consuming effort and the focus quickly turns to developing the first fully polished level.

Creating one polished level, often referred to as a vertical slice, is typically the first goal of a greenlit game project. This first level serves as the template on which all the game's features and content are built from. Once a game has passed through the vertical slice milestone, the game enters full production. Team numbers are further augmented by contracting support in areas such as cinema, art, and audio. Open world games, such as Grand Theft Auto IV, rely on hundreds of internal employees and almost just as many contractual employees in order to complete them in a timely fashion. A smaller scale game like WB Games’ Mortal Kombat vs. DC Universe (2008) was created by a core team of roughly 70 full-time employees.

It is very common for a game, even while in full production, to change direction from what its original design goals were. This altering, of course, can be slight or drastic and can happen for a variety of reasons. In the best cases, the changes are slight and the reasons are player-centric, such as adding an additional weapon or consolidating two similar levels. In other cases, teams may suffer from unplanned attrition of key personnel, miss milestone deliverables, or simply be incapable of pulling off a feature as originally promised. In these more challenging cases where creative problem-solving is required, having a tight-knit development team and a flexible publisher can be the key to avoiding disaster. Carefully constructed and well-executed focus testing can provide critical information on how actual game players are experiencing your game. More and more developers and publishers are using periodic play-testing throughout a product's development in order to ensure the players’ experience is a positive one. Focus groups, however, are by no means a guarantee for success. What they do best is uncover specific problematic areas within a game's level.

Production Studios

Many of the same production facilities with history and expertise servicing the television and film industries also provide art and audio content for the video game industry. Bigger companies often have subdivisions or groups that specialize in video game development. Production studios are capable of creating 3D characters and backgrounds, lighting effects, camera movements, particle effects, storyboards, cinemas, music, sound effects, Foley, mixing—pretty much anything content related. Typically, there is less support when it comes to game-side implementation, but even these tasks are starting to appear more and more on production companies’ lists of services.

Production studios can provide security where the independent contractor cannot. When you hire a production studio you're hiring their collective experience, their internal networks, and their experienced staff. If one of the people working on your project falls ill during the last week of mixing, for instance, the production studio will very likely have a top-notch replacement ready to keep you on schedule. When schedule is of primary importance, this depth of support can provide wonderful security. This infrastructure and security can come at a cost though. Typically production company services are substantially more expensive than those of an independent contributor, but this isn't always the case.

Individual Contributors

In today's laptop society an independent visual artist, sound designer, or composer can contribute significantly to the success of a video game project from the comfort of his or her own home. People who find the most success in this role have more often than not paid their dues by working in the industry for a number of years and by working on projects that are financially successful. In order to survive as an independent contractor you have to have a strong network of potential employers, and these relationships are also typically formed through work experience. To begin a career as an independent contractor is practically impossible since so much of this type of hiring is contingent on past work. Employers want to see a nice trail of shipped titles on your résumé. They need to know that your work has been part of products that have reached the market successfully, just like their game needs to.

As mentioned earlier, using contractual or external audio support for video game development is widespread in the industry. Typically, the smaller the developer the smaller the resources dedicated toward audio support. For many smaller developers this means only one internal sound employee or one full-time sound contractor. Depending on the scope of the game, this person could have a very wide range of responsibilities. Hiring independent, individual composers is a common outsourcing technique within the game industry, but hiring individual sound designers is rarer. This seems to have more to do with legal reasons than quality of work. In my experience, when reputable individual sound designers have been contracted to augment an existing internal sound team the results have been fantastic. With a clear plan, and the proper support structure in place, having someone in-house, on the ground level, can be a great boost for the audio team during the final few months of a game project.

The Game Development Team

Understanding who does what in the video game industry is a critical component of getting things done. Although roles in games are not as clearly defined as roles in film, patterns have emerged across the industry that allow talent to transition from company to company without any big issues. In many areas, the game industry has drawn clear inspiration from the film and TV industries, but titles are still not as clearly established yet. Having a basic understanding as to who does what on a game team will help you understand your place in the machine and, more importantly, discover ways to improve your process.

At the very top of the team sits the creative director and the executive producer. These two roles are critical in order for a game team to run efficiently. The creative director, acting much like a film director, is involved with the project from inception through completion and is responsible for creatively pushing the team leaders to create the compelling homogeneous game experience he set out to make. In contrast, the executive producer role focuses on budget, schedule, and communications between disciplines. A typical game development team for a high-end console game breaks down into six main disciplines: design, programming, art, animation, audio, and production. Each discipline is represented by tiered leadership that focuses everyone's efforts toward common goals. Visual artists and programmers typically make up the majority of the team, encompassing 60 or 70 percent.

If you are a sound professional working onsite at a development studio, it is particularly important to have an awareness of who's who on the programming, art, and animation staff. Sound in particular benefits from a basic understanding of particle effects, textures, and rudimentary artificial intelligence (AI) behaviors. These systems are often used to trigger sounds during a game. For example, having open communication with the visual effects (VFX) team can save the video game sound designer time and energy. Many times I have had the benefit of working with a VFX lead who would send me an email and point out new material that was just approved by the creative director. In one such instance, I was able to design a few sound treatments for a new car explosion before the production team even assigned me the task. Unlike film, however, the core game development team works together in the same building. This proximity allows for direct contact between disciplines on a daily basis and it ends up breeding trust and synergy.

Designers and Gameplay

In essence, the design team is responsible for the heart of a video game. The design team, led by the design director, is responsible for creating and maintaining the gameplay. For games, the core experience is the gameplay. Gameplay refers to the interactive experiences that occur between player and software while playing the video game. Put more simply, gameplay is what you do in a game. In film, the story is the core of the experience. Designers drive what the player can do. If the player can run, jump, double-jump, strafe, climb, glide, fly, grapple, swim, fight, shoot, discuss, fix, eat, and fall in love, it is because the design team wanted it to happen. Of course, each of these abilities has huge ramifications for the entire team. Attributes that affect the player's abilities also fall under design's jurisdiction. Attributes such as fatigue, injuries, power-ups, illness, and geographical locations are just a handful of parameters that could affect a player's ability to jump high, shoot accurately, or run more slowly. In fact, the entire world the player interacts with is blocked out by the design team before being “skinned” by the art team. This job falls to the level designer.

Figure 22.14 The step-by-step process of how we get from the initial concept to the Final Mix.

In an adventure game (third- or first-person) the object or character the player controls in the game is intrinsically tied to the environment it exists within. For example, if the main character can jump only 10’ high, then there should not be any 11’ tall walls that require the player to climb them in order to progress. Great first-person shooters such as Unreal Tournament, Call of Duty, and Quake rely heavily on clever level design in order to provide a rich environment, conducive to a variety of playing styles, that players enjoy competing in. Players who love first-person shooting games become extremely familiar with the game's arenas (levels) they play in because they spend hours and hours playing them again and again. A deep knowledge of the game environment gives a player enormous tactical advantage in a multiplayer game.

Great designers are masters of giving players just enough abilities and information to point them toward a solution, allowing the player to fill in the blanks with their skill. Riding that narrow area between too easy and impossible is the designer's job. When design does its job well, players achieve a wonderful feeling of accomplishment and mastery that keeps them coming back over and over again. Since players’ skills are constantly evolving, designers allow players to adjust the degree of difficulty to suit their skill level. This is a critical design mechanism that allows players of all skill levels to enjoy the game.

Programming

If design is the heart of a video game, then programming is surely the soul. Games, like all software, are held together by programming. Programming is the never seen or heard, but always felt, force on which the entire game is built. This industry was founded by programmers—brilliant and slightly crazy programmers.

Good game programmers know how to make the game look and play great to the player while maximizing the performance capabilities of the hardware. They are not out to write the deepest artificial intelligence for the enemy characters you will face as a gamer; rather, they cleverly trick the player into thinking that the AI is incredibly dynamic. It doesn't matter if the AI is truly adaptive and dynamic; what matters is the player feels like the AI adaptive and dynamic. When there are so many systems demanding CPU time, RAM, and disk resources, it's critical that game code is optimized for performance and the resources spent to perform a task are exactly what is needed and not a cycle more. I once worked with a programmer on a sports game who was absolutely brilliant. He had his master's degree in computer science from a very reputable university and had the respect of the team. He was in charge of the crucial AI system that is so important in multiplayer sports. Things, however, ended up not going so well and the AI techniques had to changed significantly. Later, another programmer commented to me that “yeah, that guy wrote the most amazing AI code that ran at 1 frame a second I ever saw.” Sometimes it doesn't matter if you're a math genius; sometimes what matters is that you can solve a problem well enough, but with great efficiency. Games are full of programming tricks that allow the player to see only what the programmers want them to see and not a polygon more. Just like on film set, if the game camera were to swing a few more inches to left you see a great big hole in the sky or that only half the car was modeled in 3D, and the carefully constructed illusion would be broken.

Programmers work by using a programming language such as C++ to express computations that can be performed by the hardware. The programming team creates and maintains the game engine, as well as the tools that are used by the designers, artists, animators, and sound folks to deliver their content into the game. Ultimately, the game, like all software, must be compiled into machine code with compiler software so that it can run on the target hardware. This process is called building the game, and software developers refer to each new compiled iteration of the software as a new build.

Not only do programmers build the infrastructure that all game content hangs on, but they also maintain the game tools. As mentioned earlier, game tools refer to the software game developers use to make the game. Typically a substantial percentage of the programming team is dedicated toward tech and tool support. Designers, sound designers, and artists are always looking for more flexibility and efficiency in their tools, and programmers are the ones who make those enhancements a reality.

Art and Animation

A game's art look cannot be underestimated. From the frontline art, such as the main characters and primary locations, to the subtler hidden gems like the hard-to-find weapons and random debris, each fragment of art is crafted to suit its purpose. From the grand vistas of games that allow you to see the enormity of the playing space (Assassin's Creed, Fallout 3) to the tiny can of beans that explodes and deforms after you shoot it, leaving a smear of bean paste on the wall, the art is everything you see. The art team for a video game project typically requires the most amount of sheer manpower, about 30 to 50 people for a big-budget console game. Today's demand for lifelike art and rendering capability means that artists have to put more detail into their models than ever before and that takes talented people—a lot of them. The art team is led by the art director whose primary function is to keep the art team marching in the right direction. Just like in film, there is so much work for the art department that the art team subdivides into specialties in concept art, characters, backgrounds, lighting, particle effects, user interface, and more. Some artists can fulfill several of these roles, but choosing a specialty and developing advanced skill helps bigger game teams work more efficiently. On larger projects, art specialists are so numerous that they are further subdivided into their own groups and managed by their own leads.

Much like in animated films, animators are responsible for making characters appear to move. Whether it's a colossus swinging a skyscraper-sized hammer or a cute hamster batting its lush eyelashes, an animator has likely brought it to life. These dynamic and delicate motions can easily be taken for granted as we fall further under a game's spell, but the effort and energy required to make smooth and efficient motion takes a massive amount of energy and planning. When motion is awkward or irregular it pulls us out of the experience and can really ruin a game—sometimes literally! Animation data are either based on a captured performance of an animation actor or are hand-keyed one frame at a time by an animator. Even when the motion-capture method is used, animators still spend hours and hours polishing and tweaking the animations before they are usable in the game itself. Another important point is that animation data, like all the other content components, must fit in RAM and on disc. Some games strive for hyper-realism in their animations, making sure that characters move in a way that is true to life, while other games may pursue a completely different aesthetic goal.

The Sound Team

As an audio professional, it's important to understand that most of the people you'll be working with outside of audio (for games this means the vast majority of the team) have no idea what you do. This is nothing personal; they simply do not know—nor do they need to. You are doing your job; they are doing their job. Everybody has an important role to fulfill on a game team and learning exactly how someone else does his or her job isn't always a good use of one's time. It is important, however, that your creative leadership care about how the game will sound. Some creative directors and leaders know a great deal about audio. More often than not, directors care tremendously about the sound, but don't have the language or terminology to express a clear direction. Unable to see it, sound can be a difficult discipline to collaborate on and direct. Let's be real: even audio professionals, competent in audio vernacular, still have difficulty directing each other! Whether games or film, being able to collaborate, interpret, and partner with your creative leadership are very important skills for an audio director and any entertainment professional to cultivate.

Being a good collaborator doesn't mean giving up on your opinions and doing “whatever the director says,” although sometimes that will happen. Good collaboration requires a respectful, open sharing of ideas and unending pursuit of a win-win outcome. It requires that you handle differences of opinion with graciousness and deference; this detached passion is a major component of professionalism. If you've put your heart into your work and you feel passionately that it serves the game well, it will be hard to hear that it needs further revision, but it's critical that you are willing to accept it when this happens, as it happens, and push to discover an alternate solution. I am constantly surprised and inspired by the threads of truth that come in the form of criticisms and suggestions from nonaudio people. These comments, the ones with complete disregard for how to create audio “the right way” are laden with cool possibilities; it's up to you to recognize their value and figure out how to pull it off!

Within video game audio, job titles and job descriptions change to suit a company's needs and philosophies. What a “sound designer level 3” is expected to be here may be called a “senior sound designer” there, but for the most part the general responsibilities listed below shadow most of how the game industry stratifies audio personnel for big-budget console titles. For smaller game companies and smaller games, not all the roles may exist, but the responsibilities remain.

In many ways, a video game audio director shares a lot in common with a supervising sound editor from the film world. The audio director is responsible for getting the game to sound great, on time, and within budget. More often than not the audio director is also in charge of the game's mix. In the early stages of a game, the audio director works with the creative director and the audio lead to define the aesthetic approach to the game, and collaborates with the technical director to define the necessary tools and technical budgets (RAM and disk space). The audio director typically manages the project from beginning to end and is usually a full-time employee at the development studio. Making sure that the right internal and external people are in place at the right times and that the audio quality is being maintained is also part of the audio director's world. This means audio contractors, composers, and production companies are usually vetted and hired by, or in conjunction with, the audio director.

At the beginning of every game, the audio director partners with the creative and technical leadership to stake out the audio design and define a set of technical constraints that the audio design must comply with. It is critical that while a game is still in this early blueprint stage an audio director is consulted. The audio director will translate the Game Design Document, the video game equivalent of a movie script, into an Audio Design Document (ADD). This early ADD is usually somewhere between 10 and 30 pages depending on the scope of the project, and more often than not lives in a wiki form on the developer's intranet. Defining the audio goals for a game is a push and pull of creative ideas versus technical costs. Balancing the creative and technical aspects requires taking a long, careful look at the specific game design goals and then coming up with audio goals that reflect the core game design but are also achievable technically. It's important to understand that although all design documents are meant to get everyone marching in the same direction, they are not unbendable directions. It's quite possible that the audio blueprints may need to shrink or expand during development as the game evolves. The audio director facilitates the most compelling audio design possible and never stops pursuing that goal throughout the development process, no matter what changes occur.

Once the basic audio blueprints are in place creatively, it is time to do a full examination of technical limitations. How much RAM can be used for audio data and the audio engine? Can audio data be streamed in real time from the disc? Will there be any load-on-demand audio functionality? In my experience, the constraints are defined in large part by the audio director's vision for the game. An audio director with a clear vision of what he wants is much more likely to attain his goal than one who is uncertain. Everyone wants the game to sound great, but they also want it to look great and play great. Since the game software must share the hardware resources, technical limitations of art, sound, and animation are always ongoing negotiations.

Audio pipelines are another major concern for the audio director. A pipeline refers to any path that must be taken when getting content into the game. Most sound designers and composers develop sound content with some type of professional digital audio workstation (DAW). These DAWs come in many varieties and prices and some of the best have been discussed in depth in the book already. After the sounds are designed in the DAW, they are printed or rendered into a mixed sound file and then named properly. These designed audio files then need to be authored into a game-ready state so that they can be introduced into the game engine. Finally, the packaged audio is associated with some kind of game trigger and the sound is now heard when playing the game. The audio pipeline refers to all these aforementioned steps, which must be taken in order to get a sound playing in the game. The quicker and easier an audio pipeline is to work with, the happier and more creative the audio team will be, and it is the audio director who needs to advocate for pipeline maintenance improvements.

While working on Mortal Kombat vs. DC Universe, a fighting game, I spent a significant amount of time setting up processes that would enable the audio team to work more effectively and create a more compelling soundtrack. Up until that point, Mortal Kombat had very little Foley being triggered by the characters movements. In previous versions of the game, programmers were triggering footsteps at key moments during animations and that was the extent of Foley. It took me a while to convince the leadership that a more involved Foley system was worth the investment. That's exactly what it is, too, an investment! An investment in money to pay the programmers to build the infrastructure, an investment of time from both programmers and sound designers, and an investment in computer resources—CPU, RAM, and disk space. It was hard to get people excited about investing in “quiet little sounds that characters make as they move around the world,” but with a little bit of persistence and some cinematic demonstrations the leadership finally understood the impact these small sounds could have in making the game world more believable, and therefore greenlit the investment. Once the Foley pipeline was in place, it was easy for the sound team to create content, author it, and attach it to character movements, making the feel much more real.

An audio lead acts as the ground-level leader of the troops. He has equal parts technical skill and creative talent and is often responsible for leading the way in ironing out new audio initiatives. Where the audio director can be more focused on the future, the audio lead is focused on the here and now and getting the job done well. His leadership and professionalism are also vitally important to the audio team as they are often seen as the role model for quality levels, work ethic, and teamwork. A skilled audio lead is typically the first hire of an audio director.

In-house sound designers are the life blood of the sound team and form the majority of the group. At some studios, sound designers (largely content creators) are separated from audio implementers (who author sound events and attach them to game events). I feel, however, that the best results are derived when the people who create the sounds are the same people who author and implement them. Having the understanding of the full path, from creation to playback, gives sound designers a perspective and understanding that they cannot get if they do not have direct experience with all these processes. Once they have spent significant time dealing with all these systems they become more clever at making impactful yet economic sounds that take advantage of the software capabilities without abusing them.

Sound designers often get to wear many hats when working on a video game. Not only will they be designing sound effects and implementing them, but they will likely have to cut dialog, go on many field recordings, mix short movies (often in 5.1), and edit music. This variety in the job is truly one of my favorite parts of working in video game audio because it provides hands-on exposure to many aspects of audio that you may not get in other fields.

Dialog leads are also becoming a common job that can fall in the audio department. A person who can see a game's dialog through from inception to final submission can have a lot of work on his or her plate. Many games rely heavily on dialog in order to coax the players through their narrative. Some games have upwards of 20,000 lines of dialog—and that's just the English-language version! It takes tremendous effort and attention to detail to track all these data and ensure their quality. Dialog leads (also called dialog coordinators) work with scriptwriters, casting directors, acting talent, studio engineers, dialog editors, and many other people to make sure every file is recorded, edited, processed, and named properly. Dialog coordinators can also facilitate audio localization needs. When you know that your game is going to be localized from the beginning, all sorts of practices can be put in place to prepare for this work. The simplest example is stubbing out all the foreign language data by renaming the existing English data. This way the game team has a clear idea of how much room on this disk these data will take up and can start preparing the pathways to switch to the appropriate language. For all this work and effort, well-planned, well-implemented dialog often goes under the radar in video games. What players do seem to notice and remember are the lines that don't fit the game well or get repeated too often, or don't fit the context of the game situation. It takes tremendous vision, persistence, and attention to detail to make natural sounding dialog, that isn't forced or simplistic, yet still impart necessary information to the player. Having one or more people on staff to create and follow through with this vision is essential.

Game Audio Techniques in Action

Putting sounds in a game is five-step process that repeats itself over and over again throughout the development of a video game. The first step is the audio design. This step mostly involves planning and imagination, as the sound artist drinks in information about the game and projects that into sonic possibilities. Some of this visioning is very straightforward, for example, “There is a laser rifle in the game, so we need a cool laser rifle sound.” Some sound ideas require more effort, such as designing the sound components needed to make a user-controlled helicopter sound like it's speeding up, slowing down, and turning as the player moves the joystick. During the design phase you are leveraging your experience against what you wish you could do and coming up with real but hard-to-reach goals.

After you've thought about how you're going to solve a particular sound need, it's time to move on to step two: sound creation. Sound creation is the art of creating sounds, music, dialog, and ambiences with quality source material, some type of multitrack device, and a good ear. At the end of the day, a game's sound package is made of thousands of little designed sound files. Most of these little guys are mono, some are stereo, and others are multichannel. Each sound needs to be crafted with its ultimate purpose in mind, then rendered (printed) and named with care. I cannot stress enough how developing and maintaining a robust naming convention for all these little sound files will help you stay organized and work quickly. Prefixes like ENV_ for environmental audio, MUS_ for music, VOX_ for dialog, and SFX_ for sound effects are a great place to start for any game. The multitudes of sound files need to be loved and tended to so they can grow up to be big strong game sounds. The sound content will change and change again over the course of the game as it does with every game; it matures and ripens, just like the art and tech. Once you have the sound content ready, the next step involves associating “control” or behavioral data with the sound (or group of sounds) and thus creating a “sound event.”

Figure 22.15 Complex sound cue inside Epic's Unreal Engine 3’s editor.

The third step, creating sound events (also called soundcues or sound containers) is done in whichever sound tool is being used by the development team in a process referred to as authoring. Game audio tools such as FMOD, the Miles Sound System, and Wwise are needed to complete this step. Here atomic sound files are imported or dragged into the sound tool and organized, and their control data are modified to suit their function (such as looping or one-shot). There is shortlist of basic control parameters that will be found in every game audio tool such as pitch, volume, priority, and whether or not a sound is meant to be 2D or 3D. These basic parameters are mostly designed to help combat the highly repetitious nature of video games. Today, most video game audio tools go way beyond these basic parameters, however, allowing for much more detailed control over the behavior of sound. Many development teams also make use of user-definable parameter control. By manipulating audio effect parameters with realtime game data you can make a car's engine feel responsive, a concussion-grenade stifle your hearing, or a bullet whiz by your head.

Just about every sound file that is created for the game—dialog, sound effects, music, and everything in between—must be packaged into some type of sound event in order to be recognized by the game engine. It is important to understand that the sound event data store information about how the sound will behave when played back by the game and does not equate to directly to the sound files themselves. In this way a video game is very similar to a sample playback device. Instead of the typical musical audio files and patch (bank) data, we have all sorts of audio files and game-centric data. And instead of the notes being performed by someone at an electronic keyboard, the performer here is a combination of player and game performing through a game console. Just like with MIDI, sound behavioral data can drastically change how the source audio sounds during playback or not change it much at all. Sound event data also answer questions like: Will the sound loop? Will the sound be randomly chosen from a group of sounds? What randomization scheme will be used? Will some sounds be weighted more heavily than others? Will the sound's volume be randomized and to what degree? What about the pitch?

For 3D games, the most fundamental question that a sound event answers is whether or not the sound event should be 3D. If a sound is intended to be 3D, then the audio content is generally mono. A 3D sound event has its volume and pan information calculated by the game based on the distance from the listener position to the sound event while the game is being played (run-time). The clearest examples can be seen in persistent ambient sounds like a campfire. To make the sound for a campfire work correctly, a sound event is associated with the location where the campfire exists (generally this is done with a sound emitter, discussed later on). The campfire sound event must contain information that defines at what distance the sound will begin to be heard; this is called the maximum radius. This sound event will also hold a value for when the sound should reach its maximum volume, also called the minimum radius. In order for this whole distance calculation to work, we need more than just the coordinates for the sound event, we need to define a listener position as well. More often than not the listener's position is the camera, but for some third-person adventure games the listener's position in the main character.

Now imagine this scenario in a first-person adventure game with the listener position fixed on the camera. When you are outside the campfire radius you will hear nothing. As you approach the campfire and break its maximum radius value you will start to hear the crackling fire quietly and in the front speakers. As you continue to walk toward the campfire, you will notice the volume gradually increasing until you finally break its minimum radius and the sound's maximum volume is heard. A 3D sound event not only has its volume adjusted in real time but also has its pan information adjusted based on the listener's orientation as well. Imagine listening to the game in your living room through a 5.1 consumer system. You are in the same scenario as before, standing next to a crackling campfire. As you begin to turn away from the campfire to the left the campfire will disappear from the screen on your right and should be prominently heard from your right front and right rear speakers. When the campfire is directly behind you will hear it coming solely from your two rear speakers. The game engine smoothly changes the 3D sound's volume and 3D pan values based on the distance and orientation of the listener to the sound event. Many authoring tools and game engines contain a number of parameters that exist to further control 3D sounds beyond a simple min/max radius, but defining the radius is at the heart of defining a 3D sounds. This is a concrete example of how a player changes the game soundtrack simply by playing the game.

Not all sounds should be 3D, however. Sounds such as user interface sounds, headset communications, players’ weapon fire, and music are three examples that are typically not 3D. These sounds, which are typically in the minority on a 3D game, are referred to as 2D sounds and do not have a location in the game's level. Once the sound designer has made the 2D/3D decisions and authored the sound events appropriately, the fourth step is to get them triggering in the game properly.

After sound content is created and authored correctly, it's time for them to be triggered by a game-side event in the game engine. Sound events can be triggered in a variety of ways in a modern 3D console game. Game-side triggers such as distance, time, button presses, animations, particle effects, and scripting are some of the most common devices for triggering sounds. Oftentimes the programming team will invest in modifying or creating additional tools to support features that are fundamental to the games experience. During Jon Woo Presents: Stranglehold the development team created robust weapon tools and breakable object tools that allowed artists, visual effects artists, animators, and sound designers to associate their game content with these all-important features in a detailed way. Once the sound events are being triggered by the game you will be able to hear the behavioral adjustments you made during the authoring process as you play the game. This leads us to the final step of the audio process: testing and revising.

Of course the final test of every sound is how it works in the context of playing the actual game. Here, during this final testing phase, is where the sound designer listens and adjusts the sounds as necessary. After listening for a while, the sound designer may decide that the triggering distance is too large, or maybe the sound volume randomization is too wide, or maybe the content itself needs reshaping. A good sound designer can usually figure out which one of the three it is within minutes and then start to modify the weak link. This testing, massaging, and retesting of the sound data occurs constantly throughout a game's development. Sometimes, as it turns out, the original concept was flawed and the plan itself needs to be reenvisioned. Don't worry though, it's right about now that the design team is realizing that they need to redo the entire third level, so all your sound will stop working!

As a quick example of sound authoring and working with game data to produce an engaging effect, let's look at the following example. Let's say that on level 6 of our third-person fantasy game the player must fight the evil boss monster. This evil boss monster draws her powers from the blue crystals in the center of the room you must fight in. When the monster is close to the crystals (the distance between the monster and crystals is small) you want her voice to be pitch-shifted down slightly, maybe even run through some light distortion as to convey a creepy power. As the monster is lured away from the crystal (and the distance from the monster to the crystal increases), however, you want the voice to return to a more human-like state—this will help inform the player that the monster is vulnerable and capable of being attacked. This is a classic example of how the game parameter “distance from monster to crystal” can be used to drive the parameters of the audio effect's “pitch amount” and “distortion wet/dry” to achieve compelling, game-centric audio support.

Sounds Good to Me

Good sound design is good sound design, regardless of the medium. Sound effects are just as crucial to a game as they are to a film, yet many video game studios will spend much more effort and budget on music and dialog than on sound design. Obviously an overly frugal approach to sound design yields results that can cheapen the entire product. A player may not be able to verbalize the fact that “since there were only two sounds being used for ‘footsteps on gravel’ when the player was outside the industrial complex, the game felt flat,” but that doesn't mean that the lack of sound design is not contributing to the bland experience. Another mistake that occurs in video game sound design is the overabundance of overly compressed (dynamically) sound files. Many sound designers are ready to make every sound huge in order that every sound is as big and important as it can possibly be. An experienced sound designer, however, understands that every sound, from the biggest explosion to the smallest computer futz, is reflecting the world in which the game is taking place and must consequently take its shape as a piece of a much larger puzzle. Sometimes this means designing quiet little sounds that fill in the cracks. We need to ask: How does the sound I'm making fit into the big picture? Will it be triggered alone or with other sounds? Will it be heard often or rarely? Most importantly, what does it need to communicate to the player? If the player has just picked up the 32 mm revolver, it better not sound as big and powerful as the .44 Magnum that he can get in level 3. It is a tremendous letdown to pick up a newer, bigger, better weapon and perceive zero to very little difference in the size of the gun as compared to a smaller caliber weapon. This overcompression mistake is common in games and it takes resolve and diplomacy to keep this monster from ruining your game.

Unlike film, where a sound editor crafts and layers sound and then hands it off to the mixer for final balancing and pan, a video game sound designer is responsible for printing the final mix of each sound. The good news is that these source sounds are typically checked into some type of version control software that allows the sound designers to modify them again and again until they are done. These game-ready sound files or “atomic sounds” are the basic building blocks of a game's sound package. Every atomic sound that is created must be made with its playback intention in mind. If you are making a simple sound that is to be played when the player navigates user interface screens, it's a good idea that the sound has a sharp, clear attack. This way, when the players press the button, they hear instant confirmation that the game has recognized their input and responded. No matter how creative and cool this sound was, if it had a long buildup, the player would likely be confused by the apparent slowness of the game's reaction and may no longer identify the button press with the sound. In this way, although the sound itself was fine, it was not a good match for its purpose in the game.

Figure 22.16 Sound FX menu.

There are two primary approaches to one-shot sound effects for video games. The first approach, called simple sounds, also referred to as prerendered or baked sounds, results from the sound designer layering, EQing, and processing source sound to create one final audio file that is then triggered by the game in a one-to-one capacity. Simple sounds put all the control in the sound designer's hands. The sound that the artist creates is exactly what the game plays back. The great thing about this method is that it allows the sound designer to sculpt a sound that fits the game event perfectly, but the downside is that it can limit variety. Complex sounds, on the other hand, are sounds that have been designed in pieces or fragments so that they can be reassembled in a variety of patterns and thus create a wide variety of sounds while the game runs.

This complex sound technique was very helpful in a third-person shooter I worked on called Stranglehold. This game was full of guns, whiz-bys, and explosions. Our explosions were long stereo sounds that came in three different flavors: small, medium, and large. Although we had four variations within each size category (a total of 12 sounds), small, medium, and large explosion categories were not a deep enough well to cover the variety of explosions seen in the game. The sound team then crafted a series of explosive sweetener categories—water, metal, glass, wood, brick, and fire. Each of these categories had 3 variations within them as well. Together these 30 sounds good produced well over 100 permutations. These sounds were then associated with various exploding objects to further define their specific explosion color. In Figure 22.16 we see a custom tool that allowed nonprogrammers to apply attributes to breakable objects. On the right lower half of the image three sound events (called soundcues in the Unreal Engine) have been associated with this explosive metal barrel.

Combining the right kinds of explosion sounds with the right kind of object would bring a distinct character to the event, allowing for a wider variety of permutations; less RAM would be consumed than the amount needed to create baked versions of all the possible variations. On the other hand, playing three stereo sounds on one explosion eats up three times the polyphony a baked file would per explosion, and this means more CPU cycles. This means that if the game designers want to fill a room with exploding barrels, there will be consequences for the audio playback. Fortunately, most tools put controls in place that allow video game sound designers to gracefully handle such situations. Whether the approach uses a simple or complex sound technique, designers have to carefully craft the sounds, test their creations in the game, and weed out the weaknesses.

Felt More Than Heard

Foley and ambiences play the same important role in games as they do in film. They impart reality and vulnerability to the characters. Foley in games, however, must be edited into small files so that it can be associated with a variety of animations. Ubisoft's Assassin's Creed series is an example of expertly crafted Foley that has been implemented and mixed with restraint and precision. The effect of a marriage this solid is a direct and personal connection between the player and the character you play as. Mortal Kombat (2011) is a game that relies heavily on a larger set of characters. As mentioned earlier, in an effort to make each character more “real,” a set of costume Foley sounds were created. Costume Foley types were broken down into categories such as cloth, leather, light armour, cyborg, and so on, and assignable through programming to the upper or lower half of each character's body. In this way, a character with cloth pants but no shirt could be accommodated with the same amount of effort as a character with leather pants and army jacket. Each of the Foley sound types was further divided into “long” (about 1 second) and “short” (about 0.3 seconds). This gave us more flexibility and variety during the implementation process. Although the sounds were not specifically sculpted to any particular animations (of which there are thousands) the categorization allowed us to bring a good level of detail and differentiation for a low amount of RAM cost—good bang for the buck.

Sometimes space limitations are self-imposed by the developer in order to keep the game from taking forever to download. Other times size limits are determined strictly by the storage medium the game ships on. In order to provide sound for a game that averages 25 hours of gameplay, it's likely that many of the sounds will have to loop. Continuously looping sounds are used for objects that exist in 3D worlds such as fountains, refrigerators, air-conditioners, car engines, torches, ambiences, and other sources that continuously emit sound.

Figure 22.17 Adding realistic ambiences gives the audio track a special envelope that gives the player the presence “in” the sound experience.

Immersive, realistic ambiences in video games occur when quality audio tools are used by experienced sound designers. Third- and first-person adventure games generally serve as the best examples of immersive ambient audio. The Skate series, published by Electronic Arts, is a shining example of quality ambient audio in action. As you skate around the world on your board, the sounds of the world around you shift imperceptibly into one another. The sounds that stand out to you as ride around are usually clues to some type of additional interactivity—a secret area you can unlock or an object you can ride on. Another game that has ambiences that are striking and engrossing is Rockstar Game's Red Dead Redemption. When playing this game, and a storm starts to roll in on the plains, and the thunder rumbles soft and low in the distance, you truly feel like a cowboy.

There are three basic ambient implementation techniques that are available in most game engines. The first tool is your basic sound emitter. As you may have guessed from the name, a sound emitter is an unseen object that emits sound.

Speaking of Video Games

Typically, the amount of audio files associated with dialog in a 3D console video game dwarfs those of sound effects and music combined. In recent history, games have made extensive use of seasoned writers and celebrity talent to help hone the storytelling aspects of a video game. Enormously popular games such as Bioware's Mass Effect 2 and Bethesda Games’ Fallout 3 make extensive use of gameplay dialog to support gameplay. Character development and narrative tools have dominated trade magazines and conference lectures as industry professionals search for ways to broaden the appeal of video games. Intertwining a well-written story, performed by talented actors around cleverly designed video game, can be an extraordinarily powerful device in drawing players further into a game experience. When dialog is poorly executed, however, it stands out in a particularly painful and obvious way. Video game history is littered with lines of dialog that are, at their best, uncomfortably awkward, and at their worst, contradictory to their intended meaning. Granted, many of these issues revolve around inaccurate translation, but some of them are just plain bad decision making.

Dialog design can require significant preproduction. If a game is going to use dialog as a device in the gameplay mechanics, then there is much to think about. There will be many questions that need to be answered that have serious repercussions on budget, schedule, and cost. From basic questions such as “What role should dialog play in supporting both the broad and specific game design goals?” to more logistic-based questions like “How many languages do we need to support with audio localization on one disc?,” typically most of the design-related dialog questions are answered by the design team while the logistics questions are left to the audio team, but this isn't always the case. Regardless of which people are tasked with the different aspects of dialog creation, ultimately it's is critical that audio and design both consciously move the effort forward. Dialog on a big-budget console game is a cooperative effort that requires care, consistency, and attention to detail.

Most video game dialog design is done using spreadsheets. Spreadsheets are great because they allow a line of dialog to be contained in a single cell. Once in a cell it's very easy to associate each line with an explicit filename and use the concatenation features of a spreadsheet to automate as much of the process as possible. Spreadsheets are enormously helpful in formatting scripts for talent reading, progress tracking, and script translation. While film and television use word processing applications to deal with script issues, for video games it's all about the functionality of a spreadsheet.

Video game dialog can be divided primarily between traditional linear cinemas (whether prerendered or real time) and gameplay speech. The reason for this division revolves around their two completely different implementation methods. Regardless of what event triggers them, traditional cinemas play back the same way every time, which makes sculpting the dialog pretty straightforward. More often than not, you cannot move your character through the world during one of these scenes, but it is common for the player to be able to move the camera slightly on a fixed point in space, and change the perspective slightly. These linear cinemas remove or limit player control and play out a story vignette (cut-scene) where acting and well-written dialog are critical. Some games rely heavily on these cut-scenes forming the backbone of the story experience and feature little else to support any story-related agenda. Writing, recording, and editing for these set pieces is generally straightforward and mimics that of an animated feature. Some games, however, use dynamic dialog and speech during active gameplay to support the ambience, impart general information, or even as gameplay mechanics itself.

Gameplay speech lines vary from the very generic, which is often repeated many times throughout a game, to the highly specific, which may only be performed once in an entire game. Gameplay speech needs to be grouped into categories based on the circumstances (also called a moment) under which the speech would be triggered. For instance, the moment, “Enemy Soldier Has Line of Sight on Character” may contain lines of dialog such as “There he is!,” “I see him!,” or “The intruder is here!” This moment now has a three-line variation. Depending on how common this game event is, the designers may choose to make this moment 30 or 50 lines deep. It can be a very challenging task to say “There he is!” 50 different ways without them starting to sound ridiculous, but a good game writer can be surprisingly clever. These lines could then be divided up among several actors in order to further the depth of variation. The better lines will likely be read by several characters.

Sports commentary is a classic example of how gameplay speech can play a prominent role in a game. While the player is playing the game the commentary engine is constantly examining the information being produced and deciding what to say. The information is flowing fast, and choosing what to say when is a job that is left up to logic and weighting systems. If a particular moment is associated with qualifying logic such as “Homerun + Lead Change + home team + 9th inning” then the game will see the highly weighted moment “Home team wins the game in the bottom of the 9th” and will randomly choose a line from the corresponding sound event: “I can't believe it, they've done it here in the bottom of the 9th! The crowd is going wild!” A moment like this would trump a more generic moment that may have also had qualifying logic (Homerun + Lead Change) because the designer has weighted this moment much higher and the commentary engine will choose the more highly weighted moment. Moments are designed to highlight exciting moments and cover up more boring ones, generally mimicking exactly how sports broadcasters work. Randomization, weighting, and logic need to be part of any gameplay dialog system in order to make the feature remotely believable.

With video game dialog, it is particularly important to discuss processing paths and reverbs. Unedited source dialog material must be saved and archived. Carefully labeling backup DVD costs a few bucks and an hour of your time, but coming up with a great alternate take without having to pull the talent back in will make your director very happy. Saving dialog at various stages of editing can also be very helpful. I'm a big proponent of saving high-resolution (24-bit/96 kHz) edited, but unprocessed dialog files. These files will save you a lot of time when the director decides to change the monster processing in the last few weeks before submission. Another important tool when dealing with video game dialog is a batch processor. Batch processing means applying a sequenced set of audio processes such as compression, normalization, EQ, pitch-shifting, and more to a large number of sound files in non-real time. Since games are made up of thousands of individual sound files, being able to process dialog files accurately and efficiently is the best way for you to stay on schedule and remain cost-effective. Products such as Sony's Sound Forge Pro 10 (comes with a batch converter) and Audio Ease's BarbaCatch (Mac only) are solid batch converters. Batch converting does not always produce the desired results. Errors can occur on some files and, therefore, every file must be double-checked by listening to it after the process is complete. If possible, this task can be delegated to an intern, but one way or another it must be done.

Musical Notes

Music for games also has to be approached with interactivity in mind. Nobody wants to hear a 30-second music loop, no matter how beautifully orchestrated or performed, go on for a half hour while he or she tries to capture the golden orb of destruction. Poorly designed music can turn a challenging portion of a game into an excruciating trial that must be endured before you're allowed to move on. When a video game has music that has been carefully crafted and implemented, however, the added emotional drama can be deeply powerful as a player. Music, more than any other audio content, can be most effective when it is used carefully and with some restraint. Having music playing constantly throughout a game makes it less effective during the times when it's really needed. Reserving music and then bringing it in to energize an exciting gameplay shift is a simple and powerful technique that is used in games. Music goes through the same processes that sound effects and dialog go through: design, create content, author, implement, and test. What types of music should be used? Heavy metal, baroque, indie, underground, trance, hip-hop, and everything in between is possible. Which type of music is right for this game? If your game has similarities with a feature film or two it's always a good idea to listen to the soundtracks to those films. While you are playing the early builds of your video game, try listening to film scores and any other commercially available music that you think might be a good fit. This is a cheap and easy way to see if what you imagined would work really well, actually does work really well. While the aesthetic direction is being chosen, the technical music design can also be worked on.

A video game's music design mostly deals with the why and how music gets triggered during the game and what makes it change once it's playing. Developing a music design starts with a complete understanding of the game design. Who are the main characters and groups in the game? What are the big-picture goals the player must accomplish to progress? What are the fundamental gameplay processes and how could they be enhanced by music? These questions need to be answered so that the necessary schedules and budgets can be created.

Much is made in the video game industry about music that changes as the player's experience changes. This type of music has been most commonly referred to as “adaptive music,” meaning music that adapts to what is going on in the game. But exactly how much will the music change? On one end of the spectrum is procedural or “generative” music. That is music created by a process or a set of rules that results in a music that is ever-evolving and does not repeat. On the other side of the spectrum is traditional “deterministic” music. This is music that has been written from beginning to end by a composer and comes out the same way every time. Most modern games’ music falls somewhere in between these two concepts but definitely more toward the traditional approach.

One popular technique for adaptive video game music involves mastering the music into a multichannel interleaved audio file that contains several stereo pairs of musical instrument tracks sometimes called stems. These music stems can then be muted or unmuted depending on the game state. The advantage of this technique is that the music layers, being from one interleaved file, never fall out of sync. Another popular technique is to have the composer score one fully orchestrated, dense music piece and then have them produce one or two remixes of that same piece extracting some instruments and down-shifting the energy. Later these stereo tracks can be cross-faded with or without beatmapping, to lower the energy, but still maintain the musical continuity. There are so many possibilities for customizing music to fit a game, it's easy to get carried away. It's important that the music further the design goals without being so complex that it siphons away valuable time, money, or energy.

A music design document doesn't necessarily have to be finalized in order to hire a composer and start the composition process. The composer must be selected carefully, and typically there is some type of demo process involved in vetting a potential composer. Ideally, the chosen composer should have a variety of existing material on his or her demo reel that fits your vision and a demonstrated history of delivering material and maintaining successful relationships (return customers). Master video game composers are capable of writing emotionally charged, nuanced music that doesn't draw too much attention to itself. When a player is getting important tactical information, such as enemy position from weapon fire, or communications from headquarters, you don't want overactive music to obstruct that information. You can always minimize this confusion with mixing techniques, but it's so much better when the problem doesn't exist in the first place!

Mixing Games

Unlike film mixing, video game mixing should start on the day the first sound goes into the game. With a reference level and reference speakers established and agreed to ahead of time, every sound designer who is working on the game is contributing to the mix as he creates the sound content and authors each sound events volume. You cannot wait until everything in the video game is done to start mixing, because everything will never be done. On a big-budget game, there will be tens of thousands of sound files and thousands of sound events. There may also be many mixer snapshots that correspond to various game states. Turning down a sound event to deal with one problem area may cause a problem in a different part of the game. The concept of leaving a video game mix until at the end of a game's production is a setup for disaster. Agreeing to reference standards at the beginning of a game, when everyone is still sane, and even harder, maintaining those standards through to shipping the title, is the single most important step that can be taken in regard to mixing. Does this mean that you should not listen to the game through a wide variety of speakers at a wide variety of volumes? No, you should absolutely listen to the game through all sorts of playback scenarios, and it should sound good, but you should make mix adjustments based on your reference standards just like the art director makes color and lighting adjustments.

Choosing a reference level at which to mix the game is the most critical step toward maintaining consistency throughout the mixing process. Feature films use – 20 dBFS pink noise calibrated to 85 SPL (c-weighting, slow response) as a reference for playback. With a speaker system turned up this loud there is plenty of headroom to rattle the seats of a movie theater during an alien invasion without fear of clipping. In the near-field experience of a living room, however, there is no need or desire for program material with dynamic range that wide. Choosing a reference level is still a controversial issue in the games industry. Should reference be 85 SPL or 78 SPL or something in between? The fact that this debate is occurring is at least a sign the game audio mixing is moving forward. For too many years video game audio directors have been guilty of producing mixes that are loud, distorted, and void of dynamic impact. Many games, games that have won awards for sound, do not seem to use a reference level as a guide for mixing. Some games rely on a final stage of CPU-based compression/limiting to prevent distortion and many others simply distort the outputs of the audio system. Having a CPU-based compressor or limiter running in real time while the game is being played can be great for a highly dynamic game looking to catch the odd scenarios of “multiple bombings during a raging battle.” If these compressors are constantly engaged, however, the mix will not yield a dynamic audio experience, rich with the detail of a feature film. A video game without volume dynamics ends up leaving so much sound potential on the table.

The game team at large will not be listening to the game at the reference volume level. Many team members will only ever listen to the game when it's absolutely necessary, preferring instead to listen to music, radio, or something else that inspires them while they work. This is not because they hate they sound of the game, but listening to the same anything all day everyday for 2 years is not everyone's cup of tea. Most team members are often in cubicles and when they do listen to the game it may be inappropriate for them to listen even at a moderate volume. When a game that is mixed at 78 SPL is monitored at 58 SPL, it sounds weak and small, especially with low-frequency contours. Now add in the complication that most team members will be listening through low-quality TV speakers and you have the recipe that bakes up into an incredibly overcompressed and distorted cake—yuck! A great way to combat this potentially distorted view of the audio mix is to select a basic set of stereo (2.1 minimum, 5.1 preferable) computer speakers that can be used as a baseline standard and then install them on all the team's leadership development systems. There are many 2.1 computer speakers out there that cost around $50 and suffice as a baseline reference speaker. In addition to a baseline system, the audio team should be equipped with professional speakers as well. Having the ability to listen to the game on built-in TV speakers, a baseline 5.1 system, and a professional 5.1 system just by tapping a few buttons will enable the audio team to keep tabs on best-case and worst-case scenarios and optimize for the average consumer. The bottom line here is that a professional video game art director is going to make color and lighting changes based on nothing but a properly calibrated TV or computer display; why would an audio director not do the same? Reasonable audio standards must be agreed to and adopted early on in the process, and the leadership should understand why. The reasoning for audio calibration is the exact same for that as picture calibration: to maintain quality and to avoid chasing windmills.

My TV at work isn't of a very high quality and it is not calibrated. The red is too strong and the contrast is too high, making the picture too dark. When colleagues come into my office for the first time to hear something that I'm working on, they will often comment “Man, what's wrong with that TV?!” It's actually not horribly off, but my colleagues have been trained to view the game through properly calibrated TV monitors and can instantly recognize when a picture is not being accurately represented, and it bothers them. Less often, however, do non-audio personnel think the same way about calibrated speakers. It is still very common to hear comments that there is not enough low boom in the explosion, even when the person is listening through a speaker that is acoustically incapable of producing frequencies below 100 Hz and facing the wall behind the TV. Video games are played on televisions and need to sound good on televisions, but they will never sound great through television speakers, because television speakers are incapable of producing anything close to a broad, flat frequency response. The design intent of a TV speaker is usually to add as little cost as possible but still produce sound. TV speakers are a great convenience, not a device for mixing audio.

In contrast to film, a game sound mixer cannot assume that their audience will be sitting in an acoustically sound movie theater with stadium seating and a properly calibrated speaker system. The computer industry services a wide variety of playback scenarios, from the tiny speakers of a handheld device to a console system feeding a high-end speaker system in a home theater. Playback formats in games also vary greatly. Most console games are capable of producing multichannel surround formats and a stereo mixdown at the same time. It's important to understand that these mixes are created as the player plays the game. The explosion that happened 2 minutes into level 6 did not happen at 9 feet and 14 frames. Although this explosion may have a high probability of occurring, when exactly it will happen (if at all) is unknown. Every time players play the game, they are going to play slightly differently and this makes mixing for games a fantastic challenge.

A film mixer knows the future down to a frame. This fact allows him to anticipate events with great precision, and this is a powerful artistic tool for sound. Think of any horror film made in the past 40 years and the use of prolonged, horribly tense silence that precedes the villain jumping out from the darkness with a deafening scream. Since it is impossible to predict the exact actions a player will take during a video game, the game sound mixer does not have this tool at his disposal. Other than cut scenes and scripted events, there are very few guarantees as to what will happen next in the game. When you play a cut scene or scripted event, you are taking the control away from the player and taking some of the interactivity away, and for this reason video game makers are usually very careful about how much they do this. There are, however, tools that are available and with a little creativity and protocol a good mix isn't difficult.

So you've got your speaker variety and you've chosen your reference level—what's next? The next most important player in the good mix philosophy has to do with the dynamics of the final sound files themselves. As mentioned earlier, not all sounds should have their dynamics compressed and limited away. I have found that it's a lot easier to compress a sound a little more than it is to expand it and make it breathe again. I like the approach of designing game-ready audio files as they appear in the world. What I mean is that if a sound is a quiet sound, like a footstep on concrete, don't compress it; allow its natural dynamics to work for you. If a sound is naturally loud in the world, such as an explosion or double-barrel shotgun, then compression of one sort or another should help shape the sound and keep the energy high. Even if your game is lucky enough to be able to spend CPU cycles on a final limiting process, you will still want to control the high-energy sound material. A real-time limiter on the final audio outputs is best used as a safety net for the chaotic moments where a lot of high-energy sounds can build up (such as a quick sequence of explosions or a massive gunfight). Using a real-time limiter as a constant state of compression flattens the mix does not serve the game.

The next level up is at the sound event. Here atomic sounds can often have their volume values manipulated individually or as the entire sound event they are a part of. Where possible I do subscribe to the method of making atomic sounds a little louder so that I can turn them down a little at this sound event level and give myself some room to change my mind later without having to adjust the volume of the atomic sound itself. After the event level, usually some type of mixer tech is involved.

Mixer tech, at its most basic, allows for the grouping of sounds in such a way that a great number of similar sound events can be turned up or down without having to change the value of every sound event in the game. Sound events are typically associated with these “mix groups” by the sound team in an audio tool. For example, in Mortal Kombat (2011) we had mix groups such as Hits, Footsteps, Special Moves, Dialog, and In-game Music. Having a nice variety of mix groups tailored to these game experiences allowed me to make global volume shifts with very little effort throughout the game's production. Most mixer tech also offers some type of real-time volume manipulation as well. If ducking the music 4 dB every time the sergeant barks his commands makes sense, you can typically use mixer tech to achieve this. Having mixer snapshots is another common technique used in game mixing. By moving between mixer snapshots, the game mixer can manipulate categories of audio to achieve cool effects such as indoor/outdoor transitions, slow-mo moments, or concussion effects. This type of mixer system is prevalent in the video game industry and can also be used as valuable debug tool.

Figure 22.18 A snapshot of the Mixer Tool from Midway's Jon Woo Presents: Stranglehold.

If you are a sound designer working on the ambience of the rain forest level, you can't be hearing the game music running all the time, covering up your work. A mixer snapshot that mutes music but allows environmental audio to continue is extremely helpful here. The same is true for Foley, music, and dialog; you need to be able to check the submixes just as if they are subgroups on a mixing console. Although a high-level mixer is a powerful and essential tool for modern game mixing, mixer tech is only a piece of the puzzle and most straight-up volume balancing adjustments end up being done at the event level where there is more detailed control over specific volume offenders.

There are many more video game techniques that have not been explored in this segment. The insights presented give a surface look at some of the more common challenges inherent in video game sound work.

by Rich Carle, director of audio, NetherRealm Studios

IT IS ALL UP TO YOU

The limitations of your creativity and your dedication to your art form are only limited by your imagination and your personal commitment to quality in whatever craft you choose.

Thank you for your patience and open mindedness in reading both this chapter as well as this entire book.

Now go forth and be creative!

¹Gary R. Edgerton, “The Germans Wore Gray, You Wore Blue,” Journal of Popular Film and Television, 2000; Gary R. Edgerton is a professor and chair of the Communication and Theatre Arts Department at Old Dominion University.