Vertical Layering by Instrument Group

The natural starting point for vertical scoring is to split your layers up based on instrument section. For example, after we have written a looping cue, we could split the cue into four layers for implementation: 1) strings, 2) winds, 3) brass, 4) percussion. These layers could then be brought into middleware and triggered based on “intensity.”

An important point to remember is that with vertical layering the ordering of layers should generally be from least to most dense. There is some ambiguity here depending on the parts you write, but if you start with a very dense layer of percussion and then add a sparse layer of woodwinds it is unlikely to make a significant impact on the mood of the cue. A far greater impact is made if you start with winds and then add the percussion on top, because the percussion will make a noticeable impact. You will be limiting the flexibility of your system if you choose to bring in your densest layers too early. This goes for just about every type of vertical layering system.

Vertical Layering by Timbral Grouping

A very close cousin to layering through instrument sections is layering through the timbral groupings we studied at the end of Chapter 8. Here, instead of splitting based on strings, winds, brass, and percussion, we would split layers based on wind group I, wind group, II, brass I, brass II, etc. This can be slightly more effective if the texture is chordal because it will help with achieving clear, balanced voicings.

Vertical Layering by Mood and Intent

This method is a combination of the first two with the added stipulation of intent. This means that you will split up layers not just based on balance in the orchestra, but on the intended emotional outcome for your player. Here we might make use of one layer that contains low strings and low woodwinds together, achieving a dark and ominous effect. We might make use of high strings and woodwinds as another layer, achieving a soaring and bright effect. The low strings alone are not as dark without the low woodwinds, and the high strings alone are not as bright as they are combined with high woodwinds. For this reason, these pairings are not split up by instruments or timbres, but are nonetheless extremely successful at evoking emotion.

Vertical Layering by Interlocking Motifs

Vertical scoring by interlocking motifs is a common and highly effective implementation technique. It can be heard in games ranging from the Zelda franchise all the way to Nier Automata, so it is important to have it in your toolkit. The idea behind this technique is not so much about the timbral grouping of each vertical layer; instead the emphasis is on how the layers interlock. With interlocking vertical layers, every group of instruments is meant to “lock into place,” as if they were each a piece of the puzzle. Each layer then has a particular space in the overall mix, and every instrument should be audible and clear even when all the layers are triggered. In fact this is an important criteria for all vertical layering systems, but it is taken to the next level with the interlocking method.

In an orchestral setting vertical layers have a tendency to blend together, which is often the intention. As a consequence this method of vertical layering works extremely well with non-traditional instruments as well as orchestral instruments. Synthesizers, samples, processed effects, and orchestral instruments are all fair game with this method so don’t hold back if you’re thinking about using it. Hybrid templates sound fantastic when interlocking in a game soundtrack. It can be compared somewhat to a fugue, or to a heavily polyphonic piece of concert music. Every melody, countermelody, and rhythm contrasts so prominently that their entrances are noticeable. But when all layers are triggered each motif becomes a dot in a pointillist painting.

Adaptive Mixing in Vertical Layers

It is possible to adjust your music mix in real time using vertical layers. This is helpful for settings in which the composition is fixed, but the mood needs to adapt. In other words, you can use this method to change the timbre or instrumentation of your layers, but not the basic elements of the music itself.

One example of this is splitting a cue up into melody, chords, percussion, and background instruments. In a way this can be thought of as vertical layering by function. Here we might have two or three alternate versions of each layer, which differ in instrumentation. For the melody layer we may have a flute layer, a cello layer, and a pizzicato string layer. These alternates contain the exact same melody, but by switching the instruments we can change the mood or tone of the music as a whole. The flute might be bright and cheery, while the cello would be darker and more somber. Pizzicato strings as a melody could be considered quirky or ironic. Based on the actions of the player, your vertical system will fade in and out of the corresponding layers.

Vertical Layering Limitations

An important consideration to make here concerns the emotional flexibility of all vertical scoring methods. Vertical scoring is powerfully adaptive and easy to control, but it has some limitations regarding mood. Because every layer has to work together in various combinations, there is a limit to the textural contrast of each layer. It is not likely that a heavily polyphonic layer with a wild harmonic progression will fit smoothly with three other layers that stay in the same key and operate homophonically. This is where we begin to see a need for horizontal scoring, which absolutely can accommodate this contrast.

What vertical scoring can do well is provide almost infinite possibilities for adaptive layering and mixing within a basic range of emotion. This range can stretch with a bit of creativity, but there is always a limit. It is important to understand vertical scoring as a means of adjusting intensity and density in an adaptive way. If vertical layers are meant to provide a spectrum of emotion from neutral to dark, then the intensity of that darkness is what we are changing by adding or subtracting layers. Equally important to note is that regardless of mood, adding vertical layers always increases the density of the music. Subtracting vertical layers always decreases density. From a compositional perspective this is incredibly important in games because there are many circumstances where increased density is not necessary or even beneficial.

One helpful example of this is with infinite runners where an increase in distance also increases the speed. It is tempting to pick a base layer and then add more layers on top as the player reaches distance milestones. However this isn’t always the best approach. For many infinite runners the key emotion is urgency. If you start a vertical system with a layer at a lower tempo you may never reach the level of urgency that the game calls for. You will however be increasing the density, but as we have learned that is a completely different quality from urgency and the intensity of that urgency. In short, know what vertical scoring is capable of and what it’s not, and then use it to your advantage where appropriate. In the case of our infinite runner, a horizontal system that transitions tempo might be a much better fit.

Vertical Applications in Middleware

Most vertical techniques are accomplished in middleware through use of RTPCs and automation (see “Audio Middleware,” Chapter 8, page 265). In short, RTPCs allow middleware to track values in the game engine and automate aspects of our musical system. Each program has a different method of application for RTPCs. Game parameters in FMOD and game syncs in Wwise (see Figure 9.3) track game data, at which point we can set volume automation to respond. These automation ramps will determine which layers the player is listening to as the game changes. The techniques described above are different ways of splitting musical cues into layers, so that they respond appropriately to the game data (see Figure 8.3, page 268).

An example of how this could be applied to middleware would be a vertical system tied to the number of enemies in a combat scene. First you would create a parameter or game sync (depending on whether you’re using FMOD or Wwise) and call it “number of enemies.” You would then split up a combat track into three layers based on their intensity level (low, medium, and high) and import them into your middleware program as separate assets. Finally, you would draw in volume automation ramps that trigger one layer at a time as the “number of enemies” value increases. We are left with a vertical layering system that increases in intensity as the number of enemies in the scene increases.

Stingers and Transitions

Although we mentioned using stingers (short, linear chunks of music) earlier, we have not yet covered how useful and far-reaching their applications can be. Stingers are usually thought of as brief, catchy melodies that help players identify treasures or valuable pickups. However their true potential can be realized when used in conjunction with other loops. In this way stingers can act as intros and outros, adding purpose and development to the musical structure. They can also fire over the top of a loop, behaving almost like a sound effect to alert players to some item being used, or a special attack being triggered.

Lastly, and most importantly, stingers can be used as transition pieces to connect larger modules of music. This is an important function because it allows composer to write complex and diverse music in disparate styles. These modules can then be attached like Legos using stinger transitions. The best part of this method is that these stinger transitions are pre-composed and linear, so they will sound extremely musical and cohesive moving into or out of another musical cue. If care is taken to write effective transitions, it’s easy to approach the tailored sound of film scores with this technique.

Horizontal Resequencing to Define Musical Structure

In traditional linear music we often take for granted its ability to take us on a journey. But how do we accomplish the same when composing for games if you can’t predict player actions? The answer is by using horizontal scoring methods. To feel satisfying, musical cues need to have a clear structure (see “Form and Structure in Game Music,” Chapter 6, page 184) that corresponds to game states. This means that any cue should have a compelling intro, a number of varying and/or contrasting sections (A, B, C, etc.), and a clear ending. It is possible to accomplish all of this using only horizontal scoring.

Our starting point will be the introduction. This is essentially a stinger which will trigger and move on immediately to the next section. The length will depend on the context of the game, but generally it will be a short (one- to five-second) introduction to a cue. Often these stingers are attention grabbing and fun. The intro needs to successfully alert the player to whatever new game context is about to occur (immediate combat, a potential threat, a mini-game, etc.), and it needs to lead into the main loop in a musically coherent way. Remember that as the introduction, this stinger is really asking a question that the player must answer with her actions. The stinger shouldn’t feel like a musical resolution; instead it has to feel somewhat open ended so that the player feels like there is more to come.

Next comes the loop. This loop serves the sole purpose of setting the mood for this new game context. It’s important that this module (we’ll call it the “A” section) loops because we won’t know how long the player will take to complete her task. On the one hand, if it’s a combat task it could be as quick as a couple seconds. On the other hand, a puzzle could take minutes to solve. The priority here is to compose a loop that exactly fits the mood of the task.

Finally, we need an ending to our horizontal system. After the player completes her task the music needs to quickly arrive at a satisfying conclusion. This is accomplished by composing another stinger. This time the stinger should feel resolved. Our player has defeated her enemies in combat so this outro is her reward. An open-ended ending stinger would feel quite unsatisfying here. In this sense we are now aiming for a period, or an exclamation point, rather than a question mark.

Horizontal Staging

The above type of setup can be made more adaptable in any number of ways. The “beginning–middle–end” structure is just one possibility. Adding other tiers of horizontal staging is a great way to add complexity and adaptivity to a game scene. Take a boss level for example. Bosses usually have different forms that emerge when a certain amount of health points are lost. This usually means the combat will be split into multiple stages. For each stage the musical cue should change correspondingly. Perhaps Stage A is meant to feel easy, but Stages B and C are meant to be insanely difficult and chaotic. In this scenario the horizontal structure may now look like this: Intro–Stage A–Stage B–Stage C–End. In this way each stage can accommodate tempo changes, instrument changes, or basically anything you can imagine that will up the ante. This is noticeably different from vertical methods because we are actually transitioning to an entirely new piece of music, rather than simply increasing density in the same piece of music.

To smooth out the cue as a whole it is also possible to insert transitions that help the piece build organically: Intro–Stage A–Transition–Stage B –Transition–Stage C–End.

It’s possible that each stage shift may be accompanied by a cutscene, in which each transition will be replaced by a linear score to fit the cutscene. In this case each stage might act as its own mini-horizontal system. By this we mean that after each introduction stinger, the stage in itself is a loop, followed by an ending stinger that transitions into another linear cue that synchronizes with a cutscene. A three-stage battle would then look like this:

In these more complex examples it can dramatically help with transitions to quantize horizontal triggers (not to be mistaken with quantizing in a DAW). This means that transitions will only occur 1) after a parameter/game sync meets a predefined value (e.g. Boss reaches < 100 hp), and 2) after a pre-defined number of bars or beats (e.g. after the Boss reaches < 100 hp and the music reaches the next downbeat). This ensures that there are no abrupt cuts or unwanted shifts in rhythm. There are times when immediate shifts may be desirable (as mentioned above), but that is left to the composer’s discretion. Volume envelopes can again smooth over these myriad transitions, as it did in our examples above.

Refer back to Chapter 8 for some specifics on transitions, and be sure to check out the Sound Lab (companion site) at the end of the Chapter 9 for some video tutorials and examples that go into detail on how to set up these systems.

Horizontal Markers

At times it will be necessary to take a longer cue and make it adaptive. For example, a developer might love a particular theme you wrote and want it used in a battle scene. In these scenarios it can be helpful to use markers to define sections of the track. It will be necessary to listen carefully to the track to find points that will loop seamlessly, and designate them with markers. Since the track is linear, it will likely already have a structure to it, but only parts of it will sound cohesive when looping. Drop a marker on these sections and use horizontal transitions to trigger each section of the track as the gameplay dictates. Note that you may have to cut up some sections and move them around in order to ensure seamless looping.

The goal for this technique is to allow adequate flexibility of a track. It should function similarly to any other horizontal scoring method, but will allow you to adapt a linear track to fit the game. Usually the markers are set as mood markers. So Marker A will dictate a subtle exploratory mood, while Marker B might be designated for combat.

Horizontal Applications in Middleware

Most if not all horizontal techniques will be accomplished using sound containers and transitions. As covered in “Audio Middleware” in Chapter 8, sound containers are playlists of music. We can randomize them, weight them differently, loop individual cues within the playlist or the entire playlist itself, and basically perform any operation on them that we want. Transitions are functions in a middleware program that jump from one sound container (or marker) to another based on parameter or game sync values. We can define the threshold ahead of time, and when the value hits that threshold, the transition will occur.

For more resources (including a downloadable version of this session) see the Sound Lab (companion site).

Playlists and Randomization

Complexity and variation can be added to the horizontal and vertical methods by using playlists (see Chapter 8). For example, instead of using one loop (A) in between intro and outro stingers we can instead randomly select from multiple loops (A, B, C, etc.). This will add some interest to the main module of a horizontal system and keep things fresh for the player. We can use playlists for stingers as well. This is especially effective when you want the player to feel excited and/or uncertain as the game state changes. Horror games in particular are great opportunities to employ playlists of stingers. Each stinger can be a new and horrifying way to alert the player of an incoming threat.

To make a horizontal system as random and varied as possible you can also use randomized playlists for the stingers and the loops. This adds some unpredictability to the scene, without changing the mood. This works well for open-world games where players will be logging hundreds of hours. It will get you the most mileage out of your system and keep players from muting their TVs.

In the case of a vertical system, playlists can be used to randomly (or intentionally) vary instrument timbre for a given melody. Rather than having one layer with the violins playing a melody, you can instead use a playlist for that melody layer. The playlist would then contain multiple loops of the same melody performed by violins, a flute, and a vocalist. The audio engine would then alternate between them either randomly or based on game data. The same process works just as well for chords, background parts, and variations for all of the above. In effect you could have an entirely randomized system where the compositional components (melody, harmony, rhythm) of a track remain the same, but the instruments that play them are constantly shifting!

Most middleware programs will allow you to set a play percentage or weight on each item in the playlist (again, see Chapter 8). This allows you to control the amount of randomness in your music system by limiting variability. Having a playlist on each module of music will vastly decrease the chances of your system becoming repetitive or stale upon retriggering.

Figure 9.2 (H) gives us an alternative look at some basic randomization settings for horizontal transitions. In this case, we can actually add a bit of chance to a horizontal trigger by altering the probability settings. For values below 100 percent, the horizontal trigger will sometimes fail to execute.

Variation for Musical Development

From a compositional perspective, it can be very effective to rearrange themes and motifs using different instruments and ensembles. This method can effectively create variation on a small scale by providing a fresh sound for frequently used themes. On a larger scale, it allows further development of your music. A theme can be used and reused many times if the arrangements are sufficiently different. A character theme for example can be arranged for a chamber ensemble during intimate moments, and then rearranged for full orchestra during a battle. In most cases the context of a theme has more of an impact on the player than the theme itself.

As you’re using this method, keep in mind the story arc of the characters and the game itself and try to update your arrangements to match the story. Characters that are descending into villainy can have an arrangement that matches this descent. Likewise, characters that begin the story in a dark situation can also have arrangements which develop into something more hopeful or triumphant.

Lastly, in terms of variation and development, there is no replacement for well-designed linear cues. By recreating a theme with some new elements it is possible to make the original theme feel fresh when it triggers again later on. In many cases these new elements can be adding or changing instruments as mentioned earlier. It is also equally as effective to compose variations in harmony, melody, and any other compositional elements themselves to match the game context.

Combining Vertical Layering and Horizontal Resequencing

A fantastic example of an extreme musical adaptivity is the game Peggle 2.² The composer, Guy Whitmore, used a combination of vertical and horizontal techniques to score the player’s actions very tightly. Peggle 2 is similar to pinball in that players shoot balls onto a two-dimensional map to hit pegs and progress through to the next level. Musically, this game uses many short, vertically scored loops in conjunction with horizontal stingers and transitions to create a sense of development and progression throughout each level.

The system as a whole can be thought of as a series of horizontal stages with added vertical layering. The vertical layers allow for subtle shifts in intensity while the horizontal elements produce more drastic shifts in urgency. These shifts include changes to the orchestration as well as modulations to closely related keys. Modulations are generally avoided in game music due to the immense difficulty of coordinating them, but with Peggle 2 composers can no longer use that as an excuse. The horizontal elements also act as natural climaxes in the music that sync to the animations and provide some structure so that it feels like a tightly scored movie.

On top of all of this, the actual pegs themselves generate musical notes when hit. These notes are tied to the scale that Whitmore uses at each horizontal juncture in the system. The notes proceed in ascending scalar order to relay a sense of anticipation onto the player. The amount of thought and planning that goes into a highly adaptive system like this is monstrous, but the end result unequivocally pays off. Peggle 2 wraps its musical system around minute player actions and gives players an experience as tightly scored as any Disney or Warner Brothers cartoon.

Nested Events: Music

Although Peggle 2 is an example of adaptivity taken to the extreme, the possibilities are far from exhausted. A complex as vertical/horizontal combinations can be, they can be made even more complex through a process called nesting. Nesting events is a process by which entire musical systems can be made modular (see Figure 9.2(G) for an image of a nested event in FMOD). This allows adaptive systems to be reference within another system or master event. This can be quite useful at times, especially if the majority of music in a game scene is simple and one particular aspect of it is highly complex.

One example of this is if an exploration game includes a mini-game on a particular level. In this scenario most of the music may be a simple loop or vertical layers. When the player triggers the mini-game the nested event can spring into action at whatever level of complexity is required. In this case the mini-game music event can be as complicated as desired, but the master event will be clear and organized.

Another more complex example of nesting is what we like to call the 3D approach to implementation. If the Y-axis consists of vertical layers, and the X-axis consists of horizontal modules, then the Z-axis (the third dimension of our music system) could be a nested event. Let’s say, for example, that all of our vertical layers are triggered, making the music as intense as possible. Our system is able to jump around to different horizontal sections as well at a moment’s notice. This is a system we’ve seen in many examples thus far. But in this example we can nest the vertical/horizontal system and use a master event to add more to it. We can add an entirely new nested event, set up with the same functionality as our first vertical/horizontal system, but with entirely new sounds. We can then use a parameter/game sync to crossfade between these two sets (or planes) of events.

The power of nested events is their ability to organize large and complex events into one or more master event. These master events allow for simplified functionality, while the nested events maintain a high level of adaptivity. They also allow a complicated event to be used in more than one game scene, or as a template to be used with other music (as we saw in the 3D approach). In more involved scenarios master events can be comprised of many nested events. The only limitations are the needs of the game and your own imagination!

Branching Music Systems

The term branching refers to music systems where the outcomes split based on previous events. The musical outcomes, or branches, the player would hear would be different depending on how she played the game. With this logic, a linear system (as we have seen before) would look like this: if A then B, if B then C, and so on in a straight line. With a branching system we would get something like that shown in Figure 9.4.

Each of these letters represent possible outcomes, like a musical “choose your own adventure.”

It’s important to note that branching systems are theoretically technique-agnostic. This means that you could device a branching music system with only vertical methods, or only horizontal methods, or a combination of both. You could even devise a system where each cue is entirely linear, but the overarching logic is not. This makes it extremely adaptable to different game genres. In practice, these systems work great for decision-based games, visual novels, and text adventures. Exploration games are another great application for branching systems – you could device a system where the outcomes correlate to different pathways in a maze. In a loose sense, many adaptive scores are branching to a small extent. “Game Over” stingers are an example of one particular branch in a score. Regardless of the application, branching systems are incredible tools for telling a story through music. At the very least they can be helpful in conceptualizing entire trees of narrative logic. At most they can be the foundation of a highly complex and adaptive musical system.

When (and When Not) to Experiment

Before we get into some techniques for composing outside the box, we must determine when to compose outside the box. The truth is that not every game needs hyper-complex adaptive music. And not every game needs the timbral density of a Mahler symphony either. Sometimes the best musical solution for a game is the simplest.

The best way to determine if this is the kind of project that calls for experimentation is to take a hard look at the game design pipeline. Ask yourself if there is room in the order of operations to insert an experiment and possibly fail. If the answer is no, then it is likely an unwanted risk. This is not always a bad thing. Some developers know exactly what they want and are therefore very clear about it. If your communications with a developer look similar to a task list with musical references, then that makes your job clearer and more organized. If you have an idea that will make the game better, float it by your client and be open to whatever response you receive. Also note that the idea should make the game better, not just your music. There is a clear difference there. Sometimes the results overlap, but not always. Be aware of what your goals for experimentation are, and are not.

If your developer has a much looser pipeline, and has given you some flexibility in terms of style, then you will likely have an opportunity to play around with some of your ideas. It is still important here to communicate before sinking exorbitant amounts of time into an experiment, so don’t skip that step. Developers with this attitude are usually working on smaller games, possibly passion projects. They often look to you as an advisor if not audio director, deferring to your judgment on audio matters. This is a great space to try something new and exciting provided it fits the mechanics of the game and does not interfere in gameplay.

It is impossible to provide an exhaustive list of game characteristics which are better suited for this style of composing. The best way to determine if your musical experiments work with the game is to implement it and play, play, play! If you’re having a hard time, ask your client if you can form a small sample group to playtest with the music. Sometimes it takes a non-musician to quickly determine if music helps or hurts the gameplay.

Experimenting with Aesthetics

The first area of experimentation with game music is the overall aesthetics of your soundtrack. The musical aesthetic usually comes down to your instrumentation and how you process those instruments – in other words (if you can remember back to Chapter 6), your sonic palette. Many composers instinctually adhere to traditional or well-trodden palettes. For example, a classically trained composer might write for string quartet or symphonic orchestra more often than not. Or a composer/guitarist might heavily utilize a rock band for most of her projects. This is perfectly fine most of the time, but it can limit the possible projects that you are qualified for. It also can be fun to experiment with your sonic palette by going beyond the norm in terms of instrumentation.

A simple way to create your own experimental ensemble is to combine elements from traditional ones. For example, how would it sound if a string quartet were combined with an electric guitar or other electronic elements? What kind of emotions would you feel when presented with this sonic palette? Is it capable of capturing the mood and emotional content required by the game? Thinking along these lines can yield some very interesting and powerful results.

Another way to experiment with the musical aesthetics of your project is to get creative with the way that you process your instruments. So in this case, instead of recording a string quartet in a concert hall and perfectly balancing the mix, try recording them each individually and adding distortion, delay, and reversed reverb. Or simply load up your string quartet into a granular synthesizer. You may come up with something totally new and eccentric. You can extend the string quartet far beyond the usual range of contexts with this method.

Experimenting with Implementation

Implementation can be a bit tricky to conceptualize as composers, but is an equally beneficial area to experiment in. In order to experiment with implementation we must go beyond the vertical and horizontal systems we’ve described so far. In these scenarios we are taking all possible player actions and selecting fragmented pieces of music to react to the player. This is true even in the most adaptive systems. To go beyond this we must design a system that acts with the player, and makes changes almost simultaneously, rather than waiting for a moment that makes musical sense. We’re now entering into the territory of generative music.

The broad definition of generative music is music that emerges from a system, process, or framework.⁴ In relation to games, generative music can be thought of as a means for music to be triggered at the level of individual notes (or even smaller grains). This can be an extremely powerful method of musical implementation simply because it has the potential to wrap so tightly around player action. Instead of using pre-composed modules of music that we see in the vertical and horizontal systems, the music is obeying a set of rules and being generated on the fly. It can make for a truly immersive player experience.

Generative music has a very rich history, one that pre-dates video games and goes well beyond the scope of this book. To put it simply, there are many, many ways to generate or automate musical systems.⁵ But there are a few of particular interest to us as game composers. In a very loose sense, most video game music is in itself generative because the game acts as a framework and the player’s actions then generate and influence the development of the music. However, when we talk about generative music within a video game we usually are referring to individual notes that are created or triggered by minute player actions, and a structure that is generated in real time. This means that basic musical elements like melody, harmony, counterpoint, tempo, and rhythm are all created the moment a player makes a choice. Even the most extremely adaptive scores are actually just reacting quickly to the player’s actions, while generative music reacts simultaneously. In some ways games that incorporate generative music are really indirect methods of musical creation for the players, since the music that they end up with is not prewritten in any way. Rather it is the collective result of each and every choice that the player makes.

It’s important to remember that games have always taken advantage of what developers call procedural generation. Procedural in the context of computer science means that the content in question has been created via an algorithm. For example, a procedural dungeon is one which is created on the fly as players are exploring, rather than being created piece by piece by artists and level designers. Often these include elements of randomness. So in this dungeon example, a new room may be created randomly, but with a set of rules or logic. In this case, dungeons will always contain the same or similar basic parameters such as room size, number of enemies, or items in order to maintain consistency and coherency in the gameplay. Each of these procedurally created rooms will also always have some kind of boundary marking the outer limits of where the player can move, otherwise they would not be rooms at all. The benefit of using procedural methods over traditional ones saves time and resources and makes processes more efficient. It also makes the experience a bit less predictable for the player. Generative music shares many of these benefits, and can be thought of in much the same way as procedural systems in games.

One iconic example of generative music is Laurie Spiegel’s Music Mouse. Spiegel is one of the original composers who helped pioneer the field of generative music. Music Mouse is not a game per se, it is more of a musical playground. In it, the player’s mouse movements are translated onto a two-dimensional piano grid. One axis triggers chords while the other triggers notes, all in the scale chosen by a player. This app is fun and can yield music that is quite beautiful without taking any time at all to master. Most importantly, it puts the power of musical creation into the hands of the player rather than the composer, which is a key distinction between traditional and generative music. Check out the Sound Lab (companion site) for a link to the game.

Another example of generative music is the game Spore, whose soundtrack was composed by Cliff Martinez, Saul Stokes, and Brian Eno. Eno is a long-time champion and innovator of generative music. This soundtrack is heavily synthesizer-based, and it has a lovely ambient mood throughout. The uniqueness of Spore, however, comes from the generative nature of the music. If a player enters the character-creation mode, she will hear a score that molds itself tightly around her actions. As more details are added to the character (color, texture, or body shape) the music will grow and evolve in timbre and harmony. This highlights a big difference between adaptive music and generative music in games. While adaptive music is capable of reacting to planned events in a musical and emotionally coherent way, generative music is often surprising and unpredictable because players are surprising and unpredictable. This is entirely necessary in the context of Spore because players are given free reign to create whatever character they want. This could be a seven-legged walking breathing lizard alien, and the music must reflect that! We as composers can plan for any event we expect a player to make, but by creating a generative system we allow for possibilities that we do not expect. This can add a great deal of depth and poignancy to a generative soundtrack.

Rich Vreeland’s score for Mini Metro is a classic example of generative music in games. In this game players can choose to play in “creative mode,” and are essentially given a sandbox to make a subway layout. Each choice the player makes yields musical results based on a set of rules. In this way players can create their own dynamic musical systems, and can listen as they change with the visuals. Exploratory games like Mini Metro are perfect for experimentation because they don’t have specific tasks that players must tackle. They are meant to be experiential rather than goal-oriented, so players have the freedom to try new things and observe the results. In this case, the results are usually ambient and beautifully meditative.

Although the examples we’ve looked at so far are mostly synthetic and ambient that does not mean that themes cannot be made through generative methods as well. Daniel Brown (Intelligent Music Systems) created an artificial music intelligence system called Mezzo for his PhD dissertation at UC Santa Cruz (2012).⁶ Mezzo composes game soundtracks in real time, including character themes. The themes, dynamics, and harmonies composed by Mezzo are adaptive, so they change based on the context of the game, just as a pre-composed soundtrack would. Although there are many ways to create generative music, the fact that Mezzo is algorithmic means that it falls squarely within the realm of procedural music.

With the advent of Mezzo and other tools like it some composers are worried that their work and creative input won’t be of value to developers because an algorithm is capable of the same contribution. While we understand these concerns, we would urge composers to embrace these technological advancements rather than denouncing them. It is our firm belief that technological advancements like these can be used to aid the creative process, and will actually create opportunities for composers. Refusal to incorporate (or even to accept the existence of) new technologies and workflows is more likely to hinder our creative output than it is to sustain it. These new technologies may be daunting or threatening at first, but change is inevitable. By making ourselves more aware and flexible in our workflow we can learn to use new technology to create even more innovative soundtracks than might previously have been possible. Ultimately, tools like Mezzo will likely make our soundtracks more malleable than ever before. This can only serve to better support the player experience, which is, of course, the most important aspect of what we do as game composers.

Programming Languages

Programming languages aren’t really tools, but they are absolutely effective ways to experiment with your game music. Knowing a programming language (especially one used commonly in the video game industry like C#) can open many doors with regards to implementation. You will find yourself better able to pinpoint music-related problems down the road and better able to propose creative solutions to them. Knowing how to program in some languages will also allow you to add custom tools to middleware, making the already streamlined implementation process flexible and powerful. You will also have fuller control over how your music sounds in its final form, which is an indispensable advantage. In Chapter 8 we offer additional information on scripting for games.

Machine Learning

Machine learning is a vast and technical field. However, it’s potential impact on game audio is equally vast and very exciting. Daniel Brown (Intelligent Music Systems) has essentially built a percussion “procedural-izer.” This tool uses machine learning to take in pre-composed tracks, and outputs new tracks on the fly in the same style. It integrates to middleware programs like FMOD and Wwise, and has been used to great effect in Rise of the Tomb Raider. This is a huge first step toward what we hope will be a much more widely used technology in game audio.

Audio for New Realities: VR

New realities are upon us, and Virtual Reality (VR) is a great example of a tool that allows us to experiment with music and sound. Just as Stephan Schütze mentions in his book,⁷ audio for VR is sometimes referred to as “The Wild West” because no one person at this point in time has all the answers, and there are few if any “best practices methods.” But this doesn’t mean we can’t find our own way. Just as game audio took some workflows and techniques from film sound, we can take methods from game sound and bring them into the world of VR, along with some new tricks which we will inevitably come across as we experiment.