4

Music Systems

Introduction

Music is one our most powerful tools to affect how players feel in games. The systems described in this chapter will give you an understanding of some of the key techniques involved in implementing music in games. You will be able to try out your music in these systems and to prototype new ones. It is unlikely that many of these systems would operate in exactly the same way in an actual game engine because you would have a dedicated programmer to implement them at a lower level. However, an understanding of how the systems and prototypes operate will enable you to describe to the programmer precisely the effect you are after. By looking at the systems you have built in Kismet, your programmer will be able to replicate them a lot more efficiently. We’ll focus here on techniques to get your music working and sounding good in a game. Although music is typically streamed from disk, it is not immune to the concerns over memory that we discussed in Chapter 2. See the section titled “Streaming” in Appendix B for some specific advice on memory management for music.

As you’re reading this section, you’re obviously interested in writing or implementing music in games, so we’re going to assume some musical knowledge. If you don’t know how to write music then we’d suggest you put this book down, and go and learn about pitch, rhythm, harmony, timbre, texture, instrumentation, and how to structure all these things into a successful musical whole.

You’re back. Well done! Now let’s move on.

If you want to write music for games, you need to learn to write music … for … games. It’s not a shortcut to that film music job you always wanted. You need to play, understand, and love games and, yes, if you skipped straight to this chapter in the hope of a shortcut, you will have to go back and read the earlier chapters properly to understand the implementation ideas ahead. Your music works alongside, and hopefully with, the sound in games, so you need to understand both.

If you take a flick through the following pages and are thinking “but I’m a composer, not a programmer,” then we can understand that initial reaction. However, in games audio you cannot separate the creation from the implementation. If you want your music to sound the best, and be the most effective that it can be, then you need to engage with the systems that govern it. As you will no doubt recognize from the discussions that follow, the production and recording methods involved in producing music for games share a great deal with those used for film, but there are some important differences. As these are beyond the remit of this book, we’ll provide some links to further reading on this topic in the bibliography.

Don’t worry. Writing good music is possibly one of the most complicated things you can do. If you’ve mastered that task, then this part is easy.

Styles and Game Genres

Certain genres of games have become closely associated with particular styles of music. That’s not to say that these traditions can’t be challenged, but you should acknowledge that players may come to your game with preexisting expectations about the style of music they will experience. This is equally true of the producers and designers who may be commissioning the music. The following list of game categories is intended to be a broad overview.

Social/Casual/Puzzle

The music for this type of game is often repetitive. Perhaps there is something in the mesmeric quality of these games that suits the timelessness or suspension of time implied by a repeatedly looping music track. The more pragmatic answer is that by nature many of these games are often fairly undemanding in terms of processing so that they can be accessible to older platforms or work on the web. They will also often be played without the audio actually switched on (in the office), so music is not thought to be a great consideration and typically receives little investment. This leads to simple repetitive loops, which are the easiest to process (and the cheapest music to make).

Platformer/Arcade

In the public’s imagination, it is this genre that people still associate most closely with the idea of “game music.” The synthesized and often repetitive music of early games has a retro appeal that makes music in this genre really a case of its own. What might be considered repetitive level music within other genres is still alive and well here, the iconic feedback sounds of pickups and powerups often blur the line between “music” sounds and “gameplay” sounds, and it is not unusual to find an extremely reactive score that speeds up or suddenly switches depending on game events.

Driving/Simulation/Sports

Driving is closely associated with playing back your own music or listening to the radio, so it is not surprising that the racing genre remains dominated by the use of the licensed popular music sound track. In the few cases where an interactive score has been attempted, it has usually been in games that have extended gameplay features such as pursuit. In a straightforward driving game where race position is the only real goal, perhaps the repetitive nature of the action and limited emotional range of the gameplay would make any attempts at an interactive score too repetitive. Simulations typically do not have music (apart from in the menu), and sports-based games aim to closely represent the real-life experience of a sporting event, which of course does not tend to have interactive music, apart from perhaps some responsive jingles. On the rare occasions where music is used, it tends again to be the licensed “radio” model consisting of a carefully chosen selection of tracks that are representative of the culture or community of the sport.

Strategy (Real Time and Non)/Role-Playing Games/Massively Multiplayer Online Games

The music in this genre is often important in defining both the location and the culture of the player’s immediate surroundings (see “Roles and functions of music in games,” presented later) although it often also responds to gameplay action. Attempting to provide music for an experience that can last an indefinite amount of time, together with the complexities of simultaneously representing multiple characters through music (see the discussions of the “Leitmotif,” below) means that this genre is potentially one of the most challenging for the interactive composer.

Adventure/Action/Shooter

Probably the most cinematic of game genres, these games aim to use music to support the emotions of the game’s narrative. Again this presents a great challenge to the composer, who is expected to match the expectations of a Hollywood-type score within a nonlinear medium where the situation can change at any time given the choices of the player.

Roles and Functions of Music in Games

There are usually two approaches to including music in a game. The first is to have music that appears to come from inside the reality of the game world. In other words, it is played by, or through, objects or characters in the game. This could be via a radio, a television, a tannoy, or through characters playing actual instruments or singing in the game. This type of music is commonly referred to as “diegetic” music. Non-diegetic music, on the other hand, is the kind with which you’ll be very familiar from the film world. This would be when the sound of a huge symphony orchestra rises up to accompany the heartfelt conversation between two people in an elevator. The orchestra is not in the elevator with them; the music is not heard by the characters in the world but instead sits outside of that “diegesis” or narrative space. (The situation of the game player does confuse these definitions somewhat as the player is simultaneously both the character in the world and sitting outside of it, but we’ll leave that particular debate for another time, perhaps another book.)

Whether you choose to use diegetic or non-diegetic music for your game, and the particular approach to using music you choose, will of course be subject to the specific needs of your game. Here we will discuss a range of roles and functions that music can perform in order to inform your choices.

You’re So Special: Music as a Signifier

Apart from anything else, and irrespective of its musical content, simply having music present is usually an indication that what you are currently experiencing is somehow special or different from other sections of the game that do not have music. If you watch a typical movie, you’ll notice that music is only there during specific, emotionally important sections; it’s not usually wallpapered all over this film (with a few notable exceptions). The fact that these occasions have music marks them as being different, indicating a special emotional significance. In games, music often delineates special events by signifying the starting, doing, and completion of tasks or sections of the game. By using music sparsely throughout your game, you can improve its impact and effectiveness in highlighting certain situations. If music is present all the time, then it will have less impact than if it only appears for certain occasions.

This Town, is Coming Like a Ghost Town: Cultural Connotations of Instrumentation and Style

Before you write the first note, it’s worth considering the impact of the palette of instruments you are going to choose. Although the recent trend has been for games to ape the Hollywood propensity for using large orchestral forces, it’s worth considering what alternative palettes can bring. The orchestral score has the advantage that it is, to some extent, meeting expectations within certain genres. It’s a sound most people are familiar with, so it will probably produce less extreme responses (i.e., not many people are going to hate it). Of course, the melodies, harmonies and rhythms you use are also essential elements of style, but instrumentation remains the most immediately evocative. By choosing to use a different set of instruments, you can effectively conjure up a certain time period or geographical area (e.g., a Harpsichord for the 17th century, Bagpipes for Scotland). If you think of some of the most iconic film music of the previous century (leaving John Williams to one side for a moment), there is often a distinct set of instruments and timbres that has come to influence other composers and define certain genres (think of Ennio Morricone’s “spaghetti westerns” or of Vangelis’ synthesized score for Blade Runner).

In addition to conjuring up a specific time and place, these instruments—and, of course, the musical language—also bring significant cultural baggage and hence symbolic meaning. Different styles of music are often closely associated with different cultural groups, and by using this music or instrumentation you can evoke the emotions or culture ascribed to these groups. This is particularly effective when using diegetic music (e.g., from a radio), as you are effectively showing the kind of music that these people listen to and therefore defining the kind of people they are.

You Don’t Have to Tell Me: Music as an Informational Device

Sometimes it is appropriate to make explicit the link between the music and the game variables so that the music acts as an informational device. Music often provides information that nothing else does; therefore, you need it there in order to play the game effectively. The music could act as a warning to the player to indicate when an enemy approaches or when the player has been spotted. The details and consequences of linking music to the artificial intelligence (AI) in such a way must be carefully considered. Depending on your intentions you may not want the music to react simply when an enemy is within a given distance, as this may not take orientation into account. Do you want the music to start even when you can’t see the enemy? Do you want it to start even when the enemy can’t see you? Again, your answer will depend on your particular intentions and what is effective within the context of your game.

He’s a Killer: Music to Reveal Character

One of the most powerful devices within film music is the “Leitmotif.” Originally a technique associated with opera, it assigns specific musical motifs to specific characters. For example, the princess may have a fluttery flute melody, and the evil prince’s motif may be a slow dissonant phrase played low on the double bass. The occurrence of the motif can, of course, inform the player of a particular character’s presence, but additionally the manipulation of the musical material of the motif can reveal much about both the nature of the character and what the character is feeling at any given time. You may hear the princess’s theme in a lush romantic string arrangement to tell you she’s in love, in a fast staccato string arrangement to tell you that she’s in a hurry, or with Mexican-type harmonies and instrumentation to tell you that she’s desperately craving a Burrito.

You might choose to have explicit themes for particular characters, for broad types (goodies/baddies), or for particular races or social groups. What you do musically with these themes can effectively provide an emotional or psychological subtext without the need for all those clumsy things called “words.”

In Shock: Music for Surprise Effect

A sudden and unexpected musical stab is the staple of many horror movies. One of the most effective ways of playing with people’s expectations is to set up a system that’s a combination of the informational and character-based ideas described earlier. In this scenario, the music predictably provides a warning of the approach of a particular dangerous entity by playing that character’s motif. By setting up and continually re-establishing this connection between music and character, you set up a powerful expectation. You can then play with these expectations to induce tension or uncertainty in the listener. Perhaps on one occasion you play the music, but the character does not actually appear, on another you again override this system in order to provoke shock or surprise by having the danger suddenly appear, but this time without the warning that the player has been conditioned to expect.

Hey Mickey, You’re So Fine: Playing the Mood, Playing the Action

With a few notable exceptions, music in games tends toward a very literal moment-by-moment playing of the action and mood of the game, rarely using the kind of juxtaposition or counterpoint to the visuals you see in films. When music is used to closely highlight or reinforce the physical action on the screen, it is sometimes referred to disparagingly as “Mickey Mousing.” This derives, of course, from the style of music in many cartoons which imitates the actions very precisely in a way that acts almost as a sound effect, with little in the way of a larger form of musical development. You can do this in a linear cartoon as the action occurs over a fixed and known time period; this way, you can plan your music carefully so that it is in the correct tempo to catch many of the visual hits. Although game music is rarely so closely tied to the physical movement or action taking place (partly because of the difficulty in doing so), it often responds to communicate the game situation in terms of shifting to different levels of intensity or mood depending on gameplay variables such as the presence of enemies, the number of enemies within a given distance of the player, player health, enemy health, weapon condition, amount of ammo, distance from save point or target location, or pickups/powerups. Along with these reactive changes in intensity there might also be “stingers,” short musical motifs that play over the top of the general musical flow and specifically pick out and highlight game events.

Punish Me with Kisses: Music as Commentary, Punishment, or Reward

Related to playing the action of a game, music can also fulfill an additional function of acting as a punishment or reward to the player. You might use music to comment on actions that the player has just undertaken in a positive or negative way, to encourage the player to do something, or to punish the player for doing the wrong thing.

It’s a Fine Line between Love and Hate

The way in which music and narrative interact could be seen as a continuum between Mickey Mousing at one end and actively playing against the apparent emotion of the scene at the other:

Hitting the action (Mickey Mousing)
Synchronising musical gestures with the physical action on the screen

Playing the emotion (Empathetic Music)
The music works in empathy with the emotions of the character or characters on the screen

Playing the meta-emotion (Narrative Music)
Here music might not directly reflect the moment to moment emotional state of the characters but instead might play a more over-arching emotion for the entire scene or indicate some larger scale narrative point

Counterpoint (Anempathetic Music)
This music will actually play against or be indifferent to the emotions that appear visually. The juxtaposition of happy upbeat music with a tragic scene for example.

Choosing the degree to which music is going to respond to the game action or narrative emotion is a difficult decision. If the music is too reactive, players can feel that they are “playing” the music rather than the game, and this can break their immersion in the game world. If there is too little reaction, then we can miss opportunities to heighten the player’s emotion through music.

Together with the game designer, you need to make emotional sense of the combination of the games variables. Music is effective in implying levels of tension and relaxation, conflict and resolution. Choosing whether to score the blow-by-blow action or the larger emotional sweep of a scene will depend on your aims. You can avoid the relationship between the game system and the music becoming too obvious by sometimes choosing not to respond, as this will reduce the predictability. However, sometimes you may want a consistent response because the music may be serving a more informational role.

Here are some questions to think about when deciding on how to score the action or mood:

•  Are you scoring what is happening in terms of highlighting specific actions?

•  Are you scoring what is happening in terms of the underlying mood or emotion you want to convey?

•  Are you scoring what is happening now, what has happened, or what is about to happen?

Making the Right Choice

Bearing in mind some of the possible functions of music just described, you and your team need to decide upon what you aim to achieve with the music and then implement an appropriate mechanism to deliver this aim. Let’s be clear, it is perfectly possible to implement music into games effectively using simple playback mechanisms. It depends on what is appropriate for your particular situation. Many people are totally happy with simple linear music that switches clumsily from one piece to another. Look at the time and resources you have available together with what the game actually needs. Some circumstances will be most effective with linear music, and others may need a more interactive approach. Your experience with the prototypes outlined here should help you to make the right choice.

Music Concepting/Spotting/Prototyping/Testing

It is not always meaningful to separate out these process into these stages because the implementation can (and should) often affect the stylistic considerations, but we will attempt to do so for the purposes of illustration.

Concepting

It’s quite likely that you will have been chosen for a composition role because of music you have written in the past or a particular piece in your show reel. Although this may seem like an established starting point, before you spend lots of time actually writing music consider putting a set of reference tracks together to establish the kind of aesthetic or tone your producer is after. This may differ from what you thought. Words are not accurate ways of describing music, especially to most people who lack the appropriate vocabulary, so some actual musical examples can often save a lot of confusion.

Editing together a bunch of music to some graphics or illustrations of the game can be an effective shortcut to establishing the style that the producers are after. You should also discuss other games (including previous games in the series) and movies as reference points. Certain genres of Hollywood film composition have become so established that they are a kind of shorthand for particular emotions (think of John Williams’ mysterious Ark music from Raiders of the Lost Ark, for example). There are many other approaches that could work; so rather than just pastiche these musical archetypes, why don’t you try to bring something new to the table that the producers might not have thought of? (If they hate it, that’s fine, as you will have also prepared another version in the more predictable style.)

Spotting

Spotting is a term taken from the film industry that describes what happens when a director and composer sit down and watch the film together to establish where the music is going to be. During this discussion they not only decide on start points and end points but also discuss the nature of the music in detail, the emotional arc of the scene, and its purpose. Unfortunately, when writing music for games you are often working on an imaginary situation with possibly only a text-based game design document to go on. Along with the producer or designers, you should discuss the roles you want music to play, the desired emotion or goal of the scenario, and how it might reflect the conflict and resolutions within the gameplay or the tempo of the action. You should establish particular events that require music, start points, end points, and how the music will integrate with different gameplay variables such as player health, enemy health, and so on. It’s important to have access to the right data so that you are able to discuss how to translate that data into emotional, and therefore musical, meaning.

During this process you should try to get involved as early as possible to act as a convincing advocate for music. Importantly, this also involves, where appropriate, being an advocate for no music!

Prototyping

As part of the concepting process, it’s important that you try not to be simply reactive, just supplying the assets as requested. To develop game music, you should be an advocate for its use. This means putting your time and effort where your mouth is and demonstrating your ideas.

It’s a continuing irony that decisions on interactive music are often made without it actually being experienced in an interactive context. Often music will be sent to the designers or producers and they will listen to it as “music” and make decisions on whether or not they like it. However, music for games is not, and should not be, intended to be listened to as a piece of concert music. Its primary goal should be to make a better game. Therefore, it should be heard in the interactive context within which it will be used. (This is not to say that there are not many pieces that are highly successful as concert music, just that the primary purpose of game music should be to be game music.) This problem is exacerbated by the fact that people’s musical tastes are largely formed by their exposure to linear music in films (and mostly of the “Hollywood” model).

Some people continue to argue that it is not necessary for composers to write music with the specifics of gameplay in mind, that they should simply write music in the way they always have, and then the game company’s little army of hacks will come along to cut and splice the music for interactive use. In some cases this process can be perfectly adequate, but there should be opportunities for the more intelligent composer to face the challenge more directly.

Although it is possible to prototype and demonstrate your music systems in some of the excellent middleware audio solutions (such as FMOD by Firelight Technologies and Wwise by AudioKinetic), an in-game example is always going to be more meaningful than having to say, “Imagine this fader is the player’s health, this one is the enemy’s proximity, oh, and this button is for when the player picks up an RBFG.”

Either by developing example systems on your own, based on the examples that follow, or by working directly with a game designer, the earlier you can demonstrate how music can contribute to the game, the more integral it will be considered. Remember, the aim is to not only make the use of music better but by extension, and more importantly, to make the game better.

Testing

The importance of testing and iteration can never be underestimated in any game design process and music is no exception. As you address these issues, keep in mind that the degree of reactiveness or interactivity the music should have can only be established through thorough testing and feedback. Obviously you will spend a significant amount of time familiarizing yourself with the game to test different approaches yourself, but it is also valuable to get a number of different people’s views. You’d be amazed how many QA departments (the quality assurance department, whose job it is to test the games) do not supply their testers with decent quality headphones. Of those actually wearing headphones, you’d also be surprised at the number who are actually listening to their friend’s latest breakbeat-two-step-gabba-trance-grime-crossover track, rather than the actual game.

In addition to determining whether people find that Bouzouki intro you love a bit irritating when they have to restart that checkpoint for 300th time, you may also discover that users play games in very different ways. Here are a few typical player approaches to “Imaginary Game”: Level 12c:

1.  The Creeper. This player uses stealth and shadows to cunningly achieve the level’s objectives without being detected at any point (apart from just before a rather satisfying neck snap).

2.  The Blazer. This player kicks open the door, a pistol on each hip, and doesn’t stop running or shooting for the next three minutes, which is how long it will take this player to finish the level compared to the creeper’s 15 minutes.

3.  The Completist. This player is not satisfied until the dimmest crevice of every corner of the level has been fully explored. This player will creep when necessary or dive in shooting for some variation but will also break every crate, covet every pickup, read every self-referential poster and magazine cover, and suck up every drop of satisfaction from level 12c before moving on.

You should also note the positions of any save points, where and how many times people restart a level. Do you want your music to repeat exactly the same each time? By watching these different players with their different approaches, you may find that a one-size-fits-all solution to interactive music does not work and perhaps you need to consider monitoring the playing style over time and adapting your system to suit it.

The Key Challenges of Game Music

Once you’ve decided on the main role, or roles, you want the music to play in your game, two fundamental questions remain for implementing music in games, which will be the concerns of the rest of this chapter:

1. The transitions question: If the music is going to react to gameplay, then how are you going to transition between different pieces of music, when game events might not happen to coincide with an appropriate point in the music? This does not necessarily mean transitioning from one piece of music to another, but from one musical state to another. The majority of issues arising from using music effectively in games arise out of the conflict between a game’s interactive nature, where events can happen at any given moment, and the time-based nature of music, which relies on predetermined lengths in order to remain musically coherent.

2. The variation question: How can we possibly write and store enough music? In a game you typically hear music for a much longer period than in a film. Games may last anything from ten hours up to hundreds of hours. Given huge resources it may be possible to write that amount of music, but how would you store this amount of music within the disk space available?

The only way to decide what works is to try the music out in the context to which it belongs—in an actual game.

Source Music/Diegetic Music

400 Radio Days

Outside of, and on the approach to, Room 400 you can hear two sources of diegetic music—that is, music that originates from “real” objects in the game world, in this case a Tannoy and a Radio. This kind of music can be used to define the time period, location, or culture in which we hear it. For example, if you heard gangsta rap on the radio it would give you the distinct impression that you weren’t walking into a kindergarten classroom!

As well as defining the place where you hear it, you can use sources of music in your games to draw the player toward, or away from, certain areas. The sound of a radio or other source playing would usually indicate that there are some people nearby. Depending on your intentions, this could serve as a warning, or players may also be drawn toward the source of the music out of curiosity.

You can interact with this radio to change channels by pressing E. This plays a short burst of static before settling on the next station. If you so wish, you can also shoot it. You did already, didn’t you?

When you enter the outer corridor, a [Trigger] goes through the [Switch] to start the first [PlaySound], which holds our first music [SoundCue]. Notice that the target for the [PlaySound] object is the [StaticMesh] of the radio itself so that the sound appears to come from the radio rather than directly into the player’s ears. This spatialization will only work with mono sound files. As explained in Chapter 3, stereo sound files do not spatialize. (If you really want the impression of a diegetic stereo source that appears spatialized in game, then you’ll have to split your channels and play them back simultaneously through “left” and “right” point sources in the game.)

When the [Trigger] surrounding the radio is used, it then switches a new track on (with a [PlaySound] in between for the static crackles) and switches all the others off. In the [Switch]’s properties it is set to loop, and so after switch output 4 has been used it will return to switch output 1. Finally there is a [TriggerVolume] when leaving the room that switches them all off so that this music doesn’t carry over into the next room as well.

The [Gate] is there so that if or when the player decides to shoot the radio the Touch messages no longer get through. The [Take Damage] event destroys the radio [InterpActor], fires off an explosion [Emitter] and sound, and turns off all the [PlaySound]s. (For more on [Take Damage] events, see Appendix C: UDK Tips).

Linear Music and Looping Music

Composing and implementing music for trailers or cutscenes is not unique to games audio. Because they last a specific length of time, you can use the same approach you would use to write music for a film or animation. We’re interested here in the specific challenges that music for games throws up, so we’re not going to dwell on linear music. Instead we’re going to work on the presumption that you want your music to react (and perhaps interact) with gameplay in a smooth, musical way.

Perhaps an exception in which linear audio starts to become slightly interactive would be in so-called looping “lobby” music that might accompany menu screens. It’s “interactive”, as we can stop it to enter or exit the actual game at any time. This is a source of many heinous crimes in games audio. Producers that will lavish time and effort on and polishing every other aspect of a game will think nothing of brutally cutting off any music that’s playing during these menu scenes (and Pause menus). We understand that if players want to start (or exit) the game, then they want an immediate response. It would be possible to come up with more musical solutions to this but the least your music deserves is a short fade out.

Using the Fade Out functionality of the [PlaySound] object in Kismet would at least improve some of the clumsy attempts to deal with this issue.

If you don’t respect the music in your game, it shows that you don’t respect your game. After all, the music is part of your game, isn’t it? Don’t cut off music for menus or do clumsy fades—it’s just not classy!

Avoid the Problem: Timed Event Music

Often the most effective, and least complex, way of using music in games is to use it only for particular scenarios or events that last a set amount of time. This way you can use a linear piece of music that builds naturally and musically to a climax. Although it appears that the player is in control, the scenario is set to last for an exact amount of time, whether the player achieves the objective or not.

Linear music can be very effective if the game sequence is suitably linear as well. An event that lasts a specific amount of time before it is either completed or the game ends allows you to build a linear piece to fit perfectly.

401 Timed Event

On entering this room you are told that the “Flux Capacitor” barring entry to the next room is dangerously unstable. You’ve got to pick up six energy crystals and get them into the Flux Capacitor before she blows. If you manage it, you’ll be able to get to the next room. If you don’t…

Around the room are a number of [InterpActor]s. Each one has a [Trigger] around it so it will sense when it is being “picked up.” As you enter the room, a [Trigger] Touch object starts the music track and a series of [Delay]s. The total value of all the [Delay]s is two minutes precisely. After the first delay time, one of the [InterpActor]s is toggled to Unhide by a [Toggle Hidden] object. At the same time, a [Gate] object is opened to allow the [Trigger] that surrounds this [InterpActor] to be utilized. This can then play the pickup sound and destroy the [InterpActor]. When the player then runs over to deposit this into the flux capacitor, a [Trigger] around this object is allowed through an additional [Gate] to play the [Announcement], telling the player how many are remaining.

Each time the pickup is “deposited,” the [Play Announcement] outputs to an [Int Counter]. As we saw earlier, this object increments its value A by a set amount (in this case 1) every time it receives an input. It then compares value A to value B (in this case set to 5). If value A does not equal value B, then the [Gate] that leads to the explosion event is left open. As soon as value A does equal value B (in other words when all five of the energy crystals have been picked up and deposited into the flux capacitor), the A==B output is triggered. This closes the explosion [Gate] and opens the triumph [Gate] so that when the music has finished, its output goes to the appropriate place. This also destroys the [DynamicBlockingVolume] that so far has been blocking our exit.

If you can still read this through your tears of bewilderment, the good news is that in terms of the actual music part of this system, it is probably the simplest you can imagine. When the events are started by the initial [Trigger] touch, this goes to a [PlaySound] object, which plays the music [SoundCue].

The [SoundCue] targets the [Player] so it is played back to the player’s ears and does not pan around. This is the most appropriate approach from music that is non-diegetic (i.e., music that does not originate from a source object within the game world).

Avoid the Problem: Non-Time-Based Approaches Using Ambiguous Music

402 Fatal Void

In this room there are three “bad things.” Each of these bad things carries a “scare” rating of 1. So if two overlap where you’re standing, then you have a scare rating of 2. If you are in the area where all three overlap, then things are looking bad with a scare rating of 3.

These scare ratings are associated with music of a low, medium, and high intensity. The music itself is fairly static within its own intensity. As you listen, notice that no real rhythmic pulse has been established, nor is any particular chord or tonality used. This makes it less jarring when the music crossfades between the three levels. (This is our homage to the music of a well-known game. If you haven’t guessed already, see the bibliography.)

On the level start a [Trigger_Dynamic] (Actor Classes/Trigger/Trigger_Dynamic) and a [Particle Emitter] are attached to the two moving lights using an [Attach to Actor] object. A [TriggerVolume] on entry to the room starts the movement of the two colored lights, which is controlled in the usual way by a [Matinee] object. The [Trigger_Dynamic]s are movable triggers, which will now follow the movement of the lights. As the lights are higher up and we want the triggers to be on the floor, we have offset the triggers from the lights by 400 units in the vertical plane.

Attach to Actor properties.

When any of these triggers (including the central light area, which does not move) are touched, it adds to a central variable; when it is untouched, it subtracts from it. This central variable therefore represents our “scare” factor. Three [Compare] objects compare this variable to the values 3, 2, or 1 and then play the corresponding sound file. If the “scare” variable is the same as their compare value, they play their sound; if it differs, then they stop their [Playsound]. Each [PlaySound] object has a Fade in time of two seconds and a Fade out time of five seconds to smooth the transitions between the different levels of intensity.

The use of music that is ambiguous in its tonality and rhythmic structure can make the transitions necessary in interactive games appear less clumsy.

Avoid the Problem: Stingers

403 Music Stingers

A stinger is a short musical cue that is played over the top of the existing music. This might be useful to highlight a particular game event without instigating a large-scale change in the music. Two bots are patrolling this area. If you manage to shoot either one, then you are rewarded with a musical “stinger.”

When you enter the room, the [TriggerVolume] begins the background music via a [PlaySound] and also triggers the bots to start patrolling.

The bots that are spawned can send out a signal in Kismet when they [Take Damage], and this is used to trigger the musical stinger over the top via another [PlaySound]. (See the Spawning and Controlling Bots section of Appendix C: UDK Tips).

The stinger [SoundCue] is made up of five brass stabs that are selected at random.

This example works pretty well because the background music is purely rhythmic. This means that whatever melody or harmony the stinger plays over the top will not clash with the music already playing. If you want to put stingers over music that fits more rhythmically or harmonically, then we need to extend the system’s complexity to take into account of where we are in the music when the stinger is played.

403a Rhythmically Aware Stingers

With certain types of musical material, it could be that the stingers would work even better if they waited until an appropriately musical point to play (i.e., until the next available beat). To do this you would need to set up a delay system that is aligned to the speed of the beats and then only allow the stinger message to go through at the musical point.

At [Level Loaded] we make a simple calculation to work out the length of one beat in seconds. A minute (60 seconds) is divided by the tempo (the number of beats in a minute) to give us the length of a beat. As this is unlikely to be a whole number (and as a [Delay] object only accepts floats anyway) we then need to convert this integer to a float before passing it as a [Named Variable] to a [Delay] object.

The [Delay] will now send out a pulse for every beat. This goes into the gate to go through and trigger the stinger (via the [PlaySound]) on the beat. It is only allowed to go through and do this, however, when the bot’s [Take Damage] event opens the [Gate]. Having taken damage, the [Gate] opens, the stinger plays on the beat (triggered by the [Delay]), and the [Gate] then closes itself.

There is also an additional system at work just to make sure that the looping background musical bed remains in time with the beat pulse. Instead of using a looping [SoundCue] for the musical bed, we are retriggering it. An [Int Counter] is set to increment by 1 every time it receives a beat from the [Delay] object. When this reaches 16 (the length in beats of this particular [SoundCue]), the music is retriggered. This helps avoid any slight discrepancies in the timing that may arise from using a looping sound.

403b Harmonically and Rhythmically Aware Stingers

Percussive-based stingers are easiest to use because they will fit with the music irrespective of the musical harmony at the time. Using more melodic or pitch-based stingers can be problematic, as they may not always be appropriate to the musical harmony that’s currently playing. For example, a stinger based around the notes C and E would sound fine if it happened to occur over the first two bars of this musical sequence, but it would sound dissonant against the last two bars.

To make this work, we would need to write a specific stinger for each specific harmony that was in the piece. Then if we can track where we are in the music, we can tell the system to play the appropriate stinger for the chord that’s currently playing.

As the system is getting more complex, [Activate Remote Event] actions are used to transmit both the beat pulse and the measure event that retriggers the bed to make it loop. Passing events like this can help to stop your Kismet patches from becoming too cluttered. (See Appendix C: UDK Tips.)

Although our rhythmic bed remains 16 beats long, now we have an additional harmonic element provided by some strings. This is in two halves of 8 beats each. To monitor which half is playing, to therefore choose the appropriate stinger, another [Int Counter] is used. This counts to 8 and is then reset. Each time it reaches 8, it equals the B variable and therefore outputs to the looping [Switch] to send either a value 1 or 2 to a named variable “Bar no.”

A [Compare Int] object is used to look at these numbers and then play either stinger 1 (if bar 1 is currently playing) or stinger 2 (if bar 1 is currently not playing).

You can see how this system could be extended and developed significantly so that we could have both harmonically variable music and stingers that fit with what’s currently playing. It would be even better if the system could transpose the stingers to play at a pitch appropriate to the current harmony. This would be trivial to do if we were using MIDI, and in theory this would be possible in UDK if one were to calculate the relationship between the pitch multiplier and “real pitch” (see the earlier Mario example if you really want to know.). Of course, the problem would be that the playback speed would also be altered.

Stingers have a role to play in highlighting game actions through musical motifs. However, when a significant change in musical state is required, we still need to tackle the issue of how we move from the piece of music that is playing now to the piece of music we want. This is referred to as the Transition Problem.

Transitional Forms/Horizontal Switching

Clunky Switching

Location-Based Switching

404 Music Switch

The idea of having different pieces of music playing in different game locations is common in genres such as MMOs (Massively Multiplayer Online game). The system in Room 404 represents the simplest (and most musically clumsy) way to implement this feature.

The floor tiles mark out two different locations in your game, and each has a [Trigger] to start the music for its particular location. (Obviously this would be happening over a much larger area in a game, but the principle still applies.) When each one is touched it simply plays the music cue through the [PlaySound] object and switches the other piece of music off.

This fails to take account of where we might be in musical terms within piece A or whether it’s an appropriate place to transition to the beginning of piece B (they may even be in completely different keys). Any of the three scenarios (and more) at the bottom of the image below are possible.

These fairly brutal music transitions to the start of piece B are pretty clunky.

Gameplay-Based Switching

404a Spotted

As you cross this room you are inevitably spotted by a bot. Fortunately, you can run behind the wall and hide from his attack. By an even happier coincidence, you notice that you are standing next to an electricity switch, and this happens to lead (via a smoldering wire) to the metal plate that the bot is standing on (what are the chances, eh?). Flick the switch (use the E key) to smoke him.

On entrance to the room a [TriggerVolume] is touched and triggers the initial ambient musical piece via a [PlaySound] object. (In case you show mercy and leave the room without killing the bot, there are also a couple of [TriggerVolume]s to switch off the music.)

A bot is spawned in the room, and when the player falls within the bot’s line of sight, a [See Enemy] event stops the music and starts the other [PlaySound] object which contains a more aggressive intense “combat” piece.

Although the inputs may be different (a game event rather than a change in location), you can see that this is effectively exactly the same system as the location based switching above and shares the same problem in that the musical transition is less than satisfying. In some circumstances this sudden “unmusical” transition can work well, as it contains an element of shock value. In others (most), we need to develop more sophistication.

Smoke and Mirrors

404b Smoke and Mirrors

Using a piece of dialogue, an alarm sound (or, as in this case, an explosion) to “mask” the transition between two pieces of music can mean that you can get away with simple switching without drawing attention to the clunky nature of the transition.

When you enter this room you’ll notice that the door locks behind you. You get a message that states, “They’re coming through the door; get ready.” The tension music starts. The other door glows and gets brighter until Bang! The action music kicks in as bots come streaming through the hole.

After the door closes, a [Delay] is set while the particle emitters build in intensity. When the [Delay] ends and the [Matinee] blows the door open, a loud explosion sound is heard, and the second music track is triggered. “Ducking” the currently playing music (lowering its volume temporarily) below a piece of important dialogue can also be useful in masking the transitions between your musical cues. We’ll discuss ducking in more detail in Chapter 6.

Better Switching: Crossfades

405 Music Crossfade

In this room there are four musical states. Enter the room and wander around before succumbing to temptation and picking up the ‘jewel’ (press E). This launches a trap. Now try and make it to the end of the room without getting killed.

Now restart the level, and this time allow yourself to be killed to hear how the music system reacts in a different way. (We’re using the “Feign death” console command here to “kill” the player because a real death would require the level to be restarted. When you “die,” simply press “fire” [left mouse button] to come back to life.)

The tracks are as follows:

Ambient. For when the player is just exploring.

Action. In this case, triggered when you “pick up” (use) the jewel.

Triumph. If you then make it to the end through the spears.

Death. If you allow the spears to kill you.

This works better than the location based or instant switching examples partly because we are now using a short crossfade between the cues and partly because the musical language here is more flexible. Writing in a musical style that is heavily rhythm based or music that has a strong sense of direction or arc can be problematic in games, as the player has the ability to undermine this at any time.

As you enter the room, the first [TriggerVolume] plays the “Ambient” [SoundCue]. When you use the jewel on the plinth to pick it up (okay, we know it’s a HealthPack, but use your imagination), this first [SoundCue] is stopped and the “Action” [SoundCue] is started. In the [PlaySound] object, the fade out time of the “Ambient” [SoundCue] is set to 1.5 seconds, and the fade in time of the “Action” [SoundCue] is set to the same, allowing these to crossfade for the transition between these two cues.

If the player is hit by any of the spears flying across the room (the [InterpActor] s Hit Actor event), then an impact sound is played, the “Action” is stopped (the fade out time is one second) and the “Death” [PlaySound] comes in (the fade in time is one second). Likewise if the player reaches the end then the [TriggerVolume] in this area will stop the “Action” (the fade out time is one second) and play the “Triumph” [SoundCue] (the fade in time is one second).

If you play the room through several times, you’ll see that the player could spend much longer exploring the room before picking up the jewel (in which case the transition to “Action” would be from a different part of the “Ambient” music) or remain on “Action” for some time while carefully navigating the spears. Or else the player could run just through the whole thing. For example, the music could be required to transition at any following points.

Or a different player with a different approach might result in this:

The more impulsive player might even get this:

When combined with the right musical language and the occasional masked transition, the use of simple crossfades can be quite effective.

UDK’s (Legacy) Music System

You may be wondering by now why we have been using the [PlaySound] object within Kismet rather than what appears to be more appropriate, the [Play Music Track]. The reason is that this offers little in terms of functionality and much that is broken or illogical as part of Unreal’s legacy music system.

The [Play Music Track] will allow you to fade your music in up to a given volume level, but this could be set in your [SoundCue] anyway. It then appears to offer the ability to fade back down to a given level, which could be useful for dropping the music down to a background bed, for example. However, the problem is that unlike the Fade Out features of the [PlaySound], which is triggered when the [PlaySound] is stopped, there is no apparent way of triggering a [Play Music Track] to apply this Fade Out feature, nor indeed is there any way of stopping a [Play Music Track] once it has been started. If you want one piece of music to play throughout your level irrespective of where in the level the character is or what situation the character is facing, then this could be a useful object for you. If that is your approach to music for games, then go back to the bookstore right now and try and get your money back for this book. (Once you’ve done that, you can go and crawl back under your rock.)

If you’ve been playing some of the Unreal Tournament 3 maps that come with UDK (C:UDK UDK(***)UDKGameContentMaps), then you will have noticed that there appears to be an interactive music system at work. In the UDKGame/Content/Sounds/Music folder you can see these music tracks grouped into Segments and Stingers. You can also a package called Music Arrangements. Open up the properties of one of these by double-clicking on it.

These arrangements govern how music used to work in UT3. If you open the map VCTF-Necropolis from the UDK maps folder and select World Properties from the top View menu, you can see that it was possible to assign one of these arrangements to your map.

You’ll also note that although you can create a new [UDKMapMusicInfo] item within UDK that appears to be the equivalent of these [Arrangement] objects, you cannot actually add these to a new UDK map because this functionality has been removed.

The music in these items is triggered by game events and allows some musical transitions by allowing you to specify the tempo and the number of measures/bars over which the music will transition between states. Although this might be potentially effective for certain styles of music, it is not working and so we won’t dwell on it anymore here.

(If you want to try it out, you can replace the map music info in one of the UDK example maps with your own by copying one of the existing [Arrangements] files. Using a new [UDKMapMusicInfo] will not work.)

Maintaining Musical Structures: Horizontal Transitions

Write in Chunks

406 Searchlight Chunks

If the cameras in this room spot you, then you must hide behind the wall. The guards will quickly forget about you and the music will return to its more ambient state.

As in Room 402, a [Trigger_Dynamic] is attached to the search light (at a Z offset so that it is on the ground), and when this is touched it changes the music to the “Action” track. When you go behind the wall, the [TriggerVolume] starts a five-second delay before switching back the “Ambient” track. (The guards have a very short attention span; we blame Facebook).

Using a simple system, this effect could be achieved with the following steps. As the player enters the room, there is no music. When the [Trigger_Dynamic] (attached to the spotlights) hits the player, the first track is played and its gate is closed so it does not retrigger. At the same time, this action also opens the second gate so that when the player hides behind the wall and comes into contact with the [TriggerVolume], this is allowed to go through and adjust the switch to play the second track.

However, now we want to improve the system by only allowing changes at musical points. Out of these tracks only the “Ambient” track is able to be interrupted unmusically. We allow this because we want the music to react instantly when the player is spotted by the searchlight.

On entrance to the room, a [TriggerVolume] triggers a [Set Variable] Action to set the variable named ‘Set’ to zero.

This action triggers a [Compare Int] object, which compares the number 1 with the variable ‘Set’. In this case, input A is greater than B, so the result comes out of A > B and triggers the [PlaySound] for the “Ambient” track.

The room entry [TriggerVolume] also opens the [Gate] so that when the [Trigger_Dynamic] (searchlight triggers) are touched they can go through and change the variable ‘set’ to 1. The [Gate] then closes itself to avoid retriggering.

When the [Trigger_Dynamic]s (searchlight) have been touched, the variable ‘Set’ is the same as input A of the [Compare Int], so a trigger comes from the == output into the [Switch]. This initially goes through output 1, plays the “Intro,” then plays the “Action” track. When the “Action”track is finished, it sets the switch to go to output 2. That means that when the signal comes through the [Compare Int] again (assuming nothing has changed and we’re still in the “Action” state), the trigger will go straight to play the “Action” cue again instead of playing the “Intro.” Although the [SoundCue] does not have a [Looping] object in it, this system effectively “loops” the “Action” cue until the circumstances change.

When players hide behind the wall, they come into contact with another [TriggerVolume], which now changes the variable ‘Set’ to 2, causing an output from A < B. This plays the “Outro,” resets the ‘Set’ variable to zero, and resets the [Switch] to 1 (so on the next switch to “Action” it will play the “Intro” first). “Outro” then opens the [Gate] for the [Trigger_Dynamic]s. (This will make more sense by examining the system in the tutorial level. Honest.)

The key to this system is that triggers are only entering the [Compare Int] object when each track has come to its end using the [PlaySound]’s Finished output (with the exception of the move from “Ambient” to “Action” already discussed). This means that the changes will happen at musical points. You could put any music into this system with chunks of any length and it would work because it’s using the length of the pieces themselves as triggers. A typical use of this sort of system would be to check whether the player was still in combat. The “Action” cue keeps going back through the [Compare Int] object to check that the variable has not changed (i.e., we’re still in combat). If you anticipated being in this state for a significant length of time, you could build in variation to the “Action” [Soundcue] so that it sometimes plays alternate versions.

Transitions and Transition Matrices

If you consider the range of musical states you might want for a typical game scenario, you can quickly see the challenges that face the composer in this medium. Let’s take a relatively straightforward example based in one general location at a particular time of day (you may wish to have different sets of pieces dependent on these variables, but we’ll leave this additional complexity to one side for now).

“Stealth 01”, “Stealth 02”:

Two stealth cues of slightly different intensities.

“Action 01”, “Action 02”:

Two action cues, one for the normal guard combat, one for the boss combat.

“Action 02a”:

One additional action intensity for the boss combat (when the boss’s health gets low).

“Death”:

The player dies.

“End”:

The player defeats the guards.

“Triumph”:

The player defeats the boss.

“Bored”:

If the player loiters on stealth for too long switch the music off.

“Stingers”:

The player lands a blow on the boss.

If we discount the “Stingers,” “Death” and “Bored” (because they don’t transition to any other cue), then we have seven musical cues. If we allow for a game state where it might be necessary to transition between any of these to any other, then there are 127 different possible transitions. Of course, not all of these would make sense (e.g. “Stealth 01” should not be able to transition straight to “End” as there would have been some combat “Action” in between), but nevertheless writing music that allows for this kind of flexibility requires a different mindset from composing traditional linear scores. Before beginning, the composer needs to consider which pieces will need to be able to link to which other pieces, and the ways in which you want to handle each of these transitions. This is sometimes referred to as a branching approach to interactive music because it can be reminiscent of a tree-like structure. At the planning stage, a flow diagram such as the one that follows can be useful.

Next you need to decide on the best way for each of these cues to transition between one another.

Transition Times/Switch Points

There are, of course, a variety of ways to move from one piece of music to another, some more musical than others. You need to make a choice that is preferably reactive enough to the gameplay to fulfill its function while at the same time maintaining some kind of musical continuity. Here are some illustrations of the types of transition you may want to consider.

The arrow indicates the time at which the transition is called by the game.

Immediate

This immediately stops the current cue and plays the new cue from its beginning.

Crossfade

This transition fades out the current cue and fades in the new cue.

At End

This transition waits until the end of the currently playing cue to switch to the new cue.

At Measure

This transition starts the new cue (and silences the old one) at the next measure (bar) boundary.

At Beat

This transition plays the new cue at the next musical beat of the currently playing cue.

At Beat Aligned

This transitions at the next beat, but to the same position in the new cue, rather than the start of the new cue—that is, we exit cue A at beat 2, and we enter cue B also at its beat 2. More about this later.

Transition Cues

Often it is not satisfactory to simply move from one cue to another. You may need a “transition cue” to smooth the path between the two main cues. This is a specific cue whose role is to help the music move from one cue to another; of course, these cues can begin at any of the transition times described earlier.

A transition cue might take the form of an “End” to the current cue before starting the new one, on an “Intro” to the new cue (or a combination of both of these options (e.g., Cue 1/EndCue1/IntroCue2/Cue2).

Here is the transition cue leading from “Main Cue 01” to “Main Cue 02”:

Here are the transition cues leading from “Main Cue 01” to “End” to “Intro” to “Main Cue 02”:

A transition cue will normally ease the shift in intensities by leading down, building up, or managing a change in the tempo between the old and the new. It might also act as a “stinger” (! - see image) to highlight an action before then leading into the new cue.

A Transition Matrix

A table like the one shown here is useful for considering these kinds of systems. There is a blank template version for your own music available to download from the website. This defines which cues can move to which other cues and the transition time at which they can move. In this example, you can see that the “Intro_01”/”Intro_02” and “End_01”/”End_02” cues serve as transition cues between the main pieces.

Reading from left to right:

When moving from “Stealth_01” to “Stealth_02” the system should do it at the next measure.

When moving from “Stealth_01” to “Action_01” you should do it at the next available beat via the transition cue “Intro_01.”

When moving from “Action_01” back to “Stealth_01” or “Stealth_02” do it at the next available bar/measure using the “Action_End” cue to transition.

Although this might seem like a slightly academic exercise when you sit down to compose, the time you spend now to work these things out will save you a lot of pain later.

407 Transitions

This series of rooms contains a “stealth” type scenario to illustrate a system of music transitions.

Because the system is reasonably complex, we’ll present it in both narrative and diagrammatic forms.

First, here’s the story of 407. Upon entering the first room, the “Stealth 01” cue begins. The player must avoid the security cameras to get to the next room undetected. If the player is detected, then “Intro 01” leads to “Action 01” as a bot appears and they do battle. When the enemy is defeated, the cue “End 01” leads back into “Stealth 01.”

In Room 02 the situation gets slightly more tense, so we switch to “Stealth 02” (a slight variation of “Stealth 01”). Here the player needs to get the timing right to pass through the lasers. If these are triggered, then again “Intro 01” leads to “Action 01” for a bot battle. This time a different end cue is used, “End 02”, as we’re not going back to “Stealth 01” but returning to “Stealth 02.”

The player needs the documents contained in the safe, so there’s nothing to do but to open the safe and be prepared for the boss battle that follows. The music in the room, initially remaining on “Stealth 02,” switches on the boss’s appearance to “Intro 02” to lead to “Action 02.” During the battle when the enemy’s health gets low, the music goes up in intensity to “Action 02a” and there’s also the occasional “Stinger.” Defeating this boss leads to the final “Triumph” cue.

In all the rooms, another two cues are available: “Death” if the player is defeated by the enemy at any time, and “Bored.”

The “Bored” Switch

“Bored” is a version of “Stealth 01” that has a fade to silence. Part of music’s power is to indicate that what is happening now is somehow of special significance or importance. If we have music playing all the time, it can lose its impact (and make your system look stupid). It’s useful to always have a “bored” switch (Marty O’Donnell;), so if the player remains in one musical state for an unnaturally long period, the music will fade out.

The diagram shown here also represents this music system.

(For the gameplay systems themselves, take a look in the room subsequences. These are simple systems based on triggers, dynamic (moving) triggers, bot spawning, bot take damage, and bot death.)

Here is a screen shot for the implementation. Sadly this is our idea of fun.

First a pulse is established that matches the tempo of the music. From this we get two outputs via [Activate Remote Event] s: the beat (407 Beat) and another signal on the first beat of a four-beat bar/measure (407 Measure). These will allow us to control transitions so that they occur on either the musical beat or Bar/Measure line.

Both these are received via [Remote Events] at our main switching system. (You can double-click on a [Remote Event] to see the [Activate Remote Event] that it relates to.)

This switching system receives a message from a game event, which comprises two parts:

Which track should I play?

(Named Variable “Track No”)

When should I switch to it?

([Remote Event]s “407ChangeAtBeat”/“407ChangeAtMeasure”)

On receipt of this message the track number goes straight to the [Switch] and changes its index so that the next message received by the [Switch] will go out of the appropriate link number of the [Switch]. You can see that these then go to further [Remote Event] objects, which then go to actually play the music when received by the [PlaySound] objects.

This message will only be sent out of the [Switch]’s link when the [Switch] receives an input, and this is governed by the second message that determines whether this will be at the next beat or the next bar/measure. Both impulses are connected to the [Switch] via a [Gate], and the choice of transition type (407ChangeAtBeat or 407ChangeAtMeasure) determines which pulse goes through by opening/closing their respective [Gate]s. (When a ChangeAtBeat message goes through to the switch, the [Int Counter] that counts the length of a measure is also reset so that the next measure is counted starting again at beat 1.)

How these transitions occur is under the composer’s control; all the game designer needs to do is to call one of the remote events named T0 to T9 (Trigger 0 to 9). You can then set these up to call a particular music cue, at a designated time.

Which track should I play?

(Named Variable “Track No”)

When should I switch to it?

([Remote Event]s “407ChangeAtBeat”/“407ChangeAtMeasure”)

At the moment they are set to do the following:

T0: Play “Intro 02” at next beat // Then play “Action 02”

T1: Play “Stealth 01” at next bar/measure

T2: Play “Intro 01” at next beat // Then play “Action 01”

T3: Play “End 01” at next bar/measure // Then play “Stealth 01”

T4: Play “Stealth 02” at next measure

T5: Play “End 02” at next bar/measure // Then play “Stealth 02”

T6: Play “Stinger” (this plays over the top and therefore operates outside of the usual switching system)

T7: Play “Triumph” at next bar/measure (this then stops the system) T8: Play “Death” at next beat (this then stops the system) T9: Play “Action 02a” at next bar/measure.

You will notice that many of these actually trigger two changes. In these cases, the second part of the system receives a [Remote Event] when the first music cue has finished playing.

Finally (you’ll be relieved to hear) we have what’s known in Yorkshire as a “Belt and Braces” system to stop all the cues apart from the one that is currently supposed to be playing—just in case. (We also did this because it makes a pretty pattern.)

As you can see, setting up a system for musical transitions is not trivial, but once you’ve got your head around the system here you can adapt it for your needs.

Postscript: The Bored Switch

When a “Stealth” track is triggered, a [Delay] starts (how long to give it before you get bored and fade out the music). If this time passes then we change at measure to a new track (track 7). This triggers the [Remote Event] “Bored,” which comes back and starts a [PlaySound]. This [PlaySound] is actually playing the “Stealth 01” cue again, but this time it has had a fade out applied. It also closes the [Gate] so that it only plays once before ending. The music will not start again until there is a change in state that flicks the playback switch to a different track.

Switch Points: Marking Up Existing Tracks

A few games have attempted to use commercial tracks interactively in games, with some success. One approach has been to apply DSP effects to the track. For example, in a racing game when the player performs a jump, apply a low-pass filter to the music while the player-character is in the air. When the player gets a speed boost, play the track back faster by pitching it up, or apply a filter to temporarily boost the bass frequencies.

Another approach would be to identify parts of the track/song that have different intensities and note the possible in and out points. Moving to different parts of the track in response to game events can work if the track’s possible transition points (or switch points) are carefully (and musically) identified.

‘Banging Track no. 87’

Having identified which sections are of what intensity, the programmer can then call these particular sections at relevant times using transition points on bar/measure boundaries.

If the track is at a consistent tempo, then you should be able to calculate these points easily and either supply the timings to your programmer or slice up the track in your digital audio workstation (DAW). (See the downloadable spreadsheet file “Markup” on the website.)

For example, Tempo = 120, 4 beats per bar.

Your DAW will have a Bars and Beats option for its timeline where you can see these timings, and some will also have the functionality to set markers at the beat or bar lines automatically.

Many audio editors have the ability to store extra information in the wav file, which contains a header with non-audio information known as metadata. Using metadata to hold your markers is better than chopping up the wav itself, as your programmer can call these but will also have the original sound file in order to enable a short volume envelope to fade the music out. This can avoid the kind of clicks you will otherwise get from editing music at arbitrary points in the Wav file (see Appendix D). Methods for marking up users’ music tracks with data automatically are already available, and this is likely to be an area of growth in game audio research. Many games already enable or encourage players to use their own music within the game, but the potential to make this music interactive using embedded metadata about the track itself has yet to be fully exploited.

You’ll be aware that musical sounds are no respecters of bar lines, and the “Decay” of a note will often fall across where you are likely to edit. This is known as the “decay problem.”

The Problem of Decay

408 Decay

We’ve seen how a matrix of transitions between pieces can be musically challenging, but there’s an additional technical challenge that we also need to consider. The following four-bar/measure loop might be our music for a “Stealth” sequence. We want to export it from our sequencer and use it in the game initially as a loop. In our sequencer it sounds fine, but when exporting it as a wav file it suddenly sounds unnatural. In Room 408, use button 01 to hear this version.

As you can see, the harp has a short descending motif just before the loop point. In our sequencer this sounds okay because the sample-based instrument allows these notes to decay naturally even though the sequence has returned to the start of the loop. However, when exporting, the decay of these notes is not preserved as they fall outside of the fourth bar.

By selecting a section to export that’s longer than the actual MIDI files, you can see that there is a significant decay to the sound. Within your sequencer you will continue to hear this as the pattern loops around to the start; by exporting the audio for just the loop, however, this is lost. In the selected area (white) you can see the natural decay of these notes. In the unselected area (black) is the actual exported wav file we get.

There are two approaches for preserving the decay for a looping sound, both of which have the same effect of mixing the decay of the end of the sample back into the start of the sample.

Copy and paste your sequence so that it is repeated and then export this to a new file.

Selecting the second half of this wav file (divide the total length by 2 and start your selection at this point, then “Trim” to keep only your selection) will give you a version of your loop that includes the decay of the Harp notes. This means that when this is looped around it will now sound more natural because these are not cut off. Use the second button in the level to hear this version looping. The other method to achieve the same effect would be to export the original sequence, but with a few following empty bars included. Then you can select the decay portion and mix-paste (blend) it back into the start of the loop.

This is fine while it’s looping, but you will have noticed that the first playing has decaying portions at the start (shown below in blue) of notes that have not actually been played yet.

The result, of course, is that you need both, either as separate files as shown here or the ability to select loop points within one file.

One version is used the first time the music is played; the other is a looping version.

These issues also apply to transitions, of course. Where one block of music switches to another the decay of the instruments from the first is lost. You can have a short crossfade so that some of the decay is there, but, of course, so are all the other notes. You might consider recording a separate decay sample for each of your blocks so that on transition this could be played to overlap with the new music.

This method is entirely possible although laborious. (Some sequencers have the ability to export with a “Capture effects tail” option that makes it slightly less tiresome). You’d also need to rework the previous transition systems to add this new decay sample of the previous piece to the first playing (and first playing only) of the new cue.

Here’s a question we’ll leave you to ponder until we return to the subject at the conclusion of this chapter: Have you ever tried getting a symphony orchestra to play in one bar’s chunks so that you can get a natural decay for each musical block? Would you want to?

Parallel Forms/Vertical Layering

We’ve looked at some of the issues and solutions to interactive music using a horizontal approach—that is, the transitioning between a series of tracks depending on the game state. Another approach is to consider that it might be possible to generate music for many of the circumstances you want by using the same piece of music. By exposing the different layers or tracks, we can bring them in and out of the mix to change the arrangement:

When tracks 1 through 6 are all playing, this could be your “very intense, many enemies, health low” musical state, whereas muting all the tracks except for track 1 might give you your “ambient sneaking” version.

The term “stems” is commonly used to describe these individual layers as these might consist of individual “tracks” in the sequencing/recording sense, or might be a group of tracks. The idea of stems is most commonly associated with film postproduction where a sound track might be grouped into the dialogue stem, the music stem, and the effects stem, although it’s also used in music production to represent the output of a particular subgroup or bus.

Simple Mute/Unmute

409 Vertical Mix Tension Layers

Press the button to start the game. Objects appear in random places/at random times. You need to collect these to score points and get bonus time (the countdown started when you pressed the button). Some health packs have been damaged—picking these up will give you a time penalty.

When the game is started by using the central [Trigger] all six [PlaySound] objects containing the six stems/layers are triggered. When the [PlaySound] object number 1 finishes, it retriggers itself and all the rest again. This ensures that they keep in time.

For the game aspect, a random [Delay] is triggered (sometime between zero and six seconds); when this is complete the first pickup is unhidden, and a [Gate] is opened allowing the invisible [Trigger] around the pickup to pass through. This now plays the pickup [SoundCue], destroys the pickup [InterpActor], and triggers an [Announcement] informing the player of the points just won. This then goes on to trigger the next randomized [Delay] to unhide the next pickup, open its [Gate], and so on.

The timing system starts with an initial value of 60, from which the value 1 (‘PickupPoints2’ named variable) is subtracted every second by a [Delay]. (For more on named variables, see Appendix C: UDK Tips.)

While the announcement is made, a value is sent to the countdown clock depending on whether this was a “good” pickup or a “bad” pickup, adding a bonus of 10 seconds or subtracting a penalty of 10 seconds. To add/take away these values from the time remaining, a [Set Variable] is sent to alter the ‘PickupPoints2’ variable to +10 or –10, which is added to the current remaining time. This is then set back to 1 to continue incrementing the countdown in seconds.

As explained earlier, all the separate tracks that make up the full intensity arrangement are playing throughout. The Time Remain value is then used to mute or unmute certain stems in order to produce different versions of the full arrangement for our lower intensity levels. The relationship between the Time Remain variable and the music arrangement is outlined in the following diagram.

And when Time Remain ≤ seconds, all the tracks play for the arrangement of fullest intensity.

A series of subsequences compare the Time Remain variable with their settings; if it is within their range, then they unmute the appropriate track. For example, this sequence checks to see if the Time Remain variable is less than or equal to 60 (≤60) and more than or equal to 51 (≥51).

This result is checked by the [Delay] pulse (every second). In theory it shouldn’t be a problem with the parts being unmuted at any given time as they are all aligned in time. If part 3 happens to be unmuted while we are three seconds into the music, it will be okay because you will hear part 3 at three seconds (which, of course, is written to fit with all the other parts at three seconds), not the start of part 3 as in previous systems. We could have several versions of each part for variety and a system that mutes and unmutes in the following ways and it would (theoretically) still function musically.

You may notice that in the UDK system although the numbers change and possibly unmute at second intervals the musical parts do not appear to actually change until they are retriggered. This is not a huge problem in this instance because the musical parts are short and therefore the system appears relatively responsive. However, if our parts were a minute long, we wouldn’t actually hear the change in the arrangement until these were retriggered after 60 seconds.

Here’s the problem. Although you have access to the Volume Multiplier variable of the [PlaySound] object, this is the volume set when it begins to play that sound. You cannot change the volume of a currently playing sound.

Mixing Layers

409a Music Layers 01

You have to sneak up on the guards in this area and silently dispose of them (press E when close behind them).

In this example the music indicates proximity from the enemy. This might be useful in a stealth-based game. There are four music stems (in this case single tracks): “Sticks,” “Shakers,” “Bell” and “Bass.” The “Bass” and “Bell” tracks are mixed up or down in volume depending on the player’s distance from the enemy, the idea being that the closer the player is to the enemy the more tense the music should be, and therefore we bring up the “Bass” and “Bell” elements to indicate this tension. In addition to the mixing of layers there’s also a stinger element that plays when we get within a certain proximity of the bot, and another when you attack him.

Instead of simply switching tracks bluntly on or off at the end of each loop as we did earlier, we’d like to be able to have smooth transitions in volume to change the mix of the full arrangement in real time in response to gameplay variables. If you like, it’s an analogue scale, using a range of numbers rather than a binary on/off switch system.

Binary

High intensity (full boss battle), Medium intensity (1 enemy), and Low intensity (sneaking)

Analogue

The [TriggerVolume] on entry to the room plays all the [SoundCue] layers, and they continue to loop (they have [Loop] nodes within their [SoundCue]s). When the bots are spawned, we get the distance between the player and the bot and do some maths on this number so that it falls into a more appropriate range of values to control the volume of our “Bass” and “Bell” stems.

This example is musically simple to illustrate the point, but we could have several stereo stems streamed from the disk simultaneously, and if they were all built over the same harmonic structure, we could dynamically mix them to produce different intensities through changes in the orchestration, texture, and instrumentation. You could say this was similar to the “Fatal Void” system (Room 402), but this time our music can be rhythmic and harmonic as long as the tracks are kept aligned in time.

The ability to control the volume of several separate music tracks that maintain their alignment in time is a fundamental requirement of interactive music. Because UDK’s [PlaySound] object disappointingly does not allow us to change the volume of a sound while it’s currently playing, we’ve had to make our own.

Introducing the [GATPlaySound] object!

This is a custom Kismet object and is found in (right-click) New Action/GATKismetActions/GATPlaySound.

The [GATPlaySound] object allows you to select a [SoundCue] and control its volume. The volume level can be set by the fade level variable. Whenever a signal is received at the Trigger Fade input, the volume will go to this fade level over a time period designated by the Fade Time variable.

Here the output of the [Get Distance] object is then scaled and sent to the Fade Level (BassVol). Each time this value changes, the [Divide Float] object is also sending a signal to the Trigger Fade input to enact a volume change, which then occurs over 0.2 seconds.

In this case, the variable is the distance from the enemy, but it could be anything, such as the number of enemies in the area, the amount of ammo you have left, player health, player hunger, proximity to pizza….

409b Music Layers 02

This room switches to an imitation 2D platformer style level. You have to avoid the lasers and the falling crates to reach the end. The music uses a vertical layering system based on two variables; the player’s distance from the objective [Get Distance], and the player’s health [Get Property].

All the music [SoundCue]s are started by the [Trigger] on entry to the room. A [Delay] is polling the player’s health every 0.1 seconds, and this is scaled by some maths to give an appropriate range for the volume of the “Health” cue. As we want this cue to increase in volume as the player’s health declines, it is divided by 100, and the result is subtracted from 1.0. This has the effect of inverting the value (i.e., health high = volume low, health low = volume high.) The [GATPlaySound] object is used again here as it allows us to fade to these changing levels while the music is playing.

The second variable (the player’s distance from the “end” objective) is also scaled and controls the volume of the three other music tracks. For these tracks we did not want a simple linear (or inverted linear) relationship between the variables’ values and the volume of our cues, so we have extended the system to allow us to define different volume curves. A switch is set up and the scaled variable reads through the index of the switch and outputs the value attached to that link. In programming terms, this is the equivalent of reading through an array of numbers. Your programmer can get you access to UDK’s distribution curves, which can perform a similar function, but for now we’ve chosen to make this system explicit so you can see what’s going on and have easy access to the values yourself.

If we plot the array horizontally, it’s clearer what this allows us to do. The Volume values are held in the [Set Variable] actions attached to each switch output link.

Having control over the relationship between the game variables and the volume of your stems is where the real power of a vertical layering approach lies.

This idea is often more elegantly implemented and referred to as “Real Time Parameter Controllers” (RTPC’s). Audio Middleware like FMOD and Wwise are very good at allowing you to tie in different volume curves for your music layers in response to the game variables in a much more composer-friendly way. Unfortunately, at the time of writing you need to already be working for a game developer to get hands-on with this middleware in conjunction with the Unreal engine. Until this is opened up to the audio community, you can have fun testing out these approaches with your music using the models we have provided here.

Leitmotifs and the Combinatorial Explosion

410 Leitmotifs

By associating a melody or motif with a character, we can be made aware of the character’s presence (real or implied) and gain insight into how that character feels by the different ways in which the character’s leitmotif is presented. A solo flute version of the motif might mean that the character is feeling lonely, whereas a version with blazing brass and drums might tell us that he or she is ready for a fight. (You can have leitmotifs for many other functions besides being attached to a character, but this is its most typical usage.)

In the context of film music where everything is predetermined in terms of time, this works well and is a powerful tool for the composer. For the game composer who wishes to utilize this technique, the problem of combinations arises. We’ve seen how the exponential growth of possible combinations can be a real help to us as audio designers for games as it can generate many new combinations from relatively few sources, but to understand why this is a problem when it comes to leitmotifs, consider the following game design brief for an MMO.

•  Each location should have a particular musical style to define it. There are five locations.

•  Each tribe in the game has its own associated melody (leitmotif) that you hear whenever that tribe is in your vicinity. There are five tribes.

With 10 variables this would, in theory, give us a potential of 1,023 possible combinations. However by using a similar idea to the vertical layering, we could write all of the leitmotifs so they fitted over the same harmonic structure. It might be difficult to avoid clashes in the melodies, but the goal is achievable (after all, Mozart seemed to manage it). The five locations could be defined through the instrumentation of the accompanying parts.

In Room 410 there is a gathering of creatures around a campfire. As each creature comes closer to the player, you can hear the creature’s leitmotif join the music. Theoretically you could have the music for all the beasts start at the same time (on level startup) and be of the same length. You could then use the attenuation curves on the [SoundCue] so that the music is only heard when they get within a given radius of the player. The trouble is that if a player is outside of the max radius, the cue isn’t actually played. This is, of course, completely logical. Why play something you can’t hear? But it means that the cue will actually be started when the player enters the max radius, not in alignment with the other musical parts.

If we keep the parts aligned, our music should fit.

If we allow the leitmotifs to start at any time (i.e., they are not temporally aligned), then you will get the musical mess that follows. You do not need to be an advanced musician to know that this is not going to sound good.

You can see that this is the same problem we have faced earlier when composing layered music, so it follows that a potential solution would be the use of the [GATPlaySound] to synchronize all the musical parts but fade their volume up or down depending on the distance from the player.

On entry to the room, the “Harp” background music starts and loops. The bots that spawn in the area wander around at random time intervals, moving toward and away from the central fire.

Within each bot subsequence the distance from the player is scaled and used to vary the volume of each bot’s leitmotif. The effect is that as each bot enters the area around the player, its leitmotif joins in the musical texture. Because all the music stems are started at the same time (immediately after all the bots spawn on level start), they are all aligned in time, and remain so even when their volume is changed.

As you can see, the ability to keep separate music tracks or “stems” aligned in time and the ability to vary their volume in response to game events can be a partial solution to the Leitmotif problem.

The “Variation” Question: Generative or Procedural Forms

(See the “Streaming” section in Appendix B for some specific advice on memory management for music.)

When composers sit down to write a piece of music, they usually feel that they have complete control and autonomy over what they are writing. In fact, most of the time, if using traditional Western harmony, they are actually doing so within the confines of a well-understood and codified system of rules. It’s just that these rules have been so internalized that the composers forget they are there.

Although most music does not need to be as immediately responsive as sound effects and can therefore be streamed from disk, it is not immune to the restrictions of memory and file space. Games can have anything from 10 to many hundreds of hours of gameplay. It is not realistic to write and to store this amount of original music. By moving away from a linear music model to using generative or procedural techniques to create, or at least vary, music on the fly it is possible to avoid a repetitive score while still keeping the storage space required within reasonable limits. Generative music or procedural music—that is, music that is generated by a system or procedure—can have totally predictable output, but to produce the variation we are after for games we need to look at a subset of generative methods usually referred to as aleatoric (from the Latin alea, meaning dice). These systems involve leaving some aspects of the output to chance or random events. For many composers, the idea of leaving some aspects of their music to chance is an alien concept. However, once they appreciate that they are just creating the opportunity for different possibilities (the range of which they still have complete control over), many find this an exciting and intriguing approach to writing music.

Vertical Recombinations

411 Vertical Recombinations

This room illustrates how four different music tracks can be randomly combined. Each combination is different and therefore creates variation. This is much like the combination of explosions and debris covered earlier in the book (Room 205). You could relate this to the simple muting and unmuting of layers in Room 409, but this time we are not trying to go up and down in levels of tension but are instead swapping layers in and out to create variation within one tension level.

The switch in this room cycles between two versions of the same music. The first version attempts to get as much variation out of the four music tracks as possible. The [SoundCue] randomly picks between these different possibilities by having 1 by itself, then 1 + 2, 1 + 2 + 3, 1 + 2 + 3 + 4, 1 + 3, 1 + 4, and so on.

The second version has been tweaked. Using the weightings within the [Random] node and by limiting the combinations available, the amount of variation has been limited but the overall effect is more musical.

If you have a piece of music that could last an indeterminate amount of time in a game, there are two solutions. The first is to have a timer mechanism so that after a given amount of time the music fades out so that it does not become repetitive and lose its impact. The second is to develop more complex [SoundCue]s that have a significant amount of variation built in through the use of controlled random processes.

Here is the old approach:

Here is the new approach:

Writing in this style takes some getting used to for the composers because they need to release some level of control, but this can be liberating! The approach that works the best for this type of generative system is to write your music over a static harmony. Writing your motifs over one chord will make their vertical combinations easier, because you don’t have to worry about them fitting with each other and with any changing harmonies. Ironically, writing over a single chord will also appear to make the music less repetitive. If we have multiple motifs or ostinati (repeating patterns) based around a single “drone” note or chord, you will avoid the repetitive feeling that a cyclical chord sequence might create. The chord sequence will become quickly predictable, and therefore feel repetitive, whereas without it people will start focusing on different aspects of the music.

Set Phasing to Stun

411a Music Phasing

Another interesting way of generating musical variation from limited resources involves the idea of “phasing loops.” This works best with rhythmic elements, because with harmonic ones the combinations are often too complex to predictably avoid clashes.

If we had one rhythmic loop that lasted 3 bars and another that actually lasted 4 bars, then we would only hear the same music repeated after 12 bars. In other words, what you would hear at arrow marker 1 will not occur again until arrow marker 5. Although the start of part 1 will always sound the same, the overall music will sound different because you are hearing this in combination with a different section of part 2 each time. Only after 4 bars of part 1 (and 3 bars of part 2) will you hear both the start of part 1 and the start of part 2 at the same time like you did in the beginning.

A 3 bar piece looping and a 4 bar piece looping.

A 6 bar and 8 bar combination would only repeat every 24 bars, a 7 and 9 bar combination every 63 bars.

The first example as you enter the room uses files that are stereo 44.1 kHz. It consists of a 4-bar loop (858 kB), a 6-bar loop (1286 kB), a 9-bar loop = (1,715 kB), and an 11-bar loop (2,143 kb). The total for these files is approximately 6 MB.

By looking at the lowest common multiple between them (2 × 2 × 2 × 3 × 3 × 3 × 11 = 2,376), we can work out that in order for them to come back in sync and for us to hear exactly what we heard when they all began to play, we would have to play the 4-bar cue × 594 times, the 6-bar cue × 396 times, the 9-bar cue × 264 times, and the 11-bar cue × 216 times to come back in sync.

The 4-bar loop lasts 4.571 seconds, so that means we get more than 35.5 hours of music before it repeats. (Listen to it for 35 hours and check if you don’t believe us.)

If we convert these to mono at a 22-kHz sample rate like the second version in this room then we get the following:

4 - bar loop (197 kb), 6 - bar loop (296 kb), 9 - bar loop (394 kb), 11 - bar loop (474 kb) = 1,361Kb

Now 35.5 hours of music for just over 1.3 MB of memory footprint is not bad (if this were one file at CD quality, it would have only given us only 8.5 seconds of music).

Obviously, although every detail will not repeat until the cycle is complete, this does not mean that it’s necessarily interesting to listen to—it is up to you to make that judgment. This is, however, a useful tool for producing nonrepetitive textures.

You can get 35.5 hours of interesting music from this? Probably not but…

Low-Level Generative Processes

412 Music Generative 01

Taking these ideas of generative music to their logical extreme, you can build systems that will procedurally create music from very small samples of individual notes or sounds. The rhythmic drumming in Room 412 is actually being entirely generated in real time.

The [SoundCue] is a randomized choice of seven samples of a large tin can being hit.

A [Delay] of 0.2 seconds is serving as a pulse that enters a looping switch. As this is sending out half beat pulses, it imitates a tempo of 150 bpm (60/0.4 = 150). This serves as a basic sequencer, outputting from each link in turn with the pulse provided by the delay.

The [PlaySound] “Tin 01” receives every pulse directly from the [Delay]. This provides the basic quaver (half note) with some [Modulation] in the volume of the hits within the [SoundCue]. On pulses 1 and 3, a second [PlaySound], “Tin 02,” is triggered. This [SoundCue] reuses one of the samples of the first, but its [PlaySound] object pitches it down (pitch multiplier 0.5) and lowers the volume (volume multiplier 0.5).

Note that it does not always play (that would be predictable) but instead goes through a [Random] switch, of which three of the four outputs are empty. This means that there is a one in four chance that this element will play. This system is spiced up a little with the occasional “Guitar being hit by a drumstick” chord that adds another element of unpredictability. This has a one in four chance of being played on pulse 1, a one in six chance of being played on pulse four, and a one in twelve chance of being played on pulse 9.

This is a simple example designed to illustrate a point, but you can see the potential. The total size for the percussive samples used here is 5.1 MB, or 1.5 MB without the longer guitar hits. By programming systems to generate musical output, it is possible to create long non-repetitive sequences with a minimal memory cost.

The Problem with Generative Music

412a Meta Generative

This room uses a similar “sequenced” approach to producing music with some randomized elements to Room 412. However, this time there is an additional switch that allows for some more global changes to the musical texture.

To add a sense of drama, musical textures should do this:

Not this:

Some calculations are made to determine the length of a half beat at the given tempo; this is sent to the [Delay] object, which provides the basic pulse for the switch sequencer. This switch is linked up in various ways to the musical [PlaySound] elements, which are organized in terms of intensity layers.

For example, at a low intensity the “Claydrum” plays on every pulse and the “Bass drum” on the first of every group of 16 pulses. At another intensity level the “Stomps” are playing on pulses 1, 5, 9, and 13.

You’ll note that the [Playsound]s have their volume multiplier exposed, so this means that we can send a variable to these to alter their playback volume, to mute or unmute them. As we know from bitter experience, this will only take effect when the [PlaySound] is triggered, not while it’s playing back. In this case, as the sounds are made from such short sampled elements the changes should appear to be near instantaneous.

Every 16 pulses, the section of the sequence that controls the volume of each musical element receives a [Remote event].

A [Random] variable is chosen, and this is used to control the index of a [Switch] (i.e., which output the switch will switch to). These outputs control various sets of variables, which will set the volume of the various [PlaySound] elements and hence govern the musical intensity.

We’ve noted how textures produced by random systems can perversely become fairly predictable. You can produce more dynamic or dramatic shapes within your music through building procedural systems that react to gameplay via “meta” overarching controls. Ideally we would be able to have a curved or variable parameter that could collate variables from the game state and then control the overall intensity level of your generative music. Have a go - and send us a copy when you’re done!

“Interactive” Music: Music as an Input

Although we’ve tried not to dwell too much on abstract theory in this book, there are several terms typically used to describe types of game music (and game sound) and a variety of definitions within the existing literature on the subject. We’d like to take this opportunity to confuse things even further by offering up our own.

Reactive Music System=Adaptive or Dynamic Music

This music responds to the player’s actions, as mediated by the game state, or varies according to the game state itself. Players get feedback from the game on their interactions, and they receive feedback from the music. But the music is simply a recipient of commands from the game engine.

The music could be a purely linear piece or could respond to the game state in complex ways using horizontal transitions or vertical layering as described earlier. This is described variously as interactive music, adaptive music, or dynamic music. Although common in many games, we would argue that this music is not interactive (which implies a cycle of feedback between the elements of the system), simply reactive.

Interactive Music System

For truly interactive music, there needs to be a feedback cycle between the game engine and the music state. Rather than simply receiving input from the game and responding accordingly, the game state should receive feedback from the music as to its current position and the time to the next musical transition point. This way the game state can decide whether to call the music immediately or whether a more satisfying overall result can be achieved by taking into account the music state within its decision making processes.

Performative Systems

This is an additional category that encompasses situations where the player has some direct control over the music state itself such as in music-based games like the rhythm-action genre.

Let’s look at a simple example to explore these principles further.

413 Music as an Input

This room contains three different musical scenarios. It is intended to be a simple demonstration of the concept that when appropriate (nearly always) the game state should consider the music state, as well as the music reacting to the game state.

Each button in this room represents one of three choices. When you press it a bot will appear and you must battle this bot to the death. As you do so, an “Action” cue will loop around playing one of three variations. When the bot dies an “End” cue will be played. The difference between the three scenarios is in the way we transition from the “Action” to the “End” cue.

(A) Instant Death and Instant Music

In this example the bot is set to die when its [Take Damage] reaches 14. When this number is reached, it dies instantly and the music instantly switches to the “Triumph” cue.

(B) Instant Death but Music Delayed

In this example the bot is also set to die when its [Take Damage] reaches 14. When this number is reached it dies instantly, but this time the music waits until the next “musical” opportunity to transition (i.e., the next bar line).

(C) Interactive Death

The bot is still programmed to die when [Take Damage] reaches 14, however this time the actual [Destroy Actor] and [Toggle] explosion that represents its death are blocked until the music is ready to transition to “Triumph” (at the bar line/measure).

(If you keep getting killed there is also an additional switch (D) to stop the bot shooting at you!)

Play the Room a few times. Which do you prefer, A, B, or C?

(A) Sudden Switch on Enemy Defeat

When the enemy is spawned, the [PlaySound] for the “Action” cue is triggered.

An [IntCounter] is incrementing every time you land a shot on the enemy via the [Take Damage]. When this reaches 14 (outputting from A==B), the [Destroy] object kills the bot, the “Triumph” cue is played, and simultaneously the “Action” cue is stopped.

(B) Death Immediate but Music Waits

This time the bot dies as normal; at the instant that the [Take Damage] = 14, however, the music only switches to the “Triumph” cue at the measure/bar.

When the enemy is spawned, the “Action” cue is triggered. At the same time, a [Delay] object one bar/measure long is started, which serves as the retriggering mechanism for the music, initially producing a looping retrigger of the “Action” music. Every time the enemy takes damage, an [Int Counter] is incremented. While this value is below 14, the A < B outputs a signal, and the [Gate] to the “Action” cue remains open and the [Gate] to the “End” cue is closed. When the damage reaches 14, however, the [Int Counter] outputs via its == and destroys the bot.

This time, instead of triggering the musical change directly, it now opens the “Triumph” [Gate] and closes the “Action” [Gate]. The “Triumph” cue will, however, only be triggered when the next [Delay] comes through, so in effect it only actually plays on the next measure/bar.

Although this is musically satisfying, it could feel “wrong” or “broken” that the music does not always coincide with the bot’s death.

(C) “Interactive” Music: Death Waits for the Musical Measure

In this final example, the whole system becomes properly interactive. In other words the feedback goes both ways, not simply that the music reacts to the game state, but the game state also takes input from the music. This may sound complex, but in fact it involves only the movement of one connector. See if you can spot the difference.

When the Damage [Int Counter] reaches 14, it outputs from the == output as usual; however, rather than instantly triggering the bot [Destroy], this object is now connected to the [Gate] of the “Triumph” music cue, meaning that the bot will be destroyed on the next musical bar/measure when the music’s [Delay] signal goes through. (Compare the input to the [Destroy] object discussed earlier with the previous image.)

This tiny change means that the game system is now involving the music system and music state in its processes. Rather than simply being reactive, this system is now interactive.

Challenges and Responses to Interactive Music

A game designer may argue that if the bot takes 14 shots to kill, then it should die when it receives the 14th shot, not later when you may have actually shot it 15 or 16 times. This is a fair argument. However, in the heat of battle who counts the number of shots that have successfully landed on the enemy?

Another scenario could be that you’ve shot it 14 times, then you stop shooting, only to see it actually die a few seconds later. This actually happens frequently in games and is actually more realistic than the enemy dying in sync with a shot. Sometimes the last shot has killed the enemy, but it takes a few seconds for the enemy to die (we don’t know this from personal experience, but we have watched a lot of movies).

Does the satisfaction produced by the synchronization of the death with the musical hit outweigh the need for strict accuracy? We would argue that in many cases it does.

Some games will work fantastically with totally linear music and some with reactive music. However, some demand that we improve the current model of interactivity. Whatever clever methods, present or future, we come up with, they are never going to work musically (and therefore will never be as effective for the game experience as they could be) unless we start to consider music as part of the system, not just as an output. The current state of the music and the available musical transition points should be inputs to the game’s decision-making system. It should not simply have to react.

Go and spread the word.

A Short History Lesson: Gains and Losses

Imagine for a moment that you have hired an orchestra to play a piece for your game. They sound fantastic, but there is a problem. Very strict new union rules have been introduced, which means that orchestras are only allowed to play one piece of music, and they also have to play this piece in exactly the same way each time. If you want to change one note within a four-minute piece, you will have to hire a different orchestra to play it. If you decide you want to have a rallentando (slowing down) toward the end of your piece, again a new orchestra will be required. Composers are puzzled. “Why can’t I just alter the score (which amounts to a very cheap box of paper) and they can play all the versions I want?”

Isn’t this dumb?

Isn’t this the equivalent of using wav files for interactive music?

There is an inherent irony at the heart of writing interactive music in that the format we use to deliver it (typically stereo or 5.1 wavs) is not as appropriate for its use as the format we probably used to write it in; MIDI. The majority of composers work using a MIDI sequencer combined with some kind of high-quality sampling or synthesis engine. With the separation of “Orchestra” (the sample banks) and “Score” (the MIDI tracks) you can play any piece you like, solve the decay problem (see the earlier discussion), produce as many variations as you like, plus add tempo changes, controller changes, and the like with roughly zero impact on the overall size of the project (MIDI files are tiny).

It’s clear that the greater the level of interactivity, the more flexible music must be and therefore the more granular (note) level control over the music we must have. But by rendering our work out to a wav file, we are essentially creating the “one-track-orchestra.” This is big, expensive (in terms of memory), and inflexible (every possible variation, even changing one note, means you have to render the piece out as a separate file). Even with the inclusion of musical states into the game’s decision-making processes, argued for above, can we really say that we have interactive music when we actually have control over so few parameters (whether it plays or not + its volume)?

So why can’t we have our sample banks in memory (tweaking them so the “Orchestra” isn’t too big) and then control our interactive score with MIDI? The answer is that we did have this ability once, but then we lost it.

Early games could use MIDI to control their music, knowing that as all soundcard manufacturers were meeting the “General MIDI Specifications” they could safely assign a track to Program No. 110 and it would be played by a bagpipe. The problem was—what kind of bagpipe? Although there was agreement in terms of the instruments that should be there (i.e., instrument 110 should always be a bagpipe), there was a wide variation in the sounds themselves as these were reliant on the local synthesis or sample-based General MIDI (GM) sound set. So depending on the system or soundcard, you might have bad bagpipes or good bagpipes (at this point you need to take a leap of faith to imagine that such a thing as a “good” bagpipe sound actually exists).

Once again, the MIDI Manufacturers Association (the ones that coordinated the GM specifications in the first place) stepped in with a solution. Initially developed by the Interactive Audio Special Interest Group (IASIG), the DownLoadable Sound (DLS) format specification allowed composers to develop their own “Orchestras”/Sample banks that would be distributed alongside the “Score”/MIDI files to ensure that the music was heard exactly as intended. A very effective development tool appeared from Microsoft called DirectMusic Producer that allowed composers to create and prototype their music and so all was good in the garden of interactive audio. Until—what’s this? A shiny silver disc blocking out the sun?

The ability to stream relatively large music files from CD, then DVD, and now Blu-ray disc has had a huge impact on the development of interactive music. In trying to solve all the problems of interactive music in one swoop, DirectMusic Producer was to an extent a victim of its own complexity and few managed to master its depths. Streaming was, and is, popular for good reason as it gives you absolute consistency, arguably a better sound quality (you can’t “master” a MIDI + DLS track in the same way as you can master a stereo or 5.1 track in a good studio with expensive racks), and most importantly, musicians.

The qualities that musicians bring to a performance have been rightly treasured over centuries of human existence and the complex interactions of an ensemble of players are never going to be replicated within a MIDI file, no matter how careful the programming and articulations. Although it’s possible to produce fantastic and realistic-sounding instrumental scores with MIDI and good-quality sample sets, there’s a reason why people still pay thousands of pounds or dollars an hour to record a professional ensemble in a top-quality studio (and no, it’s not just for the marketing department). We’re not going to argue that it’s anything but entirely appropriate that some music is written and used in this way, that sometimes the gains in musicianship and ensemble playing are worth the losses in terms of flexibility.

But where’s the alternative?

Where you still have significant memory or bandwidth restrictions (such as on the Wii, NDS, or mobile phone) or the nature of the music-based game demands note-level granularity, the techniques of MIDI + DLS are still alive, but within the mainstream of game development this technology has been rendered obsolete and its advantages (flexibility = interactivity) dismissed. You can do a lot with streaming wavs (as we’ve demonstrated in some of the examples), and the makers of current middleware tools such as FMOD and Wwise are continually developing better systems for their use, but there should also be an alternative available for composers to develop more granular interactive scores. Why do we struggle today to get the same degree of interactivity that the team at LucasArts had in the early 1990s? The iMuse system had/has most of the answers when it comes to interactivity and music, but 20 years later there is still no universal platform for writing and testing interactive music without having to first render it out as inflexible wavs.

Where we go from here obviously remains to be seen, but it has already been noted that the processing power of even the current generation of consoles should theoretically allow us to effectively run a DAW/sequencing environment in real time. Others have suggested that the answer lies in leapfrogging over any technical constraints within the console itself to produce the music in the “cloud”. Data from the game would be transmitted via the Internet to dedicated servers that could handle the processing of the music and then stream back a complete 5.1 mix. The IASIG continues to develop the iXMF format (Interactive eXtensible Music Format), which aims to be an open standard cross-platform solution to interactive audio that encapsulates both the implementation instructions and the audio data itself (using audio files or MIDI + DLS). Although the design principles of this format are now well defined and we can learn a great deal from the lessons of iMuse, DirectMusic Producer, FMOD, and Wwise, a composer-friendly composition and authoring tool remains the missing link. The question is where this will come from?

The game industry is significant enough now that the big boys and girls of the DAW world must be starting to pay attention, and perhaps even by the time this book reaches the shelves the solution may be out there. In the meantime, we hope you can hone your skills through experimentation with some of the systems we’ve included here.

One area where authoring tools of a type are beginning to emerge is in the genre where the music is the game.

(To get a taste of some old-skool approaches, check out the MIDI + DLS tutorial on thegameaudiotutorial.com website).

Music-Based Games

The nature of many music-based games is that the player has to attempt to perform the music. The player’s success is usually based on the timing of the input or, in the case of microphone-based games, both the timing and the degree of pitch accuracy. The input does not just affect the player’s score but also frequently affects what elements of the music are heard, with a correct “note” leading to the music being played as intended or an incorrect one playing back a bum note or error sound.

Given the note-level interaction with the music, most of these games use some kind of MIDI and sample-based playback or advanced stem muting/unmuting system. Some of these games are simply about rhythmic matching for fun and some are driven by the aim of making music creation more accessible to people who are not trained musicians. You can get a good feel for how the rhythm action processes work by creating your own interactive music track for “Rock Band” using the plug-in for the DAW Reaper (see the bibliography for a link).

There is another range of game applications that take a more experimental or abstract approach to both music and interaction. The most interesting ones lie somewhere in the middle.

Given their highly specific nature, we are not going to cover these in depth here but would recommend taking a look at some of the examples that follow together with those referenced in the online Bibliography for this chapter in order to get an idea of the range of possibilities in this area.

(The “Peter” and “Piano Master” systems discussed in this section were developed by two of our students on the MSc. Sound and Music for Interactive Games course at Leeds Metropolitan University. Thanks to Andrew Quinn and Phil Briddon for allowing us to use them here.)

Simon

414 Music Game 01: Peter

This room contains a system inspired by the well-known Simon game of 1978. In this game, the player needs to accurately recall and replay an increasingly long sequence of tones attached to the colored areas.

Each colored area of the game has a [DynamicTriggerVolume] placed around it that generates a [Take Damage] event in Kismet. This toggles a light around the area and plays back a musical tone. Within the Kismet patch there are 32 subsequences that record the randomly generated sequence to play it back each time and to check whether the player has correctly matched the order of sound and lights.

The randomized element of this game is what makes it particularly unique. There are still many games within the rhythm action genre that rely on memory (or a combination of memory and reactions) but mostly within a musical context and with fixed and repeating patterns.

Piano Hero

414a Music Game 02: Piano Master

In this rhythm action type game system, you have to shoot the keyboard to match the timing of the descending notes. Good timing will reveal a well-known tune.

A series of [Delay]s generates the sequence of notes for the tune. As each note is triggered, it starts a series of lights that indicate to the player when the note is coming.

The player input (shooting) is detected by a [DynamicTriggerVolume] around each note. In FreePlay mode, the note is allowed to pass through the bottom [Gate] directly to a [PlaySound] for that note.

In the game mode, the final light opens the top [Gate] to detect the correct note at the correct time (GoodNote), and the Dead Note [Gate] is temporarily closed. Both the Goodnote and the Deadnote are passed to a scoring system to subtract or add [Int]s to the total score.

Abstract Music “Game”

414b Music Game 03: Abstract

This room is an illustration of some of the more abstract things you can do within a game environment that don’t necessarily fit with the definition of a game (there are no specific goals or rules) or music (there is no organization or predetermined system of musical events) but instead fall somewhere in-between.

On entry, the player picks up a new weapon that has been adapted to shoot out projectiles that will bounce around the room. As they do so, they will hit the different colored panels on the walls, floor, and ceiling. These are attached to musical notes so the player can construct a piece of music through layering multiple bouncing sequences.

Each panel is encapsulated by a [TriggerVolume], the properties of which have been set to detect the weapon’s projectile via a touch event (Block Complex Collisions, Block Zero Extent), which then plays a [SoundCue] via a [PlaySound]. The musical tones have been selected so that they can play in any combination without being dissonant.

In the UTWeap_Musical, UTProj_MusicProjectile, and UTAttachment_Musical script files, the weapon fire has been altered and the velocity, damping, and lifespan variables to make the projectile bounce and last for longer than its default settings.

Conclusions

For most games there should not be a music system, there should be music systems. These should work in tune with the particular requirement of the circumstances of the game within different scenarios. One answer to the challenge of interactive music is certainly a deeper integration with the game design itself. This will require producers and designers to understand what music can bring, and how it operates. It should inform the design of the game, not simply be an add-on to it. Sharing this understanding and making this happen is your job.

Although this might seem like a contradiction given the contents of this chapter, we would urge you not to get obsessed about interactive music in games. This is one of our options, and it’s an option that we can’t yet satisfactorily fulfill, but remember that it is on one end of the continuum of what music can do (see section presented earlier, “It’s a fine line between love and hate”). Representing the larger scale narrative or playing against the action on screen could also be options and should have their place. Taking the contradictions one step further still, you will also remember that it is part of your duty to your craft, and to the game, to sometimes argue for no music. As powerful an effect as music can have by being there, it can have an equally powerful effect by not being there. We’ll leave you to ponder that thought.

(If we could have embedded the sound of a Tibetan gong into the book at this point, then we would have.)