Figure 7.1 © Andrew Magil (www.ominoushum.com).
Chapter 7
Fixing Audio Problems
“This is what it sounds like when doves cry.”
—Prince
In many ways, dealing with the audio content is much easier than dealing with the picture content of a production. There are fewer concerns: after all, humans are primarily visual, and ultimately our perception of a TV show or movie is driven more by what we see than what we hear. Structurally, constructing audio involves much less complexity than constructing moving images. Simply put, there is less that can go wrong.
However, there are also fewer tools available for working with audio. There's nothing to compare with the suite of color correction processes available when working with images, and certainly no real equivalent to the ubiquitous clone tool, used for hiding a multitude of visual glitches. On the other hand, there are lots and lots of filters for doing different things, but just as you wouldn't simply use a sketch video filter to make an animated feature, the vast majority of audio filters are best kept at arm's length.
I also don't want to give the impression that dealing with postproduction audio is in any way easy. There are a number of skilled people who are involved with audio on a typical production—the dialogue editors who painstakingly build fluid speech from salvaged scraps of vowels and consonants, the sound effects editors responsible for all the sounds that you'd miss if they weren't there, the mixers who take everything and make it all work together harmoniously. In fact, the process of producing great audio goes far beyond the scope of this book, let alone this chapter, which will limit itself to dealing with some of the more common audio-related problems that can arise during the mastering phase of postproduction.
Digital and Analogue Audio
Audio consists of a number of overlapping wavelengths (or sound waves, if you prefer) at different frequencies. The frequencies combine to form the unique sounds that you hear (different instruments, for instance, are said to primarily occupy specific frequency bands). Audio is primarily an analogue format, much more so than images. Unlike digital video, which can be both created and displayed on digital devices, audio must be created in an analogue form, and must be transmitted as analogue in order for us to hear it (no matter how expensive your MP3 player is, by the time the sound reaches your ear, it is just vibrating air).
Digital audio is therefore problematic for a couple of reasons. First, it potentially contains all the same problems as its analogue counterpart, and second, the fact that it has been digitized opens up a whole new set of problems.
The analogue issues are usually due to something having gone wrong during recording, such as the volume being too low or the microphone popping. The digital issues are similar to those faced on the picture side, such as aliasing (undersampling), noise, and corruption.
Anatomy of Digital Audio
Digital audio is created by sampling an analogue waveform. These samples are taken at regular intervals across a time period (determined by the sampling frequency, which is typically measured in kilohertz, or thousands of times per second). The amplitude of each sample is then quantized at a specific bit resolution. If it helps to think of it in digital imaging terms, the sampling frequency is analogous to the pixels (and the corresponding resolution of the image), while the bit resolution is like the bit depth (and the corresponding color range). The combination of these two parameters can be broadly classified as the fidelity of the audio.
Digital audio can also comprise multiple channels. For example, stereo sound consists of a left channel and a right channel. Surround sound can consist of more, and in certain formats and workflows, extra channels are often used to store additional audio for other purposes (though in general, it's considered good practice to always store each channel in separate files, even for stereo recordings).
Figure 7.2 An analogue waveform.
Figure 7.3 Anatomy of digital audio.
Decibels and Dynamic Range
Our ears have a nonlinear response to sound. That means that a sound that is measurably twice as loud as another one will not be perceived as being twice as loud. To make things simple, therefore, we measure audio level on a logarithmic scale, typically using decibels (dB). A bel is a unit of loudness; each unit is ten times louder than the previous one (a decibel is one-tenth of a bel).
In the digital realm, loudness is measured in decibels full scale (dBFS). Confusingly, this scale (as with most audio scales) is measured in reverse, with 0 being the maximum allowable value (anything above 0 will clip, becoming irreparably distorted). Audio engineers usually try to keep 3–6 dB of headroom clear (allowing for sudden peaks in the signal to exceed the expected levels without distortion), so the typical acceptible working range for digital audio centers around −6 dBFS.
The dynamic range of an audio sample is the ratio of the loudest part to the quietest part. For example, operatic music (which can go from a silent whisper to a glass-shattering shriek) has a very high dynamic range, while a jet engine, although very loud, would have a much narrower dynamic range by comparison.
High dynamic range is not necessarily preferable to low dynamic range in audio. For example, the high dynamic range of operatic music works well in the controlled environment of an opera house, but would not work so well against the backdrop of a busy street, for example. Using compression and expansion processes on digital audio can decrease and increase the dynamic range of the sample accordingly.
Figure 7.4 A digital dBFS meter.
How to Get Consistent Volume
Volume inconsistency is the most basic audio problem to solve, as it can be noticed by even the most untrained listener.
There are two parts to the problem of audio volume. The first is technical: the first step to any aspect of consistent audio is to properly meter it. The second is more subjective: the perception of audio is determined to a large degree by its context.
1. Load the audio sequence.
a. If there is just a single clip with inconsistent volume, then the problem is almost certainly a technical one.
2. First play through the sequence, carefully monitoring the audio levels as you go.
a. You'll need to rely on your ears as well as any visual aids, such as dBFS meters (volume levels should peak at around −6 dBFS and should never reach 0 dBFS).
b. If there is more than one audio track, you should check them independently of each other for any obvious problems first.
3. If the levels vary greatly between clips, adjust the volume of each clip so they are consistent.
a. Try not to exceed −6 dBFS, where possible, and never peak at 0 or the sample will distort.
4. If the levels vary within specific clips, you will need to dynamically adjust the volume within the clip itself. Displaying the waveform of the clip at this point can help identify the areas of volume shift.
5. If the volume is still inconsistent, it is probably a perceptual shift caused by surrounding audio clips.
6. Check that all the clips in the sequence have the same bit resolution, as higher bit resolutions can produce perceptually higher volume output, even though the meters may not register a change.
7. If all else fails, you can reduce the perceptual volume of the affected area by increasing the volume of the audio immediately preceding it, or vice versa (this can be especially effective with sequences using multiple tracks, as changing the volume of one of the other tracks can soften the perception of a volume shift).
8. Save the changes.
Audio Monitoring and Calibration
With experience, your ears will prove to be your most reliable equipment for monitoring audio. That said, your ears can only be as reliable as the sound you feed into them, which means it's vital that you provide them with accurate audio in the first place. Good-quality speakers are important, but there are many other factors that have to be considered. The sound you hear comprises the sound coming out of the speakers and the acoustics of the room, as well as any ambient noise.
All of these aspects need to be kept in check, if not directly controlled. It's also advisable to listen to audio in an environment similar to the one in which the audience will hear it. For theatrical productions, this means going into a dedicated screening room or dubbing stage whenever possible, rather than relying on the output of an editing system. Once there, you may discover new problems that simply weren't detectable before.
How to Retime Audio
Retiming audio is fairly simple to do with the right tools.
1. Load the audio into a sequencer.
2. If the audio does not need to sync to the accompanying picture (such as atmospheric sound), then you may be able to just trim or loop portions of the audio to get the desired duration. This may be preferable, as there will be no degradation when using this approach.
3. If your editing system has the capability to time-stretch the audio, then this is probably the best option to use, particularly with dialogue, as it will change the speed without also adjusting the pitch. Some systems timewarp audio in addition to time-stretching it, which will give you greater control over the process. Set the desired amount of stretching to get the new duration, and save the new audio clip.
a. If your system does not have the means to time-stretch audio, then you must pitch-shift the audio in order to retime it. This will, of course, also result in changing the pitch of the audio, but it may not be noticeable for very small changes.
How to Correct Sync Issues
Follow these steps to correct many types of sync issues.
Getting correct audio sync is simple in theory—you simply line up the audio with the corresponding picture. In practice, though, dealing with sync problems can be a very frustrating experience. Spotting sync problems in the first place is a bit of an art form in itself, and is something that gets easier with experience. The good news is that the really bad sync problems will be very noticeable. The other point to bear in mind is that you need to pay attention to the sync of all audio elements, dialogue, foley, and atmosphere.
1. Load the conformed sequence into a sequencer.
2. Play through it, identifying where the sync issue occurs.
a. Make note of the difference (in frames) between the audio and the picture.
b. If the audio sync gets progressively worse over the course of a single shot, this is a good indication that the speed of the audio track doesn't match the picture content. This may mean that the picture is at the wrong frame rate or that the audio sampling frequency is wrong for the project. If changing the sampling frequency doesn't work, you'll need to either retime the picture or adjust the audio pitch.
c. If the sync is fine on some shots but out on others, split the audio tracks into individual clips to separate the good ones from the bad. You'll then be able to leave the good ones unmodified while fixing the others.
3. First ensure that the conform itself is correct.
a. Check the conform against any reference material available, such as an EDL or offline cut. Make sure that the cut points all line up correctly and that the timecodes match the edit.
b. If the conform is found to be wrong, address the issue and see if there is still a sync problem.
4. Now you must decide whether to adjust the picture or the audio. Make this decision by factoring in what's possible: for example, you might not have access to the original picture footage; and by what's least destructive: for example, it makes more sense to change the audio speed slightly than to retime an image sequence.
5. To adjust the sync, you have several options (modifying either the video or audio as necessary):
a. You can slip the problem shots by the required number of frames to get them in sync.
b. You can ripple-edit (trim or extend) the start of the first shot with the issue by the required number of frames (and if need be, ripple-edit the end of the last problem shot so that the overall duration remains unchanged).
c. You can retime the shot (pitch-shift if modifying the audio) so as not to affect the position of cuts or change the overall duration of the sequence (for information on retiming shots, refer to Chapter 10).
6. After making changes, verify that the entire sequence is in sync. Sometimes when fixing sync problems, you can indirectly affect the sync of other shots in the sequence, so it's important to check everything again before outputting the result.
How to Reduce Audible Noise
There are several methods available for reducing noise in audio.
Electronic noise is one of the biggest problems encountered in audio postproduction. It's caused by any of a number of factors, such as the cables and tapes used during recording, imperfections in the components used for recording or digitizing audio, or even data corruption. As with digital image noise, it cannot be completely removed, but you can take steps to reduce the audible impact. Bear in mind that the whole process of noise reduction can easily be taken too far; you should never sacrifice audio detail in your mission to remove noise.
TIP
When dealing with audio sync problems (as with many other problems covered in this book), you should apply Occam's razor: the simplest solution is usually the correct one. The less you modify the original, the better the end result will be (and the less work you'll have to do).
1. Load the audio into a sequencer.
a. Also load alternate takes, if possible, to see whether there are less noisy sections from other audio clips you can scavenge (and clean audio segments, such as room tone, ambient sound recorded on location).
2. Identify the properties of the noise. Is it fairly consistent? Does it overpower the main recording, or is it only audible over silent portions of the audio? Is it high-pitched (like hiss) or low-pitched (like rumble)?
3. For noise that you can only detect over silence, consider just lowering the volume of the silent regions. You'll probably need to cross-fade these volume changes so that the transition is not so brutal. It may be that this process will only draw more attention to the noise within the nonsilent portions, but sometimes this simple approach can work very well.
a. An even better solution is to replace the silent portions with clean room tone rather than dipping the volume.
4. To reduce high-pitched noise, you can use a low-pass filter. Adjust the settings until the filter only just removes the majority of the noise. If it also affects other sounds in the recording, you will have to use a different approach.
5. Similarly, to reduce low-pitched noise, use a high-pass filter. As with the low-pass filter, adjust it until it only just affects the noise, and don't use it at all if it interferes with other sounds in the recording.
6. If at all possible, ultimately the best option is to try to rebuild the audio by cutting together fragments from alternate recordings that don't have the same degree of noise. This can be hard work, and probably is not possible with audio that has been completely mixed when supplied to you, but this option should always be strongly considered.
7. If all else fails, the only other method is to use dedicated noise reduction software.
a. You'll probably get the best results by applying the noise reduction process incrementally, fixing the noise in stages, rather than using a single application.
8. Save the resulting audio clip.
How to Remove Pops and Clicks
Removal of pops and clicks is best done manually.
Pops and clicks embedded in a soundtrack may be the result of a glitch during recording or some other postprocess, or part of the soundtrack itself (such as if the actor speaking dialogue has dentures). Most of the time, such noises are distractions, and so it's advisable to remove them, provided of course that doing so results in an overall improvement to the scene (on odd occasions, the noises can add atmosphere, and so can be left alone). If the noise is particularly bad and there are no suitable alternate takes you can use, your only recourse may be to get it remixed (or if it's on a dialogue track, to use ADR).
TIP
Noise is probably the single best reason to keep analogue recordings for future reference. Although the convenience of digital audio for archiving purposes cannot be understated, the continuous improvements in analogue-to-digital conversion mean that the ability to make better-sounding digital versions from analogue recordings is always just around the corner.
1. Load the audio into a sequencer and identify the point where the noise to be removed can be heard.
2. Display the waveform for the audio, zooming in as close as possible. You want to find the point where the noise can be seen as part of the waveform. Typically, this will appear as a dip or a peak in an otherwise smooth region (with experience, these points will become much easier to spot).
3. If the affected region is quite small, you should be able to fix the problem using a pen, or similar audio editing tool, by drawing over the region of the waveform where the noise occurs, smoothing it out. Depending upon the type of noise you're removing and the content of the recording itself, the noise may now be completely removed without a trace, and you're done. Sometimes, though, this process can make the situation worse, in which case you can revert back to the unmodified version.
4. In all other cases, the noise can usually be removed by replacing it with similar-sounding material—much in the same way you'd use the cloning tool to remove dust from a film frame (although with a little bit more effort on your part).
5. Identify a section of matching sound without the distracting noise and copy it. If the noise occurs on an otherwise silent section (for example, between two words of dialogue), you can probably just copy some recorded room tone (or some other silent portions you can salvage from the track).
6. Paste the copied audio onto the affected area. You'll probably need to cross-fade either side of the edited section so there is no audible transition.
7. Save the resulting audio clip.
TIP
Many audio editing applications have filters for automatic pop and click removal. However, even the most sophisticated algorithms are no match for a human with the pen tool. Such filters are best left for the occasions when time is limited and the damage is severe.
Stereophonic and Multichannel Audio
We perceive the origin of sounds in space based on the differences between what each ear hears and on the differences between sound that reaches us directly and sound that reaches us indirectly (such as sound that reflects off of walls and other surfaces). Stereophonic sound (or stereo), which directs one track of audio to the left ear and the other to the right ear, therefore appears to have much more depth than just monoaural (single track) sound. Almost every type of sound production uses at least stereo sound, from television shows to music CDs.
Beyond stereo sound is surround sound. This combines multiple tracks to give the impression of sound originating from specific locations in space. There are several different standards available for reproducing surround sound, each with a corresponding configuration of speakers, but by far the most common is 5.1 surround. This comprises five speakers in a circle around the listener, with an
Figure 7.5 Earth © BBC Worldwide Ltd 2007.
additional low-frequency emitter (LFE or subwoofer) to provide bass. Surround sound is commonly used in theatrical releases, and many home systems now have some form of surround sound capability.
How to Get Stereo from Mono
Making sound into stereo is easy; making it sound like it is stereo is a little bit more involved.
1. Load the mono audio track into a sequencer.
2. Add a second track and copy and paste the mono track into it. Depending on your particular system, you may have to adjust the audio patching so that one track is denoted as the left channel and the other is the right one (some editing systems make this assumption automatically).
a. At this point, you technically have a stereo soundtrack, but as there is no difference between the two channels, it will still sound mono.
3. The next stage is to artificially introduce subtle differences between the two channels so that it creates the illusion of stereo sound.
a. Use a comb filter to offset one of the channels by around 50 milliseconds (this will be small enough so that you will register the difference on only a subconscious level).
b. Use shelving filters to direct different frequencies to either channel.
c. Pan the audio left and right to give the impression of a moving source (this is particularly effective for shots where there is camera motion).
d. If your editing system has surround panning capabilities, you can get good results by converting the project to surround sound, using the surround panning tools, and then mixing back down to stereo.
4. Save the new clip.
The M/S Audio Model
A powerful approach to working with stereo audio is to use mid-side encoding. This converts the left and right channels into sound that can be heard in the middle and sound that can be heard at the sides by creating a track with audio common to both left and right channels (forming the mid channel) and a track featuring the differences between the two (forming the side channel). The process originates from recording audio using a combination of a cardioid and a side-facing (figure-8) microphone, but can be replicated fairly well using digital processes.
The benefit to this approach is that rather than panning left to right, you can effectively pan from the center to the sides, processing each seperately from the other, in a way that would be difficult to achieve with a standard stereo mix. For example, you can use this approach to change the perceived distance of a sound from the listener by adjusting the M/S ratio. In addition, it can also isolate specific types of sound, such as isolating dialogue from background noise.
How to Get Surround Sound from Stereo
There are a few approaches you can take to produce surround from stereo.
1. Load the audio into an editor sequencer capable of working with 5.1 surround sound.
2. Depending on the material you have to work with, you can produce a surround mix in a number of ways.
a. If you have a stereo mix with separate foley elements, you can keep the mix in stereo and surround pan the other elements to match their location in screen space.
b. If you have separate tracks for each element, you can surround pan them individually to get the desired effect.
c. If you have a single stereo mix, you can use a combination of low-pass and high-pass filters to separate the audio frequencies and work on them separately. For example, you can redirect low-frequency sounds (such as the rumbling of trucks across gravel) to the LFE channel.
3. Depending upon the capabilities of your sequencer, you may have several options available to refine the sound.
a. Most systems are able to pan tracks in 2D space (as opposed to just left and right panning available for stereo). This essentially raises or lowers the volume at each speaker relative to the panned position of the track. This can be used to give the impression of the sound being located at a particular point in space. You can also dynamically pan the audio to give the impression of a moving audio source.
b. There will usually be a separate control for LFE, which determines the amount of audio that is directed to the LFE channel rather than the surround speakers. Naturally, increasing this control will result in more bassy, delocalized sound.
c. There may be the option to adjust the width of the sound across channels. This causes the sound to be diverted to more of the speakers, resulting in sounds being delocalized. By the same token, by narrowing the width, you make the sound seem to originate from a more specific point in space.
d. It may be possible to patch sound directly into a specific channel. This can be useful if, for example, you want to patch all dialogue tracks into the center channel.
4. Save the audio. Note that this will usually entail saving it in a surround-encoded format, although you may be able to output a separate file for each speaker channel.
TIP
It's advisable to keep dialogue in the center as much as possible, rather than panning it to match the location of the actors in screen space, or else it can become distracting to the audience.
How to Automate Dialogue Replacement
There are a number of reasons why automatic dialogue recording (ADR) can be necessary, such as audio that is so distorted it is beyond repair, bad takes, script changes or additions, or even due to wanting to get a different performance from the actor.
The ADR process is not as automatic as its name suggests
Although there are systems dedicated to ADR, you can recreate the workflow in most editing systems with a little preparation.
1. Load the sequence for ADR into an editor sequencer.
2. If more than one line needs to be replaced, split up the track so that each line to be recorded is separate. You should also remove sections that are unnecessary (you can keep a short amount of footage before the line to be replaced to act as a cue for the actor).
3. Duplicate each clip, positioning the duplicate so it runs immediately after the original. Remove the audio from the duplicate.
4. Cut four beeps, evenly spaced so that the fourth beep occurs immediately where the recorded line is to begin (on the duplicate clip).
5. Remove the fourth beep.
6. With the setup complete and the actor present, mark the first pair of clips to run in a loop and record the actor until the result is satisfactory.
7. With certain systems, you will be able to record directly into the playback system, which will allow you to audition the result right away. Otherwise, you will need to import the recorded audio into the editor and cut it into takes, auditioning each one in turn.
8. Repeat with other lines until you're done.
9. Save each new take separately, and then cut the desired takes into the main sequence. Be prepared to edit more than one take to replace a single line.
TIP
ADR is sometimes referred to as looping, although this actually refers to the process of the actor trying to sync lines against a looped section of audio, rather than trying to speak in a way that ensures continuity with what's on screen.
This approach (and the ADR process in general) can and should be adapted to make the actor most at ease. For example, some actors find the rhythmic beeps distracting (in which case, cut them out); some like to record while listening to their original lines (in which case, paste the audio back into the duplicate clip and give them a pair of headphones); and some prefer to simply work from a script. Likewise, some actors like to be involved in the process of listening back to each take, while others don't.
How to Ensure Audio Continuity
Continuity is only noticeable in its absence, especially in the case of audio.
1. Load an entire scene into a sequencer.
2. Play through the scene, noting any potential problems.
a. Play through it at least once without picture sync. This will ensure that you're able to detect problems that are masked to some extent by action on the screen.
b. Pay attention to level readings, particularly the peak level indicator, which should be fairly consistent across most scenes.
c. If it's a surround mix, you should also listen to it in stereo to make sure there are no obvious problems when it gets mixed down.
3. If the audio levels fluctuate over the course of the scene, refer to the process on page 112 for getting consistent volume.
4. If there are differences in the tone and quality of the sound (but not in the audio levels), use compression and expansion processes to make the tone of different sections match. You may need to apply other filters, such as reverb, to make the audio appear to have originated in a single, specific location.
5. Make sure that background and ambient noise is consistent throughout the scene. For example, if the scene is set in a factory, do the mechanical processes in the background all occur at regular intervals? If not, you may need to replace, trim, or loop sections to make it sound more natural (and therefore, more subconscious). Also, be sure that there are sounds to cover everything that happens on-screen (and off-screen). For example, if someone leaves a room, are there accompanying door sounds, even if we don't see the door?
a. Be sure to avoid looping sounds in a way that sounds artificial, such as the same sound of a barking dog repeated over and over.
b. Occasionally, it's desirable to make background sound inconsistent. For example, when something pivotal is happening (or about to happen) in the foreground, it is common to reduce all other sounds to create an aural focal point.
6. Make sure dialogue is clearly audible at all times.
a. Again, you can make exceptions if it's in keeping with the storyline.
7. Check that the audio is in sync with the picture. Refer to page 114 for specific details on checking the sync.