9. Using the Roto Brush Tool
Lesson overview
In this lesson, you’ll learn how to do the following:
Extract a foreground object from the background using the Roto Brush tool.
Correct the segmentation boundary across a span of frames.
Touch up a matte with the Refine Edge tool.
Freeze a matte across a clip.
Animate properties for creative effects.
Track a face in footage.
This lesson will take about an hour to complete. If you haven’t already done so, download the project files for this lesson from adobepress.com/AfterEffectsCIB2023, following the instructions in the Getting Started section under “Accessing the lesson files and Web Edition.”
PROJECT: TV COMMERCIAL
With the Roto Brush tool, you can quickly separate a foreground object from a background across many frames. You can achieve professional results in a fraction of the time you’d spend performing the same task with traditional rotoscoping.
About rotoscoping
When you draw or paint on the frames of a movie, you’re rotoscoping. For example, a common use of rotoscoping is to trace an object, using the path as a mask to separate it from the background so you can work with it separately. Traditionally in After Effects, you drew masks, animated the mask paths, and then used the masks to define a matte. (A matte is a mask used to hide part of an image so that another image can be superimposed.) While effective, this is a time-intensive, tedious process, especially if the object moves a great deal or the background is complex.
If a background or foreground object is a consistent, distinct color, you could use color keying to separate the object from the background. If the subject was shot against a green or blue background (green screen or blue screen), keying is usually much easier than rotoscoping. However, keying is less efficient when you’re working with complex backgrounds.
The Roto Brush tool in After Effects is much faster than conventional rotoscoping. You use the Roto Brush tool to define the foreground and background elements. Then After Effects creates a matte, and tracks the movement of the matte over time. The Roto Brush tool does much of the work for you, leaving only a little cleanup work to be done.
Getting started
In this lesson, you’ll use the Roto Brush tool to isolate a young boy from a wet winter background so you can color-treat the background without affecting the boy. To finish up the project, you’ll add an animated title.
First, you’ll preview the final movie and set up your project.
Make sure the following files are in the Lessons/Lesson09 folder on your hard disk, or download them from adobepress.com now:
In the Assets folder: boy.mov, Facetracking.mov
In the Sample_Movie folder: Lesson09.mp4
Open and play the Lesson09.mp4 sample movie in Windows Movies & TV or QuickTime Player to see what you will create in this lesson. When you are done, close Windows Movies & TV or QuickTime Player. You may delete the sample movie from your hard disk if you have limited storage space.
When you begin this lesson, restore the default application settings for After Effects. See “Restoring default preferences” on page 3.
Start After Effects, and then immediately hold down Ctrl+Alt+Shift (Windows) or Command+Option+Shift (macOS). When prompted, click OK to delete your preferences.
Click New Project in the Home window.
After Effects opens to display an empty, untitled project.
Choose File > Save As > Save As.
In the Save As dialog box, navigate to the Lessons/Lesson09/Finished_Project folder.
Name the project Lesson09_Finished.aep, and then click Save.
Creating the composition
You’ll import a footage file and create a composition from it.
Click New Composition From Footage in the Composition panel.
Navigate to the Lessons/Lesson09/Assets folder, select the boy.mov file, and then click Import or Open.
After Effects creates a composition named boy, based on the settings in the boy.mov file. The composition is three seconds long, with a frame size of 1920 x 1080. The movie file was shot at a rate of 29.97 frames per second.
Choose File > Save to save the project so far.
Creating a segmentation boundary
You use the Roto Brush tool to specify which parts of the clip are in the foreground and which are in the background. You add strokes to distinguish the two, and then After Effects creates a segmentation boundary between the foreground and background.
Creating a base frame
To use the Roto Brush tool to isolate a foreground object, you start by adding strokes to a base frame to identify foreground and background areas. You can start on any frame in the clip, but in this exercise, you’ll use the first frame as the base frame. First, you’ll add the strokes that identify the boy as the foreground object.
Move the current-time indicator across the time ruler to preview the footage.
Press the Home key to move the current-time indicator to the beginning of the time ruler.
Select the Roto Brush tool (
) in the Tools panel.
You use the Roto Brush tool in the Layer panel, which you’ll open now.
Double-click the boy.mov layer in the Timeline panel to open the clip in the Layer panel.
Choose Fit from the Magnification Ratio pop-up menu at the bottom of the Layer panel if you don’t see the entire image.
By default, the Roto Brush tool creates green foreground strokes. You’ll start by adding strokes to the foreground—the boy. Generally, it’s most efficient to start with broad strokes and then use smaller brushes to refine the border.
Choose Window > Brushes to open the Brushes panel. Set up a hard round 100-pixel brush. (You may need to resize the Brushes panel to see the options.)
When you’re drawing strokes to define the foreground object, follow the skeletal structure of the subject. Unlike traditional rotoscoping, you don’t need to define a precise boundary around the object. Start with broad strokes, and work down to small regions as After Effects determines where the boundary is supposed to be.
Draw a green stroke from the boy’s head down toward the bottom of the clip.
TipYou can quickly zoom in and out of the Layer panel using the scroll wheel on your mouse, if it has one.
A magenta outline identifies the boundaries After Effects created for the foreground object. After Effects recognized only about half of the boy, because you initially sampled only a small area of the subject. You’ll help After Effects find the boundary by adding some more foreground strokes.
Still using a large brush, draw a green stroke from left to right across the boy’s coat, including the black strip on the right.
Use a smaller brush to add any neglected areas to the foreground.
TipTo increase or decrease the brush size quickly, press Ctrl (Windows) or Command (macOS) as you drag. Drag right to increase the brush size; drag left to decrease it.
It can be tricky to get some areas without accidentally adding background as well. It’s okay if you haven’t captured every detail in the foreground. You’ll use background strokes to remove any extraneous areas of the matte.
Press Alt (Windows) or Option (macOS) to switch to the red background stroke brush.
Add red strokes to background areas you want to exclude from the matte. Switch back and forth between the foreground and background brushes to fine-tune the matte. Don’t forget to deselect the area under the boy’s hat where the background shows through. In fact, one click may be all it takes to exclude that area from the matte.
Don’t worry about being exact about your brush strokes. Just make sure the matte is within 1 to 2 pixels of the edge of the foreground object. You’ll have an opportunity to refine the matte later. However, After Effects uses the information on the base frame to adjust the matte for the rest of the span, so you want the matte to be accurate.
Click the Toggle Alpha button (
) at the bottom of the Layer panel. The selected area is white against a black background, so you can see the matte clearly.
Click the Toggle Alpha Overlay button (
) at the bottom of the Layer panel. The foreground area appears in color, and the background has a red overlay.
Click the Toggle Alpha Boundary button (
) at the bottom of the Layer panel to see the outline around the boy again.
As you use the Roto Brush tool, the Alpha Boundary is the best way to see how accurate your boundary is, because you can see everything in the frame. However, the Alpha and Alpha Overlay options let you see your matte without the distraction of the background.
Refining the boundary across the span
You used the Roto Brush tool to create a base frame, which includes a segmentation boundary that divides the foreground from the background. After Effects applied the segmentation boundary across the entire length of the layer. The Roto Brush span appears below the time ruler at the bottom of the Layer panel. When After Effects has propagated the current boundary for a frame, the bar beneath that frame on the time ruler is green.
As you move forward and backward through the footage, the segmentation boundary moves with the foreground object (in this case, the boy). You’ll step through the frames in the span and make adjustments to the segmentation boundary as necessary.
In the Effect Controls panel, make sure Quality is set to Standard.
When Standard is selected, frames are processed at a lower resolution and the segmentation boundary propagates more quickly, but the matte may include fewer edge details. When Quality is set to Best, the matte will have more detailed edges, but the process will be slower. For most projects, Standard is sufficient.
Press the 2 key on your main keyboard (not the numeric keypad) to move forward one frame.
TipTo move forward one frame, press the 2 key on your keyboard; to move back one frame, press the 1 key.
Working from the base frame, After Effects tracks the edge of the object and attempts to follow its movement. Depending on how complex your foreground and background elements are, the boundary may or may not conform exactly to the area you hoped it would. As you progress through this clip, you may notice changes to the segmentation boundary along the boy’s right sleeve (the left edge of the clip) as more of the coat is revealed in the frame. Likewise, you may need to refine the segmentation boundary around the flaps of the cap and the edges of the hood.
Using the Roto Brush tool, paint foreground and background strokes to refine the matte for this frame. If the matte is accurate, you don’t need to paint any strokes.
NoteAs it propagates the segmentation boundary for a frame, After Effects caches that frame. Cached frames have a green bar in the time ruler. If you jump ahead to a frame further along the span, After Effects may take longer to calculate the boundary.
If you make a stroke you don’t like, you can always undo the stroke and try again. As you move through the span, each change you make affects the frames after it. The more you refine your selection, the better the overall results will be. You may find it useful to move forward a few frames to see how changes affect the boundary.
Press the 2 key again to move forward to the next frame.
Use the Roto Brush tool to add to the foreground or subtract from the background as necessary to refine the boundary.
Repeat steps 4 and 5 until you reach the end of the layer.
Pay special attention to the left earflap of the cap when it crosses in front of the tree. The dark areas overlap, making it more difficult to get a consistent edge. Remember that you are trying to get the segmentation boundary as close to the edge of the foreground object as possible.
When you have completed refining the segmentation boundary for the entire layer, choose File > Save to save your work so far.
Fine-tuning the matte
Roto Brush does a pretty good job, but there may be stray bits of background in the matte, or foreground areas that weren’t included. You’ll clean those up by refining the edge.
Adjusting the Roto Brush & Refine Edge effect
When you use the Roto Brush tool, After Effects applies the Roto Brush & Refine Edge effect to the layer. You can modify the effect using settings in the Effect Controls panel. You’ll use those settings to further refine the edge of the matte.
Press the spacebar to play the clip in the Layer panel; press the spacebar again to end the preview when you’ve seen the whole clip.
As you preview the clip, you may notice that the segmentation boundary area jumps around quite a bit. You’ll use the Reduce Chatter setting to make it smoother.
In the Effect Controls panel, increase the Feather amount to 10, and increase Reduce Chatter to 20%.
The Reduce Chatter value determines how much influence the current frame has when performing a weighted average across adjacent frames. Depending on how tight your matte was, you may need to increase Reduce Chatter to 50%.
Preview the clip again. Notice how much smoother the edge of the matte has become.
Using the Refine Edge tool
The boy’s coat and face have hard edges, but his cap is fuzzy, and the Roto Brush tool didn’t pick up the nuanced edge. The Refine Edge tool lets you include fine details such as wisps of hair in designated areas of the segmentation boundary.
Though it might be tempting to use the Refine Edge tool immediately after creating the base frame, it’s best to wait until you’ve refined the segmentation boundary across the entire clip. Because of the way After Effects propagates the segmentation boundary, using the Refine Edge tool too early results in a matte that is difficult to use.
Go to the base frame, which is the first frame of the clip, and then zoom in so that you can see the edges of the cap clearly. Enlarge the Layer panel if necessary, and use the Hand tool to move the layer so you can see the entire cap.
Select the Refine Edge tool (
), hidden beneath the Roto Brush tool in the Tools panel.
The cap is relatively soft, so a small brush size will work well. For a fuzzier object, you might have better results with a much larger brush. The brush needs to overlap the stray edges that emerge from the object.
Change the brush size to 10 pixels.
When you use the Refine Edge tool, draw strokes along the edges of the matte.
In the Layer panel, move the Refine Edge tool over the edge of the cap, straddling the segmentation boundary and including the fuzzy variations. You can use multiple strokes to move the tool around the entire cap.
When you release the mouse, After Effects switches to the Refine Edge X-ray view so that you can see how the Refine Edge tool changes the matte, capturing the detail in the edges.
Move the current-time indicator across the time ruler to view the edge. Where more of the hat is visible onscreen, extend the Refine Edge boundary to include the top areas of the hat. Just as with the Roto Brush tool, press Alt or Option to use the tool to remove areas from the matte if necessary.
Zoom out to see the entire scene, resize the Layer panel if you enlarged it, and then choose File > Save to save your work.
NoteUse the Refine Edge tool only after you have cleaned up the matte across the entire clip.
Freezing your Roto Brush tool results
You’ve put a fair amount of time and effort into creating the segmentation boundary across the entire clip. After Effects has cached the segmentation boundary so it can recall it without having to make the calculations again. To keep that data easily accessible, you’ll freeze it. This reduces the processing demands on your system so you can work faster in After Effects.
Once the segmentation is frozen, you cannot edit it unless you unfreeze it. Refreezing the segmentation is time-consuming, so it’s best to refine the segmentation boundary as much as possible before freezing.
Click the Freeze button in the lower right area of the Layer panel.
Depending on your system, this may take a while.
After Effects displays a progress bar as it freezes the Roto Brush and Refine Edge tool data. Freezing may take a few minutes, depending on your system. As After Effects freezes the information for each frame, the cache line turns blue. When it has finished freezing, a blue warning bar appears above the time ruler in the Layer panel, reminding you that the segmentation is frozen.
Click the Toggle Alpha Boundary button (
) in the Layer panel to see the matte. Then click the Toggle Transparency Grid button (
). Move the time marker across the time ruler to see the subject without the distractions of the background.
Click the Toggle Alpha Boundary button again to see the boundary.
Choose File > Save.
After Effects saves the frozen segmentation information with the project.
Changing the background
There are many reasons to isolate a foreground image from a background. Often, you want to replace the background entirely, moving the subject to a different setting. But rotoscoping is also useful if you simply want to change the foreground or background without modifying the other. In this lesson, you’ll make the back-ground blue to enhance the wintry theme and help the subject stand out.
Close the Layer panel to return to the Composition panel, and then move the current-time indicator to the beginning of the timeline. Choose Fit from the Magnification Ratio pop-up menu at the bottom of the Composition panel.
The Composition panel displays the composition, which includes only the boy.mov layer, consisting only of the foreground you isolated from the clip.
Hide the properties for the boy.mov layer, if they’re visible.
Click the Project tab to display the Project panel. Then drag another copy of the boy.mov clip from the Project panel to the Timeline panel, and place it below the original boy.mov layer.
Click the new layer, press Enter or Return, and rename the layer Background. Then press Enter or Return again.
With the Background layer selected, choose Effect > Color Correction > Hue/Saturation in the After Effects menu bar.
In the Effect Controls panel, do the following:
Select Colorize.
Change Colorize Hue to −122 degrees.
Change Colorize Saturation to 29.
Change Colorize Lightness to −13.
Choose File > Increment And Save.
If you save incrementally, you can return to earlier versions of your project to make adjustments later. This can be very useful if you’re experimenting or want to try alternative effects. The Increment And Save feature preserves the previously saved version of the project and creates a new project with the same name, with an increasing number added to the filename.
Adding animated text
You’re nearly done. All you need to do is to add the animated title between the boy and the background.
Deselect all layers, and then move the current-time indicator to the beginning of the time ruler.
Choose Layer > New > Text.
A new text layer appears in the Timeline panel at the top of the layer stack, and a cursor appears in the Composition panel.
Type WINTER BLUES in the Composition panel.
Select all the text in the Composition panel, and then do the following in the Character panel:
NoteIf you don’t have Myriad Pro Semibold on your computer, use Adobe Fonts to install it. See “Installing a font using Adobe Fonts” in Lesson 3.
Choose Myriad Pro for the font.
Choose Semibold for the font style.
Type 300 px for the font size.
Choose Optical from the Kerning menu.
Select white for the fill color.
Select black for the stroke color.
Make sure the stroke width is 1 px and Stroke Over Fill is selected.
NoteYou’ll position the text in step 8; it’s OK if it doesn’t match ours here.
Select the text layer in the Timeline panel to deselect the text. Then press T to reveal the layer’s Opacity property. Change the Opacity to 40%.
Click the Effects & Presets tab to open the panel, and then type Glow in the search box. Double-click the Glow preset under Stylize.
The text gains some texture. The default settings are fine.
Drag the WINTER BLUES layer in the Timeline panel down to position it between the boy.mov and Background layers. Then move the current-time indicator to the beginning of the time ruler, if it’s not already there.
You’ll animate the text so that it moves left behind the boy while he crosses to the right of the frame.
With the WINTER BLUES layer selected, press P to reveal its Position property. Change the Position to 1925, 540. Click the stopwatch icon (
) for the Position property to set a keyframe.
The text moves offscreen, so it’s not visible when the movie begins.
Move the current-time indicator to 3:01 (the end of the clip), and change the Position to −1990, 540.
The text moves to the left. After Effects creates a keyframe.
Deselect all layers in the Timeline panel, and move the current-time indicator to the beginning of the time ruler. Press the spacebar to preview the clip.
Choose File > Save to save your work.
Outputting your project
You’ll render your movie to complete the project.
Choose File > Export > Add To Render Queue.
In the Render Queue panel, click the blue words Best Settings.
In the Render Settings dialog box, choose Half from the Resolution pop-up menu, and make sure Use Comp’s Frame Rate is selected in the Frame Rate area. Then click OK.
If you’re using macOS, skip to step 5. If you’re using Windows, click the arrow next to Output Module, and choose Lossless.
Click the blue text next to Output Module. Then, at the bottom of the Output Module Settings dialog box, choose Audio Output Off, and click OK.
Click the blue text next to Output To. In the Output Movie To dialog box, navigate to the Lessons/Lesson09/Finished_Project folder, and click Save.
Click Render in the top right corner of the Render Queue panel.
When the project has rendered, save and close it.
Congratulations! You’ve separated a foreground object from the background, including tricky details, modified the background, and animated some text to complete the movie. You’re ready to use the Roto Brush tool on your own projects.
Review answers
1 Use the Roto Brush tool any time you would have used traditional rotoscoping. It’s particularly useful for removing a foreground element from the background.
2 The segmentation boundary is the boundary between the foreground and background. The Roto Brush tool adjusts the segmentation boundary as you progress through the frames in the Roto Brush span.
3 Use the Refine Edge tool when you need to rotoscope objects with fuzzy or wispy edges. The Refine Edge tool creates partial transparency for areas of fine detail, such as hair. Use the Refine Edge tool only after you’ve adjusted the segmentation boundary across the entire clip.