Masking, Rotoscoping, and Basic Keyframing
In Chapter 2, we discussed the creation of alpha transparency through the application of chroma key tools and custom mattes that shifted information between channels. Although these techniques are useful in many visual effects situations, there are other approaches that are commonly used. These include masking and rotoscoping. Masking is the process by which you draw or import a spline shape to define an alpha matte (Figure 3.1). Rotoscoping simply adds animation to the mask so that it changes shape over time. In addition to masks, you can animate properties in After Effects—these include transform properties and effect properties. The program provides a robust Graph Editor for fine-tuning the resulting keyframes and animation curves.
This chapter includes the following critical information:
• Creation and manipulation of masks with the Pen tool
• Application of Roto Brush and importation of masks
• Basic keyframing techniques on the timeline and in the Graph Editor
Masks come in two main forms: masks that retain a part of the scene and masks that remove part of the scene. Masks that retain are often referred to as holdout masks. Masks that remove are often referred to as garbage masks (as they remove unwanted “garbage” from the frame). Note that the words mask and matte are often interchanged. For the purposes of this book, I’ll refer to a mask as a drawn or painted shape that creates the alpha matte.
After Effects offers several methods for drawing masks. These include the Pen tool and its variants, as well as the Roto Brush tool.
You can draw a custom mask with the Pen tool, which is available via the main toolbar at the top left of the After Effects program window. The Pen tool creates a series of vertices that form a spline shape. If the shape is closed, the area within the shape is assigned 100 percent opaque alpha while areas outside the shape receive 100 percent transparent alpha values. (In terms of color vales on an 8-bit scale, transparent pixels become 0-black and opaque pixels become 255-white.)
You can follow these basic steps for the Pen tool’s application:
1. Select the layer you wish to mask. Click the Pen Tool button (Figure 3.2). In the Composition view, LMB-click to place a spline vertex. Continue to LMB-click to add additional vertices to form the spline shape. Note that the spans between vertices are linear by default.
2. When you’re ready to close the shape, place your mouse over the first vertex that you drew. A small “close” circle is added to the mouse icon. LMB-click the first vertex. The spline is closed and the mask is finished. The alpha values are automatically updated for the layer (Figure 3.3). Lower layers show through the transparent outer area. If there are no lower layers, the transparent area is filled in with empty black.
FIG 3.3 A drawn mask surrounds a lamp. The resulting alpha channel is shown on the right.
3. Access the layer properties by clicking the small expansion arrow beside the layer name in the layer outline. Expand the Masks section. The new mask is listed as Mask 1. Expand the Mask 1 section. The mask properties, including Mask Path, Mask Feather, Mask Opacity, and Mask Expansion are listed (Figure 3.4). By default, the mask is static. However, you can animate these properties over time to create rotoscoping (this is discussed later in this chapter).
FIG 3.4 Mask properties in the layer outline.
4. To soften the mask edge, increase the Mask Feather. There are two values, X direction and Y direction, which are linked by default. To reduce the opacity of the masked shape, whereby the opaque alpha area becomes semitransparent, reduce the Mask Opacity value. To “grow” the mask, so the opaque border pushes outwards, raise the Mask Expansion value. To shrink the mask, enter negative values into the Mask Expansion field.
You can manipulate a mask at any point. To do so, you can apply any combination of techniques listed here:
• To move a vertex, switch to the Selection tool (the leftmost tool in the program’s main toolbar), LMB-click the vertex in the Composition or Layer view, and LMB-drag. You can select multiple vertices by Shift-LMB-clicking them one at a time.
• You can also LMB-drag a selection marquee over multiple vertices after selecting the mask. To select a mask, click on the mask name in the layer outline or click on a mask segment (the line between vertices) in the Composition or Layer view.
• To deselect vertices, LMB-click off the mask in an empty area of the view panel.
• You can delete a vertex and thus alter the spline shape by selecting it and pressing the Delete key.
• You can transform a mask as a single unit by double-clicking on a mask segment. In this case, a transform box appears and all the mask’s vertices are selected (Figure 3.5). To move the entire mask, LMB-drag the transform box by placing the mouse in the interior of the box. To scale the mask, LMB-drag the box handles (the small, hollow squares). To rotate the mask, LMB-drag outside the box edge (the mouse turns into a double-headed arrow). To escape the transform box, double-click in an empty area outside the box.
FIG 3.5 A mask transform box.
In addition, masks provide the following functionality:
• The vertex positions and overall mask shape are stored by the Mask Path property. You can keyframe Mask Path and thereby shift the mask shape over time. This is explored in the “Rotoscoping” section later in this chapter.
• If you start drawing a mask while no layer is selected, the mask is applied to a new shape layer. Hence, the mask cuts out a solid-colored shape. See the “Importing Bitmap and Vector Masks” section later in this chapter for more information on shape layers.
• If you wish to delete a mask, select a mask and press the Delete key.
• You can copy and paste a mask between layers. For example, select a mask, press Ctrl/Cmd+C, LMB-click the name of a different layer, and press Ctrl/Cmd+V.
• You can temporarily hide the mask segments and vertices in the view panel by clicking the Toggle Mask And Shape Path Visibility button. This may be useful when judging the impact of a mask or examining the quality of the resulting matte edge.
• You can close an unclosed mask by selecting the mask and choosing Layer > Mask And Shape Path > Closed. If you are drawing a mask, you can finish the spline before it’s closed by switching to a different tool, such as the Selection tool.
• A layer can hold multiple masks with each one numerically named Mask n. You can rename a mask by RMB-clicking the mask name and choosing Rename. You can choose your own color for each mask by clicking the color box beside the mask name.
• You can create a mask with a predefined primitive shape by using the mask shape button set found on the main toolbar (to the immediate left of the Pen Tool button). You have the option to create a rectangle, rounded rectangle, ellipse, five-sided polygon, or star. Choose one of these tools and LMB-drag in the Composition view; when you release the mouse button, the mask size and rotation is fixed.
Combining Masks to Make Complex Shapes
When masking a complex shape, it’s often easier to combine multiple masks. For example, you can draw one mask for a person’s head, a second mask for a person’s torso, and a third mask for a person’s legs. In Figure 3.6, three masks are combined to separate an actress from the background.
The larger, hollow vertex on a mask represents the mask origin (where the mask was started and closed). The white squares at the mask edges represent the bounding box of the cut out layer. Note that there is no penalty for extending a closed mask outside the edge of the frame. This ability is particularly useful when you need to animate a mask over time.
FIG 3.6 Three masks are combined to separate an actress from the background.
By default, all masks on a single layer are added together. That is, all the opaque areas are added together and maintained. You also have the option to select a unique mathematical blending mode for each mask. As soon as two or more masks exist on a layer, a blending mode menu appears beside each mask’s name. The use of the blending mode menu is required for cutting holes into masks. For example, in Figure 3.7, two masks separate the head of an actress. However, in order to maintain a hole in the hair, a second mask is added with the Subtract blending mode.
The mask order, as they are stacked in the Masks section of the layer outline, affects the way in which masks are combined. With the Figure 3.7 setup, the large mask, Mask 1, is at the top of the Masks section. It’s set to Add. The small mask, Mask 2, is at the bottom of the Masks section. It’s set to Subtract. If you were to reverse the order and place Mask 2 on top, the masks fail to cut out the actress. Nevertheless, you are free to change the mask order by LMB-dragging the mask names up or down in the Masks section.
FIG 3.7 A second, smaller mask cuts a hole in the first mask when its blending mode is set to Subtract.
If you were to reverse the order and place Mask 2 on top, the masks fail to cut out the actress. Nevertheless, you are free to change the mask order by LMB-dragging the mask names up or down in the Masks section.
Aside from Add, Subtract, and None, which turns a mask off, the following blending modes are available:
Intersect keeps the overlapping section between the mask that is set to Intersect and all other masks. To make the intersection function, place the mask that is set to Intersect at the bottom of the Masks section.
Difference gives you a result that is opposite of Intersect, whereby mask areas that do not overlap the mask that is set to Difference are kept.
Lighten uses the highest Mask Opacity value in overlapping areas for the mask that is set to Lighten.
Darken uses the lowest Mask Opacity value in overlapping areas for the mask that is set to Darken.
Each mask also carries an Inverted check box. If this is selected, the mask is inverted and what was opaque becomes transparent and vice versa. Note that the None blending mode is useful while drawing or adjusting multiple masks in order to see the entire unmasked frame.
Adjusting a Mask with Pen Variants
The Pen tool is accompanied by additional tools for manipulating masks. These are available by LMB-clicking and holding the Pen Tool button (see Figure 3.2 earlier in this chapter). Descriptions of each follow:
Add Vertex Tool
This tool allows you insert a new vertex where you LMB-click on a mask segment. (You can also add new vertices on a preexisting mask by LMB-clicking with the Pen tool.)
Delete Vertex Tool
Delete Vertex removes the vertex that you LMB-click. It’s recommended that you add or delete vertices on a mask before animating the mask.
Convert Vertex Tool
This tool converts a vertex from a linear tangent to a smooth tangent or vice versa. A smooth tangent provides a tangent handle on either side of the vertex (Figure 3.8). You can LMB-drag the tangent ends to adjust the resulting curve segments. You can also lengthen or shorten the tangent handles to affect the shape. Shorter tangent handles create tighter bends while longer tangent handles create more gradual bends. While the default linear segments may be more appropriate for masking mechanical or architectural elements, smooth tangents allow you to follow organic forms more closely. You can create smooth tangents as you draw a mask if you LMB-click and drag before releasing the mouse button. As you drag, the tangent handle is formed in the direction of the mouse. (Adobe refers to the tangent handles as direction lines.)
FIG 3.8 Close-up of a masked can. A combination of linear and smooth vertices is used. Note the varying length of the smooth tangent handles.
By default, the transition from 100 percent opaque alpha pixels to 100 percent transparent pixels occurs over one pixel. You can “feather” the alpha matte edge, however, by applying the Mask Feather tool. When you LMB-click a mask segment with the tool, a feather point is created and a feather region surrounds the entire mask. You can LMB-drag the feather point closer or farther from the mask path to increase or decrease the feather region. The farther the feather point is from the mask, the more gradual the taper is from opaqueness to transparency (Figure 3.9). Such a taper is often useful for masking objects that carry motion blur or otherwise have a soft edge. You can vary a feather around a mask by inserting multiple feather points. You can place each feather point at a unique location and give each one a unique distance from the mask. You can adjust the “tension” of the feather—the way in which the feather segments run between feather points—by selecting a feather point with the Mask Feather tool and Alt/Opt-LMB-dragging left or right over the point. The action will make the feather segments “tighter” or “looser” to the points. You can delete a feather point by selecting it and pressing the Delete key. If the feather points become hidden, click on a vertex with the Mask Feather tool.
FIG 3.9 Top: A mask with four feather points placed at different distances from the mask path. Bottom: The resulting alpha channel. Note that the feather continues around the entire mask unless stopped by feather points placed directly on a mask segment (as is the case with the leftmost feather point).
After Effects provides an alternative method of controlling mask tangents via the RotoBezier option. If a significant amount of mask manipulation is necessary, whether a mask shape is intricate with a large number of vertices or the mask must constantly change shape through rotoscoping, RotoBezier may offer a time-saving method of manipulation.
To use RotoBezier, you must convert the mask. To do so, select the mask in the layer outline and choose Layer > Mask And Shape Path > RotoBezier. When the mask is converted to RotoBezier, you can interactively adjust the vertex tangents in the following ways:
• To make the segments surrounding a vertex more linear, switch to the Convert Vertex tool and LMB-drag over the top of the vertex toward the left in the Composition or Layer view.
• To make the segments surrounding a vertex smoother, switch to the Convert Vertex tool and LMB-drag over the top of the vertex toward the right in the Composition or Layer view.
To disable RotoBezier, choose Layer > Mask And Shape Path > RotoBezier so that the option becomes deselected. When using RotoBezier, the tangent handles are hidden and are not directly accessible. You can use RotoBezier with the Mask Feather tool.
Rotoscoping is the process by which you animate a mask over time to follow a moving object. Before the advent of digital compositing, rotoscoping was created by drawing and painting masks on paper in preparation for motion picture optical printing. Rotoscoping allows you to separate an object from a background so that a new background may be inserted below it. Rotoscoping is also useful for creating a garbage mask that is intended to remove an unwanted, moving object.
In After Effects, the primary method of rotoscoping is through the keyframe animation to masks created with the Pen tool. However, you can also automate the process by using the Roto Brush. In the realm of digital animation, a keyframe or key is a stored value for a property at a particular frame on the timeline. A series of keyframes form an animation curve. Positions on the curve that do not carry keyframes provide interpolated, “inbetween” values for the property. A property may be a mask shape, a transform such as Scale, or an effect slider.
As an alternative to manually keyframing a mask, you can motion track the feature isolated by a mask over time. For a demonstration, see Chapter 4.
When a mask is animated, you don’t have access to the animation curves for each vertex. Instead, the entire mask shape is given a single curve stored by the Mask Path property. To keyframe a mask, you can follow these basic steps:
1. Move the timeline to the frame where you’d like to place the first keyframe. Click the Time icon beside the Mask Path property below the mask name. A keyframe is laid on the timeline. This stores all the vertex and tangent positions of the mask.
2. Move the timeline to a different frame. Alter the shape of the mask (see the section “Altering Existing Masks” earlier in this chapter). As soon as you alter the shape, a new keyframe is automatically laid down. Optionally, you can disable the mask by setting its blending mode menu to None; this allows you to see the entire unmasked frame.
3. Continue to move to different frames and alter the mask shape. Any frame that does not carry a keyframe generates an inbetween mask shape by interpolating between mask shapes defined by nearby keyframes.
You can edit existing keyframes. For more information, see the “Editing Keyframes on the Timeline” section later in the chapter. To determine where keyframes should be placed, see the guidelines in the next section.
There are several different approaches you can take when rotoscoping. The approach you take is largely dependent on the type of footage you are working with and the style with which you feel the most comfortable. Nevertheless, here are some general guidelines you can follow:
Straight Ahead
If the object you are rotoscoping is moving slowly or moving in a predictable fashion, you can keyframe in a straight ahead manner. Straight ahead is a traditional animation technique whereby keys are set from the beginning to the end of the timeline without concern for the type of motion the footage may contain at a later point. For example, you would set a keyframe at frame 1, then 10, then 20, and so on until the end. The more predictable the motion and the less the object silhouette changes, the fewer keyframes you need to successfully follow the object.
Key Poses
If the object you are rotoscoping moves in an erratic manner or the object’s silhouette constantly changes, you can apply the key poses technique. In traditional animation, key poses are the most critical positions of the animated object or character. For example, if a character is walking, critical key poses occur where each foot touches the ground and where each foot is raised to pass the other. By studying the footage in After Effects, you can identify the most critical positions and add keyframes to those frames. Such positions often represent the largest silhouette shape changes. For example, if you are rotoscoping a baseball pitcher throwing a ball, the largest silhouette changes occur as the pitcher winds up, as he pulls his arm back to throw, and as his arm extends forward to release the ball. These changes are often referred to as extremes, where an appendage (an arm, leg, hand, foot, and so on) is at its most extreme position.
Bisecting
You can refine the straight ahead or key poses rotoscoping techniques by bisecting. Bisecting requires the insertion of new keyframes between old keyframes. For example, if you have keyframes at 1, 10, and 20, you would add new keyframes at 5 and 15. You can also bisect at the start of the animation process. For example, if the timeline is 200 frames long, you would add keyframes in the following pattern: 1, 200, 100, 50, 150, 25, 75, 125, and 175 (Figure 3.10). Each new keyframe is placed halfway between each pair of old keyframes. Bisecting in such a regular fashion is suitable for objects with slow-moving or predictable motion.
FIG 3.10 A 200-frame timeline with two animated masks. The top mask uses the key poses technique to follow erratic object motion. The lower mask uses the bisecting technique where keyframes are placed at bisected intervals.
Inbetweens
Regardless of the rotocoping approach you take, you generally need to add inbetween keyframes to take into account nonlinear variations in the object motion. These are placed wherever the mask is not accurately following. For example, an automobile may make a subtle change in speed, an actress may turn her head slightly, or an actor’s hair may form different shapes due to wind. In traditional animation, inbetween keyframes are less important poses that fall between the keys, but are nevertheless needed to create continuous, smooth motion.
Drawing and manipulating masks, a very common task with visual effects work, can be extremely time-consuming. Hence, After Effects supplies the Roto Brush tool to make the process semi-automated. Roto Brush allows you to interactively define foreground and background elements. With this information, the tool detects the foreground edge and forms a closed mask with opaque alpha in its center. After the foreground is defined, Roto Brush continues to update the mask shape as the timeline moves forward. Roto Brush lives on a layer as an effect and is bundled with the Refine Edge effect so that you can adjust the resulting alpha matte. See the following tutorial for a practical application of Roto Brush. Refine Matte is discussed after the tutorial. Refine Matte shares the same properties as the standalone Refine Soft Matte effect, which is demonstrated in Chapter 2.
Adjusting Base Frame Influence
When you draw an initial Roto Brush stroke, the current frame becomes a base frame. The mask definition, as determined by the Roto Brush strokes, is automatically propagated forward and backwards 20 frames. If you look closely at the Layer panel’s timeline, a small, arrow-laden, base frame span bar extends from the frame you are working on (Figure 3.13). The direction of the arrows (< or >) on the span bar indicates the direction of the base frame propagation. In essence, the base frames function as keyframes for the Roto Brush tool.
FIG 3.13 Close-up of Layer panel timeline. The gold line indicates a Roto Brush base frame. A span bar extends to the right with small > arrows.
If you draw corrective strokes at a later frame, the influence of the strokes is carried in the direction of the arrows. Each time you draw a corrective stroke, the base frame span bar extends. You can limit the effect of a base frame by LMB-dragging the end of the span bar. This may be useful if the object you’re rotoscoping leaves the frame or is occluded. If you move to a frame that is not covered by a base frame span bar, no mask is drawn. You can lengthen or shorten a span bar at any point.
Any frame of the timeline can hold a single Roto Brush base frame and a single Roto Brush mask. However, if you move to a frame where no base frame span exists, you can draw a brand new base frame that receives its own span bar in the forward and backward direction. This produces a brand new mask. Once again, you can determine the length of any base frame span bar by LMB-dragging its left or right end.
Note that rotoscoping with Roto Brush may be significantly slower than rotoscoping with the Pen tool. The Roto Brush tool must calculate the mask shape along the span bar whenever you update the mask and move to a different frame. That said, you might save time with the Roto Brush because you have to lay down fewer manual keyframes (again, you can think of base frames and corrective strokes as a unique form of keyframes).
The following steps represent the general approach you can take when applying the Roto Brush tool. For practice, apply these steps to the mini_roto_brush.aep project saved in the ProjectFilesaeFilesChapter3 directory.
1. Open the sole layer carried by Comp 1 in the Layer view. You can do this by double-clicking the layer name in the layer outline (which is named after the image sequence). You can distinguish the Layer view panel by its built-in timeline.
2. Click the Roto Brush Tool icon in the main toolbar. The button carries an icon of a man and a paintbrush and is to the right of the Eraser Tool button. In the Layer view, roughly LMB-drag a roto paint stroke to define what you wish to rotoscope. With this example, concentrate on one part of the actress. For example, roughly draw a line to establish where her right hand and arm are (Top) (Figure 3.11). The stroke is colored green, indicating the foreground area. When you let go of the mouse button, the stroke is converted into a mask (also called a segmentation boundary). The edge of the hand and arm are detected, although the detection may not be perfect due to the tool’s inability to distinguish some of the foreground and background pixels.
3. LMB-drag the brush again to better define where the edge should be. For example, draw smaller, shorter strokes to define the fingers. You can interactively change the brush size by Ctrl/Cmd-LMB-dragging in the Layer view.
FIG 3.11 Top: A green stroke is drawn with the Roto Brush tool, roughly indicating the position of an arm and hand. Bottom: The resulting mask is drawn in pink. Note that the mask does not encompass all the fingers at this point.
4. If the mask extends too far and encompasses the background, you can shrink the mask by Alt/Opt-LMB-dragging negative strokes. These strokes are colored red and the brush receives a negative sign in its center. Continue to refine the mask by painting additional foreground and background strokes (Figure 3.12).
FIG 3.12 Additional positive and negative Roto Brush strokes better define the arm and hand.
5. After the mask is properly formed, step forward one frame on the timeline. Note that the mask automatically updates. Should the mask become inaccurate, you are free to apply additional foreground and background strokes.
6. Play the timeline forward. The mask is calculated for each frame. If low contrast exists between the foreground and background, the mask becomes inaccurate. Heavy motion blur also confuses the tool. Hence, a significant amount of adjustment may be necessary to create a clean mask. When the mask is satisfactory, you can click the Freeze button at the lower right of the Layer view panel to prevent the Roto Brush tool from recalculating the frames as you play back the timeline.
The Roto Brush tool attaches itself to the layer as an effect. It carries properties that control the mask propagation over the timeline, which can have significant effect on the success of the resulting mask. These are discussed in the next section. A finished version of this project is saved as mini_roto_brush_finished.aep in the ProjectFilesaeFilesChapter3 directory.
Adjusting Propagation and Refining Edges
Each base frame carries its own unique set of property values. Creating multiple base frames is useful for rotoscoping footage where the rotoscoped object suffers significant changes in silhouette shape, carries different degrees of motion blur, and transitions through different pools of light. The properties for the Roto Brush are stored in the Roto Brush & Refine Edge section of the layer outline and are grouped into Roto Brush Propagation and Refine Edge Matte properties (Figure 3.14).
FIG 3.14 Roto Brush & Refine Edge options with Roto Brush Propagation and Roto Brush Matte sections expanded.
Descriptions of base frame propagation properties follow:
View Search Region and Search Radius
View Search Region, when set to on, displays a yellow area that represents the search region— the region in which the tool searches for foreground pixels that match the foreground pixels of a prior frame. If your object is moving quickly or is constantly changing shape, it may pay to expand the search region by increasing the Search Radius property value. If the rotoscoped object is moving slowly or doesn’t change its shape, you can reduce the Search Radius to avoid chatter (the fluctuation of the mask around the object edge).
Motion Threshold and Motion Damping
Together, these two properties constrict or expand the search region based on detected motion. Motion Threshold sets it namesake, where areas of the search region that have less motion than the threshold are removed (and assumed to be part of the background). This leads to smoothing of the search region shape, which is most noticeable when the Motion Damping value is low. Shrinking the areas with no or little motion can help prevent chatter. If you increase the Motion Damping value, the search region is constricted again, with areas containing little or no motion removed from the search region.
Edge Detection
This menu provides three options for calculating the mask edge on new frames. The Favor Predicted Edges option biases the overall shape of the mask on a base frame and is less sensitive to matching colors among pixels. This mode is useful when the rotoscoped object carries similar color values as the background. The Favor Current Edges option does the opposite and biases matching pixel values regardless of the resulting mask shape. This option works better in high-contrast situations. The default option, Balanced, combines these techniques equally.
If the Fine-Tune Roto Brush Matte property is selected and View Search Region is deselected, additional properties for the Refine Edge effect become available. Descriptions of key properties follow:
Roto Brush Matte
This subsection provides basic properties for refining the resulting alpha matte. The Feather property works in a similar fashion to the Mask Feather property of a standard mask. (You can return to the Composition view to see the result.) The Contrast option, on the other hand, controls the taper from 100 percent opaqueness to 0 percent transparency. High-contrast values create a sharp edge while low values create a soft edge. If the Contrast value is high, raising the Feather value will smooth the mask, reducing the number of small undulations. The Shift Edge property works as a choker, eroding or expanding the matte (Feather must be a non-0 value for this property to take effect). The Reduce Chatter property attempts to reduce mask undulations by averaging the mattes of the surrounding frame together. If Chatter is 0, the function is off.
Use Motion Blur
If selected, the property applies motion blur to the alpha matte, taking into account the matte’s prior position in the frame. This may be useful for rotoscoping fast-moving objects with blurred edges.
Decontaminate Edge Colors
If selected, this property attempts to remove the background color from the edges of the foreground. The Decontamination Amount property controls the aggressiveness of the removal.
Note that you can interactively refine small sections of the matte edge by applying the Refine Matte Tool (located in the drop-down menu visible when you click and hold the Roto Brush Tool button). When you LMB-drag the Refine Matte tool brush over the mask edge in the Layer view, it softens the alpha matte edge under the brush. Alt/Opt+LMB-dragging in the same area removes the softness.
You are not limited to masks that are generated within After Effects. You can import a variety of masks from other programs such as Adobe Photoshop, Adobe Illustrator, and Mocha AE.
Importing Bitmap and Vector Masks
You can import a bitmap in any common file format and use it as a mask in After Effects. In addition, After Effects supports vector-based Adobe Illustrator files. You can import an Illustrator .ai, .eps, or .pdf file directly into After Effects. The result is an alpha matte cut in the shape of the closed vector shapes. The vectors themselves are not immediately accessible. Whether you are working with bitmaps or vectors, you can add the imported mask to a composition as a hidden layer and transfer alpha information from it to another layer with the Track Matte option or a channel effect. (For examples using Track Matte and the Set Matte effect, see Chapter 2.)
You can convert an imported vector file into a shape layer. A shape layer combines solid colors with one or more masks (called paths). To make the conversion, place the imported mask in a composition as a layer, select that layer, and choose Layer > Create Shapes From Vector Layer. A new layer is created with one mask for each closed vector shape (Figure 3.15). The original layer is hidden. You are free to adjust the resulting paths by moving vertices and tangent handles as you would with masks created by the Pen tool. Each shape is assigned a Contents > Group section of the layer in the layer outline. The shape is broken down into a path, a fill color, and a set of transforms. You can change any of these properties.
FIG 3.15 An Illustrator .ai file is imported into After Effects and converted to a shape layer. There is one path for each closed-vector letter. The Fill Color for each path is set to black. A sample project file is saved as ai_shape_layer.aep in the ProjectFilesaefilesChapter3 directory.
Imagineer Systems Mocha AE is bundled with recent editions of After Effects. Mocha AE offers a planar-based motion tracking and masking system. Mocha AE is able to automate many tracking and masking tasks that would normally take a significant amount of time inside After Effects. You can import motion tracking data and mask into After Effects from Mocha. The following tutorial walks you through the basic steps of creating and importing a mask. (Motion tracking is demonstrated in Chapter 4.)
Mocha masks mini-tutorial
The following steps represent the general approach you can take when using Mocha AE to rotoscope. You can apply these steps to the mini_mocha_ae.aep project saved in the ProjectFilesaeFilesChapter3 directory.
1. Select the sole layer carried by Comp 1, which is named after the image sequence. Choose Animation > Track In Mocha AE. Mocha AE opens.
2. In Mocha AE, the New Project window opens and states the footage frame range and frame rate. If this information is incorrect, you are free to change it. Click the OK button to close the window. The footage appears in the Mocha viewer. You can play the footage with the viewer’s playback controls. You can zoom in and out by selecting the Zoom tool from the top toolbar (Figure 3.16) and LMB-dragging up and down in the viewer. You can scroll by MMB-dragging. To escape a tool, you can select the Pick tool.
FIG 3.16 The Mocha AE CC 2014 toolbar. The Pick, Zoom, Create X-Spline Layer, and Create Bezier Layer tools and Show Planar Surface button are highlighted. Note that Mocha AE CC 2015 uses the same tools and buttons, although there are minor variations in the toolbar layout.
3. Mocha is a planar tracker, which means that it tracks shapes that are defined as planar (a flat, two-dimensional surface that may rotate or change in scale). To define the tracked shape, you use one of several spline tools to loosely form a spline “cage” around the shape. For this tutorial, we’ll use the X-Spline tool, which is perhaps the easiest to apply. We’ll track a section of unpainted drywall surrounded by the white paint lines.
4. Return to the first frame. Select the Create X-Spline Layer tool from the top toolbar (Figure 3.16). This features a pen tip and a small X as part of the icon. LMB-click near the bottom left of the rightmost unpainted drywall area. You can place the vertex within the painted white line. An X-Spline vertex is drawn. The vertex does not have to touch the exact corner of the unpainted drywall; in fact, the tracking works better if there is a small gap (Figure 3.17). LMB-click at another corner near the drywall area. A new vertex is laid down with a spline span in between. Click two more times to form two more corners slightly out of frame at the top of the area. A rounded-corner, rectangular cage is formed. RMB-click over the shape to finish it. Note that Mocha is able to track planar shapes that extend past the frame edge. You can adjust the vertex positions after the cage is drawn by LMB-dragging them.
FIG 3.17 An X-spline cage is drawn around an unpainted section of drywall.
5. Mocha AE will attempt to track the caged shape over time. The application of tracking is similar to After Effects motion tracking (discussed in the next chapter). To activate the tracking, click the Track Forward button. The tracking buttons sit to the right of the playback buttons and feature a small T in the icons. You can also track backwards or track in either direction one frame at a time. When you click the Track Forward button, the timeline moves forward and Mocha updates the X-spline cage vertex positions to follow the chosen shape. With this example, the unpainted drywall area is tracked. Complex organic shapes can also be planar tracked if there are no drastic changes (such as large rotations, significant silhouette shape changes, heavy motion blur, and so on). Minor occlusions, such as the crossing of shadows, do not prevent the success of the tracking. Heavy occlusions, however, can interfere with the accuracy of some of the vertex positions, causing the cage to drift away from the chosen shape. After the tracking has completed, you can use the standard playback controls to gauge the tracking success.
6. If the spline cage drifts, you can correct it by creating reference points. To do this, return to the first frame and click the Show Planar Surface button in the top toolbar (see Figure 3.16 earlier in this section). A blue rectangle is drawn in the center of the cage. You can LMB-drag the rectangle corners and place them over specific reference points in the footage. For example, you can LMB-drag the bottom-left corner and place it over the bottom-left corner of the unpainted drywall area (Figure 3.18). You can choose to keyframe the rectangle corners over time by switching to the Adjust Track tab at the bottom left of the program window, moving to a later frame, and updating the corner positions. The corner positions, in turn, force the cage to be placed more accurately over time. When you move a corner, a zoomed view of the mouse area is presented at the top left of the viewer. This is labeled Current Frame. For comparison, a zoomed-in view of the same area is presented for the Master Frame—the frame on which the initial corner position was established (Figure 3.19). This allows you to better judge the precision of the corner placement. Note that the planar vertices carry a small X on the master frame.
FIG 3.18 The planar reference surface is revealed. Its corners are positioned over the corners formed by the paint lines and several circular nail hole patches. The Xs indicate that the frame is a master frame.
7. As soon as the cage is following the chosen shape in an accurate fashion, you can combine the tracked shapes with more traditional masking. To create a mask around the drywall area, choose the Create Bezier Layer tool from the top toolbar (Figure 3.16 earlier in this section). LMB-click in the viewer at one corner of the drywall area. A vertex is laid down. LMB-click at the opposite corner. A second vertex appears with a Bezier segment and tangent handles at the vertex positions. LMB-click a third time at the right top of the area outside the frame. LMB-click a fourth time at the left top of the area outside the frame. LMB-click a final time over the top of the first vertex to finish the mask. LMB-drag the tangent handle ends to better follow the drywall edges. You can shorten the tangent handles to create tighter corners (Figure 3.20).
8. Note that the new Bezier and the X-spline cage are listed as Layer 1 and Layer 2 in the Layer Controls panel at the top left of the program window (Figure 3.21). Rename Layer 2 by double-clicking the name and entering Mask into the field. Rename Layer 1 by double-clicking the name and entering Track into the field. You can attach the Mask layer to the Track layer by linking them. This prevents the need to manually update the mask shape over time. To do so, select the Mask layer in the Layer Controls panel and change the Link To Track menu to Mask. Play back the timeline. The mask follows the cage and covers the drywall area without drifting. Should you need to adjust the mask shape at a later frame, you can LMB-drag the mask vertices or tangent handles. A keyframe is automatically created and indicated on the timeline as a green triangle.
9. To transfer the mask to After Effects, select the Mask layer in the Layer Controls panel and click the Export Shape Data button in the Track tab at the bottom right of the program window. In the Export Shape Data window, select the Selected Layer radio button and click the Copy To Clipboard button. Return to the After Effects program window. Select the layer to which you want to add the mask, and choose Edit > Paste. The mask is added to the layer as an effect named after the Mocha layer. The After Effects layer is cut out, although no spline shape is visible in the Composition view. The transform properties carried by the effect are pre-animated for every frame of the timeline. The properties include Shape Data (mask shape information), Translation, Scale, Rotation, Shear, and Perspective. You are free to edit the keyframes as you would any other animation in After Effects.
FIG 3.20 An adjusted Bezier shape (red inner lines with blue tangent handles) forms a mask.
There are numerous ways to adjust the tracking and masking within Mocha AE. Unfortunately, these can’t be covered in this book due to space limitations. For more information on Mocha AE functionality, visit the Mocha AE Help files. Note that you can save Mocha AE project files by choosing File > Save Project in the Mocha program window. A finished Mocha project is saved as mini_mocha_ae_finished.mocha in the ProjectFilesaeFilesChapter3 directory. A finished After Effects project is saved as mini_mocha_ae_finished.aep in the ProjectFilesaeFilesChapter3 directory.
FIG 3.21 Layer Controls panel with renamed layers. The Link To Track menu is at the bottom of the panel.
A keyframe or key is stored value associated with a particular frame. In After Effects, the value may belong to an effect, mask, or transform property. There are two places keyframes appear: on the timeline and in the Graph Editor. You can edit the keyframes in either location. To set a keyframe, move to the frame where you’d like to place the key and click the Time icon beside the property you wish to keyframe. At that point forward, changes to the property result in an updated keyframe (if the frame remains the same) or a new keyframe (if you’ve moved to a different frame). If two or more keyframes exist for a property, an animation curve is threaded through the keyframes; this is used to determine what property values should exist on inbetween frames.
Editing Keyframes on the Timeline
When a keyframe is created, a symbol is placed on the timeline for the current frame. By default, this appears as a small diamond to indicate that the keyframe is using linear spans. You can alter existing keyframes in the following ways:
• You can delete a keyframe by selecting it on the timeline and pressing the Delete key. Selected keyframes turn yellow. The associated animation curve updates to ensure that it threads through the remaining keyframes.
• You can remove animation from a property by clicking the property’s Time icon so that it turns off.
• You can move keyframes in time by LMB-dragging them on the timeline. You can select more than one keyframe by Shift-LMB-clicking them or LMB-dragging a selection marquee around them.
• You can force a keyframe at the current point of the timeline by RMB-clicking over the property name and choosing Add Key from the menu. You can do this without changing the current property value or changing the current mask shape.
• You can enter a precise numerical value for a keyframe by RMB-clicking the keyframe icon on the timeline and choosing Edit from the menu. You can then enter the value into the property window that pops up.
• You can copy and paste one or more keyframes. To do this, select the keyframes, press Ctrl/Cmd+C, move to a different frame, and press Ctrl/Cmd+V. If multiple keyframes are selected, the group of keyframes is pasted at the current frame but maintain their original spacing. You can also copy and paste keyframes between similar properties on different layers. To do this, select the keyframes, press Ctrl/Cmd+C, move to the frame where you would like to paste, select a similar property on a different layer by clicking its name, and press Ctrl/Cmd+V.
Several After Effects properties have two dimensions. That is, they have X and Y values. Each dimensional value receives its own curve and set of keyframes. To access the individual curve, you must use the Graph Editor. Despite this, only a single keyframe is placed on the timeline for both dimensions. For example, if you keyframe the Scale property, both the X and Y Scale dimensions receive a single keyframe symbol for any given frame on the timeline. In Figure 3.22, the Position and Scale properties of a layer are animated, producing keyframes at frame 1, 2, and 3. The Scale property is unlinked so that X and Y can carry different values; however, there is only a single timeline keyframe symbol for both X and Y. (The Scale link button appears to the immediate left of the X and Y fields.)
In Figure 3.22, the Position property is separated into X and Y properties so that each carries its own timeline keyframe symbol. To split the Position into X and Y, RMB-click over the Position property name and choose Separate Dimensions. To recombine the X and Y, RMB-click and choose Separate Dimensions again so the option becomes deselected. Note that 3D layers cause some properties to take on a third dimension, Z. 3D layers are discussed in Chapter 5.
FIG 3.22 Position and Scale properties are keyframed for a layer.
The Graph Editor provides the most powerful means of altering keyframe animation. In the editor, you can access and fine-tune the animation curves, thus controlling how the inbetween frame values are generated.
To open the Graph Editor, click the large Graph Editor button at the top right of the layer outline. To see a property curve in the editor, click the property name while the editor is open. You can also click the Include This Property In The Graph Editor Set button beside the property name. You can display more than one curve in the editor at a time.
The Graph Editor carries two display modes: a speed graph and a value graph. The speed graph mode displays the rate of value change. Because this mode is difficult to interpret and adjust, I suggest that you use the value graph mode. The value graph mode displays property values and is similar to what you would find in the curve editor of 3D programs such as Autodesk Maya. To choose a display mode, click the Choose Graph Type And Options button at the bottom left of the Graph Editor (Figure 3.23) and choose either Edit Value Graph or Edit Speed Graph.
FIG 3.23 Graph Editor buttons. Choose Graph Type And Options, Fit Selection To View, and Fit All Graphs To View are highlighted.
If two or more keyframes exist for a property, an animation curve is threaded through them. Any given frame produces a value for the property. For example, if you go to frame 10 and read the value for the curve, that value is then given to the property at frame 10. To read a curve value at any given frame, place your mouse over the curve. The value is printed in a small readout box (Figure 3.24). In the Graph Editor, time runs left to right and property values run down to up. The property values are incrementally displayed along the left side of the editor.
FIG 3.24 The mouse is placed over a curve at frame 5. The yellow readout box displays the layer name, the property name, and the property value for that frame (524.53). px stands for pixels.
You can alter animation curves in the following manner:
• To move a keyframe, LMB-click on the yellow keyframe dot and LMB-drag it. Up-down motion creates a property value change and left-right motion creates a time change. By default, the keyframe “sticks” to whole frame numbers.
• To add a new keyframe along a curve at the current mouse position, Ctrl/Cmd+LMB-click on the curve. The new keyframe also appears on the timeline.
• To delete a keyframe, select it and press the Delete key. You can click on the keyframe or LMB-drag a marquee box around multiple keyframes.
• To frame a curve (or curves) so it is fully seen in the Graph Editor, click the Fit All Graphs To View button (Figure 3.23 in the previous section). To frame selected keyframes, click the Fit Selection To View button.
• If more than one keyframe is selected, a gray transform box is displayed (Figure 3.25). You can move the selected keyframes by LMB-dragging the gray area of the box. You can scale the keyframes by LMB-dragging any of the white dots on the box edges. If you scale the keyframes vertically, the value range is compressed or expanded. If you scale the keyframes horizontally, the frame range is compressed or expanded. The shorter the frame range, the faster the value change (for Position, this equates to faster motion).
FIG 3.25 A V-shaped animation curve with three keyframes appears in the Graph Editor. The left two keyframes are selected, producing the gray transform box.
One of the most significant changes you can make to an animation curve is changing the tangent types of its keyframes. By default, the curve spans are linear—that is, they run in perfectly straight lines between keyframes. This creates a mechanical motion that is not particularly suited for animation that needs to appear organic. To change the tangent type of a keyframe, place your mouse over the keyframe icon, RMB-click, and choose Keyframe Interpolation from the menu. In the Keyframe Interpolation window, change the Temporal Interpolation menu to the tangent type of your choice. Descriptions of the tangent types follow:
Linear produces straight spans between keyframes with no tangent handles (Figure 3.26). This type is suitable for mechanical animation where there are abrupt changes in speed.
FIG 3.26 Top: Complex curve with default Linear spans. Middle: Same curve converted to Bezier tangents. Bottom: Same curve converted to Hold tangents.
Bezier creates a tangent handle that extends in both directions from the keyframe (Figure 3.26). (To see the handles, you must select the keyframes.) You can LMB-drag the tangent ends to rotate the tangent handle, lengthen the tangent handles, or shorten the tangent handles to alter the curve span flow. This type of fine-tuning is not available to the timeline view and is often mandatory for creating quality animation, whether it’s for transform, mask shape, or effect property changes.
Hold forces the current keyframe value to be carried forward until the next keyframe is encountered (Figure 3.26). This type is rarely used but may be appropiate for rough key pose animation where the character or object “jumps” from one key pose to the next.
Auto Bezier creates tangent handles but keeps the handles short to force tight and smooth transitions at the “peaks” and “valleys” (Figure 3.27). This type often prevents excessive curve values from forming when a Bezier curve shoots too high or too low.
Continuous Bezier is identical to Auto Bezier but carries manual adjustments. Auto Bezier automatically switches to Continuous Bezier as soon as you edit the keyframes.
FIG 3.27 Top: Curve with manually adjusted Bezier tangents. Bottom: Same curve converted to Auto Bezier.
Each tangent type produces a different keyframe symbol on the timeline. Linear keyframes use diamonds. Auto Bezier keyframes use circles. Bezier and Continuous Bezier keyframes use an hourglass symbol. Hold keyframes use a square symbol with notched squares appearing for surrounding keyframes.
Note that the Spatial Interpolation menu, available in the Keyframe Interpolation window, sets the tangent type for motion paths that appear in the view panels. Spatial Interpolation is further explored in Chapter 4.
Graph editor mini-tutorial
The following steps allow you to practice editing curves in the Graph Editor. With this tutorial, we’ll create a mask in the shape of a leaf and animate the leaf slowing “flitting” down through the air.
1. Create a new After Effects project. Create a new composition. Make the composition 60 frames long with a frame rate of 30 fps. Choose any of the larger resolutions, such as 1920x1080.
2. Choose Layer > New > Solid. In the Solid Settings window, click the Make Comp Size button. Change the color swatch to a leaf-like color (green, brown, or red). See Figure 3.28.
FIG 3.28 A leaf is cut out of a solid layer with a mask. The red line is the final motion path created by the Position animation.
3. Using the Pen tool, draw a closed mask on the new solid layer that forms the shape of leaf. Position the mask so that the leaf stem corresponds with the layer center (the small red crosshair in the center of the frame). Use any of the masking methods discussed in this chapter to refine the mask shape.
4. On the first frame, click the Time icon beside the Position and Rotation properties for the layer. Keyframes are laid down. Proceed to the last frame of the timeline. Change the position and rotation of the leaf so that it sits at the bottom of the frame (as if it has fallen). New keyframes are automatically laid down.
5. Proceed to other frames and add additional keyframes to make the falling animation more interesting, For example, make the leaf “flit” left and right as it falls. You can use any of the keyframe techniques discussed in the “Rotoscoping Approaches” section earlier in this chapter. Even though we’re not changing the mask shape, you can apply bisecting or key pose techniques to transform animation.
6. Play back the timeline and adjust the keyframes. After the basic animation is roughed in, open the Graph Editor. Click on the Position property name and choose Separate Dimensions from the menu. If the Position X and Y are left merged, you cannot alter the tangent handles of individual keyframes.
7. Proceed to adjust one curve at a time. To see a property curve, click on the property name in the layer outline. Convert all the keyframes to Bezier-style tangents (Figure 3.29). For example, click on the Rotation property name, select all the Rotation keyframes in the Graph Editor, LMB-click one of those keyframes, choose Keyframe Interpolation from the menu, and change the Temporal Interpolation menu to Bezier. Proceed to adjust tangent handles and play back the timeline to see the result.
FIG 3.29 A final version of the animation. X Position is red. Y Position is green. Rotation is cyan. Note the different shapes of the curve spans and different positions and lengths of tangent handles.
As you alter the animation curves, keep these guidelines in mind:
• The steeper a curve span, the faster the change in value. With Position or Rotation, it means the leaf is moving faster.
• The more shallow a curve span, the slower the change in value. With Position or Rotation, it means the leaf is moving slower.
• If a curve span becomes perfectly flat, it means there is no change in value. With Position or Rotation, it means the leaf has stopped moving.
A finished After Effects project is saved as mini_graph_editor_finished.aep in the ProjectFilesaefilesChapter3 directory.
Chapter Tutorial: Keying a Difficult Green Screen, Part 2
In Chapter 2, we keyed the green screen of a shot with an actress. Although we spent time working on a clean alpha matte around her body and hair, there are still several improvements we can make to the shot.
1. Open the tutorial_2_1.aep file located in the ProjectFilesaeFilesChapter2 directory.
2. Go to the first frame. Select the footage layer and create a new mask with the Pen tool that surrounds the actress, but removes the X-shaped tracking marks. You can extend the mask outside of the bottom and top of frame. This is a holdout mask, as it retains the actress but throws away the marks (Figure 3.30). Animate the mask over time to take into account the motion of the camera. You can use the bisecting method to determine keyframe locations. Play back the timeline to make sure no part of the actress is cut off. In the finished version, keyframes are placed approximately every 8 to 10 frames.
FIG 3.30 A holdout mask, as seen on frame 8, is created to retain the actress but remove the tracking marks. The Refine Soft Matte effect is disabled.
3. Go to the Effect Controls panel for the footage layer. Turn off the Refine Soft Matte effect and Keylight by clicking the small fx buttons beside the effect names. As an alternative, we’ll create a custom luma matte to retain the fine hairs.
4. Duplicate the footage layer by selecting it and choosing Edit > Duplicate. Select the new top layer and choose Effect > Color Correction > Hue/Saturation. Reduce the Master Saturation to –100. The layer becomes grayscale. Choose Effect > Color Correction > Curves. Click the Curves line in two places to add two new points. LMB-drag the points to form a severe S-curve (Figure 3.31). This creates a large amount of contrast, which differentiates the hair from the background. Many of the fine hairs become visible.
5. Hide the new top layer by clicking off the Video eye icon beside the layer name. Change the lower footage layer’s Track Matte menu to Luma Matte. (Track Matte is described in detail in Chapter 2.) The footage is cut out by the top layer. Although the edge of the hair is acceptable, the center of the actress is adversely affected.
6. Duplicate the lower footage layer. Because the old top layer is hidden, the duplicated layer is moved to the top of the layer outline. Select the new top layer and reactivate its Keylight effect. The keyed version of the actress is thus placed over the luma matte version (Figure 3.32). This allows fine hairs retrieved by the luma matte to appear along the previously eroded hair edge. You can hide the solid layer to better judge the new hair integration.
FIG 3.32 The revised layer outline with the keyed layer on top, the luma matte hidden in the middle, and the matted layer at the bottom over the solid.
7. The new hair is tinted green due to green screen spill. Select the matted layer (now numbered 3) and choose Effect > Color Correction > Hue/Saturation. Reduce Master Saturation to –75 and increase Master Lightness to 20. The new hair is now better integrated (Figure 3.33).
FIG 3.33 Updated hair, which is more detailed and subtle than what was provided by the Refine Soft Matte effect.
We’re now ready to integrate a new background behind the actress. We’ll do this in Chapter 4. This version of the project is saved as tutorial_2_2.aep in the ProjectFilesaeFilesChapter3 directory.