Chapter 7. Working with Masks

Lesson overview

In this lesson, you’ll learn how to do the following:

• Create a mask using the Pen tool.

• Change a mask’s mode.

• Edit a mask shape by controlling vertices and direction handles.

• Feather a mask edge.

• Replace the contents of a mask shape.

• Adjust the position of a layer in 3D space to blend it with the rest of the shot.

• Create a reflection effect.

• Create a vignette.

• Use Auto Levels to correct the color of the shot.

This lesson will take approximately an hour to complete. Copy the Lesson07 folder into the Lessons folder that you created on your hard drive for these projects (or create it now), if you haven’t already done so. As you work on this lesson, you’ll preserve the start files. If you need to restore the start files, copy them from the Adobe After Effects CS5 Classroom in a Book DVD.

There will be times when you won’t need (or want) everything in the shot to be included in the final composite. Use masks to control what appears.

About masks

A mask in Adobe After Effects is a path, or outline, that is used to modify layer effects and properties. The most common use of masks is to modify a layer’s alpha channel. A mask consists of segments and vertices: Segments are the lines or curves that connect vertices. Vertices define where each segment of a path starts and ends.

A mask can be either an open or a closed path. An open path has a beginning point that is not the same as its end point; for example, a straight line is an open path. A closed path is continuous and has no beginning or end, such as a circle. Closed-path masks can create transparent areas for a layer. Open paths cannot create transparent areas for a layer, but are useful as parameters for an effect. For example, you can use an effect to generate a running light around a mask.

A mask belongs to a specific layer. Each layer can contain multiple masks.

You can draw masks in common geometric shapes—including polygons, ellipses, and stars—with the shape tools, or you can use the Pen tool to draw an arbitrary path.

Getting started

In this lesson, you will create a mask for the screen of a desktop computer and replace the screen’s original content with a TV news promo. Then you’ll adjust the positioning of the new footage so that it fits the perspective of the shot. Finally, you’ll polish the scene by adding a reflection, creating a vignette effect, and adjusting the color.

Begin by previewing the movie and setting up the project.

1. Make sure the following files are in the AECS5_CIB/Lessons/Lesson07 folder on your hard disk, or copy them from the Adobe After Effects CS5 Classroom in a Book DVD now:

   • In the Assets folder: news_promo.mov, office_mask.mov

   • In the Sample_Movie folder: Lesson07.mov

2. Open and play the Lesson07.mov sample movie to see what you will create in this lesson. When you are done, quit QuickTime Player. You may delete the sample movie from your hard disk if you have limited storage space.

   When you begin the lesson, restore the default application settings for After Effects. See “Restoring default preferences” on page 3.

3. Press Ctrl+Alt+Shift (Windows) or Command+Option+Shift (Mac OS) while starting After Effects CS5 to restore default preferences settings. When asked whether you want to delete your preferences file, click OK. Click Close to close the Welcome screen.

   After Effects opens to display a new, untitled project.

4. Choose File > Save As, and navigate to the AECS5_CIB/Lessons/Lesson07/Finished_Project folder.
5. Name the project Lesson07_Finished.aep, and then click Save.

Importing the footage

You’ll import two footage items for this exercise.

1. Double-click an empty area of the Project panel to open the Import File dialog box.
2. Navigate to the AECS5_CIB/Lessons/Lesson07/Assets folder, Shift-click to select the news_promo.mov and office_mask.mov files, and then click Open.
3. In the Interpret Footage dialog box that appears, select Ignore, and then click OK.

   There’s an alpha channel in the news_promo movie, but it isn’t needed for this project.

   

   You’ll start by organizing the files in the Project panel.

   About the Interpret Footage dialog box

   After Effects uses a set of internal rules to automatically interpret footage that you import. Generally, you don’t need to change these settings. However, if your footage isn’t standard, After Effects may interpret it incorrectly. In this case, you can use the settings in the Interpret Footage dialog box to reinterpret your footage. The settings in the Interpret Footage dialog box should match your source footage settings; don’t use it to specify settings for your final rendered output.

   When you choose Ignore, you instruct After Effects to disregard all transparency data in the file.

   For more information on the settings in the Interpret Footage dialog box, see After Effects Help.

4. Choose File > New > New Folder to create a new folder in the Project panel, or click the Create A New Folder button () at the bottom of the Project panel.
5. Type mov_files to name the folder, press Enter or Return to accept the name, and then drag the two movies into the mov_files folder.

   

6. Click the triangle to expand the folder so that you can see the footage items inside.

   Creating the composition

   Now, you’ll create the composition based on the aspect ratio and duration of one of the footage items.

7. Select the office_mask.mov footage item in the Project panel, and drag it to the Create A New Composition button () at the bottom of the panel.

   After Effects creates a composition named office_mask, and opens it in the Composition and Timeline panels.

   

8. Choose File > Save to save your work so far.

Creating a mask with the Pen tool

The computer screen currently contains a word-processing document. To replace it with the news promo, you need to mask the screen.

1. Press the Home key, or move the current-time indicator to the beginning of the time ruler.

   Tip

   You can also create a mask using the Mocha Shapes plug-in, which comes with After Effects CS5, and then import it into After Effects. To learn more about using the plug-in, see After Effects Help.

2. Zoom in to the Composition panel until the monitor screen nearly fills the view. You may need to use the Hand tool () to reposition the view in the panel.
3. Select the Pen tool () in the Tools panel. The Pen tool creates straight lines or curved segments. The monitor appears to be rectangular, so you’ll try using straight lines first.
4. Click the upper-left corner of the monitor screen to place the first vertex.
5. Click the upper-right corner of the monitor to place the second vertex. After Effects connects the two points with a segment.
6. Click to place a third vertex in the lower-right corner of the monitor, and then click to place a fourth vertex in the lower-left corner of the monitor.
7. Move the Pen tool over the first vertex (in the upper-left corner). When a circle appears next to the pointer (as in the middle image below), click to close the mask path.

   

Editing a mask

The mask looks pretty good, but instead of masking the information inside the monitor, the mask has removed everything outside the monitor. So you need to invert the mask. (Alternatively, you could change the mask mode, which is set to Add by default.)

Inverting the mask

For this project, you need everything inside the mask to be transparent and everything outside the mask to be opaque. You’ll invert the mask now.

1. Select the office_mask layer in the Timeline panel, and press the M key to see the Mask Path property for the mask.

   Tip

   Pressing the M key twice in quick succession displays all mask properties for the selected layer.

   There are two ways to invert this mask: by choosing Subtract from the Mask Mode pop-up menu, or by selecting the Inverted option.

2. Select the Inverted option for Mask 1.

   

   The mask inverts.

3. Press F2, or click an empty area of the Timeline panel, to deselect the office_mask layer.

If you look closely at the monitor, you will probably see portions of the screen still appearing around the edges of the mask.

These errors will certainly call attention to changes being made to the layer, and they need to be fixed. To fix them, you’ll change the straight lines to curves.

Creating curved masks

Curved or freeform masks use Bezier curves to define the shape of the mask. Bezier curves give you the greatest control over the shape of the mask. With them, you can create straight lines with sharp angles, perfectly smooth curves, or a combination of the two.

1. In the Timeline panel, select Mask 1, the mask for the office_mask layer. Selecting Mask 1 makes the mask active and also selects all the vertices.

   Tip

   The keyboard shortcut for the Pen tool is G. Press the G key multiple times to cycle through the Pen tools.

2. In the Tools panel, select the Convert Vertex tool (), which is hidden behind the Pen tool.
3. In the Composition panel, click any of the vertices. The Convert Vertex tool changes the corner vertices to smooth points.

   

4. Switch to the Selection tool (), click anywhere in the Composition panel to deselect the mask, and then click the first vertex that you created.

   Two direction handles extend off the smooth point. The angle and length of these handles control the shape of the mask.

5. Drag the right handle of the first vertex around the screen. Notice how this changes this shape of the mask. Notice also that the closer you drag the handle to another vertex, the less the shape of the path is influenced by the direction handle of the first vertex, and the more it is influenced by the direction handle of the second vertex.

   

   Tip

   If you make a mistake, press Ctrl+Z (Windows) or Command+Z (Mac OS) to undo your last action. You can also change the zoom level and use the Hand tool to reposition the image in the Composition panel as you work.

6. Once you are comfortable moving the handles, position the handle of the upper-left vertex as in the preceding figure. As you have seen, you can create very fluid shapes.

Breaking direction handles

By default, the direction handles of any smooth point are connected to one another. As you drag one handle, the opposite handle moves as well. However, you can break this connection to get greater control over the shape of the mask, and you can create sharp points or long, smooth curves.

1. Select the Convert Vertex tool () in the Tools panel.
2. Drag the right direction handle of the upper-left vertex. The left direction handle remains stationary.
3. Adjust the right direction handle until the top segment of the mask shape more closely follows the curve of the monitor in that corner. It doesn’t have to be perfect.
4. Drag the left direction handle of the same vertex until the left segment of the shape more closely follows the curve of the monitor in that corner.

   

   Drag the right direction handle of the upper-left vertex, and then the left direction handle, to follow the curve of the monitor.

   Tip

   Again, you may need to adjust your view in the Composition panel as you work. You can drag the image with the Hand tool. To temporarily switch to the Hand tool, press and hold the spacebar.

5. For each of the remaining corner points, click with the Selection tool, and then repeat steps 2–4 until the shape of the mask more closely matches the curvature of the monitor. If you need to shift a corner point, use the Selection tool.
6. When you’re done, deselect the office_mask layer in the Timeline panel to check the edge of your mask. You should not see any of the monitor screen.

   

7. Choose File > Save to save your work.

Feathering the edges of a mask

The mask looks good, but you need to soften the edges a bit.

1. Choose Composition > Composition Settings.
2. Click the Background Color box, and choose white for the background color (R=255, G=255, B=255). Then click OK to close the Color Picker, and OK again to close the Composition Settings dialog box.

   

   The white background allows you to see that the edge of the monitor screen looks a little too sharp and unrealistic. To address this, you’ll feather, or soften, the edges.

3. Select the office_mask layer in the Timeline panel, and press the F key to display the Mask Feather property for the mask.
4. Increase the Mask Feather amount to 1.5, 1.5 pixels.

   

5. Hide the properties for the office_mask layer, and then choose File > Save to save your work.

Replacing the content of the mask

You are now ready to replace the background with the TV news promo movie and blend it with the overall shot.

1. In the Project panel, select the news_promo.mov file, and drag it to the Timeline panel, placing it below the office_mask layer.
2. Choose Fit Up To 100% from the Magnification Ratio pop-up menu at the bottom of the Composition panel so that you can see the whole composition.
3. Using the Selection tool (), drag the news_promo layer in the Composition panel until the anchor point is centered in the monitor screen.

   

Repositioning and resizing the news clip

The news promo clip is too big for the monitor screen, so you’ll resize it as a 3D layer, which will give you more control over its shape and size.

1. With the news_promo layer selected in the Timeline panel, turn on the 3D switch for the layer.
2. Press the P key to show the Position property for the news_promo layer.

   

   Note

   You’ll learn more about 3D layers in Lessons 11 and 12.

   The Position property for a 3D layer has three values: From left to right, they represent the x, y, and z axes of the image. The z axis controls the depth of the layer. You can see these axes represented in the Composition panel.

3. Position the pointer in the Composition panel over the red arrow so that a small x appears. The red arrow controls the x (horizontal) axis of the layer.
4. Drag left or right as necessary to center the clip horizontally in the monitor screen.
5. Position the pointer in the Composition panel over the green arrow so that a small y appears. Then drag up or down as necessary to position the clip vertically in the monitor screen.
6. Position the pointer in the Composition panel over the blue cube where the red and green arrows meet so that a small z appears. Then drag down and to the right to increase the depth of field.

   Tip

   You can also enter the Position values directly in the Timeline panel instead of dragging in the Composition panel.

7. Continue to drag the x, y, and z axes until the entire clip fits into the monitor screen, as shown in the following image. The final x, y, and z values should be approximately 114, 219, 365.

   

Rotating the clip

The news clip fits better, but you need to rotate it slightly to improve the perspective.

1. Select the news_promo layer in the Timeline panel, and press the R key to reveal its Rotation properties. Again, since this is a 3D layer, you can control rotation on the x, y, and z axes.
2. Change the Y Rotation value to -10°. This swivels the layer to match the perspective of the monitor.
3. Change the Z Rotation value to -2°. This aligns the layer with the monitor.

   

   Your composition should now resemble the preceding image.

4. Hide the properties for the news_promo layer, and then choose File > Save to save your work.

Adding a reflection

The masked image looks convincing, but you can make it look even more realistic by adding a reflection to the monitor.

1. Choose Layer > New > Solid.

   

2. In the Solid Settings dialog box, name the layer Reflection, click the Make Comp Size button, change the Color to white, and then click OK.

   Instead of trying to exactly re-create the shape of the office_mask layer’s mask, it is easier to copy it to the Reflection layer.

3. Select the office_mask layer in the Timeline panel, and press the M key to display the Mask Path property for the mask.
4. Select Mask 1, and then choose Edit > Copy or press Ctrl+C (Windows) or Command+C (Mac OS).
5. Select the Reflection layer in the Timeline panel, and then choose Edit > Paste or press Ctrl+V (Windows) or Command+V (Mac OS).

   This time, you want to keep the area inside the mask opaque and make the area outside the mask transparent.

6. Select the office_mask layer, and then press U to hide the mask properties.
7. Select the Reflection layer in the Timeline panel, and press the M key to reveal the Mask 1 Mask Path property for the layer.

   

8. Deselect the Inverted option. The Reflection layer now obscures the news_promo layer.

   

9. Zoom in to see the screen, and then select the Selection tool (). In the Composition panel, select the lower-right vertex of the Reflection layer’s mask, and drag it toward the upper-left corner of the mask.
10. Adjust the other vertices and direction handles until your shape resembles the figure on the right.

    

11. With the Reflection layer selected in the Timeline panel, press the F key to display the Mask Feather property for the mask.
12. Increase the Feather amount to 90, 90 pixels.

    

    This creates an apparent reflection on the monitor, but because of the layers’ stacking order, the reflection spills onto the whole shot.

13. Select the Reflection layer in the Timeline panel, and drag it below the office_mask layer.
14. With the Reflection layer selected, press T to reveal its Opacity property.

    

15. Reduce the Opacity value to 65% to reduce the intensity of the reflection.
16. Press T to hide the Opacity property, and then press F2, or click an empty area in the Timeline panel, to deselect all layers.

Applying a blending mode

To create unique interactions between layers, you may want to experiment with blending modes. Blending modes control how each layer blends with, or reacts to, layers beneath it. Blending modes for layers in After Effects are identical to blending modes in Adobe Photoshop.

1. In the Timeline panel menu, choose Columns > Modes to display the Mode pop-up menu.
2. Choose Add from the Reflection layer’s Mode pop-up menu.

   

   This creates a hard glare on the monitor screen image and boosts the colors underneath.

3. Choose File > Save to save your work.

Creating a vignette

A popular effect in motion graphic design is to apply a vignette to the composition. This is often done to simulate light variations of a glass lens. It creates an interesting look that focuses the attention on the subject and sets the shot apart.

1. Zoom out to see the whole image.
2. Choose Layer > New > Solid.

   

3. In the Solid Settings dialog box, name this layer Vignette, click the Make Comp Size button, change the Color to black (R=0, G=0, B=0), and then click OK.

   In addition to the Pen tool, After Effects provides tools that let you easily create square and elliptical masks.

4. In the Tools panel, select the Ellipse tool (), hidden behind the Rectangle tool ().
5. In the Composition panel, position the cross-hair pointer in the upper-left corner of the image. Drag to the opposite corner to create an elliptical shape that fills the image. Adjust the shape and position using the Selection tool, if necessary.

   

6. Expand the Mask 1 property in the Vignette layer to display all of the mask properties for the layer.

   Using the Rectangle and Ellipse tools

   The Rectangle tool, as the name suggests, creates a rectangle or square. The Ellipse tool creates an ellipse or circle. You create mask shapes with these tools by dragging them in the Composition or Layer panel.

   If you want to draw a perfect square or circle, press the Shift key as you drag with the Rectangle tool or the Ellipse tool. To create your mask from the center, press Ctrl (Windows) or Command (Mac OS) after you start to drag. Press Ctrl+Shift or Command+Shift after you start to drag to create a perfect square or circle mask from a center anchor point.

   Be careful! If you use these tools without a layer selected, you’ll draw a shape, not a mask.

7. Choose Subtract from the Mask 1 Mode pop-up menu.
8. Increase the Mask Feather amount to 200, 200 pixels.

   Your composition should now resemble the following figure.

   

   Even with this large feather amount, the vignette is a bit intense and constricting. You can give the composition more breathing room by adjusting the Mask Expansion property. The Mask Expansion property represents, in pixels, how far from the original mask edge you are expanding or contracting the adjusted edge.

9. Increase Mask Expansion to 90 pixels.

   

10. Hide the properties for the Vignette layer, and then choose File > Save.

Adjusting the color

The shot is looking quite polished, but the color of the office_mask.mov layer is rather dull. Warming it up will make the image really pop.

1. Select the office_mask.mov layer in the Timeline panel.
2. Choose Effect > Color Correction > Auto Levels. The Auto Levels effect sets highlights and shadows by defining the lightest and darkest pixels in each color channel as white and black, and then redistributing intermediate pixel values proportionately. Because Auto Levels adjusts each color channel individually, it may remove or introduce color casts.

   

   Why apply this effect? Sometimes video cameras favor a certain color channel that makes the image cooler (bluer) or warmer (redder). The Auto Levels effect sets the black and white pixels for each channel, with the end result looking much more natural.

   The color of the office_mask.mov layer should look much better now.

3. Choose File > Save to save the finished project.

   In this lesson you have worked with the mask tools to hide, reveal, and adjust portions of a composition to create a stylized inset shot. Next to keyframes, masks are probably the most-used feature of After Effects.

   You can preview the clip now, if you’d like, or render and export it following the steps in Lesson 14, “Rendering and Outputting.”

Review questions

1 What is a mask?

2 Name two ways to adjust the shape of a mask.

3 What is a direction handle used for?

4 What is the difference between an open mask and a closed mask?

Review answers

1 A mask in After Effects is a path, or outline, that is used to modify layer effects and properties. The most common use of masks is to modify a layer’s alpha channel. A mask consists of segments and vertices.

2 You can adjust the shape of a mask by dragging individual vertices, or by dragging a segment.

3 A direction handle is used to control the shape and angle of a Bezier curve.

4 An open mask can be used to control effects or the placement of text; it does not define a region of transparency. A closed mask defines a region that will affect the alpha channel of a layer.