DIGITAL ELEMENT CREATION

DIGITAL MODELING

Kevin Hudson

Overview: The Importance of Modeling

The pipeline for digital asset creation begins with digital modeling. It is the birthplace for a digital asset, and it is here that the basic aesthetic form and technology are laid. If they are not done well, this is the place where most problems for the future pipeline begin. Modeling is as important to the creation of an asset as the foundation is to the construction of a house. If the model is solid, all of the other crafts from rigging to painting to color and lighting can be done with far greater ease.

Types of Modeling

Digital modeling can be broken up into four types: character, prop, hard surface/mechanical, and environmental.

Character Modeling

Character modeling assets are assets that will need to move in some fashion. For the most part, they can be classified as actors. These would easily include such models as a cuddly Stuart, from Stuart Little (1999), the hyperrealistic Dr. Manhattan from Watchmen (2009), the stylized characters from The ChubbChubbs (2002), or digital stand-ins for background people such as panicking or stampeding crowds from any number of action films.

Character modeling calls on the true sculptural talent of the modeler. In recent years, software that more and more resembles a sculpting experience has evolved. This type of software includes Zbrush, Freeform 3D, 3D-Coat, and Mudbox.

Figure 7.1 Stuart Little (1999) is just one example of a digital character asset. (STUART LITTLE © 1999 Global Entertainment Productions GmbH & Co. Medien KG and SPE German Finance Co. Inc. All rights reserved. Courtesy of Columbia Pictures.)

Prop Modeling

Prop models tend to be items that do not necessarily move on their own accord. They are handled or acted on by characters. This does not mean that they will not need to be rigged for movement to some degree, but they are often rigged to a much lesser degree than a character model.

Props are still incredibly important assets and can be either soft or mechanical. Often digital props are re-creations of real-world counterparts. In that case, gathering quality reference is vital to the success of re-creating these assets. Whenever possible, the real-world item should be made available to the modeler. Because this is often not practical, getting good photo reference with measurements is the next best thing.

Hard Surface/Mechanical Modeling

Hard surface modeling has been broken out as its own distinct type of modeling. Cars, trains, helicopters, and highly designed technical items fall into this category.

Environmental Modeling

At a certain point, the line blurs between what separates a prop from an environment. In the case of the train in The Polar Express (2004), it functions as a very complicated prop or hard surface model. But it is also a model that characters walk through and perform within. This is where it enters the realm of environmental modeling.

Figure 7.2 A motorcycle is a classic example of a hard surface/mechanical model. (Image © 2007 Sony Pictures Imageworks Inc. All rights reserved.)

Believable versus Real

In motion picture storytelling, it is more important to be believable than it is to be real. Since the software used for modeling has its roots in mechanical engineering, it is all too seductive to get caught up in attempting to make something mechanically real. Production budgets are built around what it will take for modelers to assist in the creation of something that is believable and helps to tell the story. Many people who come out of an industrial design background tend to overengineer their models yet learn that the requirements, though certainly high, are not as precise as might be needed in something that would actually be built.

Model Data Types

The three main data types for models are polygons, NURBS, and subdivision surfaces. Another emerging type is voxel technology.

Polygons: Part 1

The first data type used for entertainment purposes was the polygon data type. A polygon is a basic piece of geometry defined by three points or vertices—a triangular face. A model is made by assembling many of these triangles until a more complicated form is created. Originally, polygons were textured using a series of texture projection techniques.

Figure 7.3 Meeper is an example of a multi-patch NURBS Model. (Image © 2002 Sony Pictures Imageworks Inc. All rights reserved.)

If multiple planar projections were to be used, an alpha channel would be created to fade off one projection and fade in another. This technique had tremendous inherent limitations and obstacles. Chief among them was stretching of the texture as it approached areas where the normal of the surface would move toward being parallel with the projection plane.

NURBS

The second data type emerged out of the design industry and it solved many of the texture projection issues. NURBS (Non-Uniform Rational B-Spline) have texture coordinates (UVs) already associated with the geometry. A NURBS plane is essentially a mathematically defined surface that uses a B-spline¹ algorithm to generate a patch. A model built-in NURBS is a series of these patches quilted together. An average character model may be made up of hundreds of these NURBS patches.

Although using NURBS patches solved many texturing issues, it introduced considerable file management overhead. For each NURBS patch, a unique texture map was created. This meant that for color alone there were hundreds of texture maps to paint and manage throughout a production. Adding bump and specular texture maps would multiply the count by 3.

Polygons: Part 2

Polygons became popular again when techniques emerged to manipulate the UV structure of polygons. Modelers would unwrap an object in UV space and modify the UVs to avoid the stretching pitfalls of the traditional texture projection techniques. This technique of creating a UV layout allowed for superior texturing.

At this time, polygons also began using a tessellation² technique whereby the polygonal geometry would be subdivided and smoothed at render time. This allowed the polygonal geometry to function more like NURBS with a control “cage” model, and the tessellated, or subdivided “limit,” surface derived from it. Though different from a true subdivision surface, it is often referred to as such.

Subdivision Surfaces

A true subdivision surface is different from tessellated polygons, though it is common to hear tessellated polygons called a subdivision surface. This is a data type that is probably going to be retired in the near future because it never found acceptance and tools were never developed to support it.

Voxels

In the not too distant future, modeling may be done using volumetric pixels (voxels). This is most similar to clay in that the model is made up of actual volumetric material. All of the models described above are merely exterior shells. Since voxels are geometry independent, they will provide the most freedom to the modeler. They will allow modelers to easily cut and paste portions of a model, without having to think about how to tie the faces into the model geometrically. It is very exciting technology and has made its way into visualization and medical communities.

Development of Models

The three basic routes for the development of digital models are free-form digital sculpting, 2D reference material, and scanned dimensional information.

Free-Form Digital Sculpting

In free-form sculpting, the digital modeler functions as a concept artist/designer and interacts directly with the Director, Production Designer, and/or VFX Supervisor. There has been a recent rise of the independent digital artist as celebrity designer who can deliver a design sculpt. In addition, several software tools have come into use lately for doing digital sculpting: Freeform 3D, Zbrush, Mudbox, and 3D-Coat.

Freeform 3D is the oldest and most expensive. It relies on the use of a haptic arm³ stylus and attempts to emulate a sculpting experience by use of a force-feedback interface. This system uses voxels, and topologically frees up the artist. The completed voxel model will need to be sampled and tessellated into polygons in order to export it to another package for further use.

Zbrush is probably the most common digital sculpting tool in use today. It works with polygonal data, so it does have some topological limitations, but it requires no special interface or graphics hardware, making it a very cost-effective digital sculpting solution.

Mudbox is very similar to Zbrush, but provides some very cool open GL⁴ preview technology. As a result, however, a higher end workstation, OpenGL card, is required.

3D-Coat is a new software package that may emerge as the first voxel modeler accessible to the masses.

With each of these technologies, the design phase should be viewed as linked but still separate from the creation of the production model. In particular, since voxels are not supported in a production pipeline, they need to be resurfaced into some form of polygonal or NURBS data. Many people build a polygonal cage on top of the design model using GSI, Topogun, or CySlice software, but most of the common modeling software packages (Maya, 3D Max, Softimage, or Modo) will work for remeshing the design model into a more production-friendly polygonal layout.

From 2D Imagery

This is very similar to free-form, but it involves the creation of orthographic⁵ imagery (either photographic or illustrative), which is then loaded into the sculpting or modeling software as background image planes over which the model is built. Even when using the Freeform approach, it can be advantageous to do some work in 2D before proceeding to 3D.

When imagery contains perspective, as in photographic reference, it is important to record lens/location information so that matchmoved cameras may be replicated inside the modeling software to approximate the real-life camera. The more the camera approximates an orthographic view of the subject, the more the modeling process can be streamlined.

From Scanned 3D Information

Scanning requires the preexistence of a 3D object. It can be anything from a digital maquette to a skyscraper. It is still common to employ a traditional sculptor to build a maquette for a character prior to going into 3D. In this case, the maquette is scanned using a laser or photographic system.

Laser scanners are the standard of scanning systems. They use a laser light projected onto the surface of an object or actor. The 3D contours of the subject are recorded and sampled into polygons. The most common laser scanning systems come from a company called Cyberware. There are several companies in the Los Angeles area, most with portable laser scanning systems: Gentle Giant, Icon Studios, Cyber F/X, XYZ/RGB, and Nexus Digital Studios, to name a few.

With systems becoming smaller and less expensive, it isn’t outside the realm of possibility for artists to purchase a scanner. Next Engine makes an affordable desktop scanner that sells for around $2,500.

Photographic-based scanning is less intrusive, requiring less equipment, time, and setup than the larger laser-based systems. Though the data is lower in resolution, the simplicity and speed of acquisition can be advantageous. Most photographic-based scanning uses a structured light approach whereby a grid of light is projected onto a subject from the camera. It is recorded and a 3D impression of the object is created. The limitation of this system is the resolution of the camera. Some photographic-based scanning systems are now using feature recognition to eliminate the need for the structured light pass. As digital resolution becomes finer, they may one day eliminate the need for laser-based scanners altogether. Realscan 3D, Eyetronics, XYZ/RGB, and Mimic Studios are examples of companies that do photographic-based scanning.

Lidar scanning is generally used for large-scale scanning such as entire buildings. The only limitation in these situations is accessibility to scanning locations. Lidar scanning is done from a nodal point, with laser beams cast outward from that location. The more locations from which the scanner can cast beams, the more complete the acquisition of the subject will be. In the case of capturing a building, the ability to cast beams from every side of the building and capture the roof may not be practical. In this case, the data will have holes, and modeling will need to extrapolate. As a result, it is important to capture as much photographic reference of the location to use in addition to the lidar scanning.

Processing the scan data in these situations is best performed by the company that performed the acquisition because the quantity of data can be enormous. A company that has pioneered the use of lidar data for entertainment purposes is Lidar Services Inc. out of Seattle. Gentle Giant is also now providing larger scale lidar scanning.

Figure 7.4 Lidar, survey, and photogrammetric data are often used to build accurate building and cityscape models for Spider-Man^TM 3. (Image © 2007 Sony Pictures Imageworks Inc. All rights reserved.)

Photogrammetric Data

Photogrammetry is not a replacement for lidar data, which generates a tremendous amount of information, but for some simpler objects photogrammetry may be an option. It requires less complicated equipment and is less time consuming on location. It does, however, require more post-processing and the use of specific software. PhotoModeler is probably the most common photogrammetric modeling package.

Modeling for a Production Pipeline

When creating or building a model, it is very important to take into account the needs of the major downstream partners: texture painting, character setup, cloth simulation and color and lighting.

Texture Painting Needs

With modeling and texturing tools becoming more and more artist friendly, it is not uncommon these days for a modeler to also do texture painting. Modelers and texture painters communicate through the UV layout. If it is done well, the texture painters will be able to paint on the model with little distortion. It is not uncommon for texture painters to request alternate poses of a model to better facilitate painting.

Character Setup Needs

The Character Setup team will want the least amount of resolution possible to define a surface while still providing enough points to manipulate and define the structure of the model. While specific point counts are arbitrary and difficult to nail down, control meshes, which are delivered to Character Setup, are often in the range of 8000 to 16,000 faces. It is not uncommon for modeling to support two meshes: the lower resolution control mesh, which in turn drives the higher resolution render mesh.

Cloth Simulation Needs

The Cloth Simulation team has needs similar to those of Character Setup. However, they have a few additional requirements. Simulated cloth needs data that can be easily converted into spring meshes. This requires a more grid like (almost NURBS-like) layout. This is not to say that the hero cloth model needs to be completely built in this fashion, but at the very least, the basic layout of the clothing model needs to have a layout that is more grid like.

It is not uncommon for modeling to provide two products for clothing items as well: the basic cloth simulation model, which is the model discussed above, and a more detailed final hero cloth model, which is the model to be rendered. This version would contain additional details such as pockets, buttons, zippers, etc. It will also have the more subtle sculpted details such as seams and buttonholes. It may even contain an inner as well as outer surface to provide thickness. It is important that the hero cloth model not deviate too greatly in shape from the simulation model that is used to drive it. Close collaboration between modeling and the Cloth Simulation team is essential. It is very common to make adjustments to both models throughout production.

Color and Lighting

The Color and Lighting team will want enough resolution to make sure that the main details are clear and built out—relying too much on displacement and a variety of bump mapping techniques only makes their job more time consuming.

Client Servicing Approach

Modelers need to take a client servicing approach in meeting the needs of these different and often opposing groups. Collaboration and cooperation are essential to fostering this sort of team synergy and can be just as challenging as creating a great digital sculpture.

Engineering Aspects for Polygons

The reintroduction of polygons as the main data type of choice for motion picture production use has freed up the modeler from the NURBS-centric UV structure. It has cleared the way for more organic polygonal layouts and allowed artists to more accurately register the use of space in both 3D and UV

When modeling objects, one of the most important things to keep in mind is edge flow. This is true for both NURBS and polygons, but because polygons can be so free-form, some sense of organization needs to be in place. One of the first recognized strategies uses a technique called edge loops. Edge loops simply attempt to describe the shape of the object using a series of concentric rings. What edge looping did was instill a basic modeling strategy and order on otherwise free-form meshes. It creates a mesh that is visibly ordered and would allow Character Setup to better set up their skin weights and falloffs. Edge looping is similar to NURBS, while still allowing some of the free-form advantages of working in polygons.

UV editing opened the door for texturing to have customized UV layouts. These layouts could be made any number of ways, and efficiencies could be introduced into the UV layout that might make the texture artist’s job easier.

Engineering Aspects for NURBS

Although NURBS are not as common as they once were, they are still used exclusively in some facilities around the globe. Sometimes they are still the best way to build certain objects. Some basic guidelines for working with NURBS include proper NURBS parameterization and proper maintenance of continuity between the different NURBS patches.

Proper NURBS parameterization entails having uniform parameterization between spans. This doesn’t necessarily mean that the points are all equidistant from each other, but rather that the tension between the points is fairly even. A general guideline would be to strive toward a patch with squares that are, for the most part, square. The best mesh is one with even distribution of point resolution that approaches being square.

Achieving proper continuity between NURBS patches is still an art in itself. Most packages that support NURBS have some form of patch stitching. There are several types of continuity: C1 continuity means that points on the edge of a NURBS patch are merely coincident, whereas G2 continuity means that the patches not only line up but also that there is no visible seam or discontinuity between them.

RIGGING AND ANIMATION RIGGING

Steve Preeg

Rigging: What Is It?

Rigging can be simply expressed as a process that allows motion to be added to static meshes or objects. This is not to be confused with actually moving an object because that is left to animation. Rather, rigging is what allows animators to do their job. These rigs can be made up of many aspects such as bones, deformers, expressions, and processes external to the application in which the rig is built. Rigging is considered a relatively technical process, and riggers are often required to have some scripting and/or programming skills. Python, Perl, C++, and MEL are some of the more desirable languages for a rigger to know.

In general, rigging can be broken into two distinct types: animation rigging and deformation rigging. Each of those types may be further broken down into subcategories such as body rigging, facial rigging, and noncharacter rigging.

Animation Rigging

Animation or puppet rigging is creating a rig specifically to be animated interactively. An animation rig is rarely intended to be renderable but instead a means to get animation onto a full deformation rig, which will be covered later.

It is not uncommon to have multiple controls for the same object in an animation rig, allowing animators to decide the best way to animate a particular object. The additional control structures do, however, need to be balanced with the need for speed and interactivity. Since these rigs may never be rendered and speed is critical, the quality of deformations on these rigs can be quite crude. In fact, geometry is oftentimes simply parented to nodes giving rigid deformations instead of heavier skinning or clustering.

A character rig for animation may come in many flavors and the control structures vary depending on the workflow of the animators themselves. However, most character rigs have certain standard parts.

Figure 7.5 Example of an animation rig. (Image courtesy of Digital Domain Productions, Inc.)

The ability to switch between inverse kinematics (IK) and forward kinematics (FK) is probably the most common need in a character animation rig. Inverse kinematics happen when a target position for a joint chain is animated. For example, the animator may want the foot to be in a particular place and does not care, as much, how the leg has to bend to get there. Forward kinematics is where the control for a rotation value on each joint is discreetly given. For example, first the hip is rotated and then the knee and ankle to get a new foot position. Switching between these two types of controls and matching the resulting positions when switching are common practices. Another example: As a character is walking, IK may be desirable to lock the feet down at each step. Then, if the character jumps and does a flip, it may be desirable to switch to FK. This way the foot targets do not need to be animated during the flip. Only the rotation of the hips and knees need be rotated. Making sure that the position of the leg/feet at the point of transition is critical.

Motion capture integration is another common need for an animation rig to accommodate. Motion capture is, by its nature, an FK set of motion. It also tends to have data for every joint of every frame. This can be very difficult for an animator to modify, both on a single frame and temporally. In addition, it is difficult to animate “on top of” motion capture; in other words, adding to the motion rather than replacing it. For these reasons it is often desirable to convert the motion capture from its initial FK form onto the control structure of the rig itself. The process itself is application specific and therefore not covered here but should be part of the planning if motion capture is to be used.

Local versus global control, and the ability to switch between them, is another commonly requested feature of a character animation rig. To better understand this concept, imagine animating a character with IK that needs to walk up to, and go up, an escalator. While the character is walking up to the escalator, the feet may need to be in global space. In other words, it does not matter what the torso of the character does as the feet stay locked in world space wherever the foot is placed. Once the character gets on the escalator, some ambient motion needs to be added. But having to keyframe the feet to always stay on the escalator may be difficult, especially if the animation of the escalator changes. At this point it may be easier to have the IK controls in the space of the pelvis. This way, the artist can not only move the body as a whole with the escalator, but can also have the character move around on the step that he is riding on by moving the IK controls relative to, or as a child of, the pelvis motion.

Control structures for character rigging are as varied as the characters themselves. One may need bones to be able to squash and stretch cartoony characters, while another may need to be anatomically correct for realistic body motion. Regardless of the specifics, there are some basic rules to follow.

The controls should be easily seen and accessed by the animators. Color coding controls can be an effective way to distinguish parts, or even sides, of a character. More control is not necessarily better; it can cause confusion when animators are trying to refine animation if there are too many nodes with animation data on them. Speed is critical. If an animator can scrub through a scene in real time, the artist can make many creative animation revisions in short order before rendering the scene out. A sense of timing can be determined without having to play back images or a movie file. Animators should feel that the rig is allowing them to use their talents to animate; anything that slows down that creative process should be avoided.

Facial rigging for animation could be counted as character rigging for animation, but it has specific needs that warrant separate discussion. Motion capture for the face is quite different from motion capture for the body. Various methods are used for facial capture ranging from frame-by-frame dense reconstruction, to marker points on the face, to video analysis. Whichever method is chosen, the integration to the rig is tricky to say the least.

In contrast to body motion capture, where one can explicitly measure, for example, the rotation of an elbow joint, nothing is quite as straightforward in the face. The face may be approached as phonemes, muscles, expressions, etc. A common starting point is Dr. Paul Ekman’s system, which is described in his book Facial Action Coding System (FACS) (Ekman and Friesen, 1978). This system essentially breaks facial motions down into component movements based on individual muscles in the face. Although it is a common starting point, it is generally understood that it needs to be expanded from its academic version. Once a system like FACS has been chosen, the process of mapping motion capture data to corresponding FACS shapes is a substantial problem.

As with body animation rigs, control structures that are easy to understand, and not overly abundant, can increase the speed as well as the quality of the animation. Unlike body animation rigs, facial rigs can become slow, because it is imperative to see final quality deformations in the animation rig. It is common to see final mesh resolutions and render-ready data in a facial animation rig. The reason is simple: That level of detail is needed to make quality facial animation. Seeing exactly when the lips touch, for example, is critical for lip sync. Dynamic simulations run after animation can add believability. However, care must be taken to leave the critical parts of the face that animation has spent fine-tuning, such as the lips and eyes, unchanged.

Noncharacter rigging, or elements of characters that are mechanical, represents another area of rigging. Examples of this type of rigging would be vehicle suspensions, tank treads, etc.

Control structures for noncharacter rigging change dramatically depending on the level of detail and importance of the action in the shot. For example, there may be a close-up of a complicated linkage in some sort of transformation where animators would need control over the speed and movement of a large number of pieces. Alternatively, there could be a panel opening on a tank far from the camera that merely requires a simple rotation. Clearly the control structure would be different in these two cases. Keeping the control as simple as possible for the needs of the shot can save a lot of time and effort.

Part of keeping the control structures simple for noncharacter rigging is automating ancillary motion. For instance, an opening hatch may need dozens of screws that are turning. The turning of these screws can be automatically driven by a single parameter that opens the hatch. Many mechanical systems are designed to work in films. Taking advantage of systems that only work one way can be a huge benefit to animation. Pistons, flaps, and springs are all good candidates for automated rigging. Often it is desirable to have overrides on these types of controls, which is relatively easy to integrate most of the time.

Animation rigs are often designed not only to deliver animation to a renderable rig but also to prepare for simulations. A tank tread, for example, may have animation controls to move the individual tread pieces around the base of the tank, but a simulation may be run to add vibration and undulating motion, suggesting slack in the treads. Extra values or attributes can be passed to the simulation software based on the animation rig and its motion. Velocity, direction of travel, etc., are values that may enhance the quality of the simulation. Whether these simulations are run by the rigging department or passed off to a dynamics group, preparation for simulation can be a valuable tool supplied by rigging.

Deformation Rigging

Riggers often spend the majority of their time on deformation rigging. The goal of the deformation rig is to take the animation data from the animation rig and deform the final renderable mesh as dictated by the needs of the show. The quality here can vary greatly from far-away digital doubles to hero characters that need to have the feeling of skin sliding over muscles or animated displacement maps.

Figure 7.6 Example of a deformation rig. (Image courtesy of Digital Domain Productions, Inc.)

The first step is to accept data from the animation rig. This is done in many ways including constraining the deformation rig to the animation rig, or having an equivalent set of nodes or joints in both the animation rig and deformation rig. The second example allows the animation to be exported off the animation rig and directly applied to a matching set of objects, or nodes, in the deformation rig. Facial animation may be converted directly, vertex by vertex, or through matching nodes. There are many ways to convert animation data to deformation rigs. These are frequently application specific, so no detailed descriptions are provided here.

There are many different types of deformers and, of course, their names and exact capabilities are application specific. In general they may be grouped into a few different types. One type is linear deformers. This is the simplest and most predictable of all the deformers. However, they are often blamed for deformations looking CG. In general, that is more a misuse of them than a problem with linear deformations themselves. They also do offer some good advantages—for instance, they are extremely predictable. Through simple mathematical calculations they may be reversed and broken down into individual components when multiple linear deformers are affecting a single point. In addition, linear deformers tend to be very fast. Compared to deformation rigs, which might become slow, this can be a huge advantage.

Nonlinear deformers are another common type of deformer used in character rigging. These deformers may be tricky to implement and often have somewhat unpredictable results. For example, a deformer that bulges a surface based on a sphericalshaped object coming into contact with vertices may end up distorting the mesh significantly by allowing the vertices to slide over the spherical shape rather than push them out along their original normals. Nonlinear deformers often lack the control level to change the behavior to a directable level.

Other typically nonlinear deformers that deserve mention are muscle systems, which come in many varieties. They can be dynamic or nondynamic. They may have volume preservation, with stretch and slide attributes, etc. In general one can imagine implementing a muscle system on a character by imitating real-life muscle shapes and attachments and then binding the surface mesh to them. In practice, however, that rarely works. The algorithms used in most muscle systems do not behave the way a real muscle affects real skin. Therefore, the effort to accurately simulate an entire character’s muscle structure will not give accurate skin motion. Instead, most muscle systems are implemented using anatomically correct muscle placement as a guide for placement and behavior. Then, contrary to real muscle systems, extra muscles are placed or attachments adjusted until the skin moves in a desirable way. Some muscle systems also allow post-deformation adjustments at the vertex level. This can save huge amounts of time when trying to fix a muscle’s contribution to a skin.

Muscle systems, when used correctly, supply certain advantages such as simulated skin sliding or bulging, based on compression of a muscle and its associated volume conservation. In general, the amount of time it takes to rig a character with a full muscle system, compared to standard skinning methods, is much longer. Therefore, the quality of deformation and range of motion that a character is required to do in the project may determine whether or not to build a full muscle rig. As computers get faster and new algorithms are implemented for muscle systems, they could become a more standard rigging tool for the future. However, at this time there is still a need for the simpler rigging solutions in many cases.

Regardless of the method, or methods, finally chosen to rig a character, there is often a need to clean up what the rig generated. Dynamic simulations can be run and smoothing algorithms can help remove pinches or spikes in the deformations. In cases where vertex-by-vertex accuracy is required, as is often the case in facial animation, shot modeling may be considered.

Shot modeling is a phase of production in which a rigging artist adjusts a model specifically for a shot instead of rolling global changes back into a rig to have it affect other shots as well. Shot modeling, though powerful in terms of the level of control per vertex it allows, does have its problems. Shot modeling is dependent on the animation, so if the animation changes, the shot modeling needs to be readdressed. Shot modeling is also time consuming and, as the name suggests, is only useful for that one shot. Post-deformation cleanup, in any form, can be a huge time-saver over refining a rig for months to make sure it stands up in any and all shots.

Noncharacter deformation rigging is usually less time consuming and may have more of an impact on rendering and lighting optimization than on the deformations themselves. It is often more efficient to distinguish between deforming and nondeforming geometry and represent nondeforming geometry in static caches on disk, which can then simply have a transform applied at render time. Many current renderers support this functionality, so the deformation rigger should be aware of when it may be used. As with character deformation rigs, passing animation data can be done in the same ways, and ancillary motion can be driven automatically as in the animation rigs.

Last, there is often the need to prepare deformation rigs to go through a postprocess such as simulation. Again, this can be done on geometry caches, live rigs, etc. Sometimes these simulations are multiple rigs that are dependent on an earlier piece being simulated first; for example, a cloth simulation for clothes hanging on a drying line may need a simulation of the clothesline first. That is a simple case, and it is not uncommon to have many levels of dependencies.

Reference

Ekman, P, & Friesen, W. (1978). Facial action coding system: a technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.

TEXTURING AND SURFACING

Ron Woodall

The Importance of Texture Painting

The texture artist, along with the modeler, is at the head of the production line. Being able to successfully render and bring a shot to completion depends a great deal on the texture work being complete.

In the world of computer graphics, the texture artist can serve many roles: makeup artist, scenic painter, costume designer, model painter, or all of the above on any given project. These mind-sets can be quite different depending on the type of model that is being painted. If a machine is being painted versus a creature, or a set versus a human, the surfacing approach and methodology are different for each.

Hard Surface Models

When painting a vehicle the texture artist has to think about its direction of travel and how that might bias dirt toward one side versus another. Where is the model most likely going to have scratches or damage due to wear and tear? Generally, ground and air transport have the most damage to their finish coats on their leading edges because those are the surfaces most often hit by bugs, rocks, dirt, or any other thing that may be in the vehicle’s path of travel.

Thought also must be given to individual parts of a machine. Where would it get hot and discolored or show cracked paint from the exhaust? Where are the joints that may leak oil? Where would dirt most likely collect? Where then, would rainwater flow over the surface and wash dirt away? There are many things to consider when making a very unreal object look real. One of the most important things a texture artist provides to CG work is imperfection. By nature, computer-generated models tend to be too perfect looking and symmetrical. Generally, the human eye is quick at splitting masses in half and spotting symmetry, so providing asymmetry to a model is a good thing.

If the model to be textured exists as a real vehicle or a practical on-set model, the job is somewhat easier. These real things can be photographed to provide reference and, in some cases, texture source. These practical set pieces, therefore, need to be thoroughly documented. If the texture artist cannot be on set to document these objects, a list should be prepared of every view that is needed. Evenly lit, shadow-free photos that may only need some minor touch-ups to remove highlights can be applied directly to a model to move very quickly toward a finished paint job. However, if a spaceship, for instance, only exists as a piece of artwork, then much thought needs to be given to the previously mentioned aging factors in order to create a paint job that looks believable.

For a model that is entirely fictitious with no on-set counterpart, a photographic source is still the best reference. Photographs of oil stains, dirty cars, the back of a garbage truck, etc., can provide the artist with believable detail to add to the work. There are also many free websites or sites that for a small annual or monthly fee provide a cornucopia of high-resolution texture sources. Painting every detail from scratch is commendable, but very foolish, because it will most certainly take more time. Wherever possible, the use of some combination of painting and using source material is highly recommended.

With the constantly shrinking schedules of today’s production world, finding every possible way to get to a great end look quickly should be the goal of every texture artist.

Scale is another major factor in believability, especially with large-scale models. Having a human model in the same scale as the subject model is useful. Use the human model placed in various positions next to the subject model to create a clear sense of scale for paintwork. This practice helps immediately show existing scale problems. For instance, if surface scratches on an object need to be one-eighth of an inch wide in reality, and the scratches that have been painted are much larger, it will be readily apparent.

It is also important to do test rendering with bump and displacement maps. They are first set to whatever height is appropriate for the scale of the model chosen by the texture artist. As an example measurement ratio, 1 unit equals 1 foot. Keeping this in mind, initial bump and displacement heights for normal-sized models can be set to 0.02075 and 0.083, respectively. Knowing that details are being created with a maximum depth or height of 0.25 inch in bump and 1 inch in displacement makes the process of creating these maps happen more easily.

Certain camps believe that modeling every little detail is the only way to achieve realistic results. The majority of texture artists argues that this methodology is not only foolish, but also wrong. For example, if a modeler chooses to create a row of rivets on a surface, he or she will use algorithms that instance and place the rivets. This will create a perfect row of details, which normally is not desired. Creating imperfect fine details via bump or displacement is a much faster process. It’s also much easier to achieve synchronicity of maps for dirt, oil, and other effects with these details. If all of the small detail is modeled, the only way to get dirt in around it is to paint it in 3D. Faux details by nature are 2D and, therefore, can be painted in the various effects without having to interact with the model at all. Moreover, an incredible amount of time is also saved; creating UV layouts and texture assignments for tons of little details is not necessary.

Creature Models

Painting creatures provides an artist with a whole new list of concerns. Creatures can require thought about their age, their skin condition, how they move, and many other factors. Where is the skin most likely to have wrinkles from repetitive expansion and contraction? Where is the creature most likely to have dry skin or calluses from contact with the ground? What does the creature eat and how might that stain skin or fur if it has any? What is the creature’s native environment? Is it dirty? What color is the dirt? Where is the creature going to collect dirt? On its belly? Does it sleep on one side? Maybe one side is dirtier than another. This all may sound absurd, but small details such as these must be considered in order to foster believability in something entirely fanciful.

Like the hard surface model, if a texture artist is matching a real animal or a puppet from on set, the importance of a complete, thorough, and properly lit set of reference photographs of these practical creatures cannot be underestimated. The minimal time and money spent on gathering great photographic reference pales in comparison to the money that may be spent chasing and correcting textures painted from horrible reference. If an artist starts with improper reference photography, it can take forever to reach a point where the creature looks correct. If the artist has no other choice than to start with improperly gathered reference, additional relative imagery must be gathered from online sources. The artist could also take a field trip to the zoo or local pet store to take reference photography.

Types of Geometry: Their Problems and Benefits

Polygons

Polygonal modeling was once viewed as the dusty forgotten forefather geometry type but has come back around to being the preferred geometry type for productions. A whole host of robust UV layout software packages and options are available within modeling packages that really allow the user to achieve great texture setup with polygons.

Being able to easily texture polygons makes them very appealing to use since polygons are less data intensive and give way to faster render and interaction times. Visually complex parts comprised of one object can be made using polygons. This is a huge plus for most rendering packages. Being able to combine 50 objects into one and still get good texture layout makes polygonal modeling the way of the foreseeable future. The only major downside to polygons is that they shade too perfectly and therefore do not look real. More texture work has to be put into the flat areas and edges of polygons to give them a real level of imperfection.

Catmull-Clark Surfaces

The Catmull-Clark surface is the city cousin of the polygonal surface. Created by Edwin Catmull and Jim Clark, Catmull-Clark surfaces are primarily used for creature modeling or hard surface objects that require compound curved shapes. Catmull-Clark surfaces are used to render complex organic objects from light polygonal proxy objects. A Catmull-Clark is automatically made when a subdividing scheme is applied to a polygonal mesh. A six-sided cube, after one level of subdivision, becomes an icosikaitetragon having 24 sides and appears more like a sphere.

While staying light in interaction preview mode, subdivision surfaces can become very visually complex with a quick keystroke that applies the subdivision scheme. Most Catmull-Clarke meshes are rendered at subdivision level 3. Because this geometry type is ever in flux, so is the UV layout for it. The layout for a Catmull-Clark object looks different in preview mode versus subdivision level 3. It’s important to make sure that any part designated to render as subdivision be painted as such. If a surface is painted in polygonal mode and rendered as subdivision, all of the corners where the UV layout have been cut will show gaps in the paint over the entire model.

It may be difficult to make UV layout cuts in areas with only a few control vertices. Often when cutting in these light areas, the paint gets weird and stretchy. Adding another edge to the left and right of the cut edge usually fixes the stretching problem. More often than not, UV layout work is far more difficult on subdivision models than it is when working on polygons.

Splines and NURBS

B-splines and NURBS are somewhat extinct in production today. The one place they are still used is modeling. The painfully missed benefit of spline-based geometry is ready-made UV coordinates. By nature, spline surfaces have a UV layout the instant they are created. Sometimes the layout needs modification, or multiple surfaces need to be stitched together and assigned to a single texture but at least there is a starting point. Splines are easy to set up for painting, but very data intensive. Spline models almost always take much more information to describe a surface than a polygonal or subdivision model of the same subject.

Prepping the Model to Be Painted

UV Layout

Before painting can begin the model has to be prepped to receive paint. This is done most often by creating UV layouts. A UV layout is a 2D representation of points and edges defining texture allocation that correspond to the 3D control vertices and face edges of a model. The UV layout determines how much texture resolution each polygonal face will receive once textures are applied. Generally, texture artists prefer to create their own UV layouts. Most automatic layouts tend to be highly inefficient with regard to optimal usage of texture space. They make for poor memory usage during real-time interactions.

Longer render times are also an unwanted side effect of poor layout and map assignment. Automatic layouts also tend to be extremely visually incoherent. Often in these layouts, parts in proximity on a model tend to have no relationship to one another in 2D layout space. Painting in 2D is easier if parts are laid out in a thoughtful way with UV orientation closely matching that of the geometry and with neighboring model parts near each other in UV space. Painting in 2D is more efficient because the resolution is one to one and interaction with the model is not a factor. Using automatic layouts and painting solely in 3D can create models that have far too many giant resolution textures with overly soft paint mapped into them.

Projection Mapping

Projection mapping is a tried-and-true method of applying texture to a model. Because of the brute force nature of applying textures via projection mapping, this method only works well for hard surface models.

Projection mapping has many merits. Because each pixel of the texture is being projected on an infinite line through any geometry associated with the projector, the information stored in the texture file to describe this mapping is far lighter than a model mapped entirely via UV layout. With UV layout, every single control vertex in the model has multiple coordinate numbers associated with it to describe the vertex location within the UV layout. This can result in many thousands of lines of information just to map one piece of geometry. Most projection coordinate information equates out to less than a dozen lines of information in the texture file. Using projection maps wherever possible results in lighter files. Lighter files equal faster loading, interaction, and render times. Using projectors also eliminates the time needed to create UV layout for the parts being projected.

Because the paint is being blasted through the model from a single axis, any surfaces perpendicular to the projection plane will have streaked paint. If a surface tilts away from, or toward, the projector at a shallow enough angle, a projection map will usually still work well. Once surface pitch out of parallel with the projector approaches around 80 degrees, the paint starts to stretch too much. In cases where this occurs, those surfaces should be removed from the projector and placed in a projector that is closer to parallel with the problem area. The other option is to UV map the parts that do not work well with projections. Because of it strengths and limitations, projection mapping is best suited for models that don’t need to stand up to close scrutiny or models that need to go into the rendering pipeline quickly.

Camera Mapping

Camera mapping is very similar to projection mapping with the exception that the paint can be mapped from a camera having any field of view up to, and beyond, 180 degrees. Because the paint can be mapped from a nonorthographic view, camera mapping is ideally suited for adding detail to paint on a shot-specific basis. Models that have limited close-up screen time can be painted solely via camera mapping. For instance, a model that is seen closely in only three shots could be painted from the three shot cameras and most certainly take less time to complete than painting a model that has to stand up to scrutiny from every possible angle.

Volumetric Mapping

With volumetric mapping the texture is described in 3D, instead of 2D, with images being applied to the surface. Because this method of mapping requires no assigning of images or creation of UVs it may very well be the fastest way to get going on a paint job.

In its simplest form, volume mapping only requires the artist to select the model or parts of the model and assign a pixel density value to the selected volume. In a matter of seconds, a model can be ready to paint from any angle without fear of pixel stretching or resolution variation from surface to surface. However, because there are no 2D images, all changes great and small have to be done solely in 3D. With UV mapping or projection mapping, paint versions or changes can be made by simply referencing an alternate version of an original texture. With volume mapping this is impossible since any one given part of the model is part of the whole texture volume. There is no easy way to remove only a portion of the model from the volume and modify the paint of just that portion.

Texture Creation

Texture from Source

When matching to an existing prop, set, or costume, the reference photos should be lit as perfectly and evenly as possible so that they can be used as source texture with a small amount of cleanup. It is also very helpful to ask for painted swatches from the set painters, as well as fabric swatches from the costume department. If any fabrics have a pattern, be sure to ask for a swatch large enough to cover the repeat of the pattern. Actual painted card and fabric swatches are great because they can be easily turned into textures via a flatbed scanner.

Pure Painting

Purely painting textures doesn’t happen all that often (perhaps more often in digital animated features than for live-action films). Even then, using source texture where possible is wise because painting every last little detail is time consuming and may in the end not look as real as some combination of painting and source material.

Finding or Making Source Texture

The best weapon for a texture artist is an extensive, organized source library. In addition to whatever source or reference may come from location, gathering personal source texture is a great practice. Keep a camera on hand for the great textures that pop up in everyday life. If a certain texture is needed for a project, but can’t be found, make it! Just a half hour outside with India ink and printer paper can result in a wealth of scannable splatters, drips, and stains for use on models.

Diffuse Maps

When creating diffuse maps from scratch, the paint values should be created as the surface would appear in shadowless light. In some rare situations a bit of stage painting can help model details read better. Generally though, diffuse color maps should be free of highlight and shadow.

Small, faux-painted details, such as holes in a surface that do not require an accompanying opacity map, can be painted as zero black to help cheat them as an area that light is entering into but not returning from. These details should also be painted zero black in the specularity map.

The extreme values of light and dark in color maps should be no lighter than 80% and no darker than 20%. This helps to prevent the paint from looking too illustrative and provides some range for a technical director (lighting artist) to push the values up or down in the materials where needed.

Bump Maps

Bump maps are most easily equated with topographical maps in real life. With topographical maps certain colors represent a range of altitudes. Red might represent sea level, orange 1000 feet above sea level, and so on. With bump maps instead of colors, the height change on the model surface is described in values of gray. Most often middle or 50% gray is the pivot point. Any value less than middle gray will read as an indentation, and any value above middle gray will read as a protrusion.

Bump mapping is merely a method of making a simple surface look more complex by modifying the surface normal. The height of a bump map in the material should be rather shallow. Since bump maps don’t really change the surface topology, if the height value is much more than one-quarter of an inch, the trick gives itself away. Looking at the middle of the model, the bump map may provide great believable detail, but around the edges of the model there is a perfectly smooth profile. This instantly reveals that the surface detail is merely a cheat. Details that need to have more than a quarter-inch of relief off the surface should either be modeled or added via displacement maps.

Displacement Maps

Like bump maps, displacement maps are described with a range of gray values. Displacement maps are used to actually change the surface detail. If the pivot point in the material is middle gray, anywhere the paint is lighter or darker than middle gray, the paint will get converted to micropolygons pushing out of, or into, the surface of the model.

The result of paint displacement is an actual geometry change. Unlike bump, displaced details will read when the surfaces are in profile or edge on. Displacement mapping takes far longer to calculate at render time; therefore prudent use of displacement is advised. A model with every square inch of surface area displaced will certainly take far longer to render than one with efficient use of bump and displacement.

Specularity Maps

Specularity maps are used to control the specular reflectivity of a model. Typically these maps should have a full range of values from white to black all over, with white representing the most specularity and black presenting none.

More complicated uses of specularity maps involve setting up the material to behave like two or more materials based on a range of values in the map. For instance, 0% to 33% gray has a spec model of tire rubber, 33% to 66% like brushed aluminum, and 66% to 100% has the specularity of chrome. This way of painting helps create more believable faux details on the surface of a single mesh object because of the obvious material changes. On a single mesh object one could paint a rubber gasket surrounded by chromed fastener heads in the displacement and diffuse pass. By painting the same details in the specularity map, each within the appropriate range of gray, the details will have very divergent material looks from the base material of brushed aluminum.

Material Designation Maps

Material designation maps or area maps are another way to create the look of many material types within a single mesh object. Area maps are usually solid white on black maps, with no range of grays in between. Black represents the base material, and white represents the areas where it is desired to switch to an alternate material. By using area maps one can reserve the full range of the specularity map solely for attenuating the specularity and use one or more area maps to switch to various other material types.

The benefit of using area maps is that multiple material switches can be separated cleanly into individual area maps. This separation makes it easy to change how much of any one given material is being used without accidently changing any others.

Luminance Maps

Luminance maps can be used to create faux lighting such as many pools of light on a surface, backlit windows, or hundreds of point lights. Where it may be too render intensive to have many actual lights being calculated for every frame, luminance maps can be used. These maps can be painted or more effectively rendered with real lights and baked into texture maps. Luminance maps should be solid black everywhere except where the surface is being lit or is supposed to be a faux light source.

Various Other Map-Driven Effects

Many other separated maps can be used to add interest to a model; dirt, oil, and rust maps, just to name a few. These maps are essentially the same as area maps since dirt, oil, and rust all have their own material properties.

These maps should be painted pure white on black. White represents dirt or whatever effect is being created, and black is the base material. These additional effects should be set up in the material to use a procedural color map or a single color. Since the percentage of surface area typically covered by dirt, oil, or rust is so small, it doesn’t warrant the efficiency hit of an additional full-color map for each effect on every part of the model.

Help from Procedural Textures

Procedural textures are computer-generated images that are based on user-created code. Using mathematical algorithms to create real-looking materials like wood, stone, metal, or any surface that has a random pattern of texture, the true strength of procedural textures is infinite resolution. Use of procedural maps can allow the camera to travel from far away to within a sixteenth of an inch above the surface of a model without losing textural detail. On the whole, procedural color maps still tend to look fake, but procedural bump and displacement can be quite convincing.

Another strength of procedural maps is their efficiency during interactivity and during render time. Because no images are being stored on disk to create the look of the procedural texture, none is called up at render time or when a model is loaded and worked on. Procedural maps are a great way to add an extra high level of detail to surfaces at a relatively very low cost.

Film Look

Most visual effects facilities receive lookup tables (LUTs) from a facility that is color timing the entire film. LUTs can be used to preview on a monitor or digital projector how an image will be reproduced on the final film print. It is important to periodically check how rendered paint jobs look with a film look applied. This helps ensure that the film look is not shifting values or colors so much that things appear to be broken.

Always painting with the film look on is a bad idea, however, unless the reference images for the model being painted have been pre-processed to have the film look baked in. The other exception is when the model being painted has no real-life counterpart on set. In this case, there is no harm in painting with the film look on, since there is nothing real to match.

Texture Painting in Production

It is very important to keep a production on schedule. With that goal in mind, creating a cursory paint job in the course of a day or two that can be handed off to the look development artist gives that artist something to start building material work on. Most average paint jobs take around 2 to 6 weeks to complete. Making look development artists wait this long is foolish, since that artist can make paint work quite a bit easier with great material work.

It is also very important to keep in regular contact with supervisors and art directors multiple times a day. Any time a personal benchmark is reached, a test render of the paint should be sent to the VFX Supervisor, the Paint Supervisor, and the Art Director for input. Making a habit of touching base often will ensure that the work stays on the right track.

Other important points of focus include understanding real-world material behavior and understanding how textures affect materials. Understanding the physics of how and why solid plastic responds to light differently from cast iron will help in the creation of better textures. Also understanding expected gamma settings and pivot point defaults for the various texture map types will help make work go faster and easier. A good solid understanding of how all the various painted maps affect the materials to which they are assigned is imperative to prevent making more work than is necessary.

Making self-created backups is another important mode of working during production. Mistakes happen to everyone. Creating iterated master files, like Photoshop files that contain all of the layers unflattened, makes it easier regain lost paint or make corrections and resave from the master file later in the production schedule.

Some facilities have an automated nightly backup system, but relying solely on this type of system cannot be done. While a restore can take hours, it takes mere minutes to find one’s own organized master files. Naming the master files with the same naming convention as the textures they represent with the exception of extension and iteration number makes a quick workflow. For example, finding /car/images/master/Rlite_C.3.psd and saving a copy of it to /car/images/color/Rlite_C.tif is easy because the two locations are close together, and the names differ by only one character and the file extension.

Model Editing

It is important for painters to understand what modelers do to foster better communication between the two disciplines. Learning at least the basics of modeling skills helps a painter to better convey any needs to a modeler more easily. At the bare minimum learn enough about modeling to be able to converse intelligently with a modeler about problems.

Conclusion

Texture mapping is every bit as important as any other part of the pipeline. Often overlooked in the end, without texture maps from a skilled artist, most CG would inevitably fall far short of looking believable. Imagine a world covered in glossy smooth gray plastic and one will begin to understand the importance of good textures.

DIGITAL HAIR/FUR

Armin Bruderlin, Francois Chardavoine

Generating convincing digital hair/fur requires dedicated solutions to a number of problems. First, it is not feasible to independently model and animate all of the huge number of hair fibers. Real humans have between 100,000 and 150,000 hair strands, and a full fur coat on an animal can have millions of individual strands, often consisting of a dense undercoat and a layer of longer hair called guard hair.

The solution in computer graphics is to only model and animate a relatively small number of control hairs (often also called guide or key hairs), and interpolate the final hair strands from these control hairs. The actual number of control hairs varies slightly depending on the system used but typically amounts to hundreds for a human head or thousands for a fully furred creature. Figure 7.7 shows the hair of two characters from The Polar Express (2004), a girl with approximately 120,000 hair strands and a wolf with more than 2.1 million hairs.

A second problem is that real hair interacts with light in many intricate ways. Special hair shaders or rendering methods utilizing opacity maps, deep shadow maps, or even ray tracing are therefore necessary to account for effects such as reflection, opacity, self-shadowing, or radiosity. Visual aliasing can also be an issue due to the thin structure of the hair.⁶

Another problem that can arise is the intersection of hairs with other objects, usually between hairs and the underlying surface, such as clothes or the skin. For practical purposes, hair/hair intersections between neighboring hairs are usually ignored, because they do not result in visual artifacts.

In almost all cases other than for very short animal fur, hair is not static during a shot but moves and breaks up as a result of the motion of the underlying skin and muscles, as well as due to external influences, such as wind and water. Often, dynamic simulation techniques are applied to the control hairs in order to obtain a realistic and natural motion of the hair. In the case of fully furred creatures with more than 1000 control hairs, this approach can be computationally expensive, especially if collisions are also handled as an integral part of the simulation.

Figure 7.7 Examples of digital human hair and animal fur. (Image © 2004 Sony Pictures Imageworks Inc. All rights reserved.)

Finally, for short hair or fur, some special purely rendering techniques have been introduced that create the illusion of hair while sidestepping the definition and calculation of explicit control and final hair geometry. These techniques address the anisotropic⁷ surface characteristics of a fur coat and can result in convincing medium to distant short hair or fur shots but can lack close-up hair detail and don’t provide a means for animating hairs.⁸

The next subsection introduces a generic hair/fur pipeline and explains how a specific hairstyle for a character is typically achieved during look development and then applied during shots.

Figure 7.8 Pipeline diagram. (Image © 2009 Sony Pictures Imageworks Inc. All rights reserved.)

Hair Generation Process

Figure 7.8 shows a diagram of a basic hair generation pipeline. The input is the geometry of a static or animated character, and the output is the rendered hair for that character. The hair or fur is generated in a series of stages, from combing (modeling) and animating of the control hairs to interpolating (generating) and rendering of the final hairs or fur coat.

In a production environment, this process is often divided into look development and shot work. During look development, the hair is styled, and geometric and rendering parameters are dialed in to match the appearance the client has in mind as closely as possible. During shots, the main focus is hair animation.

Look Development

The Hair Team starts out with the static geometry of a character in a reference pose provided by the modeling department. One obvious potential problem to resolve immediately is to make sure that the hair system used can handle the format of the geometric model, which most likely is either of a polygonal mesh type, a subdivision surface, or a set of connected NURBS⁹ patches.

With the character geometry available, placing and combing control hairs are the next steps. Control hairs are usually modeled as parametric curves such as NURBS curves. Some systems require a control hair on every vertex of the mesh defining the character, whereas others allow the user to freely place them anywhere on the character. The latter can reduce the overall number required since it makes the number of control hairs independent of the model: Larger numbers of hairs need only be placed in areas of the body where detailed control and shaping are needed (such as the face). Keeping the number of control hairs low makes the whole hair generation process more efficient and interactive, because fewer hairs need to be combed and used in the calculations of the final hairs.

Figure 7.9 Left: Uncombed/combed control hairs. Right: Rendered final hairs. (Image © 1999 Sony Pictures Imageworks Inc. All rights reserved.)

Figure 7.9 illustrates a set of uncombed and combed control hairs from Stuart Little (1999). One such control hair with four control points is enlarged in the lower right corners. Similar to the number of control hairs, the number of control points per hair also has an effect on performance.

When control hairs are initially placed on a character, they point in a default direction, usually along the normal of the surface at the hair follicle locations. Combing is the process of shaping the control hairs to match a desired hairstyle. This can be one of the most time-consuming stages in the whole hair generation process. Having a set of powerful combing tools is essential here.¹⁰ For short hair or fur, painting tools are often utilized to interactively brush certain hair shape parameters, such as hair direction or bend over the character’s surface.

Interpolation is the process of generating the final hairs from the control hairs. One or more control hairs influence each final hair. The exact algorithm of interpolation depends on the hair system used, but the most popular approaches are barycentric coordinates, where the shape of a final hair is determined by its surrounding control hairs (requiring a triangulation of those control hairs or one per vertex of the mesh) or weighted fall-off based on the distance of a final hair to its closest control hairs.

The interpolation step requires an overall density value to be specified that translates into the actual number of final hairs generated. Frequently, feature maps (as textures) are also painted over the body to more precisely specify what the hair will look like on various parts of the body. For example, a density map can provide fine control over how much hair is placed in different parts of the body. In the extreme case, where the density map is black, no final hairs will appear; where it is white, the full user-defined density value applies. Maps can be applied to any other hair parameter as well, such as length or width.

It is also helpful during the combing process to see the final hairs at the same time when combing the control hairs, and most systems provide a lower quality render of the final hairs at interactive rates. This is shown in Figure 7.10, an early comb of Doc Ock in Spider-Man 2 (2004), where the control hairs are in green, and only 15% of the final hairs are drawn to maintain real-time combing.

The hair/fur is then rendered for client feedback or approval, usually on a turntable, with a traditional three-point lighting setup to bring out realistic shadows and depth. During this phase, a Lighter typically tweaks many of the parameters of a dedicated hair shader like color, opacity, etc., until the hair looks right.

Look development is an iterative process in which the hairstyle and appearance are refined step by step until they are satisfactory and match what the client wants. Once approved, shot work can start and a whole new set of challenges arises with respect to hair when the character is animated.

Figure 7.10 Control and final hairs during combing. (Image © 2004 Sony Pictures Imageworks Inc. All rights reserved.)

Shot Work

Animators don’t usually directly interact with the hair. When they start working on a shot that includes a furry or haired character, it doesn’t have any hair yet. It is therefore helpful to have some sort of visual reference as to where the hair would be to help them animate as if the hair were present. This can be in the form of a general outline of how far out the hair would be compared to the body surface, such as a transparent hair volume, or it might be a fast preview of what the final hair look will be. This is important because it limits the number of corrections that need to be done once hair is applied.

As described earlier, the hair appearance is decided in the look development phase. This includes where all the hairs are on the body and their shape and orientation. When an animator creates a performance for a character in a shot, the hair follows along with the body, and remains stuck to the body surface in the same positions it had in the reference pose. This provides what is known as static hair; even though the hair follows along with the body, the hair itself is not animated. If all that is needed is motionless hair on an animated body, then hair work for the shot is done.

Usually, hair motion and dynamics are an important part of making a shot look realistic and can contribute to the character’s performance. Hair motion can be:

•   Caused by the character’s animation: If the character folds an arm, the hair should squash together instead of going through the arm.

•   Caused by interaction with other objects: If the character is holding something, walking on the ground, or hitting an object, in all cases the hair should be properly pushed away by the object instead of going through it.

•   Full dynamics simulation: Short hair may have a bouncy motion (and take a few frames to settle), whereas long hair will flow with the movement and trail behind.

These situations are usually handled by applying some sort of dynamics solver to the control hairs, so that they react in a physically realistic way, while moving through space or colliding with objects. The final rendered hairs then inherit all of the motion applied to the control hairs.

Third-party hair software packages provide dynamics systems, but some facilities also have proprietary solvers. There are many published methods of simulating hair dynamics (superhelix, articulated rigid-body chains), but the most widely used are mass spring systems, much like in cloth simulations. Whereas these methods provide solutions for physically correct motion, animating hair for motion pictures also has a creative or artistic component: The hair needs to move according to the vision of the movie director. Therefore, the results of dynamic hair simulations are often blended with other solutions (hand animated, hair rigs, static hair) to achieve the final motion.

General Issues and Solutions

Combing

As mentioned earlier, combing can be an elaborate task during look development, especially for long human hair and complex hairstyles. It is therefore crucial for a hair artist to have access to a set of flexible combing tools. Even for a commercially available system like Maya Hair, additional custom tools can be developed to facilitate the shaping of control hairs. For example, to make it easy to create curls or form clumps between neighboring hairs as shown in Figure 7.11, left, the user selected a group of control hairs and then precisely shaped the desired clump profile to specify how those hairs should bunch together.

A fill-volume tool can quickly fill an enclosed surface with randomly placed control hairs. The hair volumes created by modelers to describe the rough hair look of a character can be used to actually generate the control hairs needed by the hair artist. An example is shown in Figure 7.11, right, where braids were simply modeled as cylindrical surfaces by a modeler and then filled with hair. Figure 7.12 illustrates intricate the human hairstyles of two characters from Beowulf (2007).

Figure 7.11 Left: Clumping tool with a user-customizable profile. Right: Combing braids (from left-to-right: three braided cylinders, control hairs generated by tool, final rendered hairs). (Image © 2007 Sony Pictures Imageworks Inc. All rights reserved.)

Figure 7.12 Final comb examples, illustrating the intricate human hairstyles of two digital characters from Beowulf (2007). (Image courtesy of Sony Pictures Imageworks Inc. and Paramount Pictures. BEOWULF © Shangri-La Entertainment, LLC, and Paramount Pictures Corporation. Licensed by Warner Bros. Entertainment Inc. All rights reserved.)

Render Times and Optimizations

One difference between short hair/animal fur and long humanlike hair is the difference in combing needs. Another is render times and memory requirements. Whereas a frame of high-quality human head hair can be rendered from minutes to under an hour depending on hair length and image resolution, rendering the millions of individual hair strands of fully furred creatures can take several hours per frame using several gigabytes of computer memory, especially with motion blur and when close up to the camera.

Optimizations applied at render time can therefore be very effective. One example is view frustum culling, in which hair outside the current view is not generated and rendered, which is very useful if most of the character is off-screen. If the hair follicle location at the base of the hair on the skin is used to decide whether to cull the hair or not, a safety margin needs to be added, because (long) hair with follicles outside the view frustum may still have their tips visible in the frame. The same holds true for backface culling methods, which do not render hair on parts of the character facing away from the camera.

Another optimization technique is level of detail (LOD) applied to the hair, where different preset densities of hairs are generated and rendered for a character depending on the distance from camera. Varying hair opacity to fade out hair strands as they move between levels helps to avoid flickering. Some studios have also developed more continuous techniques, which smoothly cull individual hair strands based on their size and speed.

A feature of many hair systems is the ability to have multiple hair layers. Examples are an undercoat of dense, short hair and an overcoat of longer sparse hair for an animal fur coat or more general base, stray, or fuzz layers. Each layer can have its own set of control hairs, or share them, but apply different stylistic parameters. Using layers can ease and speed up the development of a complex hair or fur style by breaking it up into simpler components, which are then combined at render time.

Procedural Hair Effects

After or during the interpolation step (in Figure 7.8), when the shapes of the final hairs are calculated from the control hairs, special procedural techniques can be applied so that each final hair looks unique in one way or another. Examples are a wave effect to apply possibly random waviness to the strands, a wind effect to blow wind through the hairs, or clumping, where neighboring final hairs are grouped together in clusters. Feature maps can also be painted on the model to vary these effects over the body. The power of these effects lies in the fact that they are independent of the control hairs and provide a finer granularity than the control hairs ever could. An example is shown in Figure 7.13 for a wave and wind effect, which changes the final hairs without changing the control hairs (shown in green).

Figure 7.13 Top: No effect. Middle: Wave. Bottom: Wind. (Image © 2007 Sony Pictures Imageworks Inc. All rights reserved.)

Time and Cost: What the Moviemaker Needs to Know

Generating digital hair is both complex and time consuming. It is important to understand what aspects of hair combing and animation can potentially drive up the cost of a character or a shot. The level of quality needed for a project will greatly influence the cost: It is quite easy to get 85% of the way to the desired look, but the last 15% is likely to take exponentially longer.

What Makes a Hairstyle Expensive?

Depending on the level of detail and refinement that is needed for a particular character, hair look development can take anywhere from a few days to a few months. This includes not only making sure the hair appearance satisfies the client but also anticipating what the interactions and the movement of the hair will be in a shot. The most obvious cost is measured by how much time is spent by the artist getting the hair to look as expected. However, external factors often come into play. Here are a few things to keep in mind:

•   Decide what the character should look like before starting to comb CG hair. This should be done through reference photography or artwork and be approved by the client. Hair photos for live-action movies, detailed art sketches, or maquettes with “hair engravings” for CG features are all good candidates.

•   Some iterations will be needed once the digital hair of a character is seen for the first time, but these should only be for corrective tweaks, not major artistic direction. Going back and forth over sketches or photos beforehand will be orders of magnitude cheaper and faster than combing and rendering fully furred characters, only to see that work thrown away. This is why creating photoreal digital doubles of actors is often very fast, regardless of the complexity of the hairstyle, because a strict reference must be matched and very little flexibility given to supervisors or clients to stray and experiment during the combing stage.

•   Make sure that things are changing as little as possible beneath the hair artist’s feet. This includes modifying the model that the hair is grown on (which may cause parts of the hair to be recombed or feature textures to be redone).

•   Regarding the model, it is critical to properly account for hair volume when modeling fully furred creatures. Early in production on Stuart Little (1999) the artists had little experience with how the fur coat would change the appearance of the character. It took several cycles back and forth between the Hair Team and the modeling department and eventually resulted in a considerably skinnier model, which had about the same volumetric appearance with the fur as the initial model without fur. If the extra volume a fur layer adds can be anticipated during modeling, money is saved, because every time the model changes, the hair is potentially affected to the point at which some or most of the work has to be redone. The only consistently reliable solution is to always create an anatomically correct furless model. This may look strange but will produce the desired result when fur is added and will behave much more realistically when in motion.

Figure 7.14 Left: Reference photography. Right: Digital double. (Image © 2007 Sony Pictures Imageworks Inc. All rights reserved.)

Be mindful of aspects that will increase the render times, such as the number of hairs and their transparency: The longer the render time, the less time an artist has to address comments in between renders. Try to keep hair counts as low as possible.

What Interactions/Shot Situations Are Problematic (and Drive Up the Cost)?

A hairstyle often consists of millions of final hair strands that an artist can’t control directly because they are all generated procedurally at interpolation time. This means any detailed interaction with the final hairs in a shot can spell trouble. If anything is looking incorrect, the artist can’t go in and manually adjust the final hairs to look right. What may appear to be minor adjustments to a shot can mean days of extra work in the hair department. This can include characters interacting with the hair (a hand touching or stroking fur) or thin objects that move through the hair (like backpack straps or accessories that slide on a furry character’s body, as shown in Figure 7.15).

Figure 7.15 G-FORCE characters equipped with accessories. (Image courtesy of Sony Pictures Imageworks Inc. © 2009 Disney Enterprises, Inc. All rights reserved.)

Figure 7.16 Wet beaver transitioning to dry. (Image from THE CHRONICLES OF NARNIA: THE LION, THE WITCH AND THE WARDROBE. © 2005 Disney Enterprises, Inc. and Walden Media, LLC. All rights reserved.)

Transitioning between hair styles within the same shot can also be an issue. Depending on the hair system used, it may not always be easy or possible to blend from one hair look to another if they are very different. A typical example is going from dry to wet hair, or the opposite, which can happen when a creature gets splashed, falls into water, and shakes itself to dry off. Figure 7.16 shows a digital beaver from The Chronicles of Narnia: The Lion, the Witch and the Wardrobe (2005).

In some of these problematic cases, one can get away with imperfections simply because the character is moving fast and motion blur will hide things. It is critical to be aware of these situations from the start, at the look development phase: If one knows ahead of time which situations a character will encounter, a hair artist can better strategize how to comb a character. For instance, more control hairs might be placed in areas that require detailed interactions. Control hairs can always be added later for a specific shot, but they may change the overall look of the character approved by the client, requiring extra time to be spent just getting back the original look.

Summay

The process and pipeline for generating digital hair for a motion picture have been explained earlier and some of the issues that can arise and possible solutions for them discussed.

Several commercial hair software packages are available, including Softimage XSI Hair and Maya Hair¹¹ or Shave and a Haircut.¹² Many of the effects and animation studios also have their own proprietary hair pipelines (see Bruderlin (2004) for an example), which makes it possible for these companies to quickly change or add to the functionality of their systems when a new movie requires it.

For a general and technical survey paper on various approaches and problems related to digital hair modeling, animation, and rendering, see the article by Ward, et al. (2007).

For more technical detail on issues and solutions to shading hair to make it look like real hair, see the articles by Kajiya and Kay (1989) and Marschner, et al. (2003). Examples of generating hair without explicit geometry include Jim Kajiya’s volumetric texture maps called texels (Kajiya and Kay), Dan Goldman’s fake fur probabilistic lighting model (Goldman, 1997), and Ken Perlin’s procedural texture approach referred to as hypertextures (Perlin, 1989).

References

Bruderlin, A. (2004). Production hair/fur pipeline at imageworks, in photorealistic hair modeling, animation, and rendering, ACM SIGGRAPH Course No. 9. Computer Graphics.

Goldman, D. B. (1997). Fake fur rendering, ACM SIGGRAPH Proceedings. Computer Graphics, 127–134.

Kajiya, J. T., & Kay, T. L. (1989). Rendering fur with three dimensional textures, ACM SIGGRAPH Proceedings. Computer Graphics, 23(3), 271–280.

Marschner, S. R., Jenson, H. W., Cammarano, M., Worley, S., & Hanrahan, P. (2003). Light scattering from human hair fiber. ACM Transactions on Graphics (TOG), 22(3), 780–791.

Perlin, K. (1989). Hypertextures (ACM SIGGRAPH Proceedings). Computer Graphics, 23(3), 253–262.

Ward, K., Bertails, F, Kim, T., Marschner, S. R., Cani, M., & Lin, M. C. (2007). A survey on hair modeling: styling, simulation, and rendering. IEEE Transactions on Visualization and Computer Graphics, 13(2), 213–234.

DIGITAL FEATHERS

Armin Bruderlin, Francois Chardavoine

Digital feathers in movies are not as common as hair or fur, and although there are several commercially available systems to generate hair, only proprietary solutions are available for feathers. This means a studio must always develop its own solutions. These can vary based on the level of quality required and the potential for reuse on future shows.

Computer-generated feathery characters have appeared in animated movies and shorts such as For the Birds¹³ (2000) and, more recently, One Pair¹⁴ (2008). An example of digital birds in a live-action movie is shown in Figure 7.17, Margalo and Falcon from Stuart Little 2 (2002). A few research papers have also been published on the subject of digital feathers (Streit, 2003; Chen, Xu, Guo, & Shum, 2002).

The generation process is similar to hair in that the final instanced geometry (here, feathers) is created from a few modeled key shapes. In fact, these same instancing and interpolation techniques can be more generally applied to the scales of fish, quills of porcupines, spines or scales of dragons, a field of grass, or even a forest of trees. For instance, a hair/fur system, as described in the previous section, was used to generate the grass and flowers in the animated movie Open Season (2006).

One of the main differences between creating feathery and furry creatures is that an individual feather is a more complex 3D structure than a single hair strand. The former has components of widely varying shapes, whereas the latter can simply be modeled as a curve and rendered as a ribbon or tube with changeable width and transparency.

Figure 7.17 Margalo and Falcon from Stuart Little 2. (Stuart Little 2 © 2002 Columbia Pictures Industries, Inc. All rights reserved. Courtesy of Columbia Pictures.)

Morphology of Real Feathers

For a good explanation of the anatomy of a feather, please see Section 2 of Streit’s Ph.D. thesis (Streit, 2003). That section also discusses different types of feathers such as down and contour and flight feathers, as well as feather coats and follicle distribution for real birds.

For practical purposes, a feather consists of three main sections: a shaft or quill, a left vane, and a right vane, as illustrated in Figure 7.18. The shaft is essentially a cylindrical tube with varying radius along its length. The vanes of a real feather consist of a series of branches fused together called barbs, which can have abrupt curvature changes and may contain splits. For flight feathers, barbs are locked together by branches called barbules with tiny hooks named barbicels. In some feather types, the tip and/or base of the vanes may be populated with down hairs.

Figure 7.18 Components of a typical flight feather. (Image © 2010 Sony Pictures Imageworks Inc. All rights reserved.)

Modeling Digital Feathers

The simplest feather geometry is a plate feather, a flat surface with a detailed texture applied. This geometry is fast and easy to implement but may not hold up for close-up or high-quality projects because the feathers may look unnaturally flat. Another option is to model and render the feather vanes as hundreds of hair strands representing the barbs, which will appear much more realistic by producing the desired anisotropic¹⁵ look. This is shown in Figure 7.19 where both left and right vanes were defined by hundreds of side-by-side hair strands. If a studio has an existing hair system, it may be able to benefit directly from all of its features when generating the feather vanes as hair. Going beyond the resolution of the barbs (i.e., down to the barbicels) is overkill even when modeling realistic-looking digital feathers.

Figure 7.19 Rendering the vanes as individual hairs. (Image © 2002 Sony Pictures Imageworks Inc. All rights reserved.)

Figure 7.20 Procedural parameters. (Image © 2002 Sony Pictures Imageworks Inc. All rights reserved.)

A set of feathers of different shapes, sizes, and styles can be manually premodeled for a given character. An alternative is a procedural generation module. Based on high-level parameters such as shaft length, vane width, breaking up vanes, etc., a feather is automatically generated by the system. Figure 7.20 shows two procedurally generated feathers, one with default values on the left and one with non-default values on the right for vane splitting and base/tip down hair generation.

Generating digital feathery creatures in a production environment brings up a lot of the same issues that were discussed in the previous section on hair with respect to modeling, animating, and rendering. The next subsections will take a brief look at some of the similarities and differences that feathers exhibit compared to hair.

Similarities between Hair and Feathers

The necessity of modeling a feathery character in a slightly skinny or shrunken appearance is somewhat more important than for an animal with short fur because feathers usually add more volume. On Stuart Little 2 (2002), the modeling department didn’t want to change Margalo drastically after she was approved for look (without feathers). As a result, the feathers ended up having to be combed very close to the body. This caused all kinds of problems (more problems than it should have) during animation, where feathers interpenetrated very easily with each other and the underlying skin.

Grooming a feathered character means shaping thousands of feathers in a cohesive pattern and flow over the geometry created by the modeler, in order to match the look the client desires. It would be painfully time consuming for an artist to individually position, orient, and shape every single feather, so instancing systems have been designed that are fairly identical to most hair systems.

The artist will manually place a few pre-modeled or procedurally generated guide feathers (or key feathers) on the character, and then instruct the system to automatically fill the blanks in between, smoothly interpolating the shape of the guide feathers among all of the instanced feathers. This can happen as an interactive process or on the fly at render time. A quick preview system may be implemented to get a better idea of the final look before the character is fully rendered.

Procedural effects can then be applied at render time to add randomness to the final feathers similar to the process used for hair (waviness, clumping of barbs) or even animation (such as procedural wind flowing through the feathers).

Differences between Hair and Feathers

Individual hair follicles are often lost in the forest of the millions of almost identical siblings that cover a character’s body. Feathers, on the other hand, have a distinct structure defined by the vanes and the central shaft, and there are usually a lot less of them (in the thousands). This makes every feather stand out a lot more.

To make matters worse, while having hairs intersect with each other is almost never a problem, any feather-to-feather interpenetration is quite obvious unless the feathers have a very fluffy or downy appearance that makes them look more like hair. Thus, the artist must not only take care to lay the feathers down without their going through each other when grooming the static reference pose, but interpenetrations will also need to be avoided when the character is animating in shots. Feathers that are entirely procedural (and generated at render time) will need an automated mechanism to prevent these penetrations, since the artist cannot directly interact with them (other than through the few guide feathers).

Some form of feather interpenetration solver should therefore be an essential component of a practical feather system. On one end of the spectrum, a full-blown dynamics system with collision “detection and response” can be employed. This fixes feathers going through each other during animation. However, it can be very computationally expensive, complicated to implement, and bound to not work robustly in many cases.

A simpler and often more feasible solution is trying to “prevent” collisions between feathers. The main idea is that if the initial reference pose is groomed such that it does not have any interpenetrating feathers, their relative positions with respect to the skin and each other can be maintained during animation, thus avoiding penetrations.

One way to accomplish this is by calculating for each vertex of each feather the closest point on the skin surface and the distance to that point in the reference pose. These distances are then preserved during animation, so the feathers all move as a whole exactly like the model’s surface. This may work in some situations, but the feathers will squash and stretch with the skin in a way that often looks unrealistic. An alternative solution allows the feathers to slide on the surface by only maintaining the distance while recalculating the closest point on the skin surface at each frame during an animation. This effect is shown in Figure 7.21 on the right side, which produces much better results than with no collision prevention applied on the left.

Whatever solution is adopted for the penetration problem, situations invariably come up when the result is not satisfactory or needs to be changed for artistic purposes. At that point, it becomes necessary for the artist to interact directly with individual feathers. Three options are possible, and anyone designing a feather system should consider one or more of these:

•   Be able to provide corrective hints, usually in the form of extra guide feathers to change the shapes or motions of the final instanced feathers.

•   Be able to un-instantiate problematic feathers. When a render isn’t satisfactory, the artist can decide to extract certain feathers from the instancing system and bring them directly into the animation software to modify or animate them as needed and then export them back to the rendering system.

•   Be able to deal with all of the feathers as geometry that the user can interact with in the animation package. If the system is properly designed for speed and memory usage, this is the most flexible solution. The full power of the software package is available to correct any problems. Rigs can be built, dynamic solver applied, and the worst offending feathers can simply be plucked if needed. Of course, care must be taken not to have the system grind to a halt when thousands of feathers are needed.

Figure 7.21 Feathers with (left) and without (right) interpenetration. (Image © 2002 Sony Pictures Imageworks Inc. All rights reserved.)

Finally, whereas hair usually moves in a subtle or passive way driven by the motion of the character and external forces such as gravity and wind, some feathers are often actively animated as part of a character’s performance. This is especially true for the bigger flight feathers toward the tip of the wings, which are frequently used as “fingers” for gesturing (see Margalo’s wings in Figure 7.17) or interacting with objects. Such potentially rigged and individually animated feathers can cause extra strain on the interpenetration solver.

References

Chen, Y., Xu, Y., Guo, B., & Shum, H.-Y. (2002). Modeling and rendering of realistic feathers, ACM SIGGRAPH Proceedings. Computer Graphics, 630-636.

Streit, L. M. (2003). Modeling of feather coat morphogenesis for computer graphics, Ph.D. thesis. Canada: University of British Columbia. www.cs.ubc.ca/labs/imager/th/2003/Streit2004/Streit2004.pdf.

DYNAMICS AND SIMULATION

Judith Crow

Dynamics and simulation techniques are a core component of digital effects. Broadly speaking, they are an attempt to replicate real-world physical characteristics in the digital realm such that realistic behavior and interactions may be achieved by letting the software calculate appropriate results without requiring keyframed animation.

It may be that practical costs are prohibitive (where multiple models may be required to allow for reshoots), don’t offer the desired level of control or directability, or that a new generation of supervisors is just more comfortable with digital technology than with traditional practical effects. Whatever the reasons, since the early 1990s practical effects have increasingly become displaced by digital equivalents to the extent that simulations of complex interactions of objects with fluids and fire are now commonplace. Where Dante’s Peak (1997) uses simple particle systems to digitally enhance models and miniatures together with live-action plates, 2012 (2009) simulates the wholesale destruction of an entire digital city.

While simulation techniques are most commonly associated with destruction and mayhem and encompass fracturing and deforming objects, explosions, fire, smoke, and liquids, they are also used to clothe characters, style and move their hair, and move muscles under their skin. It can be expected that realistic complex interactions may be achieved more readily via simulation than by keyframing.

Figure 7.22 Dante’s Peak (1997) uses digital effects to enhance practical effects provided by models and miniatures. (Image © 1997 Universal Studios Licensing, LLLP. All rights reserved. Courtesy of Universal Studios.)

Figure 7.23 The 2009 movie 2012 digitally simulates destruction on a huge scale. (2012 © 2009 Columbia Pictures Industries, Inc. All rights reserved. Courtesy of Columbia Pictures.)

How Is a Simulation Created?

Generally speaking, a simulation starts with a 3D computer model to which various physical characteristics are assigned, such as mass, density, elasticity, glue strength, and so on as appropriate to the type of object to be simulated. Forces such as gravity and wind are applied and one or more solvers (sets of equations) are run for each successive frame to compute the resulting changes for each of the objects in the simulation. For example, a cloth solver and a rigid-body solver may be run together so that a curtain (cloth) may be seen to both deform over the surface of a lampshade (a rigid body) while at the same time push against and move the lampshade.

Often simulations are determined by setting useful initial conditions from which the resulting animation will be derived with little or no further direction from the artist. Any control the artist has comes from varying those initial conditions and rerunning the simulation until the desired results are obtained. However, there is a growing trend toward the use of tools that allow for greater directability—both before and after running the simulation. It is expected that both a faster turnaround (with fewer iterations needed) and greater control over the outcome may be achieved.

When Is Simulation Appropriate?

A number of factors may dictate whether simulation is appropriate for a production:

•   Does the production schedule allow for a shot development period? Simulations do not typically deliver results out-of-the-box, but instead require development time.

•   Are appropriate resources available? It is still the case that this is highly skilled work, often requiring custom software development and always needing substantial hardware support in the form of fast multiprocessor machines with large amounts of RAM and disk space.

•   Can the effort be amortized over a number of shots? A one-off shot can probably be handled with judicious use of traditionally animated components, practical elements, and simpler particle systems, whereas a longer sequence will lack coherency without a simulated foundation.

After examining the scope of work, dynamics and simulation will be appropriate in the following scenarios:

Definitely Appropriate

•   Highly realistic natural phenomena are required.

•   Detailed and complex animation of liquids and gases is required.

•   Complex interaction is required among large numbers of objects too great to animate by hand.

•   Large numbers of shots of cloth or hair animation, especially non-hero shots, are required.

Possibly Appropriate

•   A small number of hero objects with complex interaction are required.

•   Elements for stereoscopic pairs cannot be cheated, such as volumes.

Probably Not Appropriate

•   Extensive art direction is expected.

•   Short turnaround on a small number of shots is desired.

Tricks and Cheats

Note that running simulations is a laborious and time-consuming process. Every time changes are made, the simulation has to be run again in its entirety, because each frame’s results are dependent on those of the previous frame. Adding additional complexity in terms of numbers of objects and the amount of detail tends to have an exponential effect on the time taken to rerun the simulation. A number of strategies can be used to alleviate these problems:

•   Once the gross motion is right, much of the smaller detail can be added in the shaders. Rerendering will generally be a great deal faster than rerunning a simulation.

•   Detailed final geometry can be substituted for low-resolution stand-in geometry. In a rigid-body simulation, approximations of the chunks can be run through the simulation and then replaced for rendering.

•   Particles can be rendered as sprites¹⁶ with detail coming from the texture maps or procedural shaders applied to the sprites.

•   Low-resolution simulations can be layered to produce complexity. For example, a particle fluid can be run with fewer particles to get the overall motion right and then rerun with subtly different initial conditions. When the results of the various iterations are added together, there will be enough detail to generate a suitably smooth surface.

•   Keyframed animation can provide the starting conditions and, if necessary, override the results coming out of the simulation so specific hero elements can be directed into place with precision.

Important Considerations

Running simulations can consume considerable amounts of time, memory, and storage so it is important to consider a number of strategies to increase efficiency:

•   Size simulations appropriately. Look at running simulations on a scale that elicits useful information in a time frame that fits the artist’s schedule and makes the best use of downtime. Consider what can be learned in an hour, a day, or overnight.

•   If possible, increase the number of iterations. Can a simulation be simplified or split into parts such that more iterations can be run and the results be used to inform the final, detailed simulations?

•   Look at ways to manage large datasets. Simulations can usually be cached to disk and restarted partway through. Time spent simulating will be traded against a greater need for data storage. Some kinds of simulations scale better than others— lower resolution geometry may produce adequate interaction and can be substituted with high-resolution geometry at render time.

•   Plan for data exchange. Simulations do not live in a void. The source models will be coming from somewhere—quite possibly another 3D package—and the output of the simulation will have to be shaded and rendered or used in some way to drive other geometry.

Planning and Preparation

Simulations benefit from careful planning. Here are some considerations to keep in mind:

•   Learn from previsualization. Don’t build more complexity than is necessary. Put detail only where it is needed.

•   Because shots rarely begin with no action under way, preroll has to be considered. How much action will need to have played out before the shot starts? Can an animator provide convincing initial conditions instead of having to run an expensive simulation for several seconds of run-up?

•   It is critical to work at a real-world scale and with correct physical characteristics. Make sure the scene represents the true size of objects and use accurate references for settings such as gravity, density, and friction.

•   Always think about how much post-processing the simulation data can add without needing to rerun a simulation. Additional motion can be layered. Detailed geometry can be substituted. Smaller surface detail can be provided during rendering.

Software Solutions: A Broad Overview of Current Options

The earliest implementations have come from academia or are proprietary systems built by some of the larger facilities. Notable early examples are Arete’s Renderworld ocean surface simulator [Waterworld (1995) and Titanic (1997)], ILM’s ocean swells for A Perfect Storm (2000) and their ongoing collaboration with Stanford University on Physbam; PDI/Dreamworks’ cloth and hair simulation for Shrek (2001); and Digital Domain’s flooding water for Lord of the Rings (2002).

Increasingly, third-party applications are available. Some stand-alone or plug-in options specialize in specific types of solvers. Next Limit’s RealFlow offers stand-alone fluids and rigid-body dynamics (RBD). Syflex specializes in cloth, whereas Afterworks’ FumeFX offers a gaseous fluids plug-in.

Yet others provide a more generalized and integrated environment for dynamics and support multiple solvers. Side Effects’ Houdini software uses dynamic operators (DOPs) that provide a wide range of microsolvers for combining RBD, fluids (liquids and gases), cloth, and wire, while also providing a framework for users to add their own solvers. Autodesk Maya’s Dynamics includes RBD, fire, liquids, and cloth, while Softimage offers PhysX, a highly optimized RBD solution. Even Blender, a free open-source application, now offers some dynamics capabilities.

Other open-source projects of interest are the Open Dynamics Engine (ODE) and Bullet Collision Detection and Physics library, both of which are finding their way into high-end applications. Often first developed for use in games engines, these specialized solvers are able to provide a surprising level of sophistication in real time and are of increasing interest to filmmakers who are willing to trade some accuracy for vastly increased computational speed.

PARTICLES

Craig Zerouni

What Are Particle Systems?

It would be reasonable to describe particles as the starting point for all computer simulations. They were among the first techniques to be explored as computer graphics evolved into digital visual effects. So what are they and how are they used?

Particles are locations in space. Since space is thought of in terms of X (left to right), Y (up and down), and Z (in and out), each particle is made up of at least an (x, y, z) triplet. Next, attributes are added to the particle. Attributes can be anything: color, direction, mass, etc. Typically, particles have at least a direction, a speed, and a life span. Depending on what they are representing, they may also carry attributes for mass, repulsion, attraction, color, or any other information that is necessary later to realize a particular effect.

The Next Step

Particles are sometimes used just as they are, as a way to put a dot of color in a certain location. But for the most part, they are used as a placeholder for something else. Once some particles are moving around, much more complex geometry can be added to those locations later to achieve a flock of birds or a swarm of bees. Or very simple geometry can be attached to them, with a complex image painted onto it. In other words, a 2D card could be attached to each particle, and then the card can carry a painting of almost anything—including a bee or a bird. Depending on the shot, that may be all that is needed to achieve a truly fantastic image.

Figure 7.24 Particles are locations in 3D space. These have motion and other characteristics assigned to them. (Image courtesy of Ryan Bowden.)

Particles in Motion

The key thing about particles is that they move. Typically, they begin with some kind of speed and direction (carried as a velocity vector), and then, for each frame, their position is a combination of their own internal motivation and external forces. External forces are mathematical simulations of common phenomena like gravity, wind, and especially turbulence. Turbulence applied to particles gives them a whirling, swirling quality and is how everything from jet wash to torpedo bubbles to sandstorms is created. Turbulence is a fundamental concept in simulations— it’s not randomness but a kind of structured unpredictability. Turbulence is what is seen when a drop of dye is dispersed into a glass of water.

To move the particles, the basic idea is simple: For every frame, each particle is moved along its direction vector, by its speed, subject to the wind, the turbulence, the gravity, and whatever else is affecting it, such as the proximity of other particles. It’s often the case that particles attract and/or repel each other, so this also affects their final position for each frame.

Often, adjusting positions at every frame isn’t good enough, especially if accurate motion blur is desired, so the positions are calculated several times in each frame, and then geometry is created along the path during the frame (that is, during the time the shutter is open in the computer’s “camera”).

The Birth of Particles

Particles are born, not made. They can be birthed as the result of some other action, such as one piece of geometry hitting another. This would typically create the puff of dust generated from one piece of a building collapsing on another. Particles can also be born continuously from locations. Think of a stream of bubbles coming up from the bottom of a glass. They have to be born someplace—at some location in space. The starting locations of the particles can be defined by bringing in geometry, such as the back end of the torpedo that is going to stream bubbles. Or the initial positions could be defined mathematically, such as “supply a 1000 points that are randomly distributed inside a sphere of a particular size.”

Then the attributes are added. Typically, the attributes are calculated as random numbers within a particular range, rather than entered by hand. So an effects artist would instruct the particle system to do the mathematical equivalent of “pick a speed for each particle that is randomly distributed between 50 and 75 miles per hour.”

Often, the particle positions and initial velocities are calculated based on some other event—for example, to generate concrete fragments from a bullet hit. In a case like this, the location of the particles is determined by where the bullet hits, and the initial velocities are calculated based on the angle at which the bullet hits the surface. After that, the particles go off on their own, subject to gravity, wind, etc.

Creating Effects

So far, several examples have been explained of ways in which particles can be used: They can generate fragments from a bullet hit or other explosion, they can give the swarming motion to schools of fish or clouds of insects, they can carry the structure of smoke and fluid simulations, and they are great at creating the framework for explosions and fire.

In the case of explosion or fire the particles are not rendered directly. Instead, they are usually replaced at render time by some kind of geometry or sometimes just an equation that renders something smoky or flame colored. In a fire simulation, for example, the color of the particles would vary with their height, so that the flame would be white at the bottom, moving through red and orange and then blue as it got near the tips. Flame particles typically have a short life span as well, so that the flame flickers out at the top. Then each particle would contribute some color and opacity to its surrounding area, and the result would blend together to look (and act) like flames.

Figure 7.25 Simple two-dimensional cards with images on them (sprites) can be attached to the particles. Often, this is good enough. (Image courtesy of Ryan Bowden.)

Figure 7.26 Smoke created by having the shader render a volume of smoke centered at each particle. These overlap and combine in visually satisfying ways. (Image courtesy of Ryan Bowden.)

It’s also possible to treat the particles as loosely attached points, and then at each frame connect them together into a solid. Because this kind of solid has points with positions that are not fixed in their distance from each other, this gives the artist one way of doing what is called soft body dynamics. If the artist wants to drop a blob of stiff Jell-O onto a table, this is one way to do it.

Figure 7.27 Geometry can be attached to each flying particle to generate scenes that appear visually complicated. These are teapots, but they could be bees, or jet fighters, or even people moving across a landscape. (Image courtesy of Ryan Bowden.)

As another alternative, the points can interact with other kinds of simulations. If particles are added to an RBD simulation, they can bounce off hard surfaces. For example, an effects artist could set up a simulation to pour a million ball bearings down some stairs this way—or do a lot of other related things.

All of these techniques are just the entry points to a world of different techniques and solutions that particle systems can be a part of. Because these systems are relatively simple to compute and visualize, they are often the starting point for much more visually complex effects. Readers who want to dig deeper into this technique can consult industry texts such as Cinefex and the ACM SIGGRAPH proceedings.

RIGID-BODY DYNAMICS

Craig Zerouni

Rigid-body dynamics (RBD) are physics simulations in which the objects being simulated do not change shape. That is to say, they are made of stiff, unyielding material, like blocks or ball bearings. In an RBD simulation, it’s still possible for objects to fracture into a number of smaller pieces—but when that happens, those pieces are thought of as new objects in the simulation, replacing the original single object. At any given time, all of the objects are thought of as rigid.

RBD simulations are being used more and more to simulate various kinds of destruction, although that’s not their only use.

How Rigid-Body Dynamics Are Created

All of these simulations begin with 3D computer models of the objects to be simulated. These can be as simple as a cube or as complex as a jet fighter.

Each object is assigned physical characteristics, such as mass, density, friction, bounce, etc. Since these are being used to create effects, often the desire is to start with reality, and then go farther. An object’s mass will need to be animated over time, for example. Or the friction of the object will need to be different at different locations along its surface.

Next, forces are defined in the virtual world: gravity and wind, for starters. But again, because this is for effects, rather than science, the rules can be altered to help solve a particular problem. Gravity does not have to pull down, for example, but can pull sideways. Gravity can be very strong, or very weak, or very strong on some objects and very weak on others. Forces can also be created that help the artist create effects, such as a generic force that starts strong and then very quickly fades out—an explosion, in other words.

Constraints can also be defined. Constraints impose limits on the motion of the objects. For example, one object can be constrained to another, so that where one goes, the other must follow. Types of motion can also be constrained. A door hinge, for example, is a constraint in which the door object is only allowed to rotate, not translate (move). So if the door is hit by another object, or by some forces, it can swing open and shut, but that’s all it can do.

After all of the geometry has been defined—the physical characteristics, the forces, and the constraints—the computer takes over. The basic idea is actually quite simple: For each frame, move each object according to the forces that affect it and its own physical characteristics. See if it hits any pieces. If not, the simulation is done. If it does hit something, compute the rebound for this piece and also for the piece it hit. Then repeat for each object.

But because each collision may create other collisions, this generally means going back through all of the objects multiple times per frame. It gets worse, because in order to get any kind of accuracy, doing these updates only once per frame isn’t enough— it generally requires anywhere from 2 to 10 subframe increments to get something reasonable. That can slow the simulation down quite a bit, especially when many objects have to be added.

Potential Problems

Apart from speed, there are other issues. One of these is stability. In real life, once an object settles onto the ground, it stays there and stays still. In RBD simulations, this can be a problem, because the calculations for deciding whether an object is on, or above, or penetrating into another object (or the ground) may not be completely precise. This can generate instability, where pieces vibrate against each other rather than stop, because at each frame the relationship comes out slightly differently.

There are also issues with accuracy, where points or edges of one piece may penetrate the face of another, because there is not enough detail for the software to realize there is an intersection. Frequently, this problem is solved by increasing the density of the object mesh. However, that makes the simulation slower still.

An effects artist can also have problems in situations where objects begin inside other objects or are moving so fast that within a single frame time they end up inside another object. Since, according to the rules of RBD simulations, objects are not allowed to be inside other objects, if this happens the results can be unpredictable—and, predictably, the results are not the ones desired.

Other Issues

Because one main use of this technique is to get realistic destruction, it would be beneficial if the simulation broke down (fractured) the geometry as it went in a natural way. But that’s generally beyond the capabilities of today’s software (though there are systems that can do this in some controlled situations). That means that in order to get an object to crumble into smaller pieces as it collides with other objects, those smaller pieces have to be modeled in advance and then glued together (mathematically speaking) to create the whole original object.

Then the simulation can decide if the forces on the object are strong enough to make the glue break, so that the subobjects fly apart. Although this works well, it means a lot more work up-front. It also forces an artist to decide where the breaks will occur and what they will look like. Of course, this can also be a benefit, since it allows the fracturing to be art directed.

Some facilities have now developed fracturing tools that will do the pre-breaking automatically. It still has to happen before the simulation, rather than as part of it, but at least it doesn’t require a modeler to slave over each piece. Future generations of simulation software will undoubtedly not have this restriction and will do the destruction during the simulation itself.

Another issue is that the geometry in these simulations is generally first converted to something called a level set (sometimes called a signed distance field). This is a volumetric representation of an object, rather than a boundary representation. This means that it represents the object as a lot of little cubes, which are fast to compute collisions with. The downside of this is that in order to accurately model complex shapes, a lot of little cubes can be required. That means more calculations are necessary and, thus, slower simulations.

Also, the more objects in a scene, the longer the calculations take. This makes it hard to scale up to really complicated simulations.

Finally, RBD simulations try to model actual physics. But actual physics is generally only the start of what the director wants to see happen. Typically, people want the simulation to look realistic, while simultaneously being able to control how certain hero pieces behave, where they end up in frame (or out of it), etc. Most systems allow this kind of secondary control, but it adds yet more setup time to the simulation process.

Tricks for Getting It Done

Given all of these ways in which creating an RBD simulation can start to stretch a schedule, there are several ways to try to shrink it back. One of these is to pre-calculate the simulation and bake out the geometry frame by frame. This is a slow process, but hopefully it will be done a small number of times. This better prepares the simulation such that the resultant geometry can be brought into a lighting pipeline for repeated lighting and shading takes.

Another technique is to simulate in layers. This means simulating the larger, lower detail pieces first and then baking out that geometry and locking off their motion. Then they are fed back into the simulation as objects that other smaller objects react to (bounce off of), but these larger objects are not affected by the smaller objects. Having these passive objects makes a simulation calculate more quickly. Then the simulation can be repeated with still smaller objects. This can generate a lot of geometry data on disk, but it also allows an incremental approach to simulations.

Lately, some effects facilities have integrated game engines into their pipelines. These are designed to work interactively for game play, and while they can’t do that with the complexity of modern effects shots, they can speed the process up quite a bit. Generally they are able to work with proxy objects and generate transforms that describe the motion of the objects in the shot. Then those transforms are applied to the actual detailed geometry, a process that is much, much faster than a simulation.

DIGITAL LIGHTING

Andrew Whitehurst

Light in Reality and in Computer Graphics

The product of simulated illumination and the reflectance of the surfaces in a scene determine the color of every pixel representing part of an object (a spaceship, for example) in computer-generated imagery (CGI). The software that performs these calculations is called a renderer and thus the process of generating the images is known as rendering.

When considering how light may hit a surface it is not sufficient to simulate sources of light alone. Light may bounce from one surface to another before hitting the eyes and thus the effect of this illumination should also be considered. Further, the surfaces of translucent objects may be illuminated by light that has passed through the volume of the object behind the surface. Until recently many of these effects have been prohibitively costly to render. The earliest uses of computer graphics in film visual effects limited themselves to simulations of light interacting with hard surfaces, because these are the simplest and quickest algorithms to calculate. As computers have become faster and cheaper, it has become possible to render more complex surfaces (fur or skin, for example) in a timely manner. The history of computer graphics (CG), especially in visual effects, can be seen in terms of a continual improvement in the physical accuracy and complexity of the rendering algorithms facilitated by exponential improvements in the speed of computers over time.

Once a ray of light, whatever its origin, hits a surface, it may be reflected or partially absorbed: The incident illumination is turned into radiant illumination by its interaction with the surface. Throughout the history of CG, algorithms of varying complexity have attempted to describe these surface effects. Some algorithms are mathematical models that create a pleasing image but are not derived from any physical understanding of real surfaces. Other algorithms are based on accurate measurement of actual materials. Many of these mathematical models of reflectance are designed to describe a particular surface effect, for example, the reflection of a light source in a shiny plastic surface (called specular reflection) or the matte surface of dry cement (diffuse response). Often these simpler models will be combined on one surface to create a more visually compelling appearance (e.g., a matte surface summed with a shiny layer).

Two of the earliest shading models in CG are Lambert shading, which models smooth matte surfaces, and Phong lighting, which describes shiny highlights that might be observed on a plastic object. Lambert’s model describes how the amount of light hitting a surface decreases if the surface area facing the light is smaller. At a completely glancing angle, the amount of light hitting the surface will be zero, and therefore dark, whereas a surface perpendicular to the incident light will receive the maximum amount of possible illumination and will be brighter. The brightness fall-off across a curved surface, like a sphere, will be smooth and is a reasonable description of a simple matte surface. Mathematically, Lambert shading can be described by computing the dot product of the normalized light direction vector with another vector that describes the direction in which the surface is facing at the point being shaded, also known as the surface normal. This is a very fast computational model, which has contributed to its popularity and longevity in visual effects rendering.

Figure 7.28 A CG render of an object shaded with Lambert diffuse lighting (left) and Phong lighting (right). (Image courtesy of Andrew Whitehurst.)

Phong lighting takes this idea of describing the amount of reflected illumination using the light direction and the surface normal one step farther. Bui Tuong Phong noticed that on some shiny objects the reflection of light sources could be seen as a bright highlight with soft edges. Mathematically this effect could be expressed as cos(R • L)^roughness where R is the normalized reflected vector of the light direction after bouncing off the surface being shaded, L is the normalized vector describing the light direction, and roughness describes how broad and soft the highlight should be.

Case Study of Reality Compared with Simple CG Simulation

Consider a white light shining on to a red plastic object. In reality, the red paint on the plastic is made up of small particles of pigment suspended in a clear medium. The white light hits the surface of the object. Some of the light is immediately reflected off the surface medium. This is the shiny specular highlight one sees, and it retains the color of the light rays hitting it, white in this instance. The rest of the light will enter the paint medium and bounce around among the pigment particles, which absorb the non-red wavelengths of the light and reflect the rest. This light, now red in color, will re-emerge out of the surface. The angle at which the ray emerges from the surface will be determined by the microscopic shape of the pigment in the paint; therefore rays of light that enter parallel may be scattered and emerge on very different paths because of this microscopically rough surface. This is why a matte surface looks different from a mirror-like one: The light rays hitting it scatter in many directions.

Figure 7.29 A cross section through a plastic paint-like surface. (Image Courtesy of Andrew Whitehurst.)

Figure 7.30 A simple plastic surface can be rendered by summing the results of Lambertian diffuse lighting and Phong lighting. (Image Courtesy of Andrew Whitehurst.)

To model this effect in CGI, the fastest method would be to use a matte mathematical reflectance model (Lambertian diffuse, for example) to simulate the light that enters the surface and bounces around among the red pigment particles and then add another model to this that computes the specular highlight (Phong).

Over time modifications and enhancements have been added to these models that increase the range of shading options available to CG artists. For example, it has been observed that many diffuse surfaces are very rough, and light hitting these surfaces often bounces back in the direction from which it came (retroreflection). In these cases, the brightness of the reflectance is not so simply related to the angle of the surface compared with the light, as described by Lambertian shading. The moon is a good real-world example of this. It is lit by the sun and looks almost flat, not spherical. In 1994 Michael Oren and Shree K. Nayar described a mathematical model to account for this, and Oren-Nayar shading is now a popular alternative to Lambertian diffuse in production.

Another example of the increasing sophistication of empirical shading models is the implementation of Fresnel equations, which describe how much light is reflected or transmitted given the angle of the surface to the light source and the viewer. In many materials, glazed pottery for example, light glancing off the surface is much more likely to be reflected, whereas light hitting the surface dead-on has a greater chance of being absorbed. Objects such as these appear shinier at glancing angles on their surface, and the specular models can be adjusted to take this observation into account.

Visual Sophistication through Texture Mapping

One of the most popular methods of disguising the simplicity of the majority of shading models used in CGI is to paint maps to modify the terms of these shading algorithms. For example, a piece of old varnished wood may have areas where the lacquer has worn away, exposing the matte wood grain. A map might be painted where areas covered by varnish were white and dull areas were black. The renderer could then look up this map and multiply the specular term of the surface by the returned value so an area that is deemed to be dull wood (returning a value of zero from the map) would have a specular component of zero. Maps may be painted for a wide variety of effects, for example, the pattern of a surface, the glossiness of a reflection, the roughness of a surface, and so on. Texture maps are the most common method used to add visual complexity to a rendered object.

Figure 7.31 Visual complexity can be added to simple shading models through the use of texture maps. (Image courtesy of Andrew Whitehurst.)

Physically Derived Shading Models

The ever-increasing speed of CPUs has made it possible to implement more complex shading models, many of which are modeled on physically measured phenomena unlike mathematically derived shading algorithms like Lambertian diffuse. Examples of these models include Cook-Torrance, a popular model for describing highly specular metallic surfaces, and Ward’s anisotropic model, which enables surfaces to be described whose reflectance varies with surface direction, for example, a vinyl record or a spun aluminum kettle.

The next step toward more accurate simulation of surface lighting effects came when researchers began to measure the reflectance of real materials and then apply this data directly in CG rendering. A common example of a material for which this approach is desirable is cloth. Because different cloths vary in the size of the threads that comprise them, what those threads are made of, how they are woven, and what techniques were used to dye them, it is difficult to craft an empirical mathematical model that works well in all cases. Indeed, the complexity of a cloth surface is such that it is problematic to create a simple mathematical shader that works for even one specific instance. One solution to this that has been implemented is to use data acquired from a reflectometer to describe the cloth surface.

A reflectometer is a device that measures the surface reflectance of materials and can be used to output a dataset that describes how the surface responds given every possible combination of light angle, surface normal, and viewing angle. This data can be used directly to instruct the renderer about how reflective a digital representation of a surface is under CG lighting. Renders using this methodology are often highly realistic, exhibiting subtle lighting responses that would be extremely difficult to create using a combination of traditional mathematical shading models. The term for quantitatively describing a surface’s response to light is its bidirectional reflectance distribution function (BRDF). Traditional CG shading models such as Lambert and Phong are also BRDFs, but the term is most commonly used in visual effects to describe the measurement of an actual surface and the direct use of that data in rendering.

The use of measured surfaces does have some practical production downsides. First, one must have access to a real example of the surface that is to be copied. This is straightforward for a piece of cloth, but more difficult for surfaces that are impractical to move (a historic monument, for example) or for which there is no real instance to measure (there are no real dragon scales to sample). The second issue is that the data produces one, fixed response, which may not tally with the look that the director wants for a particular shot. One major benefit of the empirical mathematical models is that, through adjusting the terms of the equations, modulating their responses with texture maps, and so on, it is possible to modify a surface’s appearance from an artistically driven perspective. The advantage of this is that if the director decides that a surface must be a little shinier and the reflections a little glossier (blurrier) in a particular shot, those changes are easy to implement with the more traditional shading models. A measured BRDF is unable to accommodate this.

Beneath the Surface

Thus far, several methods for describing how light behaves when it hits the surface of an object have been examined. This works well for many objects, such as concrete, metal, and wood, but it does a poor job of describing the visual qualities of objects that allow light to enter through the surface, penetrate into the object deeply, scatter, and re-emerge. This effect is called subsurface scattering.

Consider two objects, one made of wood, the other of marble. If both objects were strongly backlit, the wood surface would appear very dark. The marble, on the other hand, would show the effects of light scattering through its volume and this contributes greatly to a marble-like appearance.

Figure 7.32 Subsurface scattering effects are a significant component to consider when simulating materials like marble. (Image courtesy of Andrew Whitehurst.)

Many other surfaces exhibit extensive subsurface scattering contributions to their appearance (e.g., milk, skin, and wax). Traditional surface shading models and measured BRDFs cannot account for this visual quality because they measure only light interaction that occurs at the surface. Developments in measuring the amount of light that scatters within a surface by pointing a laser at a scattering medium, for example, a cube of marble, and photographing the patch of light that is produced by the laser light bouncing around within the surface before re-emerging, have enabled scientists to create functions that describe the subsurface behavior of many materials, from fruit to human skin. These models, when combined with the surface reflection contribution, enable improved visual realism in rendering.

Goals of Lighting in Visual Effects

The goal of all visual effects is to enable the viewer to suspend disbelief and watch a story unfold without having the technical aspects of the film’s creation draw attention to themselves. To achieve this using computer graphics, models must be created with sufficient detail that they are indistinguishable from what would occur in reality, they must be animated with such a level of precision and care that every movement feels natural and believable, and they must be lit in a manner that seamlessly blends the CG into the live-action plate.

While shooting, the Director of Photography (DP) will have made many decisions about how the film is to be lit and photographed to determine what emotional effect the lighting should have on the audience. In many cases the job of the digital lighting artist working on photographed plates is to mimic the on-set lighting as perfectly as possible by either using on-set lighting data directly or by placing CG lights within the scene to match the on-set lighting rig. It is clear that in order for digital lighting artists to be able to perform this task, they need excellent reference photography of the lighting conditions on set. Such reference allows the confident placement, coloration, and intensity adjustment of lights to ensure a precise match with the plate.

Equally as important as ensuring that the lighting matches, it is necessary to craft surface shaders (the code that describes the surface behavior: Lambert, Phong, etc.) that accurately mimic the real-life counterparts to these digital creations. Even if the digital lighting is crafted to match reality accurately, if the surfaces that the digital lights strike do not respond in the same way as they would in reality, the match between the photographed plate and the digital object in the finished render will be poor. Lights and surface shading must work together. This process of balancing shaders, maps, and lighting until they match the reference is called look development, often shortened to look-dev.

An example of a situation that demonstrates this need for precision would be a shot where an actor on set is required to deliver a performance and then do something it is not safe or possible for her to do practically. Here it is common practice to construct a digital double, a CG version of the actor. Shaders and texture maps will be crafted that mimic the response of her skin, clothing, and hair while on-set lighting is also copied.

The filmed plate will consist of the first part of the shot where the performance is delivered after which a compositor will paint the real actor out of the plate (using a clean plate if one exists). An animator will duplicate the last few frames of the real performance using the digital double before then animating the impossible feat. The few frames of overlap between the real performance and the digital one allow the compositor to blend the real and CG performances more easily than if the transition took place over a single frame.

The finished animation is then lit by the lighting artist so that the match with the real footage is as close as possible, especially around the frames where the real and animated performances overlap. If the lighting reference and look development have been done well, the digital element should match the plate almost perfectly. The compositor should then be able to craft a seamless blend between the actor and the digital double.

In addition to creating the digital element itself, the lighting artist will also have to create interaction elements for real objects in the plate. Consider the digital double example again. If her feet are seen, it is almost certain that she is casting shadows on the ground. Once the real actor and her shadow have been painted out, it is not sufficient to replace just the performer—her shadow must also be rendered. Reflections in shiny, on-set objects may also need to be digitally crafted and composited. For this reason, in addition to having a digital representation of the actor, it is necessary to have correctly proportioned models of the environment in which the action occurs. If the surface on which the real actor stands is flat, it may be possible to use a simple plane in CG to catch the digital double’s shadow. The more complex the environment, the more important it is to have survey data or a laser scan of the set.

There are often instances when replicating on-set lighting is not sufficient to take a shot to completion. Although a DP will often set a general lighting rig for a scene, extra lights for each setup are often added to improve the visual impact of every shot. Clearly the DP cannot set up these extra lights for an object or character to be added later in CG, so these additional per-shot lights will have to be positioned by the digital lighting artist. Occasionally, the DP will still be able to assist in this process and offer feedback to the lighting artist though it is more usual for the DP’s intentions to be overseen by the VFX Supervisor of the project during post-production. It is clear that good lighting artists must have not only a finely honed technical understanding of visual effects lighting technology but also a keen artistic sense to enable them to match the aesthetic style developed by the DP

A good foundation in art theory and practice can help a digital lighting artist gain and improve this visual literacy: the ability to understand what effects the DP is seeking to create through lighting and the facility to produce sympathetic lighting within the digital realm. An understanding of the lighting techniques employed by the great masters of painting, photography, and cinematography can greatly assist in the acquisition of these visual skills. Areas that the digital lighting artist should take time to study would be techniques of lighting design, color theory, composition, staging, and basic film grammar.

Additionally, knowledge of art history has benefits for the lighting artist. First, it fosters a mental repository of great images from which inspiration can be drawn when lighting shots. Second, when discussing matters of aesthetics with directors and VFX Supervisors, the ability to refer to a piece of art as a common frame of reference is invaluable. Discussions of artistic matters inherently lack the cold rigor of a technical briefing. For this reason it is common for all creative departments on a film to reference painting, photography, and the work of other filmmakers. If digital lighting artists are to make a full contribution to these conversations, then they will need to understand the same lexicon of terms and references.

Work Flow for Successful Creative Digital Lighting

While there are many intangibles in the aesthetics of lighting, it is possible to develop working practices that can help ensure high-quality lighting. The first step is to recognize which lighting effects should be achieved in 3D and which should be left to the composite. As a rule of thumb all effects that occur once light hits a camera lens should be done in the composite, which means that lens flares, blooms, halation, chromatic aberration of the lens, and the film’s grain pattern should not be rendered. Often depth of field and lens defocus effects (bokeh) will also be left out of the 3D render. The 3D artist should aim to supply the compositor with a render that, when composited directly over the background, already feels as if it is part of the plate. The compositor will finesse this blend and can correct minor discrepancies between CG and the real footage but the best results are achieved when the render already matches the real material.

The process of matching a real scene’s lighting has become dramatically easier since the adoption of high-dynamic-range (HDR) image-based lighting (IBL). This will be discussed in detail later in the chapter. IBL is a technology that enables accurate onset measurement of lighting conditions to be recorded and then used to illuminate the CG objects directly. Thus the correct application of HDR IBL will produce a render that, if the look development has been done well, will feel like it belongs in the plate. However, as noted earlier, the placement of extra digital lights is often required to match the mood of other shots in the sequence.

When placing extra lights it is good practice to place them one at a time and adjust each until it provides the result sought before placing another light. By adopting this approach the artist limits the number of variables from one test render to the next, enabling a better judgment to be made regarding the next step in the creative lighting process. If an artist places all of the lights he thinks he will need at once and launches a render, it can be difficult to detect which light, or combination of lights, is responsible for any undesired effects seen in that test render. By adopting an iterative approach and adjusting one light per test render, the effect of each light is clear to the artist and he will maintain a clearer understanding of the shot.

Another technique that helps to ensure high-quality lighting is producing test renders over the plate into which the CG is eventually to be composited. Most render viewing software defaults to rendering the image over a black background. This is useful for checking edge artifacts but gives a poor representation of the finished shot. The human eye sees every color relative to the colors that surround it. By judging a render against black, the artist will not see subtle variations in color and how well the CG fits into the photographed element. It is usually possible to load a background image into a render viewer and for the reasons outlined above it is good practice to do so when testing lighting.

The majority of films that utilize digital visual effects now go through a creative grading process at the end of the postproduction pipeline. This stage results in the production of a digital intermediate and the process itself is now generally referred to as DI. The DI involves creatively changing color and contrast in filmed footage to create the finished look that the Director and DP designed. The artists who specialize in this process are often required to modify the photographed plate considerably by introducing shifts in color, contrast, and saturation. This process can exaggerate small differences between colors so it is clear that the CG rendered by the digital lighting artist must match the plate so that it reacts in the same way as the plate itself to any changes applied in the DI.

Because most production facilities now use a floating point pipeline to enable greater color fidelity and flexibility, it is possible for lighting artists to examine their work under a variety of extreme grading situations. A good and simple test is to use compositing software to view the CG layered over the background at two exposure stops above and then two stops below the exposure level of the scan. This will enable the artist to see whether the colors in the shadows and the highlights of the CG are sympathetic to those in the photography. At normal levels of exposure the highlights may be blown out to white on a monitor and the shadows can appear black. This is merely a limit of the viewing technology: They do in fact have color information, and by raising and lowering the exposure in a test composite, it is possible to see whether these too match the plate. If they do not, and the final composite were to be heavily graded, these differences between the CG and the plate could become very apparent and break the realism of the shot.

The Technologies of Lights in Computer Graphics

Several methods are used to simulate illumination in computer graphics, some being more physically accurate than others. As noted earlier, the primary limiting factor in terms of which technologies are used in digital visual effects is generally the speed of the computers rendering the frames. Until recently it has only been possible to use the most basic and, therefore, fastest lighting methods.

The three main types of lighting technologies used in visual effects are direct lighting; IBL, often in conjunction with ambient occlusion; and physically based rendering. There is some overlap between these methods but generally they represent a progression toward more accurate lighting simulations at the expense of longer render times.

Direct Lighting: Source to Surface to Camera

The simplest light source that can be described in CG is a point light. A point light is an infinitely small dot that outputs light intensity in all directions. It is a simple matter for the renderer to compute the light direction by creating a vector from the surface point being shaded to the point light source itself. From there it is possible to perform the calculations for the diffuse and specular models being employed by the surface shader.

The point light may be made more sophisticated by enabling it to observe the inverse square law of illumination fall-off. This law describes how if light falls on two surfaces, one twice as far from the light source as the other, the farther point will receive only one-quarter of the illumination that hit the nearer point.

The next direct light source to consider is the spotlight. This is probably the most commonly used light source in computer graphics. A spotlight consists of a single point of illumination, but rather than radiate out in all directions like a point light, the spotlight’s illumination is contained by a cone with the source of illumination at the point. The size of the cone may be varied, enabling narrower or broader areas of light to be described.

Figure 7.33 Four direct CG light sources (from left to right): point light, spot light, parallel light, and area light. (Image courtesy of Andrew Whitehurst.)

In addition to implementing the inverse square law, spotlights may also be engineered to have a fall-off across the section of the light cone. This produces a softer edge to the circular pool of light on the surface. Spotlights are used more often than point lights because the ability to adjust the size of the cone and the fall-off affords the lighting artist more options in controlling which parts of a model receive illumination and which do not.

A third form of CG light is the parallel ray or solar light. These lights are so named because the light that they generate is described only by direction: It does not emanate from a single point; hence the rays of light from it are parallel. This type of light is most often used to simulate the rays of the sun because, even though the sun is a finite-sized light source, it is sufficiently distant from the earth (roughly 150 million kilometers) that its rays are almost parallel, and this is certainly a good enough approximation for visual effects purposes.

The fourth type of direct illumination is area light. This is a light source whose origin is not an infinitely small point but is instead a piece of geometry, usually planar. Algorithms that compute area lights typically do so by considering points on the surface of the geometry as a set of points or spotlights and computing the effect of an area light as the sum of all of these small light samples. It follows, therefore, that the more samples the light considers in simulating the area effect, the better the quality of the finished result at the expense of longer render times.

Negative Lights

All of the lights described above emit an intensity of illumination. This is used by the surface shader to return a value for the reflectance of the object. Because many of these models are mathematically derived, it is possible to put values into these lighting equations that could not occur in reality but can have visually interesting results. One common trick is to use a negative light. By supplying a negative value to the intensity of the light a proportionately negative amount of reflectance will be returned by the surface shader. In other words this will make the surface appear darker. Negative lights are most commonly used to reduce excessively bright areas in a render when an artist does not want to adjust the settings of the other light sources in the scene.

Negative lights are a handy trick but they must be used with caution. As more visual effects use lighting techniques based on physically accurate models of illumination, laws of physics such as the conservation of energy become important in producing realistic renders. Negative lights do not obey these laws and can therefore produce undesirable results in physically based renders.

Reflections

The directional light sources examined so far in this chapter are used for diffuse illumination of surfaces and specular reflection. Specular reflections are the reflections of the light sources themselves. It is often necessary for other objects within a scene also to be reflected in a surface. These reflections can be computed via two main methods.

One method is to use ray tracing. In this technique the surface shader will fire a ray along the reflection vector of the point currently being shaded. If that ray hits nothing, it will return black; there is nothing to reflect. If the ray does intersect another surface, the surface shader for that point is computed and the reflected color of that surface is returned as the color of the reflection. This produces very accurate reflections but is expensive to compute, because it must be calculated for every reflective surface point on every frame, even if the reflection remains unchanging throughout the shot.

Because of this inefficiency another technique for simulating reflections is commonly used. This is a reflection map-or environment map-based reflection. These maps differ in the means of their creation but are referenced in the same way once they have been made.

Figure 7.34 A reflection map rendered in the cross format and a render computed using lookups to this map. (Image courtesy of Andrew Whitehurst.)

To create a reflection map, the renderer pre-computes a set of six maps, each of which is a render of the scene from an arbitrary point, and each is aligned so that the six renders can be stitched together to make a cube that represents a complete view of the scene from that arbitrary point. These maps are usually “unwrapped” so they appear as T-shaped or cross-shaped. When computing the final render, the reflective surface shader, rather than firing rays, uses the reflection direction vector to compute which part of the pre-rendered map represents that direction and returns the value in the map as the reflection color.

Environment maps are computed by rendering a single map of the scene in the panoramic latitude-longitude format (often shortened to lat-long), a format that is akin to the commonly used Mercator projection maps of the Earth’s surface, which show areas near the poles as being disproportionately large compared with the equator.

Photographed Reflections

In addition to rendering reflections of other digital objects, photography can also be used to generate reflection maps. This is especially useful in visual effects where a digital object often needs to be incorporated into a real environment. If possible, reference photographs should be taken on set to provide material for these reflection maps. These photographs may be taken using either a chrome sphere or as two sets of images, 180 degrees apart, photographed using a fish-eye lens on the camera. From this data an environment map can be constructed that will create plausible reflections of the set in a CG object.

Figure 7.35 A photographic reflection map in the latitude-longitude format. The image on the right is a render computed using this map for reflection calculations. (Uffizi Gallery Light Probe Image ©1999 courtesy of Paul Debevec, www.debevec.org/probes.)

When considering reflective surfaces that are glossy rather than mirror-like in their reflectivity, an extra property of map-based reflections comes to the fore. To compute blurry reflections with ray tracing, multiple samples must be rendered by randomly offsetting the ray direction around the reflection vector and calculating the average returned reflectance. These blurry reflections will be very accurate but they are also very expensive to render. If maps are being used for reflection it is a simple matter to convolve (blur) the map being used for the reflection to get a result that is often visually indistinguishable from the vastly more costly ray-traced method.

Shadows

Shadowing in CG may also be divided into ray-traced and map-based approaches. A shadow map is an image rendered from the point of view of the light source. In a spotlight shadow map the frustum of the view matches the cone angle of the light. Each pixel in this image represents a value for the depth of the first object seen by the light. During the main render each surface point rendered computes the distance to the light source that may illuminate it and compares this value against a lookup to the shadow map. If the number is greater than the lookup, it means that an object occludes the surface’s view of the light and thus it is in shadow. Otherwise the light’s illumination effect is calculated for the shaded point.

Shadow maps are prone to several artifacts, such as a blocky appearance, caused by a shadow map of insufficient resolution. Surface self-occlusion problems are also caused when the value returned by the map is very similar to the distance computed by the render and rounding errors can result in dark aliasing patterns. Blocky maps can be improved by rendering a higher resolution map or performing multiple, jittered lookups on the map to produce a blurred shadow. Self-occlusion artifacts are fixed by adding a small offset to the value returned by the map. This is referred to as a depth-map bias parameter and must be a value large enough to remove the artifacts from the render but not so large that shadows no longer appear to be connected to the surfaces casting them.

Shadow maps may be rendered for point lights by computing six images, each with a 90-degree field of view that covers all directions from the light. Solar lights may render axonometric projection-based shadow maps to maintain the parallelism of the light rays.

Figure 7.36 Comparison of different shadow calculation technologies. (Image courtesy of Andrew Whitehurst.)

A simple ray-traced shadow is computed by each surface shading point attempting to fire a ray at the light source. If the ray hits another surface en route, then the point shooting the ray must be in shadow from that light source. This process creates very accurate but sharp shadows. Softer shadows may be rendered by assuming that the light source has an area and shooting several rays from the shaded point at different parts of the light’s surface and averaging the number of ray hits or misses to compute the amount of shadow on the surface. A larger light source will yield softer shadows, but more ray samples will be needed to create a noiseless, smooth result.

Image-Based Lighting

In visual effects, the lighting artist is often called on to illuminate a digital object so that it blends into a photographed background. To do this, the on-set lighting must be meticulously copied. While it is possible to position direct lights to mimic the position, color, and intensity of every real light source on the set, this is laborious and, assuming that infinitely small sourced spot or point lights are used, does not take into account the different areas of light, perhaps coming from an on-set soft box, for example.

Already mentioned is the fact that by taking reference photographs on set with a fish-eye lens or mirrored ball, maps can be created that can then be used to create realistic reflections of real objects on digital surfaces, thus aiding integration between the CG and the photographed plate. These same maps may also be used for diffuse lighting calculations by convolving the images greatly to create a blurred map that approximates how a diffuse surface would react to the scene lighting. For this diffuse reflection, the surface shader looks up a value from the map based on its surface normal, not the reflection vector that is used in the reflection lookup. The diffuse reflection map lookup result is then multiplied by any maps or colors that the surface has in exactly the same way as the reflectance of a direct CG light would be computed.

Although it is not always absolutely necessary to do so, HDR environment maps should be used, because these will provide a much more accurate, and realistic, representation of the lighting on set. Light sources are often extremely bright, so an image with a higher dynamic range will capture the range of light intensities and colors more accurately. This enables more photorealistic renders.

Rendering Occlusion

The use of maps for reflection or diffuse lookups is subject to an inherent problem: Map-based lookups do not take into account any objects that might occlude the path of a ray fired off the surface, and these objects would prevent at least some of the environmental light from illuminating the surface. To account for this visible effect, two prerender passes are computed that simulate the effect of this occlusion. The diffuse environment map lookup is attenuated by an ambient occlusion render and the reflection map lookup by a reflection occlusion render.

Ambient Occlusion

An ambient occlusion render is one in which every visible shaded point fires a series of rays in a hemisphere around the surface normal. The number of rays that hit another surface is divided by the total rays fired. This value represents how occluded the surface is: A value of zero would represent a surface that is not occluded at all, whereas a value of one represents a point that is in total shadow. It is common practice to invert this map so that zero (black) represents complete occlusion and one (white) is not occluded. The reasoning behind this is that it is more easily understandable for artists working with the image and it enables the diffuse lookup to the environment map in the beauty render to be multiplied by this inverted ambient occlusion map to produce a realistic rendering of the effect that the environment’s illumination would have on the surface.

Reflection Occlusion

Reflection occlusion renders are produced in a manner similar to that of ambient occlusion and for the same reason: to prevent areas of a CG surface that would not reflect any of the environment from doing so. While ambient occlusion uses surface normal-based hemispherical sampling, the reflection occlusion process samples across a cone centered around the reflection vector. Larger cone angles create occlusion for blurrier reflections, a brushed metal surface, for example. A cone angle of zero (i.e., firing rays down the reflection vector only) will yield perfectly sharp reflection occlusion, which would be ideal for simulating a mirrored surface. It is important when computing reflection occlusion to choose an appropriate cone angle for the surface material being rendered.

Figure 7.37 Render produced using simple IBL with ambient and reflection occlusion. (Image courtesy of Andrew Whitehurst.)

Average Light Direction Vectors: Bent Normals

Attenuating a convolved environment map lookup with an ambient occlusion render adds considerable realism to the lighting of diffuse digital objects. However, it does not accurately represent the direction from which the majority of the light striking a surface came.

Consider rendering the inside of a cave using IBL and ambient occlusion. The normals of the cave floor will be pointing up and the environment map lookup for these normals will return illumination values for the sky. In reality, however, the floor would not be lit by the sky above the cave because the cave ceiling occludes it. Instead the cave floor would be lit by light entering through the mouth of the cave and hitting the surface at an oblique angle.

An additional pre-rendered pass, computed at the same time as the ambient occlusion, can help correct this anomaly. For each ray that does not strike a surface in the ambient occlusion render, a record is made of its direction vector. After all ray samples have been completed, the recorded lighting vectors are averaged. Thus, not only is there data representing how much ambient light strikes the surface, but also from what direction most of it came. This vector, which is the average light direction vector, is stored for use in the main render along with ambient occlusion data.

The reason this data is colloquially referred to as bent normals is due to the way the data is applied. When computing the diffuse reflection of an object, the surface normal is not used for the environment map lookup. Instead, the normal direction is chosen from the pre-rendered data, so the normal is bent to face another direction, one that better represents the direction of the main lighting contribution to that point. In the cave example the bent normal would point toward the mouth of the cave.

Creating Light Sources from Environment Maps

Simple lookups to pre-convolved environment maps coupled with occlusion can create very realistic renders efficiently. However, there are many desirable lighting qualities that this simple technique cannot reproduce. For example, an environment map with areas of high-intensity lighting will not cast appropriate shadows on the digital model. Each surface point will receive appropriate illumination, but the effect of the interaction of digital surfaces on one another when lit by IBL will not be accurately modeled. Several techniques are available for simulating these effects and these vary in complexity, accuracy, and computational efficiency.

One method is to create many direct light sources (usually solar or parallel ray lights), which can be arrayed around the digital model. The color and intensity of these lights are derived from the environment map. Provided each of these lights is shadow casting, a reasonable representation of the lighting interaction effects is possible.

Many algorithms are available to assist in determining the optimum number and position of these direct light sources. An example of this would be to use the median cut algorithm to sample the environment map, recursively dividing the map into areas of equal illumination intensity. Some areas will be larger than others but each will be assigned one direct light source, located at the light intensity emission centroid for that area of the map. This technique has the benefit of placing more light sources in areas of greater lighting importance. The process of locating limited numbers of light samples to maximize the rendering efficiency is called importance sampling.

A further development of this concept would be to use raytracing techniques to perform lighting lookups on light samples from the maps.

Not only do these techniques afford improved realism in the illumination of the objects themselves, they also enable more sophisticated lighting interaction with other surfaces. For example, some of the more sophisticated IBL approaches, like importance sampled lighting, can enable more accurate shadows to be cast on the digital representations of the set, thus facilitating better integration between the digital model and the plate photography.

Physically Based Rendering

Most lighting algorithms, especially those that are efficient to render and have thus proved popular at visual effects houses, are based around the interaction of each point on a surface and the light sources in the scene. In its simplest form each shaded point is utterly unaffected by the presence or surface properties of any other object. The ability to create realistic renders with this lighting pipeline is highly limited. The introduction of shadows on direct light sources is an improvement, because occluded objects no longer receive illumination from lights that could not “see” them. Ambient occlusion performs the same function for IBL sources. With these technologies a surface is aware of the physicality of other objects in the scene but does not take into account the surface properties of those objects. In reality, a blue painted wall standing on a white floor in bright sunlight will produce a blue area of illumination on the white floor. The sunlight hits the wall, and some of the reflected blue light will strike the floor before being reflected once more. Some of that reflected light would strike the retinas, enabling one to see the phenomenon.

The most obvious way of computing this effect is, for every pixel rendered, to trace rays through the scene, detecting and accounting for every surface interaction until a light source is hit. From the intensity value of the light source, one can work back through the sequence of ray hits to compute the ray’s effect on the intensity of the light until the ray returns to the camera where a color value for the pixel can be produced. Because diffuse surfaces scatter light that hits them in many possible directions, it is necessary to perform this sampling approach many times for each pixel to produce a render that is free from noise artifacts. Because of the high number of samples required, this technique, although highly accurate, is very time consuming to compute. This approach to rendering is called path tracing.

A less precise but computationally faster evolution of path tracing is to make the assumption that the changes in reflectance over a diffuse surface being hit with light are usually quite gradual. (Think of a matte plastic ball being lit by a single light source.) To leverage this observation, rays must be fired into the scene, and the irradiance (reflected color) of each surface point hit en route to the light source is stored in a cache. Although the sampling may be sparse, it is possible to interpolate between the values stored in the cache, so that renders that appear noise free may be computed with considerably fewer samples than were required in the path tracing approach.

Irradiance caches may also have additional processing performed on them to calculate subsurface scattering because the cache contains a representation of the objects in the scene and their surface irradiance values. This additional cache is then referred to by the renderer to obtain a value for the subsurface scattering at any point in the scene.

Irradiance cache-based approaches work well for complex diffuse lighting calculations but sophisticated specular effects such as caustics cannot be reproduced. To simulate such phenomena, a different rendering technique, known as photon mapping, is used. Photon mapping is a two-stage process. The first pass is the production of the photon map itself for the scene. The second pass is the main render that uses the data contained in the photon map to improve the speed of its computation.

To produce a photon map, particles of light are fired from light sources in the scene. Whenever a particle encounters a surface, its direction vector and the location of the point hit are stored in the photon map. From there, by taking into account the reflectance properties of the surface, the particle may be absorbed by the surface or reflected back into the scene to continue until it hits another surface when the calculation is performed again. A third option for the particle is that it is transmitted through the surface. This could happen in the case of a material like glass. Subsurface scattering effects may also be computed with photon maps.

Figure 7.38 A render produced using an irradiance cache shows the indirect bounce light from the spotlight illuminating the wall on the other objects in the scene. (Image courtesy of Andrew Whitehurst.)

The main render uses the photon maps to help compute the soft indirect lighting by providing the ray-tracing algorithm with pre-computed information about how light has been reflected in the scene. Photons that were refracted through participatory media (like glass) onto a solid surface can be used to render caustic effects. Specular reflection and direct illumination are usually ray traced in a traditional manner.

Volumetric Lighting Effects

Rendered representations of light encountering volumetric media such as smoke, cloud, or fog cannot be calculated using any of the techniques outlined above. This is because all of the previous approaches have relied only on performing rendering calculations when a light ray hits a surface. A smoke cloud has no surface; it is a volume.

To render these types of objects, a technique known as ray marching must be used. In previous forms of ray tracing discussed in this section, the ray only becomes interesting when it hits a surface; otherwise it passes unaffected through the scene. With ray marching the ray’s condition and its interaction with the scene are sampled at uniform steps along its path. If, for example, the ray were passing through fog, each step along its march would return the amount by which it has illuminated an area of fog. The cumulative effect of all of these samples can be accounted for in the color of the final rendered pixel.

Figure 7.39 Volumetric lighting effects visible in a render of a smoke-like volume. (Image rendered using Double Negative’s DNB renderer, courtesy of Double Negative Limited.)

Ray marching is computationally very intensive, and rendering software engineered to perform the more common solid surface rendering is often not optimized to perform ray-marching calculations. For this reason it is not unusual for a special case renderer to be used by visual effects facilities to render volumetric effects, while a more traditional renderer is utilized for the other solid objects. These two renders are then combined in the composite.

Conclusion

Digital lighting and rendering technologies are continuing to evolve. Academic conferences such as SIGGRAPH and Eurographics present new papers every year that offer new techniques and insights into reproducing the effects of light’s interaction with the world. Many of these ideas are then incorporated into the software used to produce visual effects.

As machines become faster, techniques that were once only theoretically feasible become practically useful—initially for the companies with large rendering farms and eventually for everybody. The many different approaches to simulating lighting with computers is, in part, the ongoing story of the advancement of computer power harnessed by ever more sophisticated algorithms. The techniques may continually change, but there is always the need for artists with good visual skills to utilize these tools to produce the believable visual effects that today’s cinema-going audiences expect.

SHADER BASICS

Derek Spears

What Are Shaders?

Shading is the part of the rendering pipeline that gives each sample on a surface a value. In the design of a rendering system, the shading system, also called the shading machinery, is called into play once the renderer discovers visible surfaces at each pixel on the rendered image. The shader is then invoked for the point on that surface corresponding to that pixel. How and why this surface is found is unimportant to the shader.

Ray tracing and global illumination are ways to intersect surfaces and define how much light gets to them. What happens with that information on the surfaces is the responsibility of the shader. What the shader is concerned with is how to color, or shade, the object. A shader, as will be seen later, may also alter the object’s geometry. When applied to a CG object, a shader and all of its associated settings that define the look of an object are often referred to as a material. The focus of this section is shading in a software context. Hardware shading (which is done on dedicated graphics cards instead of a computer’s CPU) is rapidly becoming more powerful and is conceptually similar to the constructs described here.

Shaders can not only modify the color of an object, they can also change the shape of an object. Surface shaders deal primarily with color. Displacement/bump shaders alter the appearance of the geometry of the surface. They provide a way for the lighting artist to add visual detail to a CG object without adding complexity and weight to the model.

Implementation of shaders can either be via an application interface, such as might be seen in Maya or Houdini, or via a software programming interface such as might be found in Renderman. (Houdini and other packages also offer a programming interface.) The advantage of an application interface is ease of use. Nontechnical artists can quickly assemble the look of a material via a graphical user interface. However, they are constrained by the tools that are presented. Software interfaces allow more technical users to write program functions that describe mathematically how to shade an object. The package that allows a developer to write code for a shader is often referred to as a software development kit (SDK). SDKs allow for greater flexibility and implementation of features that might not be allowed by a user interface (UI). This construct is very similar to that of a software plug-in.

Shading Models

The most basic building block of a shader is its shading model. The shading model describes how the object responds to light at the most basic level, whether it is matte or shiny, or if the reflections are dependent on more than just the viewing angles. Much research has been done on shading models, and many of the earlier models are approximations of how light reacts with a surface rather than being mathematically or physically correct. Lighting models, such as Lambert lighting, simulate only the diffuse, or nonreflective, aspects of a surface. The Phong and Blinn lighting models allow for specular highlights to imply a shiny surface. Other more advanced models, such as Cook-Torrance, attempt to more accurately model reflectance. In fact, many of the earlier models can be expressed within the framework of the Cook-Torrance model.

More complex shading models might involve ray tracing of actual reflected light to simulate mirrored surfaces. In addition to using contributions from lights, the shader could ask for the reflected light from another object at a given angle. For more realistic lighting, the shader could model the diffuse reflection of light via a global illumination scheme (i.e., modeling the way in which a red wall spills light onto a white wall). Physically based rendering models try to accurately compute how light is transmitted and reflected through the scene. Given an appropriate rendering infrastructure, the shader could implement these shading models in a similar fashion, as described above.

Bump and Displacement

Color is not the only aspect of a surface available to the shader. Surfaces can be perturbed, or displaced, by shaders. Displacement can be thought of as bumps on the surface of the object. Rather than modeling extremely high levels of detail, CG objects often have bump and displacement shaders added in order to increase the level of detail without increasing the model complexity.

Bump mapping, the simplest method, does not actually alter the geometry, but instead alters the normal (or, more simply, the apparent direction in which the surface faces) of the surface to make the surface appear displaced. This is a shading illusion; the surface geometry has not actually been changed. A grayscale texture map representing desired detail is used to perturb that normal.

An easy way to visualize what is happening is to consider this bump map to be a height field or elevation map. Much like a topological map in the real world, colors represent altitude. In this case, black would represent the level of the original geometry and white would represent a maximum new height. The bump shader looks at this map and computes a new normal based on how high the bump map raises the point. This is based on the slope of the new surface whose height would have been raised by the bump map. The normal at the current point on the geometry is then added to this new normal, which no longer corresponds to the geometry. Since the lighting calculations are entirely based upon the normal for the surface, the surface now appears bumpy.

This process has the advantage of being simple to compute; the disadvantage is that since the surface has not actually been changed, the edges will still remain the same, and the bump detail is not there (see Figure 7.40a) Normal mapping is an extension of bump mapping, whereby instead of using a single-channel gray image, a 3-channel image is used to completely re-define the normal direction instead of just moving the apparent height of the point up and down.

Figure 7.40 (A) A surface with no bump mapping applied. (B) The bump map, with black representing no change and white representing the highest amount of bump value. (C) The surfaces are rendered with the bump map applied. (Image courtesy of Derek Spears.)

Displacement mapping goes one step farther. Using the same height field texture map, instead of just altering the normal, the displacement shader will actually move, or displace, each point on the object being shaded along its normal using the height field as an amount. The advantage of true displacement is that the geometry is actually altered, whereas before the bump map left the object’s original edges unaltered. The displacement shader will create new detail. Also, the displaced geometry can cast shadows, unlike the bump shader, which only shadows using the original unaltered geometry. All of this comes at a price. The renderer has to do significant computations to understand the visibility for this new displaced geometry, leading to higher render times. Also because, in effect, the shader is actually creating more geometry, it will lead to higher memory usage.

Figure 7.41 (A) Bump shading is used. (B) Displacement Shading. With displacement, the profile has detail due to the geometry being moved as opposed to the smooth edge of the bump shader (see inset). Also, note the shadows cast by the displaced ridges in the lower right of the image that are absent in the bump shader. (Image courtesy of Derek Spears.)

Map-Based Shaders

Once the shading model has been established, adding texture is a way in which detail is added. Adding texture to an object is analogous to painting an object in the physical world. The added complication is understanding how to apply a flat, 2D image (or texture) to 3D object. This process of transforming the 2D image space to 3D object space is known as mapping. To differentiate between the object and the image, the texture’s X and Y coordinates in the image are referred to as u and v, which prevents confusion between the objects X and Y. This is analogous to u and v in parameterized surfaces, and in fact, the same u and v parameterization is often used for the texture coordinates.

Surface parameterization is a method of defining surfaces. Instead of defining a surface traditionally whereby each vertex coordinate is specified, a parametric surface is defined by a mathematical equation. The values for the X, Y, and Z coordinates are determined by a function whose parameters are u and v. A surface defined by these parameters is called a parametric surface. The values of u and v typically range from 0 to 1. Assume Fx, Fy, and Fz are equations whose variables are u and v. Hence, each coordinate X, Y, and Z is defined by solving those equations. This is a simplification; for something such as a Bezier surface, there would be a set of control points that help define these equations.

X = F_x(u,v)

Y = F_y(u,v)

Z = F_z(u,v)

For polygonal objects that have no parameterization, some type of parameterization must be created for the surface. That is, for every vertex that has an X,Y,Z coordinate, a u,v coordinate needs to be supplied that will tell the shader what part of the image should correspond to this part of the surface. This process is referred to as UV mapping. In Figure 7.42, a 3D surface is shown with its coordinates X, Y, Z and a corresponding u,v. To determine interior UV coordinates, the values would be interpreted from the neighbors. For instance, a value halfway between u = 0 and u = 1 would have a value of 0.5.

UV mapping can happen in several ways. The simplest way is to use a projection map. A projection map is analogous to projecting a slide onto a screen (or a shaped object for that matter). This is the basis of the 2.5D projection map described in the next section of this chapter, 3D Compositing. The basic types of projection maps are planar, cylindrical, spherical, and camera.

Figure 7.42 (A) A texture map can be thought of as going from 0 to 1 in the horizontal or vertical direction. (B) The u,v texture coordinates are assigned as shown on the surface geometry. Each vertex will have its own coordinate; only the corners are shown for clarity. (C) The final rendered image has the texture map from part (A) applied to the geometry using the u,v coordinates in part (B). (Image courtesy of Derek Spears.)

In planar mapping, the coordinates are projected orthographically out from a plane along the normal direction (similar to a slide projector onto a screen). In cylindrical mapping, the coordinates are projected from a cylinder inward. Similarly, spherical mapping projects the coordinates inward from a sphere. Camera mapping is similar to planar mapping, but instead of projecting orthographically from a plane (where the projection lines are parallel), the coordinates are projected along a camera frustum¹⁷ and diverge.

Figure 7.43 (A) A texture is projected using two different methods. (B) The u,v coordinates are using a planar projection in the Y-axis. Note the stretching of the green texture and the repetition of the C, D, and E rows. (C) The texture has been projected using a more complex unwrapping scheme, which yields better results. (Image courtesy of Derek Spears.)

The choice of which method to use largely depends on the shape of the object that is being mapped. Flatter objects respond well to planar mapping, tube-like objects to cylindrical, etc. The advantage of projection mapping is its simplicity. However, it becomes limited because of the difficulty of dealing with surfaces that don’t line up with the projection. The less perpendicular a surface is to the projection, the more texture stretching can occur. Also, surfaces that overlap with respect to the projection source will have the same u,v coordinates, leading to repeated textures.

The solutions to these problems lie in either breaking up the object into smaller, less problematic objects, or groups, or using a more sophisticated mapping method. Surface relaxing and unwrapping methods can deal with these more complex shapes and unwrap them to a flat shape. This flattened shape can then be easily be projected into a u,v space. The methodology, although mathematically complex, is analogous to smoothing out a wrinkled sheet of paper in order to make it lay flat.

Now that the surface has u,v space, it is possible to apply an image to the surface. As the shader is called for any given point on the surface, the corresponding u,v coordinates for that point are available to the shader and it can then look up the corresponding pixel in the texture map. Texture maps can then be used to control all aspects of the material’s appearance, such as the material’s color or the color of the specular highlight. Maps can even be used to control noncolor parameters, such as the roughness component in the Phong shading model.

Procedural Shaders

Texture maps place a tremendous amount of design power in the hands of the CG artists, but there are times when painted maps are not sufficient. Maps are typically static images. It is possible to apply an animated set of maps to a shader, but that requires either painting many different maps or having another piece of software render a sequence of images. Rather than have an external program generate a map, it is easier to just move that code into the shader. This code-driven creation of the texture map within the shader is referred to as a procedural shader. Procedural shaders are often used to add noisy detail that is easier to compute than to paint. Additionally, procedural shaders offer the added benefit of being resolution independent. A painted texture map exists at a certain resolution. For example, suppose an artist had a scene where a camera was zooming in to a painting on a wall where the painting was a texture painted at 1024 × 1024. Once the camera got to the point where the rendered image was 1024 × 1024, getting closer to the image would not give more detail; the image would only get softer and softer. With a procedural shader, if the camera gets close to the texture, additional detail in the shader can be computed to fill in the detail, providing the possibility of adapting to varying resolution requirements.

A common use of procedural shaders is to produce noise patterns. One example of a procedural noise function is the Perlin noise function developed by Ken Perlin. The Perlin noise function allows for the quick computation of noise of a known frequency and amplitude based on a 3D point in space. The function itself might look simplistic in its appearance, but by layering increasing frequencies of the noise function, it is possible to create realistic and interesting noise patterns. These noise functions can either take the u, v space of the surface as an input (2D shaders) or the X, Y, Z coordinates of the surface as an input (3D shaders). Many different types of noise functions exist that can be layered together to create useful textures.

Figure 7.44 Different types of procedural noise patterns. (Image courtesy of Derek Spears.)

Shader Design

With a basic understanding of how shaders work, let’s take a quick look at how they are used in readily available animation packages. In Maya, the interface is streamlined to allow the user to layer up components to build a shader. This model is common across other packages as well. The user first chooses a shading model (such as Blinn or Lambert). Then, all of the different components of that shading model, such as diffuse color, specular color, roughness, etc., can be attached to procedural shaders or texture maps. In turn, all these shaders can have each of their parameters attached to another shader.

For example, the diffuse color from the Phong shader could be attached to a texture map. The base color of that texture map could then be attached to a noise shader in order to modulate the texture map with noise. In this way, it is possible to build up complex shaders with the combinations of simpler shaders.

Figure 7.45 In Maya, each parameter can have another shader, or material, as an input. The color in the Lambert shader is linked to a noise field. The amplitude of the noise is linked to a grid with bulges, which produces a shader with noisy sections only in the white areas of the grid. (Image courtesy of Derek Spears.)

Figure 7.46 Houdini’s node-based shader system. (Image courtesy of Derek Spears.)

Another approach to shader design is used in the Houdini animation system. Instead of using a list of functions and attaching parameters in a list, Houdini also allows the user to connect the shader function in a flowchart-style view. This is analogous to a node-based compositor. It has the advantage of being more powerful but at the expense of being somewhat more complicated.

Most artists will find the graphical interface sufficient for most tasks, but some problems go beyond what a prepackaged shader can provide. In this case, many renderers provide an interface for writing custom shaders. These shaders can take the form of compiled C code, such as Mental Ray, or a higher level language that has its own special compiler, such as Renderman. These frameworks allow for low-level access to the renderer. The programmer can then create new shading models or implement shaders with functionality completely outside the scope of original program.

Antialiasing Considerations

Antialiasing in shaders is slightly different from antialiasing in rendering. The latter might be referred to as edge antialiasing. When a render samples a surface, it must do so enough times to smoothly determine where the surface is and isn’t. With too few samples, jaggy edges are apparent. This is often referred to as pixel samples. However, depending on the implementation of the renderer, it is possible that, no matter how high the pixel samples are, the shader will only get called once per pixel. This may seem fine conceptually but can give rise to shading artifacts, as will be seen.

Figure 7.47 (A) No antialiasing results in jaggy image edges in the shader. (B) Correct shader sampling results in smother, correctly aliased edges. (Image courtesy of Derek Spears.)

Imagine a shader that renders a black-and-white grid on a surface. It could either be a texture map or a procedural shader. If the renderer only samples the shader once per pixel, it will only ever get a black or white pixel. No antialiasing will take place because the shader only returns either black or white, inside or outside the grid, as seen in Figure 7.47. Two steps can be taken to help minimize this. First, the shader can be sampled multiple times per pixel. Although a pixel seems like an infinitely small area, the pixel actually is a square, the corners of which define an area on the screen. If the shader is sampled multiple times within this area in a random pattern, the average of these values can generate an antialiased pixel for us, much in the same way edge antialiasing works. The control for this is referred to as the shading rate. Second, in designing a shader, it is important to minimize methodologies that give rise to aliasing. For instance, functions that access textures should do so in ways that adequately filter the image. The shader should appropriately filter procedural functions that generate noise or other images.

This has only been a very high-level look at shading. Each individual rendering package will approach the problem in a slightly different fashion. The focus here is to provide the reader with a basic understanding of how shaders can be implemented for any given rendering scheme.

3D COMPOSITING

Derek Spears

Color Representation

To fully understand how compositing works in a digital world, it is important to understand some of the basics of color representation. Color is a complex topic and the details are much broader than can be covered in this chapter. However, some basics are necessary in order to give a framework to the understanding of how and why some of the esoteric elements of compositing work.

The primary ways in which imagery is represented fall into two camps: linear or logarithmic (log). These are not color spaces per se, but are ways in which one chose to store data. CG imagery is typically stored in linear space. That is to say, there is a linear relationship between the value and the brightness of the pixel. In the real world, this is all a bit more complex; a linear gradient does not appear linear on a monitor. Human eyes and display devices have a more logarithmic response. To view a linear ramp as perceptually linear it is necessary to take into the account the response of the monitor/viewing device. This is where expressions like 2.2 gamma¹⁸ (or 1.8 for Mac) come into play. Math operations in linear space are straightforward, that is, multiplying a value by 2 doubles its value and, therefore, makes it appear twice as bright—thus the desire to work in a linear world and for nonlinear devices to do the machinations to make linear imagery appear correct.

Log encoding is typically used as storage for scanned film imagery. The reason has to do with the dynamic range of film. The Cineon file format was originally built with the intention of capturing 10 stops of film exposure using 10 bits. The data is effectively compressed using a log encoding. By using a log function, large changes in exposure value can be expressed with a smaller range of numbers.

Instead of having a large linear range, in log the same amount of brightness data may be represented by a much smaller range of numbers. Since larger numbers require more bits, log allows one to compress the large exposure range of film into a smaller format. Log scaling also more correctly models the response of film. In Cineon log encoding, each 90 code values corresponds to one stop of film. In linear space, the number of values varies depending on the brightness of the image. However, log tends to be impractical for CG compositing because it can give rise to artifacts in compositing CG images that are typically represented in linear space. Because of this, there exists a need to expand the film representation into a usable linear range, which is discussed next.

Bit Depth and Dynamic Range

Having log data and a CG image with which to composite, the next step is to define the mechanisms to relate the two. In linear space there are several choices. These choices revolve around how big the numbers to be used are. These formats are often referred to by the size of the number being used, for instance, 8 or 16 bit.

The 8-bit format, which has been used historically in CG because of its compact size, stores data between 0 and 255. It does not give a lot of range and does not get used much outside of standard-definition video. Two major problems exist with this space. First, the lack of precision (the low number of values represented) does not allow for a lot of choice in color values. Although using 8 bits for red (R), green (G), and blue (B) provides 256 shades per channel, and 16 million possible colors, problems occur when gradients try to be represented. For example, assume a smooth gray gradient from a dark gray (value 65) to a medium gray (value 125) over 500 pixels is desired. Because 500 pixels can only be covered with 60 possible values,¹⁹ some areas are going to be represented with the same value. This is referred to as banding and is a very undesirable artifact. Some modern video formats also allow for 10- and 12-bit linear images, which, although they have a larger range of numbers, still have a more limited dynamic range than is desirable.

Figure 7.48 An example of banding due to low bit depth. Left: A gradient having a large number of values over a large area. Right: A small number of values represented over a large area. Having a higher bit depth would help eliminate the banding by allowing more values to be represented. (Image courtesy of Derek Spears.)

Moving up from 8 bits, the 16-bit format provides values between 0 and 65,535. Just by doubling the bit size there is now 256 times more range with which to work. This helps address the banding problem; however, just expanding the log values into this linear space is not the answer. Since most of the image data lies between black and what is perceived as white, it is desirable to have the most precision in this range. From this need to specify a visible range of values comes the concept of a white point and black point. The white point is the value that is perceived as white. Values much brighter than perceived white can exist. For instance, in a photo, a white sign and the sun may appear to be the same value of white. In reality, the sun is orders of magnitude brighter. Preserving these relative brightness values, or high dynamic range, is critical as will be seen later. So with well-defined 16-bit space, there is an allowance for the headroom to store those extra-bright white values.

The Cineon spec is a 10-bit log number, with perceivable black defined as 95 and perceivable white defined to be 685. With 8-bit, there is not much workable space so 95 to 685 log is expanded into 0 to 255 linear. With 16-bit linear, there is much more precision available, so it becomes easier to preserve those bright overexposed values in the Cineon file. A common mapping is to map 0 to 685 in log space to 0 to 4095 in 16-bit linear for the Cineon log function. Then the values from 689 to 1023 in Cineon (1023 is the maximum value in 10 bit) map to 4096 to 65,535. This mapping is not quite as intuitive as the simple 8-bit mapping, but there is a lot more precision, and the overexposed bright parts of the film are preserved.

To use the example with the sun and the sign, the sign could be represented with a value of 4095 in 16-bit linear, and the sun could have a much brighter value, say, 40,950. This would make the sun 10 times brighter than the white sign. On the monitor, they both would appear bright white. However, if the compositor chose to darken the image, the sign would become less white with the sun staying white—that relationship would be preserved.

So 16 bits gives a workable space with a little manipulation. There is still some loss in precision and although 16 bits seems like a lot, there are still possible issues with banding. The next step up is floating point. Instead of calling an arbitrary number such as 4096 white, a more mathematically suitable number such as 1.0 becomes white. Everything brighter than white can be represented by a number greater than 1.0. And since fractional numbers, like 0.986, can be represented, there is virtually unlimited precision to circumvent the banding problem. (In reality, there is a limit to precision, but that limit is reasonably high and unlikely to be reached. For computer representation of fractions, refer to the IEEE floating point standard.)

So, log 0 to 685 values are expanded into 0.0 to 1.0, and 686 to 1023 expands into 1.0 to 13.0. Now this provides a usable linear space with good precision and overexposure capability. Formats can be represented as 16-, 32-, or 64-bit floating point, either having the same or 2× or 4× the storage requirements of a 16-bit integer. However, computation typically needs to happen in at least 32 bits, so the memory footprint used in computation will be higher. However, the benefits in computational precision far outweigh the negatives.

Mattes

Traditionally, when a matte is pulled²⁰ by a keyer, that matte is multiplied against the foreground. The background has the inverse of the matte removed from it, and then the foreground is added. This can be more clearly represented by the following formula:

Result = Fg *(A) + Bg *(1 − A)

where Fg is the foreground, Bg is the background, and A is the alpha, or matte.

In CG mattes, the matte can be more appropriately thought of as the percentage coverage of a pixel. Obviously, a jaggy line is not desirable, so in order to antialias the line, the edge pixels have their intensity reduced based on the percentage of the pixel they cover. If the line covers 50% of the pixel, then the pixel brightness is reduced by 50%, or multiplied by 0.5. That percentage of coverage is then stored in the matte. For pixels entirely inside or outside of that line, their coverage values are 1 or 0, respectively. Because that coverage is already multiplied into the edge pixels, CG mattes are referred to as premultiplied. This is important because the compositing equation is changed accordingly. No longer should the matte be multiplied to the Fg CG. It now looks like this:

Result = Fg + Bg *(1 − A)

This operation is often referred to as an over.

The biggest practical side effect of premultiplied images is that a compositor must be very careful when doing color correction. Because the matte is already multiplied in, color corrections not only affect the image but also the relation to the matte as well as the edges and in transparency. If a gamma correction is applied to a CG image, the antialiasing quality at the edges could be adversely affected, and because the CG image is no longer being multiplied (or, more simply, held out) by the matte in the composite, anything done outside the CG image in the black area of the matte will then be added to the entire image (see Figure 7.49).

The solution to these issues is to un-premultiply the image, do the color correction, and then multiply them back together. When an image is un-premultiplied, the areas that are outside (or in the black area) of the mask will usually become strangely colored. Those areas are undefined due to dividing by zero, resulting in random values. The actual values here are not important. When the images are premultiplied back, those areas will get multiplied by zero again and go back to black.

Compositing CG

In the most ideal case, a CG artist lighting a shot would hand a compositor the perfect CG image that just comps directly into the shot, leaving the compositor very little to do. However, in visual effects, as in real life, there are never really any ideal cases. There might be smoke in the shot, or the film might be grainy, or the lighting in the CG might not quite match the shot. All of these problems can be handled in the composite.

Figure 7.49 (A) The CG image to be composited. (B) The CG image composited over a background with no color correction. (C) The same image composited with a color correction respecting premultiplication. Note how the background image is not affected. (D) The same color correction without paying attention to premultiplication. Note how the background image is affected. (Image courtesy of Derek Spears.)

CG renders tend to be long and the process not very interactive. Getting iterations of varying light intensities or the color of specular highlights takes long iterations. Computers evolve and become faster, but the problems grow to consume the available resources. So, although in the past it might have taken 1 hour to render a simple CG object, now it takes 1 hour to render a CG object with proper global illumination and ray-traced, motion-blurred reflections. Conversely, compositing is comparatively cheap. It is much easier and faster to explore different lighting levels and looks in the composite than it is in the render. The issue then becomes how to get the right pieces to do it.

Enter multiple lighting layers. While artists can deliver large monolithic CG elements with all of the light, diffuse, and specular values balanced in, there is more recently a trend toward taking care of these balance issues in the composite. This is not to say that multiple-layer lighting is done exclusively throughout the industry, but it is becoming a viable option used in many pipelines.

Each setting in a shader for a CG object that might be adjusted can be broken out into a separate layer. Since the effects of these layers are combined in simple, associative mathematical ways, combining these layers is just as easily done in the composite using the same math. For example, specular or reflective highlights are typically an add operation in the shader, so it is very easy to break out the specular layers as a separate element to the compositor that can be combined with an add node. This can be extended to just about any value that a lighter would control in a shader. Diffuse, specular, reflectivity, and ambient occlusion are just a few layers that are typically broken out.

Lighting is not replaced by this layered approach. There are still many things that need to be taken care of in the lighting setup. Special shading effects that require knowledge about the 3D environment, like refraction, bump/displacement mapping, anisotropic reflection, and light fall-off, will have to be adjusted by the lighter. Shading layers are not the only things broken out. Each individual light or groups of lights are typically broken out as well to give the compositor the ability to rebalance key-to-fill ratios, and alter the light colors in order to more closely match the environment. The ubiquitous gray and chrome spheres are often used against CG render images of those spheres to check those light balances.

All of these layers can give rise to massive organizational problems. Rather than rendering out each of the layers as separate files, modern image formats allow for arbitrary numbers of channels. Whereas artists were once limited to R, G, B, and A in a Cineon image, now TIFF, EXR, and other proprietary image formats allow for all of the rendered lighting layers to be stored in the same file, making all the passes much more portable. This is often referred to as deep rasters.

Some rendering options may even be handled more efficiently in the composite. Motion blur tends to be expensive in the render, and invariably, unless the quality settings are turned up high enough, there are artifacts. When those settings are turned up, the render time grows explosively. However, it is possible to closely emulate proper 3D motion blur in the composite, and that typically equates to large savings in render time for small losses in the correctness of the blur.

To use composite-based blur, the render needs to be able to generate a vector for each pixel describing the direction and amount of blur in screen space. This 2D vector can then be used by the compositor within a specialized node to blur the image along the vector in screen space. Although an effective method, problems may arise when a semi-transparent object with motion blur overlies another object. The resulting vectors are then poorly defined because it is hard to associate which object the blur is meant to apply to. The objects could be moving in opposite directions but there is only one set of motion vectors.

Just simply placing a CG object in the scene is not necessarily going to make a convincing shot. Usually a CG creature that is being composited is touching or interacting with an object in the scene. Making that integration work is often what sells the shot. Cast shadows will be provided, as mattes, for the compositor to darken the scene. Care must be taken while darkening the scene with a shadow matte not to darken below the latent shadow value of the scene. Additionally, contact shadows may be provided. Contact shadows are small shadows at the contact points between the CG object and the scene. These provide subtle darkening to aid in grounding the object. Additionally, CG renders are often too sharp, making it necessary for the compositor to mimic the natural softness of the photographed plate. Even in fully CG shots, it is worthwhile to add some blur to better match surrounding live-action footage.

One of the key areas for making CG imagery fit into existing photography is matching black levels. Atmospheric haze and smoke are just some of the plate considerations the compositor will have to deal with. Responsibility lands on the compositor to lift, or lower, the blacks of the CG element to match the blacks of the scene. Often, photographic elements will have a color cast in the blacks that needs to be matched as well.

Perceptually, it is hard to judge the accuracy of these black level matches when they are dark, so a common compositing technique is to view the final composite at different brightness and gamma levels to ensure the black levels track. Although black levels are important, that does not make white levels unimportant. When a shot is worked on in the composite, a choice has to be made before the shot is started about what the color grading will be. This does not mean that this is where it is going to land when it is balanced for the film. The shot could be printed brighter, darker, contrast added or removed, and/or be pushed into a completely different color. If that shot is printed darker, it is going to uncover white level mismatches. For example, if the CG artist has created a CG car at night with headlights, and those headlights are only at a value of 1.2, while real headlights in the scene are at a value of 3.0, the shot will look fine in dailies or on the monitor—because of the limited dynamic range of these display devices, both of those values might appear to be white—but when the colorist prints the shot down two stops, what appeared to be a white headlight is going to appear gray. It is important to pull the final composite down in brightness to check (and measure) the highs, not just the darks. Along the same line of thinking, while the shot is pulled brighter and darker, moving the shot around in color can also be a good check to ensure the CG in the shot tracks correctly through other possible grading choices.

Live-action photography also has grain or noise. Compositing formulas exist that model the exact structure of film grain via mathematical and statistical functions. More empirical methods involve scanning different exposed film stocks to use to comp in film grain. Grain is defined by both size and density and this varies with each color channel, especially in film. In film, three layers of color-sensitive dyes are sandwiched together, red being at the front, green in the middle, and blue at the back. Since light has to pass through the other two layers to reach the blue record, it needs to have larger grain structure to compensate for less light reaching it. Although digitally acquired images have no film grain per se, there will still be some noise present in the sensor that needs to be matched. The amount of grain may change with the exposure. Underexposed areas have more prominent grain structure than overexposed areas. So it is important, when grain matching, to look not only at each individual channel, but to look at different areas of brightness. Whatever grain addition method is chosen, it should not be constant. It also needs to vary from frame to frame.

With CG images, the compositor has the luxury of much more information per pixel than just color or matte. Depth information may also be rendered, giving the compositor the ability to perform depth compositing. A typical encoding scheme might be 0 (black) for objects at the camera plane and 1 (white) for objects at maximum distance. With integer numbers, near and far distances need to be defined at render time. With floating point, the far distance can effectively be infinity (with expected loss of precision at great distances). Using these depth values, it is then possible to accurately composite CG objects in correct depth relation to each other.

Figure 7.50 Example of a depth image (right) corresponding to a rendered CG image (left). (Image courtesy of Derek Spears.)

Depth compositing is not without problems. Motion blur and antialiasing are two particular cases in which depth compositing has problems. In the case of antialiasing, unlike a matte, a depth matte represents distance from camera, not coverage. If an artist attempts to antialias a depth matte, the result will change depth values at the edges. Different depth values will then be introduced at these edges, creating depth inconsistencies. Likewise, motion blur may cause similar problems.

2.5D Compositing

Increasingly, elements in a scene are created with something referred to as two-and-a-half-D (2.5D). This 2.5D is used to create the appearance of a complex 3D environment with a much simpler and faster 2D toolset. Currently, most modern compositing packages include 3D tools. Cameras, as well as lights, can be defined and introduced in the compositing package. Simple geometry may be either imported or built. Even animation data can be brought in, such as the tracked CG camera for a shot. With these tools it is possible to create backgrounds and other effects within the compositing environment.

One of the most common uses of 2.5D is for projected matte paintings. Instead of modeling a highly detailed scene, a simple model of the scene is created that represents the outline and basic relief. This geometry is typically very low in resolution— on the order of a few hundred polygons. Instead of texturing and developing shaders, a matte painter would then paint a fully rendered environment onto this scene. All light and shading information is painted into this image—no actual CG lighting takes place. The matte painter’s image will then be projected from the CG camera onto the modeled terrain. Given the low poly count and the power of modern graphics cards, this setup can be previewed in real time in the compositing package. Often cracks will be revealed as the background moves, uncovering areas unseen by the camera projection. The matte painter will go back in and fill those gaps so that they may be re-projected onto the painting from another camera position. Because the terrain has been modeled with relief, the image exhibits parallax, giving the appearance of a complex 3D surface. Minute detail is not modeled, because the viewer is provided with enough detail in the texture to believe the result.

Using the same tools it is also possible to take an existing plate and create a new move on it. Using a simple model of the terrain again and a 3D track of the camera, the original photography can be projected onto the geometry. That image will now appear to be locked to the geometry as if it were a still texture just like the matte painting example above.

For moving footage, instead of a static image, the sequence is projected from the moving camera per frame from the camera track. Once projected the artist can create a new camera with an entirely new move. If the new move is significantly different from the original plate, unprojected areas in the geometry may be uncovered. Just as above, a matte painter can go in and fill those areas with detail because the projected texture is locked to the geometry. Still plates can be given full 3D camera moves, and existing moves can be changed. The uses for 2.5D continue to grow because it is becoming a common method for constructing digital backgrounds.

CROWD GENERATION AND SIMULATION TECHNIQUES

John Knoll

Live-Action Replication

The earliest and simplest technique for generating large crowds with a limited number of extras is to shoot multiple passes with the group moved about to fill the needed area. The separate plates can then be composited to create the illusion of a larger crowd. For example, this technique was used to assemble the large crowd around the Washington Mall in Forrest Gump (1994).

This technique is easiest to execute when the camera is stationary and motion is limited to pan/tilt; it becomes progressively more difficult as additional axes of motion are added. Particularly complicated multiple-axis camera moves may require motion control to repeat moves, especially where unambiguous perspective cues are present.

Usually groups of extras are arranged to minimize visual overlap per pass to reduce the matting complexity in the composite, especially if it is not practical to use process screens to provide an extractable edge. For example, it is usually a much easier compositing task to arrange extras in left, middle, and right groups as opposed to foreground, midground, and background groups.

Sprites

Another technique is to photograph smaller groups of extras in front of a process screen (or computer animate the extras, for that matter) and then use them to texture cards that are distributed throughout a scene with time offsets. This is inexpensive to create and render but limits camera motion. Such a system was used for cheering crowds in the Pod Race arena in Star Wars Episode One: The Phantom Menace (1999).

Computer-Generated Crowds

With the advent of commercial software tools like Massive,²¹ it has become increasingly popular to create large crowds using computer graphics. Computer-generated crowds offer numerous advantages over comping together groups of live-action extras:

•   Crowd size is unlimited (to a large extent).

•   Plate photography can proceed more quickly because it is not necessary to shoot multiple plates or coordinate and move groups of extras.

•   Behavior can be more complex and free because it is not necessary for one to police layering issues at the boundaries of the groups.

•   Camera motion is not limited in any way.

•   Anything that can be represented with computer graphics can be part of the crowd, such as alien creatures or a herd of elephants.

•   Action can include events that are extremely dangerous or impractical to shoot.

Modeling for Replication

When building characters for computer-generated crowds, it is not practical to build each member of the crowd individually. A small finite number of individual models are usually built with some mechanism to generate variations of each. In this way one can populate an entire crowd with a limited number of specific models and variations of those models.

Variation

To minimize the number of unique models to be constructed, it is usually a good idea to have a method for creating variations of each to provide sufficient visual diversity to the crowd. Four primary techniques are typically used to create variations:

•   Shader parameters such as brightness, hue, etc., may be randomly varied per individual.

•   Characters can be modeled with multiple wardrobe and prop types, such as different coats and hats, for example. These props and multiple wardrobes may be turned off and on in combination to make a large number of individuals.

•   A number of different textures may be prepared per model to give them different looks.

•   Different body types (fat, thin, long/short torso, etc.) may be sculpted onto the same base mesh as vertex position deltas or morph maps. These can be dialed in randomly per individual. Simple scale variation is also often employed to provide additional individuality.

Mesh Density

Because a large number of instances of each model will have to be processed to make each resulting frame, it is important to keep the geometric complexity of the source models as low as is practical. Some systems alleviate this issue by automatically switching between multiple levels of model complexity based on screen size. This means that very lightweight (and fast to process and render) models are used in the far distance, and more detailed models are used closer to the camera where that detail is needed because it will be visible to the audience.

Animation Cycles for Replication

Most crowd control systems require the characters to have a library of predefined animations to cover their expected behaviors. Exceptions to this are where simple parametric biped and quadruped locomotion models are sufficient.

An effective but limited method to provide motion complexity in a crowd is to have one or more long motion cycles that can be applied to each character with random time offsets.

A much more flexible but complex method involves having a library of different motion types (running, walking, fighting, etc.) and a mechanism to make transitions between them. The ability to assemble strings of motion in nearly arbitrary order (usually called a motion graph) allows characters to be directed to respond appropriately to their surroundings and is very powerful when coupled with an artificial intelligence system.

Motion Capture

Because humans or domesticated animals are the most common members of computer-generated crowds, the most direct method to create sufficient realistic motion is via motion capture. (Please refer to Chapter 4 for more information about motion capture.)

Keyframe Animation

When the characters, vehicles, or objects that make up the crowd are inconvenient or impossible to motion capture, traditional keyframe²² animation can be provided.

Dynamic Motion Synthesis

Endorphin²³ is an example of a system with an understanding of biped locomotion and ranges of motion that can generate realistic movement without example input. Its built-in physics model allows it to realistically simulate dangerous stunts like long falls or tripping down stairs. Endorphin also is able to mix synthetic motion with motion capture or keyframe animation, allowing users to begin with provided motion and then transition into simulation.

Behaviors and Crowd Control

A crowd control system manages the size and behavior of a generated crowd. Two methods are commonly employed to do this, particles and artificial intelligence.

Particles

Members of a crowd can be treated as particles. In its simplest form, particles are generated across a surface and are used to instance model variations driven by random time offsets in long animation cycles. Such a system was employed to create background battle action in the Droid/Gunga battle at the end of Star Wars Episode One: The Phantom Menace (1999).

A slightly more complex and flexible version of this is to have a particle system directly control the positions and movements of the characters. The particles have rules to constrain their motion, for example, avoid collisions with barriers or other characters or move away from repellers and toward attractors. Appropriate offsets into animation cycles are chosen based on the particle’s position along a path. Such a system was employed to generate crowds gathering before and dispersing after the Pod Race in Star Wars Episode One: The Phantom Menace (1999), as well as in the Geonosis Arena in Star Wars Episode Two: Attack of the Clones (2002).

Artificial Intelligence

A more powerful system employed by Massive is to have an artificial intelligence (AI) system control the behavior of the crowd. Massive users construct a “brain”—basically a parametric behavior decision tree that controls how characters respond to their surroundings and other characters. The brain then drives a motion graph to calculate detailed character motion. This system was developed for the battle scenes in the Lord of the Rings trilogy (2001-2003) and has been commercialized to great success. Massive-generated crowds can be seen in numerous films and television shows.

Note that live-action photographed crowds and computergenerated crowds can be, and often are, used in combination to improve realism. Usually live-action photography is used in the foreground with computer-generated characters added to create background action.

CG PROSTHETICS AND ACTOR ENHANCEMENTS

John Knoll

It has become increasingly common in recent years to employ digital techniques to create effects that change the look of an actor that in previous years was solely the province of makeup. These techniques are employed where makeup is insufficient or incapable of the task. Some examples of this might be the removal of limbs on a wounded soldier or adding an effect to a creature to allow it to change or grow during a shot. The “Two Face” effect in The Dark Knight (2008) is a particularly good example because the effect creates negative spaces—subtracting from the actor’s face—to create a burned away appearance. Makeup can only be additive by its nature.

Examples where a director chose to use visual effects rather than rely on traditional makeup include Gran Torino (2008), where a teardrop was added to an actor, and Invictus (2009), where the ability to precisely control the bruising on the players could be done in post. This reduced some of the set time but more importantly allowed the order of shots to change in editorial and permitted the director of both movies, Clint Eastwood, to precisely determine the amount required for telling the story.

On-Set Tracking and Capture Considerations

Tracking and lighting are the two most important areas to focus on when shooting material that will receive this treatment. Relatively minor investments in gathering good data on set can vastly reduce the effort, and therefore the cost, required to get good results in post.

Eye Enhancements

The use of contacts lenses to change the color of an actor’s eyes or to completely change the look of the eyes is a common makeup approach. However, with the advent of advanced digital tracking, productions may choose to do these as visual effects to avoid issues for the actors, especially the full-eye contacts that can be troublesome to wear. A color change requires isolation of the area and a selection of the existing iris color. A full eye replacement is more complicated and requires tracking a 3D CG eye and rotoscoping for the eyelids.

3D Techniques

As in any 3D work, the camera position(s) must be known, and the usual set survey and matchmoving techniques apply here.

All 3D enhancement techniques require deriving a model of the surface in question, which could be deforming during the shot, as a first step. If the subject to be tracked does not already possess satisfactory trackable detail, additional tracking marks should be added sufficient to the task. For enhancements that are fixed to the skull, common tracking spots are the cheekbones, forehead, nose, and ears. Deforming surfaces such as the area around an actor’s mouth are especially difficult and require additional marks. A minimum of three reference points always need to be visible to provide a 3D plane to work with. Due to the complexity of body parts and deformations, multiple points may be required to lock on with the accuracy required.

If tracking markers are not available, then the shot may require tracking of any specific natural features as well as a certain amount of manual tracking. Having a 3D scan of the real body part certainly makes this easier.

Additional views of the subject from one or more witness cameras²⁴ can be immensely helpful in improving the accuracy of the solution by further constraining the position of points through triangulation. A sync relationship between the taking camera and the witness cameras must be known, and the witness camera footage is easiest to use if the frame rates and shutter open times are synchronized. This is not a strict requirement since frame rate and shutter phase adjustments can be made using optical flow retiming tools if necessary, and unsynched witness cameras are better than none at all.

Because these CG enhancements are often blended with the original photography, the lighting match must be as accurate as possible. Image-based lighting techniques are useful here because of their ability to faithfully reproduce complex light fields with relatively little effort. Good texture reference of the skin to be matched is also very helpful. Images that show diffuse color should include a gray card for calibration. Images that show the specular reflectivity and bump level will help when it comes time to texture the CG asset.

2D Techniques

Cosmetic Wrinkle and Blemish Removal

Cosmetic skin cleanup, such as the removal of acne blemishes, wrinkles, or tattoos, while usually faster and somewhat less expensive to handle on set with makeup, has become a common part of post-production. These removals generally do not require special on-set consideration. They are done in the 2D realm and are handled much the same way as other wire and rig removals:

•   An offset brush clones nearby texture to hide the artifact.

•   A matte is produced of the offending area, and a procedural fill tool blends across it.

•   A still-painted patch is made and tracked over the offending area.

One company that specializes in this type of work is Lola Visual Effects where they have developed special wrinkle removal tools.

2.5D Techniques

Optical flow techniques can be used to make painted artwork track to a performer even when the performer is moving in complex ways, such as while an actor is talking, for example.

Silhouette Changes

Geometric distortion tools can be used to alter the silhouette of a performer or change the proportions of different parts of a body or face. These techniques have been used for years in advertising with still photos and are now being applied to running footage for similar reasons. Recent commercials have featured enlarged heads on bodies as well as enlarged eyes where only a portion of the original image is scaled. In Star Wars Episode One: The Phantom Menace (1999), a key shot is when the Sith kicks one of the Jedi off a platform. Because this stunt kick did not make physical contact, the image of the foot was distorted and manipulated to make the visual connection it required.

Re-Projection

3D techniques can help to provide stable coordinate systems for animation. For example, a 3D model of a performer can be matchmoved to follow a performance, the photographed plate can then be projected onto the matched model, and shape changes can be applied to the model. This technique produces results that have a closer match to the correct perspective than a simple 2D method and create a more convincing result.

3D PRODUCTS, SYSTEMS, AND SOFTWARE

Richard Kidd

Digital Element Creation Process

In the early days of digital visual effects, most computer programs used were proprietary software that had been developed in house. Although visual effects studios still use proprietary software for specialized tasks, most of these studios also use off-the-shelf software for general 3D work, including modeling, rigging, texturing, animation, lighting, and rendering. To see how these tasks fall into place in the post-production workflow, see Figure 7.51. The process and workflows mentioned in the following paragraphs are also discussed in greater detail in other chapters of this handbook. Please consult the contents for the specific chapters.

A 3D computer model is a mathematical representation of any 3D object. The 3D software used varies from studio to studio, depending on their operating system platform and pipeline. The major 3D graphics software packages used in visual effects are discussed in the next subsection.

Rigging is the process that replicates a skeletal system for the purposes of animating. Rigging is used primarily with 3D character models but may be used in noncharacter models.

Figure 7.50 An overview of the digital production workflow. (Image courtesy of Richard Kidd.)

While the rigging process is under way, textures are created in a 2D paint and editing application, such as Adobe Photoshop. These textures are then applied to the 3D model within the 3D graphics software using UV coordinates or projection methods.

The next process is to animate the 3D computer model if necessary. Animation describes how a 3D object moves through space and/or deforms over a period of time. Various methods can be used to animate a 3D computer model including keyframing, inverse or forward kinematics, motion capture, and simulations. Several of these techniques can be used in combination with one another.

Lighting in 3D graphics software follows the same principles of real-world lighting. It helps to emphasize or highlight objects in a scene or may be used to push them back with the use of shadows created by the lights. Likewise, lights can be used to convey a certain mood or evoke an emotion or intended response from the viewer.

Once the lighting process is complete, the 3D objects that make up a scene are rendered. Rendering refers to generating a 2D image from the 3D objects that have been created within the 3D graphics software. The rendering, or drawing of the 2D image, is done with a renderer that is included with the 3D graphics software, like Maya’s Software Renderer, or through a stand-alone renderer like Pixar’s Renderman. When creating the 2D image the renderer takes into account the textures, lights, and relative position to other objects. Rendering is usually done in layers, also called passes, in order to be manipulated in the compositing stage.

3D Graphics Software

As the visual effects industry has grown and matured, so have the 3D packages in use today. The current versions of 3D software are the result of many revisions and updates and in some cases of mergers and acquisitions of companies that blended together multiple 3D packages. The packages discussed next are the ones used most often in today’s 3D workflow.

Houdini

Houdini, version 10 at the time of this writing (early 2010), is a 3D graphics software package with a node-based workflow for modeling, texturing, animation, character rigging, lighting, and particles. This graphics software also includes the ability to perform dynamics simulations such as fluid, rigid-body, wire, and cloth simulations. Rendering is supported through the internal renderer, Mantra, as well as support for third-party rendering. Houdini also provides built-in compositing and Python scripting support.

Although used in all applications of 3D, including character animation, Houdini is best known for its procedural systems. It excels at procedural-based modeling, particle systems, and dynamics. It also gives free unlimited Renderman-compliant rendering through its Mantra renderer.

Houdini is available in noncommercial and commercial versions. The noncommercial versions are Houdini Apprentice and Houdini Apprentice HD. Both of these noncommercial versions have all of the features available in the full commercial version but with minor limitations including a noncommercial file format and maximum resolution limitations. Houdini Apprentice has a max render resolution of 720 × 576. Meanwhile Houdini Apprentice HD has a max render resolution of 1920 × 1080. Additionally, Houdini Apprentice has a small watermark while Houdini Apprentice HD does not.

The commercial versions of Houdini are Escape and Master. Houdini Escape features modeling, animation, character, lighting, rendering, compositing, scripting, and third-party rendering and has no limitation on rendering resolution. Houdini Master includes all of the features found in Houdini Escape but also includes particles, fluid dynamics, rigid-body dynamics, cloth dynamics, and wire dynamics and has no limitation on rendering resolution.

Maya

Autodesk Maya started off as a combination of three different 3D software packages used in the early 1990s. It is now considered one of the top 3D modeling, animation, and rendering packages used in the digital industry today.

At the time of this writing, the current version of the 3D software package is Autodesk Maya 2010 (which coincidently is the first release to unify the Maya Complete and Maya Unlimited versions from previous years). Maya 2010 includes traditional modeling, rigging, animation, texturing, lighting, and rendering modules. It also includes the Maya Nucleus dynamic simulation framework, which includes two fully integrated Nucleus modules: Maya nCloth and Maya nParticles. In addition, the graphics software features Maya Fluid Effects, Maya Hair, and Maya Fur. This version also brings Maya Composite, advanced matchmoving capabilities, five additional Mental Ray for Maya batch-rendering nodes, and a software network render queue manager.

Maya’s strengths lie in its advanced modeling tools and customization. The intuitive polygon workflow includes a host of selection and editing tools. Furthermore, the ability to transfer polygon attributes from one mesh to another, regardless of their separation in space or size differences, gives computer modelers extra flexibility. Another reason Maya is widely used is its ability to be customized. This has made it easier for companies to tailor the software for their specific needs and internal proprietary pipeline.

Softimage

Softimage Creative Environment was developed by Softimage Corporation and released for Silicon Graphics workstations at SIGGRAPH in 1988. At that time it was fast and highly regarded for its animation capabilities and was first used for advanced character animation.

In the past decade Softimage was being developed by Softimage, Co., a subsidiary of Avid technology, and was known as Softimage|XSI. At the end of 2008 Autodesk acquired the Softimage brand and 3D animation assets from Avid. Soon after, Softimage|XSI was rebranded as Autodesk Softimage.

Softimage is currently available as Autodesk Softimage 2010. This 3D software package includes modules for 3D modeling, animation, rendering, and compositing. Softimage made its mark as an animation tool, and today still offers very strong animation capabilities. Animators have access to the following tools: Animation Mixer, Animation Layering, Audio Syncing tools, F-Curve editor, Dopesheet, Shape Manager, and Motor. The animation layering, for example, allows animators to interactively, and nondestructively, add key frames on top of existing key frames. Furthermore, Softimage’s F-Curve editor is a powerful tool that allows animators to edit high-density curves from motion capture data using a simplified sculpting tool. The Face Robot toolset, a solution for rigging and animating 3D facial expressions quickly and easily, is also now available.

Softimage is also available as Autodesk Softimage Advanced 2010. This version has all of the features mentioned above but also includes five Softimage batch-rendering licenses and crowd simulation software called Softimage Behavior.

Autodesk Softimage incorporates a node-based workflow called ICE (Interactive Creative Environment), which is an open platform that allows the artist to extend the capabilities of Softimage quickly.

3D Studio Max

3D Studio Max is a full-fledged 3D graphics program that includes tools for 3D modeling, texturing, rigging, animation, and rendering. Historically, 3D Studio Max has been used extensively for video game development. It has a long history of being developed for the Microsoft Windows platform and is currently available only for Windows operating systems. It is now known as Autodesk 3ds Max, and the current version is Autodesk 3ds Max 2010. This 3D package is also used for television, film, web, and multimedia work.

New features in Autodesk 3ds Max 2010 include PFlow Advanced, a Graphite Modeling toolset, Containers, and xView Mesh Analyzer. The PFlow Advanced feature is a particle flow toolkit that builds additional functionality into the previous version. The Graphite Modeling toolset allows modelers to create models quickly with 100 new advanced polygonal modeling and freeform design tools. Containers allow users to collect multiple objects in a complex scene into a single container to be treated as a single element. Another new feature, xView Mesh Analyzer, allows users to view meshes for common errors such as overlapping UVs, duplicate faces, and isolated vertices.

3D Tracking

In the early days of visual effects, most 3D camera tracking was done by eye using manual techniques. As time went on tracking software applications were developed in house at various visual effects facilities. As digital visual effects grew, commercial applications began to be sold. 3D Equalizer was one of the early tracking packages to be widely used in visual effects. Additionally MatchMover Pro was extensively used but was acquired by Autodesk and is now a part of Maya 2010. Today the primary tracking packages are Syntheyes, Boujou, and PFTrack.

Syntheyes

Syntheyes is matchmoving software, developed by Russ Andersson, that made its commercial debut in late 2003. It offered features such as object tracking, lens distortion handling, and light solving. Its early success came not only because of its production-ready toolset but because its price was significantly lower than that of the competition and accessible to freelancers and small and large visual effects facilities.

Since its initial release Syntheyes has had a number of major re-releases, each time adding novel features such as mesh building (2005), special greenscreen analysis (2006), 3D stabilization (2007), and camera holds (2008).

One of Syntheyes’ strengths is that it has always been a technical leader. It has been available on 64-bit platforms starting since 2005. Furthermore, multithread and core support has been incorporated in Syntheyes for years as an aid to performance, as has the utilization of CPU features such as MMX and SSE2. The main advantage to moving to full 64-bit operation is the ability to address more than 4 GB of RAM. This enables the artist to store longer HD and film shots with the ability to see tracking data over the clip playing in RAM-based playback.

Syntheyes allows users to create their own customized scripts for exporting that allow the user to export to approximately 25 different industry applications.

Boujou

The primary product of the company, 2d3 Ltd., Boujou, is used by many film and video post-production companies to generate 3D tracking data used in the creation of visual effects.

Boujou’s process involves calibrating the camera’s internal parameters, such as focal length and film back dimensions, as well as its motion in 3D space. The algorithm that drives Boujou looks at hundreds, if not thousands, of distinctive points that appear in areas of high contrast or high texture. Tracking the motion of these feature points in two dimensions through a sequence of images allows their 2D trajectories to be computed.

The resulting structure, or point cloud, generated from the tracked features enables the 3D movement of the camera to be inferred. The quality of the camera solution is dependent on the quality of the 2D feature tracks it uses in the calibration process. One should also note that the points that are tracked within the image sequence are identified through intensity and not color. Therefore, this algorithm applies to sequences of color, monochrome, and thermal imagery.

PFTrack

PFTrack, developed by the Pixel Farm, is a comprehensive image analysis tool with both camera reconstruction and object tracking. PFTrack, in addition to camera tracking, may also be used for image-based modeling and automatic Z-depth extraction. PFTrack version 5 is the ninth major version and now supports stereo cameras.

Reference frames may also be loaded to assist with tracking difficult shots. Camera focal length information may be imported via a simple XML data format, and depth and optical flow data can be exported as OpenEXR image sequences.

Other GUI improvements include better access to feature tracking buttons, additional feature-track filtering tools, and improvements to the F-Curve editing tools. Integrated Z-depth extraction tools assign depth data to sequences for exporting as a gray map. The program is now fully 64 bit on all platforms including PC, Linux, Mac-Intel, and PowerPC.

Special Effects

Within the 3D workflow it is sometimes necessary to create certain 3D effects that require specialized effects software products. These include Massive, Mental Ray, Photorealistic Renderman, Photoshop, BodyPaint 3D, and Enzo 3D Paint, among others.

Massive

Massive is a stand-alone 3D package specifically developed to generate crowd simulations. It was originally developed by Stephen Regelous in Wellington, New Zealand, to be used on Peter Jackson’s Lord of the Rings trilogy (2001-2003).

Massive Prime is used for authoring, directing, and rendering custom autonomous agents for animation and visual effects. It is made up of a node-based interface and allows artists to interactively create AI-enabled agents. An agent is simply a 3D character brought into Massive, which gives a performance based on control key frames or motion capture clips, called actions. It includes tools for agent authoring, scene setup, agent control and direction, agent technology, rendering, and integration. The agent authoring is done through a brain editor, which is a nodebased interface that allows an artist to interactively create an AI-enabled agent without programming.

Massive Prime also integrates with Maya by allowing an artist to import skeletons and motion. It has the ability to import XSI skeletons and motion and be customized through code using the C/C++ API as well as Python scripting.

Massive Jet is a streamlined version of Massive Prime. Massive Jet is designed to make AI-driven animation available to less technical animation and visual effects professionals.

Additionally, prebuilt ready-to-run agents are available for purchase. These agents have performances built into their AI and simply need to be added to the scene and tweaked accordingly. Available agents include Locomotion Agent, Stadium Agent, Ambient Agent, Mayhem Agent, and Combat Sword Agent.

Massive Software provides an educational package, called Learning Massive Software, which is an education bundle that includes 10 licenses of Massive Prime; 12 months of upgrades and support; the educational versions of Ambient, Locomotion, and Stadium Agents; and comprehensive courseware.

Rendering

Once the modeling, animation, texturing, and lighting are done, a 2D image is generated from the 3D scene. This process is called rendering. The rendering process is performed by a renderer, which is either a module in the 3D graphics program or a standalone application designed to generate the 2D images. The renderer calculates a 2D image using a certain type of algorithm, or procedure (i.e., ray tracing, radiosity, etc.). Two of the most widely used renderers in the digital industry are discussed next.

Mental Ray

Mental Ray is a production-quality rendering application developed by Mental Images in Berlin, Germany, and is available both as an integrated render engine for Maya and SoftImage, as well as a stand-alone render engine.

As the name implies, it supports ray tracing to generate images. Ray tracing mimics the natural flow of light rays. This is done by tracing the path of the light ray from the camera back to the light source. Mental Ray’s feature set is comparable to that of Photorealistic Renderman. Global illumination was a feature found in Mental Ray long before it was implemented in Renderman. Global illumination takes into account all illumination, both indirect and direct, that occurs in a 3D scene. Nvidia acquired Mental Images in December 2007.

Some notable films that have used Mental Ray are Hulk (2003), The Day After Tomorrow (2004), and Poseidon (2006).

Photorealistic Renderman

Photorealistic Renderman, or PRMan for short, is a proprietary photorealistic renderer that primarily uses the REYES algorithm but is also capable of doing ray tracing and global illumination. Loren Carpenter and Robert Cook at Lucasfilm’s Computer Graphics Research Group developed the REYES rendering algorithm in the mid-1980s. It was first used in 1982 to render images for the Genesis effect sequence in the feature film Star Trek II: The Wrath of Khan (1982). REYES is an acronym that stands for Renders Everything You Ever Saw. This algorithm was designed with a number to handle complex and diverse models, handle complex shaders, make use of minimal ray tracing so as to have fast speeds, and have unsurpassed image quality and flexibility for customization.

REYES achieves several effects that are deemed necessary for feature-film-quality rendering: smooth curved surfaces, surface texturing, motion blur, and depth of field.

PRMan is used to render all of Pixar Animation Studios’ in-house 3D animated movies. It is available as a commercial product as part of a bundle called Renderman Pro Server. In addition, Renderman for Maya is a full version of PRMan that is designed to be completely integrated with Maya 3D graphics software.

Texturing

In 3D workflow, it is necessary to apply textures to the 3D models in order to render a production-ready asset. This texture application is done in the 3D package. However, before a texture can be applied, it must be created. The primary program used in the creation of textures is Adobe Photoshop. Two other specialty pieces of software that allow for the creation of textures and their application in one step are BodyPaint 3D and Enzo 3D Paint.

Photoshop

Adobe Photoshop is 2D image-editing software that is used throughout various stages of post-production and was developed by John and Thomas Knoll.

Artists can use Adobe Photoshop to create 2D imagery from scratch using a combination of paintbrushes and filters. Artists can also create imagery by editing existing images or photographs to create textures that will be applied to the surface of a 3D computer model within the 3D graphics software. The 3D computer models are then lit and rendered.

BodyPaint 3D

Maxon’s BodyPaint 3D software is texture creation software. It differs from other texture creation software because it allows artists to paint detailed textures directly onto 3D computer models. Similar to Adobe Photoshop, it has a workflow that contains layers, filters, and tablet support. With the included renderer, called RayBrush, artists can view the result of their texture painting in real time as a rendered image.

In addition, this software package has traditional UV editing tools normally found inside 3D software packages. For example, there are three automatic UV unwrapping algorithms to help in painting across seams. The software includes free plugins to exchange models and textures with popular 3D software packages.

Enzo 3D Paint

Enzo 3D Paint, by Electric Iris, is a paint system that is fully integrated into Adobe Photoshop. Enzo is not a stand-alone software package, but rather a set of plug-ins that turns Adobe Photoshop into a 3D paint program. Similar to BodyPaint 3D, artists can move around in a 3D viewer and paint textures directly on a 3D computer model. The user can also paint a flat traditional 2D texture if they choose. The workflow allows for 2D and 3D layers.

¹ B-spline: a generalization of a Bézier curve.

² Tessellation or tiling of the plane: a collection of plane figures that fills the plane with no overlaps and no gaps. One may also speak of tessellations of the parts of a plane or of other surfaces. Generalizations to higher dimensions are also possible.

³ Haptic arm: a force-feedback system. One manufacturer is Sensable Technologies (www.sensable.com).

⁴ OpenGL (Open Graphics Library): a standard specification defining a cross-language, cross-platform API for writing applications that produce 2D and 3D computer graphics.

⁵ Orthographic: representing a 3D object in two dimensions.

⁶ This is especially true in the case when applying regular shadow maps instead of deep shadow maps and when using ray-trace renderers.

⁷ An anisotropic surface changes in appearance as it is rotated about its geometric normal. An example is brushed aluminum.

⁸ See Bruderlin (2004), Goldman (1997), Kajiya and Kay (1989), Marschner, Jenson, Cammarano, Worley, and Hanrahan (2003), Perlin (1989) and Ward, et al. (2007) at the end of this section for more detail.

⁹ Non-Uniform Rational B-Spline.

¹⁰ For examples and further discussion, see the General Issues and Solutions subsection that follows.

¹¹ See www.autodesk.com.

¹² See www.joealter.com.

¹³ See www.pixar.com/shorts/ftb/index.html.

¹⁴ See www.olm.co.jp/en/olm/2008/05/one_pair.html.

¹⁵ An anisotropic surface changes in appearance as it is rotated about its geometric normal. An example is brushed aluminum.

¹⁶ Sprite: a 2D/3D image or animation that is integrated into a larger scene. In 3D graphics, it is a technique whereby flat images are seamlessly integrated into complicated 3D scenes.

¹⁷ The camera, or view frustrum, is the pyramid-shaped volume defined by the camera at its apex. The edges are defined by points running from the eye through the four corners of the visible edges of the visible area. Objects inside the volume are visible to the camera or viewer, while objects outside are not. Since it is easily defined mathematically, the view frustrum is often used to cull out unseen objects in order to improve render efficiency.

¹⁸ Gamma: a log compensation to correct the viewing of linear data (an image file) on a nonlinear viewing device (a monitor).

¹⁹ The values of the colors subtracted from each other: 125 − 65 = 60.

²⁰ The term pulling a matte refers to the act of generating a matte from a green or blue screen. This is typically done using a keyer, either a hardware device (as used in live television for placing the maps behind the weatherperson) or an effect in a composite that would generate that matte. Keyers also typically provide services to reduce green/blue spill.

²¹ Please refer to the Massive Software website for more information (www.massivesoftware.com).

²² Keyframe animation utilizes a key frame, that is, a drawing that defines the starting and ending points of any smooth transition. They are called frames because their position in time is measured in frames on a strip of film. A sequence of key frames defines which movement the spectator will see, whereas the position of the key frames on the film, video, or animation defines the timing of the movement. Because only two or three key frames over the span of a second do not create the illusion of movement, the remaining frames are filled with inbetweens (Wikipedia, http://en.wikipedia.org/wiki/Key_frame).

²³ An animation tool from NaturalMotion Ltd.

²⁴ Witness camera: video camera used to record the same scene from a slightly different position or angle. See more details in the On-Set Animation Capture: Witness Cam section in Chapter 3.