In the previous chapters, you’ve been equipped with a choice of facial motion capture methods, be it using an iPhone or a webcam with Faceware. You’ve also learned how to acquire custom body motion capture with DeepMotion. One common thread in all of this is the Level Sequencer.
In this chapter, we will come back to the Level Sequencer to manage our mocap animation data. You will learn how to manage both body motion capture and facial motion capture, fix timing issues, and merge takes.
In addition, we will look at some of the more advanced industry-standard rendering techniques and explore additional items that we can use with the Level Sequencer.
So, in this chapter, we will cover the following topics:
In terms of computer power, you will need the technical requirements detailed in Chapter 1 and the MetaHuman plus the Unreal Engine Mannequin that we imported into UE5 in Chapter 2.
If you have followed along with all of the chapters, you will also have Mixamo and DeepMotion data from Chapter 5 and Chapter 6, respectively. We will be using that data here.
In this section, you will once again add your MetaHuman to the Level Sequencer so that you can apply both body capture data and facial capture data to it.
To start, first, create a new Level Sequencer by right-clicking anywhere within your content browser, finding Animation in the menu, and clicking Level Sequencer.
Once your new Level Sequencer has been created, click on +Track and add your MetaHuman Blueprint as per Figure 10.1 (if you don’t see it, use the search function and type in BP for Blueprint):
Figure 10.1: Adding the Blueprint to the Level Sequencer
As soon as you have your MetaHuman Blueprint in your Sequence, you’ll notice that both the Body and Face control rigs are present, which is standard when you add a MetaHuman Blueprint to a Level Sequencer:
Figure 10.2: The Body and Face control rigs
For the moment, we don’t need the control rigs because we’re just learning how to manage our animation data, which we will add in the very next section. So, delete both rigs highlighted in the Level Sequencer in Figure 10.2.
With the Level Sequencer still open, we are now ready to add our body mocap data.
If you have followed along with all of the chapters, you will have created body motion capture files with Mixamo and DeepMotion
To add body animation, just click on +Track, then go to Animation, as shown in Figure 10.3:
Figure 10.3: Adding an animation
If you don’t see your animation in the list, use the search function. Remember, any track you are adding will have to have been retargeted, so you can always use the Retargeted suffix when running a search. That way, you’ll be able to see all the available mocap in your project.
In Figure 10.4, you can see that I’ve added a track. You can even see that it has the track title, which can confirm at a glance that you are using the correct mocap track:
Figure 10.4: An added track
You completed this step at the end of Chapter 8, but in the next section, we will look at ways to bring in additional mocap tracks for the body.
There are two ways to bring in additional mocap tracks:
Let’s take a look at them now.
To add an animation clip as a track in the Level Sequencer, click the + icon next to Body. This will create an additional animation track that you can reposition along the timeline and blend with the first track using the weight tool using keyframes.
In Figure 10.5, you can see that I now have two animation tracks:
Figure 10.5: Adding another track
Adding animation to an existing track doesn’t create an additional track but adds new animation to the same track. Rather than clicking on the +Track icon beside Body, click on +Animation under Body instead, as you can see in Figure 10.6:
Figure 10.6: Merging two animations into one track
In Figure 10.6, I’ve already added my animation. Notice how there is a transition between the two tracks where I’ve drawn the red square. Effectively, I have merged one animation track into another, indicated by a descending and ascending curve.
To merge a track in this way, you must go to the end of the first track in the Level Sequencer with your playhead. If your animation is 100 frames, be sure to go to a higher frame, such as 120; otherwise, you won’t be able to merge your animation to the track. Then, you must drag the second clip to the left so that it intersects with the first clip.
Note that it automatically blends from the first animation into the second animation on the same animation track. Effectively, this method has merged the two animations into one track. Take note of the number between the two animation (in the red box) clips that have merged. It reads as 0093. This indicates that the transition length from one animation to the next is 93 frames, which is approximately 3 seconds.
In the next section, we’ll take a look at a handy tip to help align animations.
Sometimes, when we try to merge animations, we don’t get great results. For example, if the first animation is of the character walking and the second animation is of the character running, often, the character will appear to have jumped into a new position dramatically from one frame to the next.
To make such an adjustment, we need to add a Transform track to the body in the Level Sequencer. You can do this as per Figure 10.7:
Figure 10.7: Adding a transform track
This will create a new transform track in the Level Sequencer, allowing us to create keyframes in the Location parameter. Now, we can move the character around to fix any character position issues without impacting any of the overall animation properties. Any movement of the character that we apply will only set keyframes on the new transform track, so we won’t inadvertently lose any of the original motion capture data.
For example, imagine an animation of a character standing and waving an arm. If there is an issue with the mocap data where the body rises upwards randomly but we are happy that the character is waving, we can fix the random upward movement by adding keyframes to the Location parameter. We can do this without negatively impacting the waving motion of the arms.
However, it can be a little cumbersome to see the subtle adjustments when it comes to moving the character from A to B. To get around this visibility issue, Unreal has a function where we can see the MetaHuman skeleton. This allows us to focus on the exact position of our character without being confused by the mesh.
By right-clicking on the timeline, you will see the Show Skeleton option. Clicking on this will reveal a bright pink skeleton, which helps to make subtle adjustments, as shown in Figure 10.8:
Figure 10.8: Show Skeleton
Note
When using the Show Skeleton function, both animations will be displayed with skeletons of different colors, which makes lining them up a little easier.
A common example of when to use the Transform track and the Show Skeleton functions is when you want to loop animation, such as a walk cycle.
However, if you have a character walking from position A to B, and you wish to loop the animation so that the character continues walking, we’ll come across an issue. The issue is that as soon as the character reaches position B, it will return to position A if we wish it to loop.
Using the Transform tracks, and placing keyframes on the locator, you can create a continual animation using a loop:
Note
To further edit the character’s motion, you can always bake the animation to the control rig and add Additive Section for further editing.
Moving on from the convenient tool of visualizing your skeleton, there are other options for editing the animation clip. By right-clicking anywhere in the track and going to properties, you will see that you have more options when it comes to editing your clip:
Figure 10.9: Further options
When it comes to merging animation, the most important thing is the frame rate. You may find that you are trying to blend two animations that have different speeds. The property function is a good way to tackle this problem by changing the frame rates so that they match. You can also experiment with different frame rates.
Now that we have looked at how to add mocap data related to the body, in the next section, we’ll cover the same principles of adding and merging animation tracks, but this time, we will focus on the face.
To add and merge the face animation requires you to follow the the exact same steps as you did for the body, by either adding a whole new Faceware track or by adding the animation to the end of the existing animation track and moving the tracks. Let’s take a look in more detail.
In this section, we are going to add the Faceware facial motion capture file to the Level Sequencer. If you are having trouble locating your Faceware facial capture file, which was recorded by Take Recorder in Chapter 8, just remember that Take Recorder added the _Scene suffix to the character Blueprint’s name. So, in my case, I just searched for BP+Glenda_Scene and all my takes were listed for me to choose from.
In Figure 10.10, you can see that I merged Take 6, BP_Glenda_scene_106_0, with Take 1, BP_Glenda_Scene_1_01_0:
Figure 10.10: Adding Face animation to one animation track
I have selected BP_Glenda_Scene_1_01_0. You can see that the keyframes have appeared in blue, including 0341 and +19. Take note of the + icon – it means that I have moved the clip 19 frames to the right, so it will start 19 frames later.
The process for editing facial capture animation clips is practically identical to that of the body data, so we can use the same tools and follow the same steps from the previous section.
In addition, we can bake the track into keyframes in a control rig and add an Additive Section to tweak the animation, as we covered in Chapter 8. You can see from Figure 10.11 that I’ve taken the same steps of baking the control rig, which is indicated by the triangular keyframes:
Figure 10.11: Baking the track to the control rig
A huge part of making the facial capture work with the body capture is timing. It may require just the easy step of sliding the facial mocap animation track to either the left or right; on the other hand, you may have to spend more time using the facial control rig, or even the body control rig, to make both animations match. The only way to get good at working with control rigs for the face is to experiment with them. A good practice would be to get as far as you can with the raw facial motion capture and then refine the animation using an Additive section with a control rig.
With some of the advanced animation features covered, it’s now time to delve into some advanced rendering.
I’ve said this before and I’ll say it again: Unreal Engine is a very powerful tool, and when it comes to rendering, it doesn’t disappoint. Unreal is capable of delivering outstanding results that are comparable to traditional path-tracing renderers such as Mental Ray, Arnold, V-Ray, and RenderMan. Unreal now supports path tracing, which is the foundation of how these other renderers work.
In the context of filmmaking, Unreal has real-time rendering capabilities that should be explored to unlock its full potential. The camera and lights have features that we find in 3D programs such as Maya, Houdini, and Blender that use path tracing renderers. It’s possible to export whole scenes from either of these applications with the camera and lighting settings intact and compatible with Unreal.
So, without further ado, let’s take a look at some of these great rendering features.
In Chapter 7, we used a method of creating a new camera directly from the Level Sequencer; however, in this chapter, we will create the camera without using the Level Sequencer.
In Figure 10.12, I went up to the Create Actor button to quickly add an actor to the scene. From there, I chose Cinematic, followed by Cine Camera Actor. This creates a camera in the viewport of my scene:
Figure 10.12: Quickly adding a camera to the scene
Next, go to the Level Sequencer and add the camera by clicking on the + Track button and choosing the camera you just created.
In Figure 10.13, the camera cut features the grid background, as does the viewport above. This allows us to preview any camera animation from the Level Sequencer directly in the viewport:
Figure 10.13: The camera cut being automatically created
Unlike the body and face of the MetaHuman Blueprints, the Cine Camera automatically generates a Transform function. This will allow you to change the location of the camera in the X, Y, and Z axes. You can also change the rotation of the camera in the X, Y, and Z axes, which change the Roll, Pitch, and Yaw, respectively.
As shown in Figure 10.14, you have keyframe attributes on the right-hand side. Typically, when animating the camera, you’ll need to adjust the Location setting and edit the Focus Distance setting to ensure your subject is in focus:
Figure 10.14: The camera transform
Rather than changing values in the Transform function within the Level Sequencer every time you want to set a keyframe in the Level Sequencer, you can always just pilot the camera inside the viewport, much like playing a first-person game using your keyboard and mouse.
Be sure to engage the autokey, as illustrated in Figure 10.15, which will create a keyframe every time you move the camera:
Figure 10.15: The Autokey function
In addition to just animating the camera position, you can also animate what the camera is focusing on. A very convenient tool for focus is the Debug Focus tool, illustrated in Figure 10.16. This tool gives us a clear visual indication of what is in focus. With the camera selected, tick the Draw Debug Focus Plane option; you’ll see a pink plane appear. Anything on that pink plane is going to be perfectly in focus:
Figure 10.16: Camera debug focus
You can control the focus and animate focus distance within the Level Sequencer but use the debug plane for precision focusing.
Now, with some of the more advanced camera features out of the way, it’s time to move on to lights.
For the most part, actors are added to the Level Sequencer with some attributes being animatable. It is possible to animate more attributes via the Level Sequencer, but we need to manually add them to the Level Sequencer.
Lights are also actors that we can animate. We can add a light to the Level Sequencer much the same as we added a camera earlier. By doing this, the only animatable attribute by default is the light intensity. But there are a lot of attributes for the light actor that we can add for animation purposes.
In Figure 10.17, you can see that by clicking on + Track next to the camera component in the Level Sequencer, I have access to nearly all of the attributes featured in the Detail panel for animating. Click on any of these attributes in the list for them to become an animatable component within the Level Sequencer:
Figure 10.17: Editing lights
Generally speaking, animating lights is rare, much the same as moving lights in a live-action shoot is rare, but not unusual – perhaps you want a flashing light from a police car or a lighthouse lamp spinning around, or you want to tweak a light’s intensity for when a character walks under it. I could spend a whole chapter on lighting in Unreal, if not a whole book on the subject, but we must move on to a very important feature, Post Process Volumes.
Post Process Volume is an interesting tool as it gives us massive control of the final picture. Typically, Post Process Volume would surround your entire scene and is a volume in the sense that it is an area within your scene that is affected by input from an artist.
In essence, Post Process Volume is an effect and grading tool with much of the toolset aimed at the camera. For instance, we can control flares, blooms, and chromatic aberrations, which are all very specific lens characteristics, along with film grain, change exposure, white balancing, and so on. If you look at any settings inside a modern video camera or digital cinema camera, Post Process Volume pretty much has a matching feature. Epic Games have done incredible work giving artists real-world tools when it comes to their scene for real-time rendering.
In Figure 10.18, I have utilized some of these features:
Figure 10.18: Lens effects with Post Process Volumes
Let’s look at some of the features that I used:
As the name suggests, Post Process Volume is a post-production tool. In terms of Unreal Engine, most of the difficult things such as lighting direction, shadow, skin shading, reflections, and so on are already set, but Post Process Volume adds an extra layer of effects over that. In a way, the Level Sequencer is like the camera and Post Process Volume is like Photoshop.
As a rule of thumb: it is best to get all of your lighting and camera work as close to the final look as possible, and rely on Post Process Volume for things you couldn’t achieve otherwise.
There are quite a lot of settings in Post Process Volume and unfortunately too many to cover in a book about MetaHumans. However, they will make your MetaHumans look amazing when it comes to rendering.
Note
If you would like to learn more about advanced lighting and rendering techniques, I recommend you take a look at the following video tutorial, which you can purchase from the Gnomon Workshop: https://www.thegnomonworkshop.com/tutorials/cinematic-lighting-in-unreal-engine-5.
This brings us to the next section, where we will take a close look at rendering out files from the Movie Render Queue for compositing and color grading.
When we are setting our renders from the Level Sequencer, we are given a choice of which render queue to use. To see what render queues are available to you, click on the three dots next to the clapper icon, as per Figure 10.19:
Figure 10.19: Movie Render Queue options
It is quite likely that you will only see Movie Scene Capture (Legacy), but this is somewhat dated and limited. The other option that you can see is Movie Render Queue; however, at the time of writing, it is still only in Beta, so isn’t enabled by default.
To enable it, you need to head over to the plugin section and search for Renderer. In Figure 10.20, you can see that I have already enabled both, which you should do too:
Figure 10.20: Enabling both Render Queue plugins
Now, when you click Movie Render Queue, as per Figure 10.21, you’ll see the following screen:
Figure 10.21: Movie Render Queue dialog box
The + Render button in green is for determining which sequence to render. Initializing the Movie Render Queue will assign the current Level Sequencer for rendering. However, should you have multiple Level Sequencers in your project, you can add them to the Render Queue using the + Render button. In this dialog box, you’ll see three columns:
Let’s go through each of these columns.
Job is the sequencer name that has been chosen for rendering. By default, it will be titled New Level Sequence. Keep in mind that parameters such as Range (how long the animation is) and Frame Rate are determined in the Level Sequencer.
Should you have more than one Level Sequencer in your project, they would be available under Job and you can select which Level Sequencer you want to render.
Settings refer to what render settings you want to apply to your sequence.
The Output area displays the folder directory where your images or videos will be rendered, as set in the output setting.
There are quite a lot of options to choose from when it comes to rendering settings. Ultimately, if you are already committed to rendering a movie out of Unreal using a Level Sequencer, then you are not likely to be concerned with the real-time rendering capabilities of Unreal and are planning to do more work on the rendered sequence in post-production.
The file type, be it a movie file or an image sequence, is an important setting to consider. For example, should you plan to do additional post-production such as advanced color grading, then you should opt for an image sequence setting such as Open EXR that provides higher color fidelity. If you plan to just do a test render or commit a final render directly from Unreal, then a movie file such as a Quicktime Prores would suffice.
Let’s look at the default settings, as shown in Figure 10.22:
Figure 10.22: Default settings in Movie Render Queue
There are three types of default settings:
Figure 10.23: Available settings in Movie Render Queue
When rendering the Level Sequencer to a hard drive, we have various options to choose from to improve its quality:
You can see the settings I have selected in Figure 10.24:
Figure 10.24: Settings selection
Note
The EXR file format is capable of storing multiple channels. Most image files store just three channels: Red, Green, and Blue, such as JPEG files. PNG files can store four channels: Red, Green, Blue, and an Alpha channel for transparency.
Your default Output settings will match your Level Sequencer's default settings. However, the Output column is a means to override them, allowing you to control the final resolution and frame rate.
In Figure 10.24, I have used three render settings that will render out into one single EXR image sequence as separate channels, one channel for each of the following:
You can see the result of that render in Figure 10.25. This figure represents the three extra channels that have been rendered and stored in the EXR file – the top left is Deferred Rendering, which appears normal, the top right is Reflections Only, and the bottom left is Object ID (in a compositing program, I can make further adjustments by manipulating these extra channels):
Figure 10.25: Example of rendering
The render settings in the new Movie Render Queue are very powerful and give the artist a little more control over the final images than that of a real-time render. This aligns with the results expected from traditional 3D animation productions, where compositors are given the same or more control over the 3D renders. As developments progress in Unreal, we expect that more and more 3D animation studios will adapt Unreal into their pipelines.
In the next section, we will look at some of the professional color standards that now ship with Unreal that allow MetaHumans to work within a professional studio environment.
As mentioned in the previous section, Open EXR is the professional industry standard file format used by visual effects and 3D animation studios for 3D rendering and compositing. One reason for this is that it can store a huge amount of pixel data, particularly color.
You might remember seeing .jpeg sequence [8 bit] as an option to render in. Well, that just doesn’t cut it in the professional world simply because it doesn’t represent even half of what the human eyes can see. An 8-bit image can only store 256 shades of red, green, and blue, respectively. In total, that amounts to just over 16 million colors versus the 280 trillion-plus colors in a 16-bit image.
If you take a look at Figure 10.26, where the widest area of color is represented by the ACES colorspace, you’ll see the significant difference when compared to sRGB. The JPEG format of 8-bit is only capable of working within a small colorspace such as sRGB. Therefore, utilizing the enormous data space of the EXR file format and the enormous colorspace of ACES, Unreal Engine is capable of incredibly photorealistic renders, which is why choosing Exr Sequence [16bit] is preferable:
Figure 10.26: ACES colorspace versus sRGB
Note
For further reading on the ACES colorspace, go to https://acescolorspace.com/.
In this section, we uncovered some pretty advanced color theory relating to the ACES colorspace versus sRGB and how this can help you improve your MetaHuman renders.
In this chapter, we got the opportunity to revise some key elements of bringing a MetaHuman to life by applying both body and face motion capture to the Level Sequencer. We looked at alternative ways to edit movement and fix common movement issues, such as fixing the position of a character when looping an animation. We also learned how to use Transform, Location, and Show Skeleton to help with creating longer animations using looped tracks.
After that, we covered cameras, lights, and Post-Process Volumes, reviewing some of the very powerful and professional rendering capabilities of Unreal Engine.
In the next and final chapter of this book, we will take a look at a new feature (at the time of writing) that allows us to use our likeness or someone else’s to fashion a MetaHuman around a real human!