In the previous chapter, we took our body motion capture to the next level by learning how to make adjustments to the animation and render video using the Level Sequencer. The Level Sequencer plays an integral part when it comes to managing animation and we will use it again here as we explore facial motion capture.
In this chapter, we are specifically looking at facial motion capture for our MetaHuman, taking advantage of the iPhone depth camera and facial tracking features. MetaHumans come with dozens of intricate facial controls that respond to facial capture tools, giving you the power to create incredibly realistic facial performances including speech. This can capture your own performance or an actor’s performance to such a degree that even very subtle nuances are reflected, thus making the overall result incredibly realistic.
So, in this chapter, we will cover the following topics:
In terms of computer power, you will need the technical requirements detailed in Chapter 1 and the MetaHuman plus the Unreal Engine mannequin that we imported into UE5 in Chapter 2.
You will also need an iPhone and a stable Wi-Fi connection to install plugins.
Let’s look at that iPhone requirement a little more. You will need to have an iPhone X or newer, or an iPad Pro (3rd generation) or newer. This is because (unlike other phones, such as Android ones) they have a built-in depth camera, which is necessary for motion capture.
You will also need a long cable – approximately 2 meters – because it is best to have your iPhone on charge during motion capture sessions as the process is quite battery consuming.
The Live Link Face app is a free app that you can download onto your iPhone or iPad directly from the App Store. Its sole purpose is to utilize the iPhone’s built-in depth camera, along with facial recognition and pixel tracking technology, to produce data that works within ARKit. In addition to collecting facial motion capture data in real time, it also transmits the data via Internet Protocol (IP) to your desktop computer using a Wi-Fi connection; this data is then picked up by the Live Link utility in Unreal Engine, which we shall delve into later.
Figure 8.1 shows the Live Link Face app in the Apple App Store:
Figure 8.1: The Live Link Face app in the App Store
Once downloaded and installed, open the app to test that it is working. While looking at the display, you should immediately see your face. By default, the mesh overlay is enabled as seen in Figure 8.2, and you’ll see that the mesh very accurately tracks your face.
Figure 8.2: Opening up the Live Link Face app
As soon as our face is being tracked, we know the app is working. There is no need to do any more with the application for the time being. In the next section, we will install a number of Unreal Engine plugins, such as Live Link, which will allow us to receive the motion capture streaming data from the iPhone.
As just mentioned, there are a number of plugins that you will need to have installed. The good news is that upon downloading and importing your first MetaHuman character, you will have inadvertently and automatically installed them without realizing it. However, you will need to make sure that the following plugins are enabled:
To easily find the plugins, go to Edit, then Plugins, and we’ll start installing the required plugins.
To find the Live Link plugins, search for Live Link as demonstrated in Figure 8.3. You can see that I have enabled the following plugins, which are indicated by the checkboxes and the blue ticks:
Figure 8.3: Enabling the Live Link plugins
Some of these plugins are essential for using the app and I will now briefly outline their importance.
Live Link is the core plugin that allows third-party applications to stream animation data directly into Unreal Engine. Applications such as MotionBuilder and Maya permit users to make changes to these applications and see the results updated in the Unreal Engine viewport in real time. This functionality in the context of this book permits users to use other applications such as the Live Link iPhone app to stream data directly into Unreal Engine.
Live Link Control Rig allows the incoming data from a third-party application to communicate with the animation controllers of a character. For instance, a MetaHuman has a control rig that allows users to manually animate it. Live Link Control Rig harnesses the same control rig protocol.
For the purpose of debugging and viewing the connection from a third-party animation data source into Unreal Engine, we will use the Live Link Curve Debug UI plugin.
Note
Live Link Curve Debug UI, while not essential, is a convenient interface for troubleshooting issues when it comes to advanced refinement; however, discussing it in detail is beyond the scope of this book as it looks at every single control point of the face.
Next, we need ARKit and ARKit Face Support. ARKit is an AR solution designed to work exclusively on an Apple iOS camera device. It ensures that Unreal is able to understand the incoming data. ARKit Face Support connects specifically to the face tracking solution via depth and RGB cameras on the iPhone.
So, in the Plugins tab, run a search, but this time for ARKit. You will see a list of the available ARKit plugins that you need to enable:
Figure 8.4: Enabling the Unreal Engine ARKit plugins
We have one more plugin to enable, which is Take Recorder. Try to think of Take Recorder as a video camera. Effectively, it records the motion capture data received by Live Link and stores it for playback, which in turn is imported into the Level Sequencer. Just like a video camera, we can record multiple takes until we are happy that we have got the best one (or at the very least, as close as possible to the best!).
To enable Take Recorder, search for the plugin as per Figure 8.5:
Figure 8.5: Enabling the Unreal Engine Take Recorder plugin
Note
You may need to restart Unreal Engine if you have enabled any of these plugins for them to take effect.
With all of the necessary plugins enabled, we can now move on to the next section where we will configure the Live Link Face app so that it can send data into Unreal Engine.
With the Live Link Face app installed on the iPhone and the Live Link plugin enabled in Unreal Engine, we now need to get them to talk to each other (figuratively speaking, of course). Our goal is to send motion capture data from the iPhone to the PC and/or the Live Link app to send this data into Unreal Engine.
To do this, we first need to configure the app by finding our computer’s IP address:
192.168.100.155
Make a note of that number, which will be unique to your machine.
Figure 8.6: Clicking on the settings icon
Figure 8.7: Clicking on Live Link iPhone
Figure 8.8: Clicking on the IP target
Once you have set your IP address in the app, you need to switch over to Unreal Engine again to see whether you are receiving a signal.
Figure 8.9: The Live Link interface
In Figure 8.9, you can see that Live Link has automatically picked up a signal. It recognizes that the type of signal is Apple AR Face Tracking and shows the name of the subject that the signal is coming from, in my case, iPhone. As it happens, iPhone is just the name that I gave to the phone after I purchased it. Whatever you have called your phone, that name will appear under the subject name.
Also in Figure 8.9, next to where it says iPhone, you’ll see that under Role it says Basic and to the right of that is a yellow dot. This dot indicates the status of the signal, which can be interpreted as follows:
Knowing that both apps are effectively speaking to each other means that we can go on to the next step and configure our MetaHuman to receive data.
We must now tell the MetaHuman Blueprint to receive data from Live Link. As it happens, this is very simple:
Figure 8.10: Creating a Level Sequencer
Figure 8.11: Enabling the Llink Face Head option
Next, click on Face within the Components tab. Then, in the Details panel, go to Animation, as seen in Figure 8.12:
Figure 8.12: Changing Animation Mode and Anim Class
With these changes made to the blueprint, it’s time to test the mocap data as it streams from the Live Link Face app and into your MetaHuman Blueprint. Before testing, make sure you go through the following checklist:
If you have followed the checklist, the next thing you need to do is to run a simulation by hitting Alt + P. At this point, you should see a live facial capture of your character in the viewport.
If you have continued on from the previous chapter and have the Level Sequencer open, you will see the new facial animation on top of your character’s body as acquired through DeepMotion Animation 3D. Note that you can even play the body animation from the Level Sequencer while previewing the Live Facial capture simultaneously.
It won’t take long to realize that the iPhone motion capture result is not perfect. However, it is a very powerful and complex tool with plenty of scope to refine the data as it comes in and refine it again after the data has been recorded.
Note
If you are experiencing problems with the facial animation, such as the head rotation working and nothing else, there was a known bug that was fixed with an update of Quixel Bridge. If you are running Unreal Engine 5.0.3 or above, ensure that you have updated the Quixel Bridge plugin to version 5.0.0 or above. You will then need to re-download and re-import your character.
In the next section, we will look at calibrating the Live Link Face app to get us a better result.
To get a better result, the ARKit capturing process needs to acquire data that takes the performer’s characteristics into account. As you can see from Figure 8.13, the app has done a good job of capturing the performer’s facial proportions, clearly outlining the eyes, nose, mouth, and jaw. However, giving the app a baseline of information (such as a neutral pose) would really help to refine the overall outcome.
Figure 8.13: Calibrating the Live Link Face app
To understand how and why baselines and neutral poses work, let’s take a look at the eyebrows as an example using an arbitrary unit of measurement. If in the resting or neutral position the eyebrows are at 0 and a frown is at –10, and a surprised expression raises the eyebrows to +15, then knowing what the neutral pose is would give the app a better idea of where to determine the frown or surprised expressions to coincide with its tracking capability.
The calibration process in the Live Link Face app is incredibly simple as it relies only on a neutral expression. In Figure 8.13, I’ve highlighted the face icon on the right of the record button – click on the face icon, hit Recalibrate, and a short countdown will commence before taking a picture of your neutral pose for calibration.
Once calibrated, it’s time to prepare for capturing a facial performance. For this, there are three main considerations:
I’ll guide you through the process of capturing the facial capture with your phone step by step:
Figure 8.14: Scalability set to Low and Unlit
Figure 8.15: Adding sources to Take Recorder
As soon as you run the simulation, the motion capture will begin, along with a 3-second countdown, similar to Figure 8.16:
Figure 8.16: The 3-second countdown
Immediately upon recording, the Blueprint track and the iPhone track will be written as a take and a reference of this will temporarily occupy the Level Sequencer to show which tracks are actively recording:
Figure 8.17: The tracks being recorded
When you are happy with your mocap session, hit the stop button on Take Recorder and stop your simulation by hitting Esc.
Figure 8.18: Importing the character Blueprint into Level Sequencer
Note that in Figure 8.18, when you bring in your character, you also bring in a rig for both the face and another for the body. However, because we are driving the character with mocap, you need to delete these rigs.
Figure 8.19: Adding an actor to the Level Sequencer
If you still have the character Blueprint from the previous chapter with the retargeted animation, you can import the body animation. In addition, if your performance requires speech, the Level Sequencer is capable of playing back audio if you wanted to test your performances inside Unreal Engine. This is useful when ensuring that the facial capture is in synchronization with the audio of the speech recorded.
Note
It is advised that you capture the body and face simultaneously because most performers lead with their heads and the head leads the body. Getting both the body and head in sync can only really be done properly when both elements are captured at the same time.
The iPhone solution can work simultaneously with DeepMotion Animate 3D. To do this, you need a facial capture rig to ensure that the iPhone’s depth camera is facing you at all times and that your hands are free. Ideally, the camera needs to be at a consistent distance at all times. Any movement of the camera separate from the head will be understood as a head movement and will lead to inaccurate data.
Companies such as MOCAP Design specialize in creating helmets that work with iPhones. You can see some of their iPhone and GoPro camera solutions at https://www.mocapdesign.com/m2-headcam.
This solution may be expensive to most so, alternatively, a DIY solution or 3D printed solution may be more cost-effective. Additionally, there are now solutions for live streamers to keep their phone cameras at a consistent distance. While not perfect, they are certainly better than holding a phone in one hand or on a tripod.
In this chapter, we have explored the iPhone’s facial motion capture solution for MetaHumans in Unreal Engine. We used the Live Link app, along with Live Link and ARKit plugins, to capture live data. Additionally, we got to use the Take Recorder plugin in conjunction with the Level Sequencer as a way to record and manage our mocap.
In the next chapter, we will continue our facial motion capture but on a much more professional level, using Faceware.