Facial animation is the magic that brings inanimate characters to life. It involves animating various parts of the face to create facial expressions. Facial animation is used extensively in the animation industry. It is the technique that enables intricate emotions to be conveyed by characters and dialogues to be synced with their lips and mouths. Without it, the feature animation industry wouldn’t be able to create as extensive stories and detailed character interactions as live-action films.
How is Facial Animation Created?
What is known as facial animation is primarily referred to the computer techniques used to create facial expressions for animated characters. To create 3D facial animation, characters’ faces should have different animatable parts as well. In a process called rigging, eyes, lips, eyebrows, chins, and any facial parts are given bones that act as manipulation handles to mold and deform them. These handles allow animators to shape these parts into desired poses and animate them over time.
Facial animation for 2D characters is created slightly differently. In 2D, mouth or eye shapes are not usually manipulated with handles in real time. 2D artists draw multiple mouth or eye shapes. Different mouth shapes are shown for the corresponding sounds in voice overs. Eyes are also replaced in animation to match the characters’ emotions.
What Tools Are Used to Create Facial Animation?
Facial animation is often created using 3D or 2D animation software. For 3D characters, Maya, Blender, 3ds Max, and Cinema 4D are the primary software used to animate faces. These programs are full-featured 3D animation tools that allow artists to create characters through a process called modeling and apply bones to them in a process we mentioned earlier as rigging.
2D facial animation is created using various 2D animation and motion graphics software like Toon Boom Harmony, Adobe Animate, and Adobe After Effects. Creating 2D lip syncing is relatively easier with these programs. Most 2D computer animations created today are cut out animations created for web or TV series that do not require highly accurate facial or mouth movements. 2D animation programs come with tools that can automate lip syncing. Although not as accurate as manually assigning mouth shapes for each corresponding sound, automated lip syncing saves animators a lot of time.
Challenges of Facial Animation
The human face is a complex part of our body. It consists of multiple bones, muscles, and joints. It even holds other important structures, such as the eyes. Deforming a 3D geometry to create facial animation is a tricky process. Understanding how facial skin and texture adapt to joint movements and how every part of the face, such as eyes and lips change accordingly is complicated.
Facial animation in 3D requires specialized knowledge of rigging, facial geometry, texturing, and of course animation. The process is so time-consuming and expensive. Even 2D facial animation requires constant replacement of dozens of mouth and eye shapes.
Blend Shapes in Facial Animation
Another method of creating facial animation is using blend shapes. The technique is relatively easier than creating facial rigs. Blend shapes are a set of predefined positions for the vertices that create a 3D object, and interpolate between these positions over time to blend the geometry to a new state. Imagine adjusting the vertices on the surface of a 3D face to make the mouth wider or shut. Blend shape tools in 3D software use sliders to interpolate between these two states over time to create animation. Blend shapes are also used in combination with complex facial rigs to expedite or facilitate the animation process.
Facial Motion Capture
Motion capture is a technology created to address the time-consuming nature of animation. Motion capture involves recording real actors’ movements by attaching multiple markers to their bodies. These markers record even subtle movements in detail. This recorded information is transferred to game or animation software to be applied to 3D characters, thus, removing the need to manually animate characters. Also known as Facial Mocap, this method is one of the primary ways VFX-heavy films like Avatar and AAA game animations are created today. Virtually any feature film that combines animated characters with live-action relies heavily on this method.
Facial Animation in Games
Due to its laborious process, facial animation is not featured extensively in games. Games often emphasize mechanical interactions more than storytelling. Thus, animation priority is given to main character movements like walk cycles, run cycles, jumps, and object interactions.
There are not many games that rely on realistic cinematics to tell stories. Even AAA games tend to skip detailed facial animations. Nevertheless, when facial animation is absolutely necessary in games, developers usually try methods that require minimum work. Using software like Reallusion’s iClone or Epic Game’s Metahuman, which automate facial animation is a common practice in game development. Blend shapes are also used to create simple facial animation for game characters.
The Role of AI in 3D Facial Animation
Recent advances in artificial intelligence have paved the way for easier creation of facial animation. 3D software developers rely on the power of artificial neural networks to mimic facial features. In a process called model training, these systems are fed a large amount of labeled animation data that enables them to learn the location and formation of vertices for different facial poses.
After being trained, these systems, which are also known as models, can apply facial animation to new characters and objects. Programs like iClone and Metahuman actually rely on these AI models to create facial animation and lip syncs for characters. The convenience of using AI models for complex tasks like facial animation highlights the potential of artificial intelligence in shaping the future of animation and creative industry.
Generative AI and 2D Facial Animation
Generative AI models like Runway and Midjourney started a hype in the content creation industry. These models were trained on massive datasets including billions of images taken from films, animations, cartoons, and virtually any type of video and image available freely on the internet. Their large volume of information and the ability to associate words with the contents of an image, allowed them to manipulate pixels in still images to create facial animation across a sequence of frames.
Although not 100% accurate, these models can create detailed and realistic facial animation that is sometimes hard to distinguish from authentic live-action videos. One problem with these types of videos is that AI models generate a new drawing for each frame, thus, the image consistency is not good sometimes. Some developers have created workarounds for this problem and have achieved better results, but further advancement seems to be needed.
Facial Animation in VR and AR
Another use case of facial animation is in the VR and AR industry. VR headsets track facial movements and expressions via built-in cameras. This information is then transmitted to other users in a virtual experience to animate the corresponding character’s face. Although very futuristic and not widely adopted, realistic VR experiences incorporate this technique to simulate facial expressions on virtual avatars that are nearly identical to the user’s real expressions.
Facial animation is also seen in Augmented Reality (AR) technology. Mobile applications like Snapchat and Instagram track facial landmarks to apply various filters and animations to faces, which is a very popular feature for young people.
Final Words
Facial animation is an evolving field that is constantly changing with new inventions and better workflows. The advancement of recording and tracking hardware, as well as software innovation has made its workflow more approachable, However, it still remains one of the most laborious and specialized tasks in animation. It is one of those animation areas where skills alone are not enough to create wonderful results. Even when skillful artists are available, the time-consuming nature of facial animation has forced innovators to look for innovations that could accelerate the workflow. For a long time, using expensive motion capture suits, cameras, and complex software have been the most reasonable way of creating high-quality facial animation. However, in recent years, with the advancements in AI and tracking tools, creating facial animation has become easier. Time will show us how these advances can lead to better workflows and allow more people to create facial animations.
