Animation Pipeline

The animation pipeline provides an end-to-end pipeline including facial animation, body animation, and rendering.

The animation pipeline provides an end-to-end pipeline including facial animation, body animation, and rendering.

Welcome to the documentation of the animation pipeline workflow! This is an end-to-end guide for understanding and utilizing this cloud-based avatar animation streaming solution. The animation pipeline, comprised of multiple microservices, empowers users to control interactive avatar animations in real-time, and to stream an avatar video over the network.

It leverages the Avatar Configurator to create a custom avatar, Audio2Face to animate facial expressions and lip movements, and offers a collection of APIs to control body motion and position based on the Animation Graph technology to deliver immersive avatar visuals.

Animation pipeline microservice connection overview.

Animation pipeline microservice connection overview.

Key Features

  • Avatar Customization: Create a custom avatar by customizing accent colors and a selection of outfit variants that best match your use case.

  • Real-Time Streaming: Our real-time animation pipeline is suited for live interactions. Unlike offline video creation tools, our solution is optimized for instantaneous reactions and responsiveness.

  • Speech-Driven Animation: By analyzing speech audio inputs, the animation pipeline animates the avatar’s facial expressions and lips accordingly.

  • Command-Based Customization: Users can send commands to the animation pipeline to trigger a variety of actions, including body postures, gestures, facial expressions, and avatar positioning, enabling extensive customization and interactivity.

  • RTP Video Stream Output: The output of the animation pipeline is delivered as an RTP video stream, allowing seamless integration with diverse platforms and applications, such as web browsers or video conferencing systems.

Use Cases

  • Voice-Based Systems Enhancement: Integrate the animation pipeline with Large Language Models (LLMs) or dialog managers to overlay animated avatars onto voice-based systems, enriching user interactions with visual feedback.

  • Interactive Video Streaming: Enable users to puppeteer avatars using voice commands and gestures in video streaming contexts.

  • Virtual Meeting Augmentation: Combine the animation pipeline with a digital assistants and a video conferencing systems to introduce animated avatars as additional participants in meetings, enhancing communication and collaboration.

This documentation provides a comprehensive guide to setting up and utilizing the animation pipeline in Docker or Kubernetes on Ubuntu 22.04. At the end of this guide, you will be able to create a custom avatar, render an avatar in a scene and stream the resulting video over RTP. You will also be able to programmatically send speech audio and gesture commands for the avatar the speak and perform.

An animated screen capture of the animation pipeline. An avatar is rendered, and performs gestures controlled by API calls made in a terminal.

Animated GIF illustrating the rendered output of the animation pipeline. The avatar performs various gestures and facial expressions controlled by API calls made in the terminal.

First, make sure to go through the steps in the Development Setup guide to ensure your system meets the requirements to run ACE workflows.

The next steps depend on what environment you want to run the animation pipeline: