Docker & Unreal Animation Pipeline Workflow#

Local docker workflow setup overview (1x Animation Graph microservice, 1x Unreal Renderer microservice, 1x Audio2Face-3D microservice).

In this setup you will run the microservices in docker containers, render an avatar using Unreal Renderer and view it in a browser.

The workflow configuration consists of the following components:

  • Audio2Face-3D microservice: converts speech audio into facial animation including lip syncing

  • Animation Graph microservice: manages and blendes animation states

  • Unreal Renderer microservice: renderer that visualizes the animation data using Unreal Engine (UE)

  • Unreal Engine Pixel Streaming client: allows to view the animation from a browser

  • Coturn TURN server: enables WebRTC communication between the browser and the Unreal Renderer microservice

  • Avatar scene: Collection of 3D scene and avatar model data that is saved in a local folder

Prerequisites#

Before you start make sure to have installed and checked all prerequisites in the Development Setup.

Additionally, this section assumes that the following prerequisites are met:

  • You have installed Docker

  • You have access to NVAIE, which is required to download the Animation Graph and Unreal Renderer microservices

You also need to link a Github account to EPIC games. To do so:

Finally, this workflow relies on the Unreal Renderer Microservice which is currently only available as Early Access. If you do not have access to this resource yet, reach out to your NVIDIA Account Manager.

Hardware Requirements#

Each component has its own hardware requirements. The requirements of the workflow is the sum of its components.

This guide has been tested on a Ubuntu 22.04 workstation with a NVIDIA GeForce RTX 3090 Ti GPU, streaming the avatar to a Chrome 129 browser on a Windows 11 computer.

Create Directory#

This guide requires downloading some files. We recommend creating an empty directory to store them. The commands in this document assume you are running them from this directory.

mkdir ~/docker-unreal
cd ~/docker-unreal

Create Avatar Scene#

To create a customized MetaHuman avatar follow the Avatar Customization guide to create and build a UE project.

Once you have the new UE project build, upload to its own unreal-scene directory in your ~/docker-unreal:

mv <your_unreal_project>/linux ~/docker-unreal/unreal-scene

We will use this project when running the renderer in the Run Unreal Renderer Microservice section.

Pull Docker Containers#

docker pull nvcr.io/nvidia/ace/ia-animation-graph-microservice:1.0.2
docker pull nvcr.io/nvidia/ace/ia-unreal-renderer-microservice:0.1.1
docker pull nvcr.io/nvidia/ace/audio2face:1.0.11

Start TURN Server#

A TURN server is required to allow the Unreal Renderer microservice to stream data to the browser through WebRTC.

To start a TURN server, download the TURN and Signalling server docker-compose file to your ~/docker-unreal directory. Then, open a new terminal and run the following:

cd ~/docker-unreal
UNREAL_ENGINE_RELEASE_SHORT=5.3 PUBLIC_IP=<CHANGE_WITH_YOUR_IP_ADDRESS> docker compose --file turn-and-signalling-server-docker-compose.yaml up --force-recreate

Starting the TURN server should take no more than a few seconds. The TURN server is ready when the output prints TCP listener opened on : <SOME_IP>:<SOME_PORT> or Http listening on *: 30080.

Note

If you are using a firewall, you may need to open ports:

sudo ufw allow 31720:31820/udp
sudo ufw allow 30080/tcp
sudo ufw allow 30080/udp
sudo ufw allow 8888/tcp
sudo ufw allow 8888/udp
sudo ufw allow 3478/tcp
sudo ufw allow 3478/udp

If you get a Permission Denied error while running these images, check the “Permission Denied For EpicGames Docker Images” chapter of the troubleshooting section.

At this point, you should be able to see the pixel streaming UI at http://localhost:30080, with a message “click to start”. Clicking this button will not work at this stage. Follow the next steps to stream the avatar.

A screenshot of the pixelstreaming web UI's initial screen, which contains a "Click to start" message.

The Pixel Streaming initial screen.#

Download Avatar Scene from NGC#

Download scene from NGC and save it to a default-avatar-scene_v1.0.0 directory in your main work directory.

Alternatively, you can download the scene using the ngc command.

ngc registry resource download-version "nvidia/ace/default-avatar-scene:1.0.0"

Run Animation Graph Microservice#

Run the following command to start the Animation Graph Microservice. This service manages and animates the avatar’s position.

docker run -it --rm --gpus all --network=host --name anim-graph-ms -v $(pwd)/default-avatar-scene_v1.0.0:/home/ace/asset nvcr.io/nvidia/ace/ia-animation-graph-microservice:1.0.2

The service is up and running from the moment you see the log line app ready.

Note

If you encounter warnings about glfw failing to initialize, see the troubleshooting section.

Run Unreal Renderer Microservice#

docker run --env IAUEMS_ANIM_SOURCE_HOST=<CHANGE_WITH_YOUR_IP_ADDRESS> --env IAUEMS_ANIM_SOURCE_PORT=51000 --env IAUEMS_RESOURCE_DOWNLOAD_DESTINATION="/home/unreal-renderer" \
  -v $(pwd)/unreal-scene:/home/unreal-renderer/unrealEngineProject \
  --rm --gpus all --network=host --name renderer-ms --runtime=nvidia  nvcr.io/nvidia/ace/ia-unreal-renderer-microservice:0.1.1

Note

You should see now a slightly moving avatar rendering at http://localhost:30080 like below. If the UI is stuck in “WebRTC Connection Negotiated”:

  1. Ensure the PUBLIC_IP you are using is accessible from both your Linux workstation and the computer running the browser. This might not be the case if you are in a private network (VPN). If this is the case, restart the renderer and the TURN server using a private IP address.

  2. Check the “Glfw Warnings” chapter of the troubleshooting section.

A screenshot of the Pixel Streaming UI rendering an avatar

Run Audio2Face-3D Microservice#

Run the following command to run Audio2Face-3D:

docker run --rm --network=host -it --gpus all nvcr.io/nvidia/ace/audio2face:1.0.11

Generating TRT models:

./service/generate_trt_model.py built-in claire_v1.3
./service/generate_a2e_trt_model.py

In service/a2f_config.yaml, make sure to set the IP (default 0.0.0.0) under grpc_output to your system’s IP.

nano service/a2f_config.yaml

Launch service:

./service/launch_service.py service/a2f_config.yaml

The following log line appears when the service is ready:

[2024-04-23 12:44:33.066] [  global  ] [info] Running...

Note

More details on Audio2Face-3D deployment can be found in A2F-3D NIM Manual Container Deployment and Configuration.

Create stream ID#

Back in the main terminal, run the following commands to generate a new stream ID and create it in the Animation Graph and Unreal Renderer Microservices:

stream_id=$(uuidgen)
curl -X POST -s http://localhost:8020/streams/$stream_id
curl -s -X POST http://localhost:8021/streams/$stream_id -d ""

Test Audio2Face-3D#

Let’s now take a sample audio file to feed into Audio2Face-3D to drive the facial speaking animation.

Normally, you would send audio to Audio2Face-3D through its gRPC API. For convenience, a python script allows you to do this through the command line. Clone the ACE repo and follow the steps to setup the script.

The script comes with a sample audio file that is compatible with Audio2Face-3D. From the microservices/audio_2_face_microservice/scripts/audio2face_in_animation_pipeline_validation_app directory of the ACE repo, run the following command to send the sample audio file to Audio2Face-3D:

python3 validate.py -u 127.0.0.1:50000 -i $stream_id ../../example_audio/Mark_joy.wav

Note

Audio2Face-3D requires audio to be in 16KHz, mono-channel format.

At this stage, you should see the avatar’s lips move, and hear it speak the sentence from the audio file.

Clean up: Remove Streams#

You can clean up the streams using the following commands:

curl -X DELETE -s http://localhost:8020/streams/$stream_id
curl -X DELETE -s http://localhost:8021/streams/$stream_id
On this page