Kubernetes & Unreal Animation Pipeline Workflow#

Kubernetes workflow setup overview (1x Animation Graph microservice, 1x Omniverse Renderer microservice, 1x Audio2Face-3D microservice).

In this setup you will run the microservices in a Kubernetes cluster, render an avatar using Unreal Renderer and view it in a browser.

The workflow configuration consists of the following components:

  • Audio2Face-3D microservice: converts speech audio into facial animation including lip syncing

  • Animation Graph microservice: manages and blendes animation states

  • Unreal Renderer microservice: renderer that visualizes the animation data using Unreal Engine (UE)

  • Unreal Engine Pixel Streaming client: allows to view the animation from a browser

  • Coturn TURN server: enables WebRTC communication between the browser and the Unreal Renderer microservice

  • Avatar scene: Collection of 3D scene and avatar model data that is saved in a local folder

Prerequisites#

Before you start make sure to have installed and checked all prerequisites in the Development Setup.

Additionally, this section assumes that the following prerequisites are met:

  • You have a Kubernetes cluster available

  • You have installed Kubectl

  • You have installed Helm

  • You have access to NVAIE, which is required to download the relevant microservices

Note

  • Note that driver issues have been reported with minikube and microk8s versions 1.29 and 1.30. If you encounter the error Cuda failure: CUDA driver version is insufficient for CUDA runtime version, consider switching to microk8s 1.24, which has been verified to work.

You also need to link a Github account to EPIC games. To do so:

Finally, this workflow relies on the Unreal Renderer Microservice which is currently only available as Early Access. If you do not have access to this resource yet, reach out to your NVIDIA Account Manager.

Hardware Requirements#

Each component has its own hardware requirements. The requirements of the workflow is the sum of its components.

Create Avatar Scene#

To create a customized MetaHuman avatar follow the Avatar Customization guide to create and build a UE project.

In the subsection How to adjust quality / scalability settings you find also information on how to upload it to NGC and how to update the UCS application to point to your new resource.

Download UCS App Files#

Download the following files and place them in an empty ucs_apps directory:

In the animation_pipeline_with_unreal_renderer_params.yaml file, update the unreal-renderer.signallingServer.peerConnectionOptions.iceServers.urls field with your own IP address (hostname -I will show you your IP address).

Configure the Resource Downloader#

The Unreal Renderer microservice uses a Unreal project and optionally packed MetaHuman and scene files. By default the UCS app configures the microservices to download the avatar scene from NGC. However, there are also alternative methods to download resources and you can also create your own resource downloader init container as described in the Resource Downloader section.

Build UCS App#

rm -rf _build/animation_pipeline*
ucf_app_builder_cli app build ucs_apps/animation_pipeline_with_unreal_renderer.yaml ucs_apps/animation_pipeline_with_unreal_renderer_params.yaml -o _build/animation_pipeline

Start TURN Server#

A TURN server is required to allow the Unreal Renderer microservice to stream data to the browser through WebRTC. The TURN server must run outside of kubernetes because it needs to be accessible from both the microservice and the browser.

To start a TURN server, download the TURN server docker-compose file (note: this is not the same docker compose file as in the Docker & Unreal workflow) to your working directory. Then, open a new terminal and run the following:

PUBLIC_IP=<CHANGE_WITH_YOUR_IP_ADDRESS> docker compose --file turn-server-docker-compose.yaml up turnserver --force-recreate

Starting the TURN server should take no more than a few seconds. The TURN server is ready when the output prints TCP listener opened on : <SOME_IP>:<SOME_PORT> or Http listening on *: 30080.

Note

If you are using a firewall, you may need to open ports:

sudo ufw allow 31720:31820/udp
sudo ufw allow 30080/tcp
sudo ufw allow 30080/udp
sudo ufw allow 8888/tcp
sudo ufw allow 8888/udp
sudo ufw allow 3478/tcp
sudo ufw allow 3478/udp

If you get a Permission Denied error while running these images, check the “Permission Denied For EpicGames Docker Images” chapter of the troubleshooting section.

Deploy UCS App#

name=animation-pipeline
namespace=$name

kubectl create namespace $namespace
kubectl create secret docker-registry ngc-docker-reg-secret --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password=$NGC_CLI_API_KEY -n $namespace
kubectl create secret generic ngc-api-key-secret --from-literal=NGC_CLI_API_KEY=$NGC_CLI_API_KEY -n $namespace

Set the Github credentials from the prerequisites section:

GITHUB_USERNAME=<CHANGE_WITH_YOUR_GITHUB_USERNAME>
GITHUB_ACCESS_TOKEN=<CHANGE_WITH_YOUR_GITHUB_ACCESS_TOKEN> # see Prerequisites section

kubectl create secret docker-registry ghcr-docker-reg-secret --docker-server=ghcr.io --docker-username=$GITHUB_USERNAME --docker-password=$GITHUB_ACCESS_TOKEN -n $namespace

And start the deployment:

helm upgrade --install --cleanup-on-fail --namespace $namespace $name _build/animation_pipeline/ -f _build/animation_pipeline/values.yaml -f ucs_apps/animation_pipeline_with_unreal_renderer_values.yaml

Check the pod states:

watch kubectl get pods -n $namespace

Starting all the pods will take up to 30 min. You need to wait until they are all indicated as ready.

Prepare Streaming#

Create a stream:

stream_id=$(uuidgen)
kubectl exec -n $namespace -c ms ia-animation-graph-microservice-deployment-0 -- curl -X POST -s http://localhost:8020/streams/$stream_id
kubectl exec -n $namespace -c signalling ia-unreal-renderer-microservice-deployment-0 -- curl -s -X POST http://localhost:8021/streams/$stream_id -d ""

If successful, the output of this command should contain “OK” and “Stream ID successfully created!”.

Then, verify in the logs that Output animation data | Stream ID: <stream_id> is present:

kubectl logs -n $namespace -c ms ia-animation-graph-microservice-deployment-0

At this point, you should be able to view the avatar in your browser at http://<CHANGE_WITH_YOUR_IP>:30080.

Test Audio2Face-3D#

In separate tabs, activate port-forwarding:

kubectl port-forward -n $namespace a2f-with-emotion-a2f-deployment-XXX 50010:50010
kubectl port-forward -n $namespace ia-animation-graph-microservice-deployment-0 8020:8020

Note that the Audio2Face-3D pod has a random suffix, which must be adapted in the above command.

Let’s now take a sample audio file to feed into Audio2Face-3D to drive the facial speaking animation.

Normally, you would send audio to Audio2Face-3D through its gRPC API. For convenience, a python script allows you to do this through the command line. Follow the steps to setup the script.

The script comes with a sample audio file that is compatible with Audio2Face-3D. Run the following command to send the sample audio file to Audio2Face-3D:

Note

Audio2Face-3D requires audio to be in 16KHz, mono-channel format.

python3 validate.py -u 127.0.0.1:50010 -i $stream_id Mark_joy.wav

Clean up: Remove Streams#

You can clean up the stream using the following commands:

kubectl exec -n $namespace -c ms ia-animation-graph-microservice-deployment-0 -- curl -X DELETE -s http://localhost:8020/streams/$stream_id
kubectl exec -n $namespace -c signalling ia-unreal-renderer-microservice-deployment-0 -- curl -X DELETE -s http://localhost:8021/streams/$stream_id
On this page