Tokkio LLM-RAG - Unreal Engine#

Introduction#

Tokkio LLM-RAG - Unreal Engine app integrates Unreal Renderer Microservice to Tokkio pipeline to support realtime animation rendering from Unreal Engine Pixel Streaming through WebRTC. It enables users to easily plug in their MetaHuman avatar created with Unreal Engine into the Tokkio app and stream on the Tokkio Frontend with high definition quality and realtime.

This reference app is a variation of Tokkio LLM-RAG using Unreal Engine as its renderer. Other workflows such as Tokkio Retail can also be used with this particular rendering option.

Minimum GPU Requirements#

Minimum Concurrent Streaming GPU Requirements#

T4

L4

A10

1 stream

2x

2x

2x

3 streams

4x

4x

4x

Prerequisites#

  • A coturn server for webrtc communications.

  • An avatar created with Unreal Engine. Please Refer to Avatar Customization for more information on creating and customizing avatar.

  • A GitHub account which its access token will be used for setting the required K8 secret for UE MS.

  • An Epic Games account which will be linked to the Github account.

  • Access to the following NGC team(s) and org(s):

    • nvidia/ace

  • UCS Tool 2.5+ (Optional - for building helm chart with UCS app spec)

Note

Please ensure Nvidia driver 535.x.x is installed

Architecture#

Tokkio UE architectural diagram illustrates various ACE components and their communications between VST, SDR, and Tokkio UI

Tokkio Unreal Engine Architecture

Detailed interactions and API calls for rendering UE avatar can be found in below sequence diagram.

Tokkio Unreal Engine Architecture

Pre-steps#

Pre-step 1. Linking Github and Epic Games Accounts#

1. To link the Epic Game account to your Github account, follow the steps under Getting started with Unreal Engine source code access
Linking Github account to Epic Game Account

Pre-step 2. Obtain Github Access Token#

2. To generate access token for Github account, follow the steps from Creating a personal access token (classic) section
Create Github access token

Please refer to the Prerequisite section from Docker & Unreal Animation Pipeline Workflow for more details.

Pre-step 3. Set Kubernetes secret to access Epic Games resources#

Note

Refer to the Deployment for setting below secret with one-click script.

Use the access token created from Pre-step 2. Obtain Github Access Token, run the command below to create the kubernetes secret ghcr-docker-reg-secret. Replace the placeholder <github_access_token> with the generated Github access token.

$ kubectl create secret docker-registry ghcr-docker-reg-secret --docker-server=ghcr.io --docker-username=<github_username> --docker-password=<github_access_token>

Pre-step 4. Deploy coturn server#

Make sure the coturn server is reachable from the k8 cluster running the app and the machine running the UI (e.g. whitelisted).

Deployment#

Note

Ensure you have completed the Pre-steps before proceeding.

There are 3 options to build and deploy the Tokkio UE app:

  1. One click script deployment, start from Step 3. Deploy the UCS Tokkio UE Helm Chart

  2. Helm chart deployment, start from Step 3. Deploy the UCS Tokkio UE Helm Chart

  3. UCS app spec deployment, start from Step 1. Configure the UCS Tokkio UE app spec

Step 1. Configure the UCS Tokkio UE app spec#

  • Replace the placeholder <META_HUMAN_PROJECT_PATH> with the path where the Meta Human avatar project is created (see Prerequisites)

  • Replace the placeholder <COTURN_SETTING> with the details from the coturn server deployed from the Pre-steps

unreal-renderer:
  ...
  resourceDownload:
    remoteResourcePath:
      unrealEngineProject: <META_HUMAN_PROJECT_PATH>
    secretName: ngc-api-key-secret
    ...
  signallingServer:
    peerConnectionOptions: <COTURN_SETTING>
  deployment:
    additionalStartupArgs: "-PixelStreamingEncoderCodec=VP9"

  • Refer to Microservice Parameters for all the configurable Unreal Engine microservice parameters from UnrealEngine microservice

Step 2. Build the UCS Tokkio UE app#

Note

Check Integrating Customization changes with rebuild to learn about using the UCS CLI to perform the steps below

Run the UCS command below to build the helm charts and the output helm charts will be generated

$ ucf_app_builder_cli app build <ucs_app>.yaml <ucs_param>.yaml

A sample UCS app spec for UE app can be found in NVIDIA/ACE

Step 3. Deploy the UCS Tokkio UE Helm Chart#

  1. [Skip this step if you started with UCS app spec deployment] Create a yaml file to specify the coturn setting (e.g. ue_settings.yaml)

ia-unreal-renderer-microservice:
applicationSpecs:
  deployment:
    initContainers:
    - env:
      - name: REMOTE_RESOURCE_PATH
        value: "{'metahumanPak': '', 'scenePak': '', 'unrealEngineProject': '<AVATAR_PROJECT_PATH>'}"
      - name: DESTINATION_RESOURCE_PATH
        value: /home/interactive-avatar
      - name: NVIDIA_VISIBLE_DEVICES
        value: 1,2,3
      envFrom:
      - secretRef:
          name: ngc-api-key-secret
      image: nvcr.io/eevaigoeixww/staging/ngc-resource-downloader:1.1.4
      imagePullPolicy: IfNotPresent
      name: init
      volumeMounts:
      - mountPath: /home/interactive-avatar
        name: asset-volume
  configs:
    IAUEMS_SIGNALLING_SERVER_PEER_CONNECTION_OPTIONS: "{\"iceServers\": [{\"urls\": [\"turn:<TURN_IP>:<TURN_PORT>\"], \"username\": \"<TURN_USERNAME>\", \"credential\": \"<TURN_PASSWORD>\"}], \"iceTransportPolicy\": \"relay\"}"

  • Replace the following placeholders as described below:

    • <TURN_IP> == COTURN server IP address

    • <TURN_PORT> == COTURN server port

    • <TURN_USERNAME> == COTURN server username

    • <TURN_PASSWORD> == COTURN server password

Note

If you are deploying with one-click script, refer to the Deployment page on how to apply the yaml file from this step to the deploy-template.yml based on your CSP. When finish, go to the Step 4. Run Tokkio UI to experience the UE app to run the UI

  1. Deploy the sample Tokkio LLM-RAG - Unreal Engine helm chart from https://helm.ngc.nvidia.com/nvidia/ace/charts/ucs-tokkio-app-base-3-stream-llm-rag-3d-ue-4.1.4.tgz with the command below based on the deployment option you started with

Note

Refer to Integrating Customization changes with rebuild for instruction on installing helm chart from NGC. Add -f <override> at the end of the helm command if you started from the Helm chart deployment approach to use the override yaml created from this step above

Step 4. Run Tokkio UI to experience the UE app#

Refer to Front-end for details on configuring and running the Tokkio UI

Tokkio UE running on Tokkio UI
On this page