Tokkio LLM-RAG - Unreal Engine#

Introduction#

Tokkio LLM-RAG - Unreal Engine app integrates Unreal Renderer Microservice to Tokkio pipeline to support real-time animation rendering from Unreal Engine Pixel Streaming through WebRTC. It enables users to easily plug in their MetaHuman avatar created with Unreal Engine into the Tokkio app and stream on the Tokkio Frontend with high definition quality and real-time.

This reference app is a variation of Tokkio LLM-RAG using Unreal Engine as its renderer. Other workflows such as Tokkio Retail can also be used with this particular rendering option.

Minimum GPU Requirements#

Minimum Concurrent Streaming GPU Requirements#
	T4	L4	A10
1 stream	2x	2x	2x
3 streams	4x	4x	4x

Prerequisites#

A coturn server for webRTC communications.
An avatar created with Unreal Engine. Please Refer to Avatar Customization for more information on creating and customizing avatar.
A GitHub account which its access token will be used for setting the required K8 secret for UE MS.
An Epic Games account which will be linked to the GitHub account.
Access to the following NGC team(s) and org(s):
- nvidia/ace

Note

Please ensure Nvidia driver 535.x.x is installed.

Architecture#

Tokkio UE architectural diagram illustrates various ACE components and their communications between VST, SDR, and Tokkio UI.

Detailed interactions and API calls for rendering UE avatar can be found in below sequence diagram.

Sequence Diagram Interactions Explanation#

Pixel Streamer plugin is loaded when UE MS is started
Once the Tokkio UI is launched, a websocket connection is created between the Tokkio UI and VST MS. The connection will be utilized for establishing the webRTC connection for streaming the avatar.
Tokkio UI sends stream/start websocket message to VST MS to initiate the webRTC connection
SDR distributes requests to available UE MS replica as well as notifying the newly added stream from the Tokkio UI by VST to the UE MS itself
UI MS makes POST /ue HTTP call to the Renderer Adapter (sidecar) initiate the webRTC p2p connection negotiation (e.g. SDP / ICE Candidates exchanges)
After the p2p connection between Pixel Streamer and the Tokkio UI is established, the avatar should be streaming on the Tokkio UI
Before the Tokkio UI is closed, it sends a websocket message stream/stop to VST to terminate the webRTC connection
SDR notify UE MS to remove the stream and terminate the webRTC connection from Pixel Streamer via the Renderer Adapter (sidecar) by calling its DELETE /ue HTTP endpoint

Pre-steps#

Pre-step 1. Linking GitHub and Epic Games Accounts#

To link the Epic Game account to your GitHub account, follow the steps under Getting started with Unreal Engine source code access.

Pre-step 2. Obtain GitHub Access Token#

To generate access token for GitHub account, follow the steps from Creating a personal access token (classic) section.

Refer to the Prerequisite section from Docker & Unreal Animation Pipeline Workflow for more details.

Pre-step 3. Set Kubernetes secret to access Epic Games resources#

Note

Refer to the Deployment for setting below secret with one-click script.

Use the access token created from Pre-step 2. Obtain GitHub Access Token, to populate the k8s_secrets for the One-Click deployment of this app. Refer to Deployment for options on deployment script. A snippet of the relevant section in the config-template.yaml provided below.

app:
  configs:
      k8s_secrets:
        - name: 'ghcr-docker-reg-secret'
          type: 'dockerconfigjson'
          registry_name: "ghcr.io"
          username: "{{ lookup('env', 'GITHUB_USERNAME') }}"
          password: "{{ lookup('env', 'GITHUB_ACCESS_TOKEN') }}"

Deployment#

Note

Ensure you have completed the Pre-steps before proceeding.

The sample Tokkio LLM-RAG - Unreal Engine helm chart is available here https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ace/helm-charts/ucs-tokkio-app-base-3-stream-llm-rag-3d-ue. Follow the steps below to deploy this helm chart.

Create a yaml file to specify the coturn setting (e.g. ue_settings.yaml)

ia-unreal-renderer-microservice:
  applicationSpecs:
    deployment:
      initContainers:
      - env:
        - name: REMOTE_RESOURCE_PATH
          value: "{'metahumanPak': '', 'scenePak': '', 'unrealEngineProject': '<AVATAR_PROJECT_PATH>'}"
        - name: DESTINATION_RESOURCE_PATH
          value: /home/interactive-avatar
        - name: NVIDIA_VISIBLE_DEVICES
          value: 1,2,3
        envFrom:
        - secretRef:
            name: ngc-api-key-secret
        image: nvcr.io/nvidia/ace/ngc-resource-downloader:1.1.4
        imagePullPolicy: IfNotPresent
        name: init
        volumeMounts:
        - mountPath: /home/interactive-avatar
          name: asset-volume
    configs:
      IAUEMS_SIGNALLING_SERVER_PEER_CONNECTION_OPTIONS: "{\"iceServers\": [{\"urls\": [\"turn:<TURN_IP>:<TURN_PORT>\"], \"username\": \"<TURN_USERNAME>\", \"credential\": \"<TURN_PASSWORD>\"}], \"iceTransportPolicy\": \"relay\"}"

Replace the following placeholders as described below:
- <TURN_IP> == COTURN server IP address
- <TURN_PORT> == COTURN server port
- <TURN_USERNAME> == COTURN server username
- <TURN_PASSWORD> == COTURN server password

Note

Refer to the Deployment page on how to apply the yaml file from this step to the deploy-template.yml based on your CSP. Next, go to the Run Tokkio UI to experience the UE app to run the UI.

Deploy the sample Tokkio LLM-RAG - Unreal Engine helm chart from https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ace/helm-charts/ucs-tokkio-app-base-3-stream-llm-rag-3d-ue with using the guide available in the Tokkio deployment

Note

Refer to Customization for instructions on re-deploying the helm chart with customization changes.

Run Tokkio UI to experience the UE app#

Refer to Front-end for details on configuring and running the Tokkio UI.