Quickstart Guide#

Introduction#

Note

For a first use experience we highly recommend that you start at NIM Agent Blueprint: Digital Human for Customer Service to get started with Digital Humans for Customer Service.

Welcome to NVIDIA Tokkio! While there are multiple variants of Tokkio, this guide will walk you through setting up a basic digital human avatar with baremetal machines. At the end of this quick start, you will have a digital avatar you may speak with.

If you do not have access to a baremetal machine with NVIDIA GPUs to deploy Tokkio, please refer to Deployment for cloud deployments.

You may expect to see the following visualization of an avatar that responds to a user based on LLM processing. This quick start guide does not detail all the customizations available to Tokkio.

At a high level, we will guide you through installing tokkio with one-click scripts. Running a one-click script from a terminal abstracts the complexity of configuring the machine, pulling, and deploying Tokkio.

Prerequisites#

This quick start guide requires having two machines:

Controller Instance: The controller instance is where you will run the one-click script.
Application Instance: The application instance is where Tokkio will be installed.

1. Configure Controller Instance#

As previously mentioned, the controller instance is where you will run one-click scripts. This machine must follow the following requirements:

Operating System: Ensure that the instance is running Ubuntu 22.04.
SSH Key Pair: Generate an SSH key pair to be used in later phase of one-click scripts. You can follow these steps:
- Open a terminal on your controller instance
- Run the following command to generate a 4096-bit key replacing your_email@example.com with your email address:
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
- You will be prompted to specify a file location to save the key:
  
  Press Enter to accept the default location (~/.ssh/id_rsa).
  
  Alternatively, specify a custom path.
Passwordless Sudo Access: Ensure the user on the controller instance is enabled with passwordless sudo access.
- You can test this by running sudo ls /root and this command should not prompt you for a password.
- If this is not setup, follow the steps below or reach out to your company’s system administrator for support.
  Edit the sudoers File Safely Use the visudo command to edit the sudoers file. This ensures syntax checking and prevents errors.
  
  sudo visudo
  
  Add Your User for Password-less Access Scroll to the section where user privileges are defined. Add a line to grant password-less sudo access to your user. Replace username with your actual username:
  
  username ALL=(ALL) NOPASSWD:ALL
  
  ALL=(ALL): Grants administrative rights for all users and groups. NOPASSWD:ALL: Disables the password prompt for all sudo commands.

2. Configure Application Instance#

The application instance is where the Tokkio Pipeline will run. This machine must follow the following requirements:

Operating System: Ensure that the instance is running Ubuntu 22.04.
GPU: You will need at least 2xT4 or 2xL4 to deploy this single stream Tokkio example.
Partition Size: You will need at least 700 GB of memory allocated for the application instance.
Kubernetes Considerations: This instance should not have any prior Kubernetes installations running before you run the one-click scripts for the first time.
Passwordless Sudo Access: Ensure the user on the app instance is enabled with passwordless sudo access.
- You can test this by running sudo ls /root and this command should not prompt you for a password.
- If this is not setup, follow the steps below or reach out to your company’s system administrator for support.
  Edit the sudoers File Safely Use the visudo command to edit the sudoers file. This ensures syntax checking and prevents errors.
  
  sudo visudo
  
  Add Your User for Password-less Access Scroll to the section where user privileges are defined. Add a line to grant password-less sudo access to your user. Replace username with your actual username:
  
  username ALL=(ALL) NOPASSWD:ALL
  
  ALL=(ALL): Grants administrative rights for all users and groups. NOPASSWD:ALL: Disables the password prompt for all sudo commands.
IPv4 Address: Please take note of the IP address of the application instance for later use. On the controller instance, run the following command to copy the ssh key to the application instance’s authorized keys to allow passwordless communication

ssh-copy-id <app-instance-username>@<app-instance-ip-address>

3. Obtain NVIDIA API Key#

Before moving any further, you will need an NVIDIA API KEY to be able to access and pull the Tokkio resources needed for this quick start installation.

To obtain the api key, navigate to the NVIDIA API Catalog. A pop-up should appear with the title “Develop with this Blueprint”. If the pop-up does not appear, select the download blueprint in the top right of the screen to see the pop-up. Select Generate API key to create a key. For now, save this in a secure location.

As a reminder, you should now have the ssh key from the controller instance, IPv4 address from the application instance, and your NVIDIA API KEY readily available.

4. Obtain the One-Click Script from Github#

The following steps should be done on the controller instance.

To get the one-click script clone the ACE Github repository from NVIDIA/ACE.git and navigate to the baremetal one-click script directory.

$ git clone https://github.com/NVIDIA/ACE.git
$ cd workflows/tokkio/scripts/one-click/baremetal

5. Configure the One-Click Script Deployment#

We need to configure the controller instance environment. Navigate to the config-template-examples/llm-ov-3d-coturn-1x-stream directory and open the my-config.env file.

$ cd config-template-examples/llm-ov-3d-coturn-1x-stream/
$ nano my-config.env

You will need to fill in the information below in the .env file.

Note

The default LLM is accessed via a NIM. For this tutorial we will be using a NIM model, no additional steps are needed, and you may leave the OpenAI API Key unchanged for this quick start guide.

Note

Currently we are supporting installation of turn servers (coturn/twilio) on same machine as the application instance, so we only need one machine. Therefore, the values of APP_HOST_IPV4_ADDR and COTURN_HOST_IPV4_ADDR should be the same. Similarly, the values of APP_HOST_SSH_USER and COTURN_HOST_SSH_USER are the same.

export OPENAI_API_KEY="<replace-with-openai-api-key>"
export NVIDIA_API_KEY="<replace-with-you-NVIDIA-Personal-API-Key>"
export APP_HOST_IPV4_ADDR="<replace-with-the-ip-address-of-application-instance>"
export APP_HOST_SSH_USER="<replace-with-the-username-of-application-instance>"
export COTURN_HOST_IPV4_ADDR="<replace-with-the-ip-address-of-application-instance>"
export COTURN_HOST_SSH_USER="<replace-with-the-username-of-application-instance>"

Then you will need to source this to the environment depending on your shell.

$ source my-config.env

The command above with source the environment variables to the current shell session and will need to be redone every time you would like to manipulate your deployment (install, uninstall).

Next, in the same directory, there is the config-template.yml file. Copy this file with a specific name and move it up two levels in the directory.

$ cp config-template.yml <my-new-config-template-name.yml>
$ mv <my-new-config-template-name.yml> ../..
$ cd ../..

The config-template.yml and my-config.env may be found for reference here

Deployment#

Follow the below steps to get one-click scripts ready for installation.

1. Run the One-Click Script#

To install Tokkio, run the command below with your config-template.yml. Capture the URLs from the expected output.

$ ./envbuild.sh install --component all --config-file ./<my-renamed-config-template.yml>

Expected Output

***(Several output lines omitted for brevity)***
.
.
access_urls:
   app:
      app: http://<application-instance-ip>:80/
      grafana: http://<application-instance-ip>:32300/login
      kibana: http://<application-instance-ip>:31565/app/kibana
      prometheus: http://<application-instance-ip>:30090/graph
    turn: {}
ssh_command:
app:
    master: ssh -i /home/metroadmin/.ssh/id_rsa.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -i /home/metroadmin/.ssh/id_rsa.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -W %h:%p ubuntu@<bastion-host-ip>" ubuntu@<application-instance-ip>
turn:
    master: ssh -i /home/metroadmin/.ssh/id_rsa.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@<turn-server-ip>

2. Verify the Installation#

At this point, you can log into the application instance with the ssh command below check watch for all the kubernetes pods are up and running.

$ ssh <application-instance-username>@<application-instance-IP-address>

$ kubectl get pods -n app

Expected Output

NAME                                                        READY   STATUS    RESTARTS      AGE
a2f-a2f-deployment-98c7fb777-lnp6w                          1/1     Running   0             72m
ace-agent-chat-controller-deployment-0                      1/1     Running   0             72m
ace-agent-chat-engine-deployment-5f8db599f6-6mkh9           1/1     Running   0             72m
ace-agent-plugin-server-deployment-645cf865b5-vvcvw         1/1     Running   0             72m
anim-graph-sdr-envoy-sdr-deployment-5b7cc55b6b-cbbz6        3/3     Running   0             72m
catalog-rag-deployment-67547fd54-lrcf2                      1/1     Running   0             72m
chat-controller-sdr-envoy-sdr-deployment-78b54b7f86-fnvql   3/3     Running   0             72m
ds-sdr-envoy-sdr-deployment-85bbfdb4c4-lbksh                3/3     Running   0             72m
ds-visionai-ds-visionai-deployment-0                        1/1     Running   0             72m
ia-animation-graph-microservice-deployment-0                1/1     Running   0             72m
ia-omniverse-renderer-microservice-deployment-0             1/1     Running   0             72m
ia-omniverse-renderer-microservice-deployment-1             1/1     Running   0             72m
ia-omniverse-renderer-microservice-deployment-2             1/1     Running   0             72m
mongodb-mongodb-666765487c-tzxxw                            1/1     Running   0             72m
occupancy-alerts-api-app-84576db5c9-qb7dr                   1/1     Running   0             72m
occupancy-alerts-app-5cfcc9f75-b84xb                        1/1     Running   0             72m
redis-redis-79c99cdd97-kl77x                                1/1     Running   0             72m
redis-timeseries-redis-timeseries-69bb884965-fnn44          1/1     Running   0             72m
renderer-sdr-envoy-sdr-deployment-55df9ccc6f-v66rr          3/3     Running   0             72m
riva-speech-547fb9b8c5-7w8h4                                1/1     Running   0             72m
tokkio-cart-manager-deployment-65d8547cbc-vplnq             1/1     Running   0             72m
tokkio-ingress-mgr-deployment-65dfdc79f6-4pvcg              3/3     Running   0             72m
tokkio-ui-server-deployment-79c9749686-ztxrq                1/1     Running   0             72m
tokkio-umim-action-server-deployment-674cccc898-qgz26       1/1     Running   0             72m
triton0-bbd77d78f-rz62w                                     1/1     Running   0             72m
vms-vms-67876bcb9b-q655s                                    1/1     Running   0             72m

3. Access Tokkio Application#

Browser settings: This installation does not setup any CA certificates. To overcome browser restrictions, we need to enable some flags in Chrome.
- On the machine where you want to access the UI, open a Chrome browser and navigate to chrome://flags.
- Search for the Insecure origins treated as secure falg
- Select Enable
- Add the following chrome flags under it. http://<application-instance-ip>:80, http://<application-instance-ip>:30888, ws://<application-instance-ip>:30888
- Follow the Chrome instructions to Relaunch the browser and complete this step.
Once all the pods are running, you should be able to access Tokkio using one of the URLs presented in the terminal output after running the one-click script. You may enter the ui_endpoint URL in a chrome browser to interact with Tokkio. e.g. http://<application-instance-ip>:80
- You may encounter a Site is not secure Warning. This warning arises since we are not using any CA signed certificate. Confirm the IP address and port are correct then read and follow the instructions in the browser to bypass this warning.
- When accessing the UI for the first time, the browser should prompt you for permissions such as Mic, Speaker or Camera necessary for UI to operate. Upon accepting the permissions, the UI should load.

Review Installation Details#

To review the installation output at a later time, run the command below. You will receive the same output from when Tokkio was initially installed.

$ ./envbuild.sh info --config-file ./<my-renamed-config-template.yml>

Expected Output

access_urls:
   app:
      app: http://<application-instance-ip>:80/
      grafana: http://<application-instance-ip>:32300/login
      kibana: http://<application-instance-ip>:31565/app/kibana
      prometheus: http://<application-instance-ip>:30090/graph
    turn: {}
ssh_command:
app:
    master: ssh -i /home/metroadmin/.ssh/id_rsa.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -i /home/metroadmin/.ssh/id_rsa.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -W %h:%p ubuntu@<bastion-host-ip>" ubuntu@<application-instance-ip>
turn:
    master: ssh -i /home/metroadmin/.ssh/id_rsa.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ubuntu@<turn-server-ip>

Un-install the Tokkio Application#

If you decide to uninstall the entire setup, run the command below. Ensure you have the variables from the my-config.env sourced to your shell.

source my-config.env
$ ./envbuild.sh uninstall --component all --file config-template.yml

Troubleshooting#

If you are unable to follow the steps outlined in this quick start guide, please refer to Troubleshooting for Troubleshooting.

What’s Next#

Now that you have a working Tokkio avatar, there are different ways to deploy or customize the application. The diagram below provides a user journey flow chart of what to expect as you progress through the documentation.

Cloud Deployments#

In addition to deploying this basic Tokkio example on baremetal machines, you may alternatively deploy Tokkio in the cloud with AWS, GCP, or Azure.

AWS: Please refer to AWS for AWS deployment.
GCP: Please refer to GCP for GCP deployment.
Azure: Please refer to Azure for Azure deployment.

Reference Workflows#

The version of the Tokkio application deployed in this quick start guide is the simplest version of the LLM-RAG reference workflow. If you would like to deploy a different reference workflow, refer to Reference Workflows.

Customization#

The version of the Tokkio application deployed in this quick start guide is the simplest version of the LLM-RAG reference workflow. If you would like to customize your Tokkio deployment, refer to the guides below:

RAG: Please refer to RAG Endpoint Customization to implement a custom RAG server rather than the default LLM.
TTS: See Third party TTS Voice Customization for TTS voice customizations.
Avatar: Check out Avatar and Scene Customization for avatar customization options.
Other: More customization options for the UI, vision possibilities, and other microservices may be found in the Customization section.