Troubleshooting#

This section will cover some common issues that you may encounter when using the Tokkio UI, and how they can be resolved. Please search the below sections for the error that was encountered.

Issues with the UI not loading#

Note

If the UI fails to load in the browser, there should be an error code on the browser error page. A full list of possible errors (for Chrome) can be found on the chrome://network-errors page in a Chrome browser; you can search for common causes of the error code online.

UI deployed in Kubernetes#

If the UI fails to load in the browser, here is how you can troubleshoot the issue:

  1. Ensure that the Tokkio deployment is running.

    This can be done by checking the liveliness of the pods in the cluster. From the command line on the machine which has access to the Kubernetes cluster on which you are deploying Tokkio, run the following command:

    kubectl get pods -n app

    Confirm that all pods are running and healthy.

  2. Confirm that you have access to the network on which the cluster is deployed.

    This can be done by pinging the cluster’s external IP address. From the command line of the machine which you are opening the browser on, run the following command:

    ping <cluster_external_ip>

    If you receive a response, then you have access to the network on which the cluster is deployed. If you do not, please reach out to your network administrator.

  3. Confirm that the UI is opened using https://<cluster_external_ip>:30111. Note that the UI is opened on HTTPS, and not HTTP.

  4. Confirm that you do not have any browser extensions or security settings which are blocking the UI from loading. It is recommended at this point to try opening the UI on a different browser or on a different machine to rule out these possibilities.

UI deployed locally using Docker#

If the UI fails to load in the browser, here is how you can troubleshoot the issue. Please follow the below steps in order.

  1. Ensure that the Docker container for the UI is running and healthy, and that port 3000 is open.

    This can be done by checking the output of the following command:

    docker ps

    You should see the UI container, and the text 0.0.0.0:3000->3000/tcp under the PORTS column. If you do not see this, please ensure that you have followed the instructions in the Docker Deployment section.

  2. Confirm that you have access to the network on which the container is deployed (not applicable if the container is deployed on the same machine as the browser).

    This can be done by pinging the machine on which the container is deployed. From the command line of the machine which you are opening the browser on, run the following command:

    ping <container machine ip>

    If you receive a response, then you have access to the network on which the container is deployed. If you do not, please reach out to your network administrator.

  3. Confirm that the UI is opened using the URL https://<container machine ip>:3000. Note that the UI is opened on HTTPS, and not HTTP.

  4. Confirm that the machine on which you are running the UI container does not have a firewall blocking the port 3000.

    This can be checked by hitting the UI endpoint on the container machine using the following command:

    curl https://<container machine ip>:3000

    If you receive a response, then check if the same command works on the machine on which you are opening the UI.

    If the command only works the first time, then likely a firewall is blocking the port 3000. On some Ubuntu machines, this can be fixed by running the following command (run this on the machine that the UI container is running on):

    sudo ufw allow 3000

    If the command does not work, reach out to your network administrator to get the port open.

  5. If you can curl the UI endpoint from the machine where your browser is open, but cannot open the UI, then ensure that you do not have any browser extensions or security settings that are blocking the UI from loading. It is recommended at this point to try opening the UI on a different browser or on a different machine to rule out these possibilities.

  6. If none of the above works, try opening the UI in the browser on the same machine on which you are running the container. If this does not work, please reach out to NVIDIA for support.

UI Error Codes#

This section will cover some common error codes that you may encounter when using the Tokkio UI, and how to resolve them.

INGRESS_NETWORK_ERROR#

Why does this happen?#

This error appears when the UI is unable to connect to the deployment.

How can this be resolved?#

First ensure that the Tokkio deployment is healthy and reachable. This can be done by hitting the /health endpoint, using the below cURL command:

curl http://<cluster IP>:30888/health

If the deployment is healthy, you should see an empty list for unhealthy services in the response payload. For example:

$ curl http://10.128.34.39:30888/health
{"unhealthy_services":[],"healthy_services":["....", "....."]}

If the endpoint does not respond as expected, you must ensure that the deployment is running and the machine you are running it on is reachable by the client.

If you are running the UI outside of the Kubernetes cluster (see Deployment for the different ways to run the UI), ensure that the deployment endpoint is configured correctly in the UI. This is configured as the INGRESS_ENDPOINT config option; it should be set to the same endpoint as the one you used to check the health of the deployment.

INGRESS_503_ERROR#

Why does this happen?#

This error appears when the UI cannot start the session because all of the available avatar streams are being used by others.

How can this be resolved?#

Ensure that there are less people using the system than there are streams available. If you have multiple browsers open running the UI, ensure that you have closed all of them. After closing the UI, it will take a minute or two for the stream used by that UI to free up.

INGRESS_400_ERROR#

Why does this happen?#

This error appears when the UI is unable to start a session because the user does not have a valid session token.

How can this be resolved?#

Ensure that cookies are enabled in your browser. Clear your cookies and history, and do not open Tokkio in incognito mode. If this does not fix the issue, try on a different browser, and make sure you do not have any extensions or security settings blocking the cookies.

INGRESS_401_ERROR#

Why does this happen?#

This happens when the Session token used by the UI to authenticate with the deployment has expired.

How can this be resolved?#

Refresh your browser to get a new session token. If that fails, clear your browser cookies and try again.

INGRESS_403_ERROR#

Why does this happen?#

This happens when the user’s session has become invalid.

How can this be resolved?#

Refresh your browser to get a new session token. If that fails, clear your browser cookies and try again.

VST_GET_USER_MEDIA_ERROR#

Why does this happen?#

This error appears when the UI is unable to access the user’s microphone.

How can this be resolved?#

  1. Ensure that the machine on which you are opening the UI in the browser has a working microphone. A website like https://webrtc.github.io/samples/src/content/getusermedia/audio/ can be used to test that the microphone is recognized by the browser. The UI should not be opened on a remote desktop.

  2. Ensure that the user has granted access to the microphone in the Tokkio UI.

  3. If the user has granted access, try refreshing the page.

  4. If the issue persists, try on a different browser, and make sure you do not have any extensions or security settings blocking the microphone.

VST_INBOUND_STREAM_ERROR#

Why does this happen?#

This error appears when the UI is unable to connect to the inbound (avatar) stream. There can be many reasons why the inbound stream might fail to connect. The most common causes are related to the TURN server. Both peers should be able to generate relay ICE candidates on their side (not applicable in the case of a Reverse Proxy). If one of them is unable to generate relay candidates, then the inbound stream will fail to connect.

How can this be resolved?#

Close the UI window, then restart the Renderer and VST services in the Kubernetes cluster. See How to Restart the Renderer Pods in the Kubernetes Cluster and How to Restart the VST Pod in the Kubernetes Cluster for more information.

Once the pods have been deleted, respawned, and have all come back up to healthy, open the UI again.

View the Troubleshooting section in the VST documentation to continue troubleshooting if the issue persists.

VST_OUTBOUND_STREAM_ERROR#

Why does this happen?#

This error appears when the UI is unable to send the user’s audio stream to the deployment.

How can this be resolved?#

Close the UI window, then restart the VST service in the Kubernetes cluster. See How to Restart the VST Pod in the Kubernetes Cluster for more information.

Once the pod has been deleted, respawned, and has come back up to healthy, open the UI again.

View the Troubleshooting section in the VST documentation to continue troubleshooting if the issue persists.

VST_INVALID_PARAMETER_ERROR#

Why does this happen?#

This error appears when the VST streaming library is misconfigured.

How can this be resolved?#

If you have not modified the UI source code, reach out to NVIDIA for support. If you have modified the UI source code, ensure that the VST streaming library is configured correctly.

VST_WEBSOCKET_ERROR#

Why does this happen?#

This error appears when the UI is unable to connect to the VST service, or if the maximum number of concurrent streams in the VST service is reached.

It can also appear if tokkio is opened from multiple browser tabs in the same window, exceeding the maximum number of concurrent streams allowed by the deployment.

How can this be resolved?#

  1. First, try closing all open browser tabs and windows, and reopen the UI in a single tab after 1 minute.

  2. If the issue persists, try restarting the VST service in the Kubernetes cluster. See How to Restart the VST Pod in the Kubernetes Cluster for more information.

    Once the pod has been deleted, respawned, and has come back up to healthy, open the UI again.

VST_BUSY_ERROR#

Why does this happen?#

This error appears when the VST service is busy.

How can this be resolved?#

  1. First, try refreshing the page.

  2. If the issue persists, try restarting the VST service in the Kubernetes cluster. See How to Restart the VST Pod in the Kubernetes Cluster for more information.

    Once the pod has been deleted, respawned, and has come back up to healthy, open the UI again.

  3. View the Troubleshooting section in the VST documentation to continue troubleshooting VST issues.

VST_WEBSOCKET_TIMEOUT#

Why does this happen?#

This error appears when the UI’s connection with the VST service goes down.

How can this be resolved?#

  1. First, try refreshing the page.

  2. If the issue persists, check the health of the VST service in the cluster; it is possible that the VST service has crashed and will be respawned momentarily.

  3. If the issue persists, try restarting the VST service in the Kubernetes cluster. See How to Restart the VST Pod in the Kubernetes Cluster for more information.

    Once the pod has been deleted, respawned, and has come back up to healthy, open the UI again.

ACE_CONTROLLER_WEBSOCKET_ERROR#

Why does this happen?#

This error appears when the UI cannot establish a connection with the ACE controller, or the connection is lost. If the ACE controller is restarted, this error will appear in the UI.

How can this be resolved?#

  1. If you are making changes to the ACE controller using the ACE Configurator, this is expected behavior. Refresh your UI once the ACE controller has restarted with the changes.

  2. If the issue persists, close the UI and restart the ACE controller pods in the Kubernetes cluster. See How to Restart the ACE Controller Pods in the Kubernetes Cluster for more information.

    Once the pods have been deleted, respawned, and have all come back up to healthy, open the UI again.

UNKNOWN_ERROR#

This error code is returned when the UI encounters an error that is not handled by the other error codes. In this case, it is recommended to check the browser console logs for more information, and reach out to NVIDIA for support.

Other UI Issues#

Start Button is Greyed Out#

Why does this happen?#

The start button is greyed out when the UI is still trying to connect to the ACE controller.

How can this be resolved?#

Keep waiting. The UI will attempt to connect to the ACE Controller for 30 seconds. If it is unable to connect after 30 seconds, it will show the ACE_CONTROLLER_WEBSOCKET_ERROR error page in the UI.

Note

If you are making changes to the ACE controller code or creating your own custom pipeline, the start button could also be stuck greyed out after clicking start if the UserPresenceProcesssor frame processor is not present in the pipeline. The UI must receive confirmation from the ACE controller that the session start was successful before re-enabling the start/stop button, and this frame processor is responsible for generating that event.

Avatar is not Appearing, Black Screen Appears Instead#

Why does this happen?#

The avatar stream is not getting passed to the UI properly. Either the stream is not rendering, or the connection is not working.

How can this be resolved?#

  1. Refresh the UI.

  2. Ensure that the network connection is high-speed and stable. See Recommended Network Bandwidths for the recommended network bandwidth for each resolution.

  3. Restart the Renderer and VST pods in the Kubernetes cluster. See How to Restart the Renderer Pods in the Kubernetes Cluster and How to Restart the VST Pod in the Kubernetes Cluster for more information.

Other Errors#

View the top-level troubleshooting section at Troubleshooting for more information on how to resolve other issues. Some issues with the avatar, audio, VST connection, or deployment may manifest in the UI but are caused by an underlying issue with the deployment, connection, or specific microservice.

Appendix#

How to Restart the VST Pod in the Kubernetes Cluster#

kubectl get pods -n app
// find the vms-vms-xxxxx pod

kubectl delete pod <vms-vms-xxxxx pod name> -n app

How to Restart the Renderer Pods in the Kubernetes Cluster#

kubectl get pods -n app
// find the ia-unreal-renderer-microservice-deployment-x pods

kubectl delete pod <ia-unreal-renderer-microservice-deployment-x pod name> -n app
// repeat for all Renderer pods that exist

How to Restart the ACE Controller Pods in the Kubernetes Cluster#

kubectl get pods -n app
// find the ace-controller-ace-controller-deployment-x pods

kubectl delete pod <ace-controller-ace-controller-deployment-x pod name> -n app
// repeat for all ACE Controller pods that exist
On this page