UI Customization Guide#
Overview#
This guide will walk you through how to create your own custom UI design with Tokkio. By the end of this guide, you will be able to understand the different customization options available, and implement a customized UI for your use case.
To follow this guide, it is recommended that you have some web development experience, and are comfortable with the JavaScript language.
Customization Options#
There are two ways to create a custom UI depending on the level of customization required:
Create a custom layout in the Tokkio reference UI
Create a new UI from scratch
Most of this guide is focused around the first option, which is the recommended way to create a fully customized UI.
To learn more about the first option, the Creating a Custom Layout section will describe the steps to do this.
For the second, visit the Creating a Custom UI From Scratch section at the bottom of this page.
Creating a Custom Layout#
Prerequisites#
This guide will show you how to update the Tokkio reference UI code to add a custom layout. Before going through this section, ensure you have done the following:
Deployed your Tokkio application.
Downloaded the UI code and run it once to make sure that everything is working end to end. You should be able to see and hear the avatar. Ensure that you are able to have a conversation with the avatar before starting. Docs for setting up the UI from source for the first time are here.
Familiarized yourself with the UI codebase. It is recommended to read this section in its entirety and familiarize yourself with the relevant parts of the UI code before beginning customization.
Ensure that you are using the version of the UI that matches the version of this document. This document assumes you are using the 4.1 version of Tokkio.
The Tokkio UI uses several technologies which would be useful to read up on before beginning development. These are:
React
React Redux
Material UI
Node Package Manager
This guide assumes that you have met the prerequisites and have a basic understanding of the listed technologies, especially React.
UI Code structure#
The UI source code is contained in the src/
directory from the downloaded NGC artifact. It will have the following structure:
├── App.jsx
├── applications
│ ├── AbstractTokkioApplication.jsx
│ ├── custom
│ └── qsr
├── config
│ └── config.js
├── connection
│ ├── axios-interceptors
│ └── websocket
├── features
│ ├── asrText
│ ├── avatar-window
│ ├── backdrops
│ ├── conversation-state
│ ├── error
│ ├── feedback
│ ├── manage-roi
│ ├── mic-button
│ ├── reconnect-dialog
│ ├── start-conversation
│ ├── top-bar
│ ├── ttsText
│ └── ui-window
├── index.css
├── index.jsx
├── layouts
│ ├── FullscreenLayout.jsx
│ ├── KioskLayout.jsx
│ └── layoutSlice.jsx
├── logo.svg
├── ngc-background.png
├── nvidia-background.jpeg
├── pages
│ └── ErrorPages.jsx
├── serviceWorker.js
├── session
│ ├── AccessAndIdTokenWrapper
│ ├── AuthorizationCodeWrapper
│ └── sessionSlice.jsx
├── setupTests.js
├── store
│ └── store.jsx
├── theme
│ ├── themeSlice.jsx
│ └── TokkioThemeProvider.jsx
├── trianglify.svg
├── utils
│ ├── Countdown.jsx
│ ├── Snackbar.jsx
│ ├── useInterval.js
│ ├── useQuery.js
│ └── utils.js
└── white-diamond-dark.png
The below table lists the different code files and directories, and explains what the code in them is responsible for:
Code Section |
Description |
---|---|
|
|
|
The code in the |
|
This file initializes the UI layout and the application. Depending on the configuration, it will either initialize the Kiosk layout + Retail application, or the Fullscreen layout + LLM application. |
|
This contains the available layouts for the UI. The two layouts available by default are Kiosk and Fullscreen. Additional layouts can be added for custom use cases. |
|
This contains the available applications for the UI. The two applications available by default are the custom application (used by the LLM app) and the retail application (called |
|
This contains the theme used by the rest of the application. The default theme uses the NVIDIA color scheme. Updating the theme is not a use-case that is tested to work out of the box, but would be a good start for updating the theme for the app. |
|
This contains features that are common between any application or layout. For example, the feature to show TTS and ASR transcripts can be used by any application. |
|
Initializes the Redux store, used to manage the Redux state tree of the entire application. |
|
Used to process incoming WebSocket messages from the UI server. |
|
Contains pages that are common to any application. Today this only contains the error pages. |
|
Sets up the default configuration for the app, and processes any configuration that is passed from an ENV variable via |
|
Contains various utilities used by the UI. |
Creating a Custom Layout#
To create a customized UI, it is recommended to create a third layout in the layouts
directory in addition to the two that are already there. Here are the basic steps to get started:
In the
layouts
directory, create a file namedMyCustomLayout.jsx
(you can call this file whatever you want).In this file, add a React component that returns
<p>Hello World!</p>
. This will eventually contain the custom code for your application. For now, this can be a very basic component.In
App.jsx
, replace the code forKioskLayout
andFullscreenLayout
with the custom component you made in the previous step. You might notice that an application parameter is passed into these layouts. For the purposes of your customization, this variable will not be needed in your custom layout, you can safely ignore it.Once you do these steps, you can run the UI using the steps from here and see
Hello World!
appear in your browser when it starts up.
Adding Features to the Custom Layout#
Now that you have set up the custom layout, it is time to start adding features to it! The Tokkio reference UI contains code for available features in the features
directory.
The below table contains the features that are currently available for the Tokkio UI. Please note that only features that are listed below are supported. There are some directories in the features
directory that do not work, and these are excluded from the below list.
Component Name |
Location in |
Description of Feature |
---|---|---|
|
|
Shows transcriptions of the user’s speech |
|
|
Initializes the WebRTC connection, and shows a video with the avatar |
|
|
Adds an indicator for the user’s state (is the user looking at the camera, or distracted?). Note that the bot state (avatar state) in this directory is no longer supported. |
|
|
Used to mute/unmute the user’s microphone |
|
|
If the user’s camera is disabled, you can use this to show a start/stop button to start and stop the session manually from the UI |
|
|
Shows transcriptions of the avatar’s speech |
By default, the available features match the overall style requirements of the reference UI. However, the appearance of these components can be easily updated by changing the CSS code present in the code. If you are unsure about how a feature should work, you can search the code for the component name to see how it is used in the existing retail and custom reference applications.
At this point, you should be able to put together your custom layout by combining the existing components for the available features with your own React code.
Useful Tools for Development#
The Redux DevTools Chrome Extension can be used to view the redux state tree of the application as the UI is running.
Creating a Custom UI From Scratch#
Overview and Warning#
This guide will help get you started with creating your own UI from scratch, but it is not comprehensive documentation.
This approach is not recommended for the following reasons:
The interface between the UI and the rest of the app is subject to change without warning for most releases. A UI built from scratch requires one to keep up with these interface changes when you update your version of Tokkio. Building from a custom layout within the reference UI will shield you from these interface changes.
A UI built from scratch will not be able to take advantage of pre-built features available in the reference UI out of the box. Each of these features will needed to be re-implemented in the new UI.
You may run into unexpected bugs related to an incorrect use of the interfaces with Ingress, the Tokkio UI Server, or VST.
In short, the reference UI gives you a quick way to set up a custom design and maintain it across releases. When creating a UI from scratch, additional effort will be required to both create and maintain the UI.
Prerequisites#
Before going through this section, ensure that you have done the following:
Deployed your Tokkio application.
Deployed the reference UI to make sure that everything is working end to end. You should be able to see and hear the avatar. Ensure that you are able to have a conversation with the avatar before starting.
Familiarized yourself with the reference UI codebase. Even if you will not use this code, some of it will likely be duplicated in your custom UI. Read the Creating a Custom Layout before following these steps.
This section is for advanced users only. To follow this section, you must be familiar with the VST, Ingress, and UI Server microservices, and have a working understanding of their interfaces. You must also be an experienced web developer.
Minimal Components for a UI to Work#
For a UI to connect to Tokkio and access all the features, it must implement the following:
Recurring call to Ingress to fetch session cookie
Create an instance of the VST Streaming Library for the avatar video connection
Connect to the UI server over WebSocket for bi-directional communication with the Tokkio deployment (use to get all of the Tokkio features, not required to see avatar video)
To get the session cookie from Ingress, make the following API call: GET http(s)://<ingress endpoint>/token
. This API will return a response that looks like this: {"token_ttl":"30","token":"77234b6a-89e0-4b37-8bb7-c33e0d2a7183"}
. Take the token and add it as a cookie to all subsequent requests to the Tokkio deployment. The cookie should be of the following format: Session=77234b6a-89e0-4b37-8bb7-c33e0d2a7183
. To keep your session alive, call this endpoint every 15 seconds, and ensure that the session cookie is present in these subsequent API calls.
Once you have your session cookie, initialize the VST connection via the VST Streaming Library. See the code in the reference UI (src/features/avatar-window
directory) for an example of how this is done. The vstWebsocketEndpoint
parameter should be set to ws(s)://<ingress endpoint>/vms/ws
.
Next, make a WebSocket connection to the Tokkio UI server through Ingress. The connection endpoint is ws(s)://<ingress endpoint>/ws
.
Now that you have made the necessary connections, you should be able to see the avatar video and communicate with the UI server. For any additional features, please refer to the reference UI for how these should work.