UI Server

The UI server provides a layer of segregation between the web UI and all the backend logic of the Tokkio App pipeline. With this approach, Minimal or no web UI change would be needed for the future evolution of the pipeline logic and its components as long as the web UI adheres to the API standards and requirements.

The overall architecture and the pipeline components are modularized with efficient connections between each other. More details about the interconnections between each component to the UI server is explained along with the logics and requirements behind them.

Connections Between Microservices and UI Server

REST API and gRPC

  • UI server establishes gRPC connection with ACE Agent Chat Controller to receive ASR transcriptions and subsequently provide this info to UI for rendering.

  • UI Server calls Cart API to retrieve and manipulate cart content based on the user interactions.

  • UI Server calls Menu API for all menu related queries, and UI server calls Cart Manager for all cart management queries. For user feedback analysis

  • [Optional] UI Server calls VMS API to handle streaming related operation and WebRTC connection establishment

Redis Messages (Incoming and Outgoing)

  • To detect FOV entry/exit, UI server listens to the “UserPresence” Redis message published by UMIM, and a session id would be provided for the corresponding stream id that the FOV event took place. This session id is required to make subsequent API calls to the UI Server.

  • UI Server listen to VMS camera remove event to clear stale or disconnected stream from its internal cache

  • UI Server subscribes to SDR error reporting event to monitor POD failure and notify UI to render the error to user

  • UI Server provides an API to trigger FOV entry / exit programmatically, once the API is called, a FOV event Redis message will be published by the UI server to simulate a FOV event.

UI and UI Server Websocket Connection and API Interaction

UI calls Ingress REST API to obtain session token. Only a set number of sessions would be allowed at a given time, and the session management is governed by Ingress. Ingress relays all UI Server API calls from UI to the UI Server after verifying the session token. Any REST API provided by UI Server would be called in this manner to ensure single point of entry.

In addition, UI calls VMS API to establish WebRTC connection via TURN server. It also obtains stream id from VMS API call and use it for the websocket connection request to UI server.

At the same time, Tokkio UI will initiate websocket connection to the UI server, through Ingress, to establish a channel for the two ways communications. UI provides the stream id it receives from VMS and to make websocket connection request, and Ingress relays WebSocket connection requests from UI to UI Server after verifying the session token.

When a FOV entry is detected from UMIM’s Redis message, UI server will receive a session id provided by this message. UI server will then send the session id along with the action through websocket message to instruct UI to display the home view and start the session.

On the other hand, when an FOV exit is detected from UMIM’s Redis message, UI server will send the corresponding websocket message to instruct UI to update the rendering accordingly to end the session

All the ASR transcripts are transported to the UI for rendering through websocket messages. Speech controlled UI renderings are also communicated through websocket.

There is a new API available which allows user to render custom view with predefined custom components based on the JSON payload provided in the API call.

UI Server Source

Tokkio UI Server source can be downloaded from NGC. It includes the source for UI as well.

$ ngc registry resource download-version "nvidia/ucs-ms/tokkio_ui_source:4.0.4"

UI Server APIs

There are total of six sections from the Tokkio UI Server REST API:

  • Cart (Cart Manager MS)

  • Menu (Menu MS)

  • VMS (VST MS)

  • View (Speech and UI Interaction, Custom View)

  • Health (Health Check)

  • Manage (FOV Entry / Exit)


See the API doc for the complete list of APIs.

Error Reporting

SDR agent provides notification to the user through web UI when there is an issue on the Backend. When issue is detected (e.g. POD crashes), the kubectl would generate a Redis message which the UI Server subscribes to, and the UI server would then construct and convey the message to the UI for rendering to the user for suggestions on possible actions that can be taken from the client side.

Below demonstrates an example of the error message notification showing on the UI page

Connection between Microservice and UI Server