UI Server#

The UI server is a controller layer that sits between the web UI and all the backend logic of the Tokkio App pipeline. With this approach, minimal or no web UI change should be needed for the future evolution of the pipeline logic and its components as long as the web UI adheres to the API standards and requirements.

The overall architecture and the pipeline components are modularized with efficient connections between each other. More details about the interconnections between each component to the UI server is explained along with the logics and requirements behind them.

Connections Between Microservices and UI Server#

REST API and gRPC#

  • UI server establishes gRPC connection with ACE Agent Chat Controller to receive ASR transcriptions and subsequently provide this info to UI for rendering.

  • UI Server forwards calls to the Cart Manager API to retrieve and manipulate cart content based on the user interactions.

  • UI Server forwards calls to the Catalog RAG RAG for all catalog related queries.

  • UI server keeps track of the view currently shown in the UI, to give context to the backend language models for when a user asks a question about something displayed on their screen.

Redis Messages (Incoming and Outgoing)#

  • To detect FOV entry/exit, UI server listens to the “PresenceUserActionStarted” and “PresenceUserActionFinished” Redis messages published by UMIM. A session id would be provided for the corresponding stream id that the FOV event took place. This session id is required to make subsequent API calls to the UI Server.

  • UI Server listen to VMS camera remove event to clear stale or disconnected stream from its internal cache.

  • UI Server subscribes to SDR error reporting events to monitor POD failure and notifies the UI to render the error to user.

  • UI Server provides an API to trigger FOV entry / exit programmatically, once the API is called, a FOV event Redis message will be published by the UI server to simulate a FOV event.

  • The UI server listens to “AttentionUserActionStarted”, “AttentionUserActionUpdated”, and “AttentionUserActionFinished” Redis events, and forwards the user attention state from these events to the UI.

UI and UI Server WebSocket Connection and API Interaction#

The UI calls the Ingress REST API to obtain the session token. Only a set number of sessions are allowed at a given time, and the session management is governed by Ingress. Ingress relays all UI Server API calls from UI to the UI Server after verifying the session token. Any REST API provided by UI Server would be called in this manner to ensure single point of entry.

Tokkio UI will also initiate WebSocket connection to the UI server through Ingress to establish a channel for communicating events to the Tokkio UI. UI provides the same stream id it uses to connect to VST and to make WebSocket connection request, and Ingress relays WebSocket connection requests from UI to UI Server after verifying the session token.

When an FOV entry event is detected from UMIM’s Redis message, UI server will receive a session id provided by this message. UI server will then send the session id along with the action through a WebSocket message to instruct UI to display the home view and start the session.

Similarly, when an FOV exit event is detected from UMIM’s Redis message, UI server will send the corresponding WebSocket message to instruct UI to update the rendering accordingly to end the session.

All the ASR and TTS transcripts are transported to the UI for rendering through WebSocket messages. Speech controlled UI renderings are also communicated through WebSocket.

There is an API available which allows user to render custom view with predefined custom components based on the JSON payload provided in the API call.

UI Server Source#

Tokkio UI Server source can be downloaded from NGC. It includes the source for UI as well.

$ ngc registry resource download-version "nvidia/ace/tokkio_ui_server_source:5.0.1"

UI Server APIs#

There are total of five sections from the Tokkio UI Server REST API:

  • Cart (Cart Manager MS)

  • Menu (Catalog RAG MS)

  • View (Speech and UI Interaction, Custom View)

  • Health (Health Check)

  • Manage (FOV Entry / Exit)

See the API doc for the complete list of APIs.

Error Reporting#

SDR agent provides a notification to the user through web UI when there is an issue on the backend. When an issue is detected (e.g. POD crashes), a Redis message describing the error is generated, which is picked up by the UI server, which then constructs and conveys the message to the UI for rendering to the user for suggestions on possible actions that can be taken from the client side.

Below demonstrates an example of the error message notification showing on the UI page:

UI Error Reporting Message
On this page