UI Functionality Overview

Initialization

The UI starts by retrieving a session token, using an IDP system. Once the UI has the session token, it will generate a random UUID to be used as the stream ID. It will start a WebSocket connection with VMS using this connection ID. After making WebSocket connection to VMS, it will start a WebRTC connection with VMS which is used to send audio and camera streams to the avatar, and receive the avatar video in a single WebRTC peer connection. With this, the UI will also use the connection ID to connect to the UI server WebSocket. The UI server uses this to associate which UI is sending which camera stream.

Once Tokkio receives an fovEntry (user presence event), the UI will display the catalog, and the user’s session will begin, opening up the following interactions to the user.

User Presence and Attention

The UI displays an indicator for the user attention state. The default value for this indicator is NOT PRESENT. When a user presence is detected, this event is received by the UI server and send to the UI, which updates the state to PRESENT. Similarly, when a user’s attention status changes, this is also received by the UI which updates the state indicator to the received user attention state. If the user’s attention state is disabled, the UI falls back to the PRESENT state. Finally, when the user leaves the field of view of the camera, this state is set back to NOT PRESENT. This indicator provides a way for the user to understand the effect of their head position on the conversation.

Catalog Interactions

Catalog interactions are only available for the Retail UI. When the UI first connects to WebSocket, it will immediately make an API call to GET /menu/category, and display the list of menu categories returned from this call to the user. When the user selects a category, a call is made to GET /menu/category/{category}, which displays each of the catalog items in this category. When the user selects a catalog item, a third API call is made, this time to GET /menu/items/{itemId}. This gives information about a specific item, and is used to display the item page to the user.

Cart Interactions

Cart interactions are only available for the Retail UI. In the UI, all of the information about what is in a user’s cart is stored in the front end in a redux store, named “cart”. Every time the user updates their cart, either through touch or voice, the store is updated with the most recent cart, using the GET /cart API call. The UI will update the cart based on a voice interaction when it receives a WebSocket message with the field "is_update_cart" set to true. The user can add cart items from the ‘Item’ page (or through voice), and can further modify their cart from the ‘My Order’ page.

View Interactions

View interactions are only available for the Retail UI. When the user navigates around the touch menu, we update the UI server with what the UI is displaying at a given moment, using the PUT /view/items/{id} call. This is done to notify the bot of what the user is looking at, giving context to a user’s query.