Tokkio QSR Application Overview

Introduction

Tokkio Quick Service Restaurant (QSR) reference application acts as an example for one of the many applications where Tokkio can be deployed. This application can then be customized based on the end user’s requirements. This is ensured by the microservice based UCS deployment on which Tokkio is built. It ensures that any application specific customization can be containerized to only a few microservices. An end user can interact with the Tokkio QSR application through a combination of both speech and touch. Details about the architecture and the individual application specific modules used in this application are presented in the following sections.

Architecture

The figure below provides an overview of the microservices used in Tokkio QSR and their interactions. Application specific microservices used in Tokkio QSR includes: ACE Agent Plugin Server, Menu, Cart Manager, UI server and UI front end. These application dependent micro services are orchestrated through ACE Agent and Chat Controller microservices.

All the micro-services within Tokkio QSR interact with each other over HTTP through REST APIs. The user’s input query is passed onto ACE Agent which then orchestrates the ACE Agent plugin (and eventually the rest of the micro-services) to fulfill the request.

ACE Agent QSR Plugin

The QSR plugin for the ACE Agent plugin server is responsible for formulating required metadata and events for driving the conversation and the behavior of the bot. It interacts with different microservices to understand the intention of the user’s query, extracting necessary information about the items and finally formulating necessary information to control the bot behavior.

The ACE Agent plugin microservice takes inputs from the below microservices when required based on the user query, to formulate different kinds of metadata. The input sources of the plugin server are:

ACE Agent microservices - Plugin module receives the query from ACE Agent.
Menu microservice - Plugin module calls the APIs exposed by menu microservice to fetch all the details associated with an item like sizes and price. These details are used to validate user requests as well as formulate required metadata for clarifications if needed.
Cart Manager microservice - Plugin module calls the API exposed by cart manager to get current state of user cart. Example: Add cheeseburger to my cart.
User Interface Server - Plugin module calls the API exposed by UI server for servicing queries related to UI. Examples: What entrees do you have?.

On taking inputs from these microservices, the ACE Agent plugin module formulates below metadata which are sent out to ACE Agent microservices and Cart Manager microservices. The metadata includes:

Information about different actions on a user cart like adding an item, removing an item, replacing an item etc. is sent out to APIs exposed by the Cart Manager.
Information about different actions on User Interface when a user speaks out certain requests like showing all sides on screen, displaying the cart, going back to the previous page etc. to APIs exposed by the UI Server.
Information to formulate the spoken responses of the Avatar to ACE Agent.

Cart Manager

The entire cart management for an active user is handled by the cart manager. It is a light weight service whose primary function is to update a user cart and communicate with the ACE Agent Plugin and UI regarding the updates. Main responsibilities of CM include:

Storing and retrieving user cart related data.
Communicate with the Menu service to get information regarding the items to be added to the cart.
Send out order details over Redis stream
The Cart Manager requires a healthy MongoDB instance to run. It uses MongoDB to store user cart information.

Menu

Menu API provides a HTTP API that interfaces with the backend menu DB. Menu API is designed to decouple the query logic from the backend solution. Currently, we leverage MongoDB as the persistence solution for all menu items, and it can be swapped out with other technologies if the need arises.

Menu API at launch downloads a menu from a remote repository as specified by the environment variable: NGC_MENU_REGISTRY in the tokkio-app-params.yaml file.
Menu API supports a wide range of searching and filtering capabilities on menu items, such as REGEX pattern matching, logic chaining, ordering, etc.
Menu API allows users to batch export and import menu content on the fly through API calls (modifications to the menu will not be persisted between restart).
Menu API requires a healthy MongoDB instance to run.

UI server and UI Front End

UI Server acts as an abstraction layer to decouple the presentation layer and all the backend AI microservices logics; hence, it provides the flexibility to integrate customized UI front-end seamlessly and minimize necessary code changes. The main features of the UI server include:

APIs for managing (adding/removing/viewing) cart items from the Cart API
APIs for providing menu contents from the Menu API
APIs for Ace Agent Plugin to manipulate UI rendering for speech input
APIs for UI Front End to communicate touch inputs for menu navigation ad food ordering
APIs for UI Front End to register information of the current view for ACE Agent Plugin (or any other MS) on demand gRPC communications with ACE Acent Chat Controller to receive ASR and forwarding to UI Front End for rendering
APIs for rendering custom views on the UI by providing JSON request payload
Redis events monitoring for FOV entry/exit, Camera add/remove, and error reporting
Multi-channel logging (file, console, etc.) with rotation
HTTP and HTTPS supported

UI front end presents the menu items in a user-friendly manner. It enables the user to interact with the UI through touch and speech inputs. It communicates with the UI server through well-defined REST APIs and web socket connection. The front end UI features includes:

Showing categorized menu items
Navigating through different categories
Adding-removing items to cart and display the cart
Stream the animated Tokkio avatar using WebRTC protocol
Display ASR messages using web socket connection
Support audio-only pipeline to use Tokkio app without camera
Render custom view components based on the provided JSON payload
Production build configuration and dockerized image

Tokkio Interaction Flow

Touch Based Interactions

A typical workflow for a touch-based interaction for Tokkio with respect to the QSR apps is shown in the figure below. When a user interacts with touch, the UI registers the request and acts on the user intent of browsing the menu (display updates) or adding items to the cart (cart updates). For display updates, the UI server simply queries the items it needs to display and communicates with the UI client to update the display accordingly.

For interactions involving cart updates, the UI server communicates with the Cart Manager. The CM in turn retrieves the session related cart information from the datastore and updates the cart. The display is then updated accordingly. Note that a session identifier is used for all communications between the QSR components to ensure that the retrieved and the updated session information is relevant.

Speech Based Interactions

A typical workflow for a speech-based interaction for Tokkio with respect to the QSR apps is shown in the figure below. When a user interacts with speech, the ACE Agent Plugin passes down the relevant request to the Cart Manager for any cart related updates or the UI server for any display related updates. In this case too, the CM retrieves the session information from the datastore and updates the cart. The display is then updated accordingly.

Session Identification

A unique token associated with a user session is used for all communication between QSR apps. The sessionId is generated by the Chat Controller. It identifies the dialog session from a FOV entry trigger to dialog completion (FOV Exit or browser closure) to maintain a user context. Note that the connectionId passed down to the UI server from the ACE Agent plugin for starting as well as stopping the session is an id associated with the incoming video stream.