Overview
Tokkio is an interactive avatar virtual customer service assistant product SDK. It can be adopted for virtual assistant use case in any domain. The face of the product is avatar. The avatar can see using camera and perceive the situation using vision AI. It can interact intelligently using AI. The SDK includes a reference app, an end-to-end pipeline that integrates the following microservices:
Microservice |
Functionality imparted in Tokkio |
Functional Category |
|---|---|---|
VMS |
The Video Management Service (VMS) is an audio-video stream provider |
Media Service |
Avatar Renderer |
The Avatar Renderer is used for rendering bot animation |
Graphics and Rendering |
Audio2Face |
Accepts streaming audio, and injects facial expression into bot animation |
Graphics and Rendering |
Animation Graph |
Animation graph compiles and buffers the animation data provided by all the animation sources and send the current chunk of audio and the current avatar pose/frame to the renderer |
Graphics and Rendering |
UMIM (Unified multimodal Interaction Management) |
UMIM service accepts events from vision, audio, and UI pipeline and instructs the animation pipeline and UI server to perform certain actions |
Graphics and Rendering |
Chat Engine |
Accepts textual query, events, multimodal metadata and generates detailed bot response which can be used to drive TTS, avatar, UI visualization. |
Speech AI |
Chat Controller |
Accepts realtime audio, passes the audio through bot-pipeline and returns synthesized speech to downstream |
Speech AI |
Riva Speech Skills |
Imparts NLP (Natural Language Processing), TTS (Text to Speech) and ASR (Automatic Speech Recognition) related features to Tokkio pipeline |
Speech AI |
SDR (Stream Distribution & Routing) |
Provides a way to distribute media streams to the individual pods and responsible for the routing and stream state management. A separate instance of SDR is created for each stateful microservice |
Foundational / Scaling |
Tokkio Cart Manager |
Part of a reference app for Quick Service Restaurant (QSR). Used for user cart management |
App-Specific |
Metropolis Occupancy and Alerts |
Detection of a user within the Field Of View (FOV) |
AI Analytics |
Metropolis Occupancy Alerts API |
WebAPI for Metropolis Occupancy Alerts |
Web API |
Tokkio Ingress Controller |
A specialized load balancer for Tokkio deployment |
App-Specific |
DS Vision AI |
DeepStream Vision AI component imparts Face Detection and Tracking |
Vision AI |
Tokkio Menu API |
Part of a reference app for Quick Service Restaurant (QSR). Used for menu database management |
App-Specific |
Tokkio UI Server |
Part of a reference app for Quick Service Restaurant (QSR). Web UI Server for Tokkio |
App-Specific |
Plugin Server |
Part of a reference app for Quick Service Restaurant (QSR). Handles food ordering for QSR bots by interacting with Menu database, Cart Manager and UI Server |
App-Specific |
MongoDB |
Data storage used to persist information from various components |
Storage |
Redis Timeseries |
Message bus used to facilitate inter-service communication, and temporarily store timeseries data |
Message Bus |
Redis |
Used as a cache to store session stats and other routing related information. |
Cache |
For deployment, the SDK includes reference helm chart to deploy in Azure, AWS and GCP. Once you are onboarded with NGC, you will be able to generate API keys, with which grants you access to all the Tokkio software components.
Prerequisites
To deploy the Tokkio helm chart user needs an NGC account (ngc.nvidia.com)
Obtain an NGC API key by following the set up instruction provided in ngc.nvidia.com
NVAIE access to ACE is required. Please contact NVIDIA to request NVAIE access to ACE.