Architecture#
The Tokkio system is an integration of multiple pipelines, each of which consists of multiple microservices. Tokkio is a typical event-driven system facilitated by a message bus. The streaming pipeline drives the lifecycle of a Tokkio transaction by firing events when a WebRTC stream has been connected or disconnected. Subsequently, all pipelines start running in parallel and serve the incoming requests from the user.
The diagram below depicts an overview of the interconnections between various microservices within Tokkio system. Each component within Tokkio is modeled as a microservice, giving it the flexibility for changes or customizations as suited.
Tokkio can be divided into 6 pipelines: Streaming, Vision, Speech, Interaction, Fulfillment, and Animation.
Streaming Pipeline
The streaming pipeline facilitates seamless video and audio communication between a user’s webcam and the Tokkio backend, and subsequently from the Tokkio backend to the Tokkio UI. This process begins with a web client, specifically supported by Google Chrome, where the Tokkio UI is rendered. Users must grant permissions for both camera and microphone access to initiate the pipeline. Once permissions are granted, video and audio data are transmitted bidirectionally between the user’s device and the Video Storage Toolkit (VST) using the WebRTC protocol. This transmission occurs after an initial signaling procedure is completed via a REST API at the ingress gateway. To ensure reliable data transmission, especially in environments with stringent security measures that might block direct connections, a reverse proxy or TURN server is employed.
Vision Pipeline
The vision pipeline processes video streams in real-time, providing immediate analysis of user presence and attention levels. It generates vision alerts to notify relevant systems or users based on detected activity and engagement, ensuring timely and accurate responses to user interactions.
Speech Pipeline
The Chat Controller Action Server is a sophisticated audio inference microservice designed to operate in real-time. It utilizes NVIDIA’s Riva technology to perform Voice Activity Detection (VAD), Automated Speech Recognition (ASR), and Text-to-Speech (TTS). This microservice seamlessly integrates with the system by providing events that comply with the UMIM (Unified Multimodal Interaction Management) standards, ensuring compatibility and efficient communication across different components.
Interaction Pipeline
The interaction management system orchestrates the avatar’s responses by intelligently interpreting and reacting to user events detected through vision and speech analysis. This dynamic process ensures that the avatar engages in meaningful and contextually appropriate interactions.
Fulfillment Pipeline
The fulfillment pipeline is designed to facilitate interactions with third-party applications or APIs via a REST interface, as detailed in the Tokkio Plugin Server documentation (Plugin Server). This component seamlessly integrates with the Tokkio UI through the UI Server, enabling direct communication and interaction. The pipeline is highly flexible and can be customized or entirely replaced to suit specific use cases, offering adaptability to meet diverse requirements.
Animation Pipeline
The animation pipeline is responsible for generating and animating avatars based on inputs from the interaction management system. It contains the Audio2Face-3D and the Animation Graph microservices. In addition, Tokkio offers multiple rendering microservices that can be switched out according to specific needs: