Components and Microservices#
Take a moment to read the diagram below. This diagram details a simplified view of the components that make up a typical Tokkio workflow.
Simplified conceptual diagram of the components that make up a Tokkio workflow.#
Below are the core components required for Tokkio to function. However, based on the target environment we choose, such as AWS or Bare-metal, we might need some additional components to make the end-to-end pipeline work. For example, when deploying Tokkio on AWS, we must set up an Application Load Balancer to access the Tokkio backend in a secured way. This Application Load Balancer is not needed, when we deploy Tokkio with Bare-metal machines.
Client#
A browser application on an end user’s device, such as mobile or laptop where we interact with the application.
Tokkio UI#
A web application UI comprising necessary components to interact with the Tokkio backend. More details about the Tokkio UI are covered in the Tokkio UI section.
Tokkio Backend#
Tokkio backend is deployed as a Helm chart in a Kubernetes environment. In addition to Kubernetes, the backend machine consists of the software stack below.
Media Relay Service#
This component is responsible for relaying media between the client browser and the Tokkio backend. Especially when the client browser and the Tokkio backend cannot connect directly, they send data to an entity such as a TURN server, which then relays this data to the other device. This process ensures that communication can occur even if the direct connections are blocked for reasons such as when they are behind NAT or firewalls. The Tokkio deployment supports the options below as of today.
COTURN service
ReverseProxy service
Twilio
LLM RAG / NVIDIA NIM#
This component helps Tokkio backend get responses for User queries. While NVIDIA NIM comes configured with every default installation, you can easily replace it with your own LLM RAG.
Microservices#
See below for a complete list of the microservices that are deployed in a Tokkio workflow.
Microservice |
Description |
Stateful/Stateless |
|---|---|---|
A specialized load balancer for Tokkio deployment |
Stateless |
|
Audio-video stream provider |
Stateless |
|
Pipecat based orchestrator of the avatar pipeline which drives the interaction and integration |
Stateful |
|
Riva Speech Skills |
Imparts NLP (Natural Language Processing), TTS (Text to Speech) and ASR (Automatic Speech Recognition) related features to Tokkio pipeline |
Stateless |
NVIDIA LLM or NVIDIA enterprise RAG blueprint |
LLM or RAG provider deployed outside Tokkio by default |
Stateless |
Accepts streaming audio, and injects facial expression into bot animation |
Stateless |
|
Animation graph compiles and buffers the animation data provided by all the animation sources and send the current chunk of audio and the current avatar pose/frame to the renderer |
Stateful |
|
Unreal-based renderer as an alternative to the default Avatar Renderer |
Stateful |
|
Provides a way to distribute media streams to the individual pods and responsible for the routing and stream state management. A separate instance of SDR is created for each stateful microservice |
Stateless |
|
Redis Timeseries |
Message bus used to facilitate inter-service communication, and temporarily store timeseries data |
Stateless |
Redis |
Used as a cache to store session stats and other routing related information. |
Stateless |
Used for easy developer workflow to allow making changes to a running ACE pipeline |
Stateless |