Tokkio 3.0#

Notable updates#

Complete refactoring of the animation and rendering pipeline into microservice components to be easily swapped out on customer demand
Improve rendering latency and avatar facial expression and gesture

NVIDIA NeMo Guardrail and Colang-based dialog management and guard-railing support
Support for plugging in custom Pythonic dialog management code
NvBotMaker CLI tool for building your bot in a native Python environment
Docker compose-based deployment support
Deployment support on Kubernetes environment
Integration with NVIDIA NeMo LLM and NeMo Inform EA services
Integration with NVIDIA Riva translation
Standalone NLP server with dedicated endpoint for common NLP tasks
Connectors for custom NLP model and retrieval pipelines
Standalone Fulfillment server with ability to support custom endpoints
Conversion script and guidance for porting previous release-based bots to Colang
Concurrent deployment of multiple bots with different configs
Support for plugging in 3rd party TTS pipeline
OpenAI LLM backend support on QSR bot on previously difficult queries
Add support for nemollm inference server engine in dialog manager
Automatic Speech Recognition improvements
Moved to a new bot architecture based on Colang to provide better control
Various bug fixes

Introduce Reverse Proxy as a replacement for the co-turn server to simplify the Infrastructure.
Support 1-click deployment script on OCI (Oracle Cloud Infrastructure)
Support more types of GPUs: T4, A10, L4 (subjected to cloud vendor availabilities)
Base OS now is Ubuntu 22.04.3
UCF 2.0

Tokkio Reference Application

Some menu related queries with speech like “Can I have something without onions”, “What are the non vegetarian options available” might give inaccurate results.
LLM prompt tuning might be required to tune some of the responses appropriately for menu, cart, IR or open domain related queries. Non-optimal results observed more frequently with NemoLLM/gpt-43b-905 model compared to OpenAI text-davinci-003.
Some queried items don’t show images as the image location was pointing to “default.png” which did not exist.
Exit message of “Thanks for visiting, goodbye!” is played when the user exits the camera view with or without placing the order.
Adding/removing some toppings to items via speech might not work correctly.
Items and topping replacement might give inaccurate results
Longer response times for queries that require LLM involvement
Intermittent FOV exits observed

BotMaker

Sub-intent classification is not enabled, leading to queries like “repeat order” and “repeat last response” not distinguished clearly by the intent-classification model.
Memory leak and pod-restart observed after prolonged use
It is recommended to return string responses instead of numeric values in the Fulfillment server when there is a possibility of 0 as the response. This is because BotMaker interprets 0 and empty strings as None when getting values from the Fulfillment server, which can lead to unexpected behavior.

MongoDB and redis-timeseries

Increase in memory usage over time when the system is in actively used for a long duration

OV-Renderer

ML-Ops

General

Tokkio deployment must be restarted after restarting the cloud instance used for deployment
After several hours of deployment, the users can observe that the bot state on Tokkio UI is stuck in initializing state OR the Avatar is not visible on UI. When this happens, inside the vst pod or ds-vision pod, the nvidia-smi command will give error “Failed to initialize NVML: Unknown Error”. To overcome this issue, the user can bounce the vst pod and wait for sometime for the issue to get resolved; user can also try to restart ds-vision, ov-renderer pod after sometime if the issue is still observed.