NVIDIA ACE

NVIDIA ACE is a suite of real-time AI solutions for end-to-end development of interactive avatars and digital human applications at-scale. Its customizable microservices offer the fastest and most versatile solution for bringing avatars to life at-scale, based on NVIDIA’s Unified Compute Services, full-stack AI platform and RTX technology. Leverage ACE to seamlessly integrate NVIDIA AI into your application, including Riva for speech and translation AI, Voice Font for transferring your voice to a digital avatar, NeMo LLM for natural language understanding, and Audio2Face, Audio2Emotion, Animation Graph, Omniverse Renderer or Live Portrait for AI powered 2D and 3D character animation. Or, further optimize for cloud scale with microservices and domain-specific AI workflows like NVIDIA Maxine, NVIDIA Tokkio, Animation Pipeline, and NVIDIA Kairos, to deliver advanced avatars across video conferencing customer service use cases and gaming.

Release Notes

• ACE Release Notes
  • 24.06 Release
    • NIMs for Digital Human and ACE Microservices
      • NVIDIA AI Enterprise Supported
        • Riva ASR 2.15.1
        • Riva TTS 2.15.1
        • Riva NMT 2.15.1
        • Audio2Face 1.0.11
        • Omniverse Renderer Microservice 1.0.1
        • Animation Graph Microservice 1.0.1
        • ACE Agent 4.0.0
      • Early Access Microservices
        • SpeechLivePortrait 0.1.0
        • Nemotron 4.5B SLM 0.1.0
        • VoiceFont 1.1.1
    • ACE Reference Workflows
      • Customer Service Workflow
        • New Reference Components
      • Game Character Workflow
        • New Reference Components
        • New Samples
    • Tools
      • UCS Tools 2.5
      • Avatar Configurator 1.0.1

Getting Started

• First Steps
  • Find Resources
  • Try Out ACE
  • Development Setup
  • Workflows
• Development Setup
  • Prerequisites
  • Install NVIDIA driver
  • Install NGC and set API key
  • Install UCS Tools
  • Install Docker
  • Login to the nvcr.io container registry
  • Install NVIDIA container toolkit

Workflows

• Animation Pipeline
• Tokkio
  • Tokkio Getting started
    • Release Note
      • Documentation updates
        • Tokkio 4.0
          • Notable updates
            • Tokkio LLM app
            • Tokkio QSR app
            • Animation and Rendering
            • VMS
            • UMIM
            • UI and UI server
            • ACE agent
            • RIVA
            • SDR
            • Architecture
            • Security
            • MLOPS
          • Known Issues
        • Tokkio 3.0.1 patch
        • Tokkio RAG
        • Tokkio 3.0
          • Notable updates
            • Animation and Rendering
            • BotMaker
            • Vision AI & EMDX
            • MLOps
            • Infrastructure
            • Miscellaneous changes
            • Known Issues
        • Tokkio 2.0
          • Notable updates
            • Reference applications and Customization
            • Scaling
            • BotMaker
            • MLOps
            • K8s Cluster Logging
            • Stream Distribution and Routing
            • Easy Deployment
            • Miscellaneous changes
            • Updating to Tokkio 2.0
        • Documentation update: 1.5.3
        • Documentation update: 1.5.2
        • Known Issues
    • Acronyms and Terms
    • Overview
    • Prerequisites
    • Tokkio Architecture
      • Overview
      • Deployment & Scaling
        • Types of Microservices
        • Communicate Pattern & Message Bus
        • Scaling and Dynamic Routing
        • Deployment
          • Limitation
    • Quickstart Guide
      • UCS Tool
      • Building the Tokkio App With UCS Tool
        • Prerequisites
          • Steps: Sync UCS registry and Download Tokkio App
          • Steps: Using UCS Studio
          • Steps: Using UCS CLI
      • User Guide
        • Session Management
        • Audio & Video
        • Bot Interaction
        • Customization
  • Reference Applications
    • Tokkio LLM-RAG
      • Introduction
      • Quickstart
        • Step 1. Deploy the RAG server
        • About the Nvidia Enterprise RAG
        • About the Nvidia Generative AI API catalog RAG
        • Step 2. Connect the RAG and deploy Tokkio
      • Customizing the Pipeline
        • Step 1. Get the Tokkio-LLM-RAG ACE bot
        • Step 2. Customize the Tokkio-LLM-RAG ACE bot
        • Introduction to the Tokkio LLM-RAG ACE bot
        • Connect to an LLM or RAG
        • Gesture generation
        • Interruptions
        • Small Talk
        • Proactive bot
        • Publish the bot
        • Step 3. Connect a custom ACE bot to Tokkio-LLM-RAG
    • Tokkio QSR Application Overview
      • Introduction
      • Architecture
        • ACE Agent QSR Plugin
        • Cart Manager
        • Menu
        • UI server and UI Front End
      • Tokkio Interaction Flow
        • Touch Based Interactions
        • Speech Based Interactions
        • Session Identification
  • CSP Setup Guides
    • AWS
      • AWS CSP Automated Setup Guide
        • Overview
          • Infrastructure Layout
          • Prerequisites
        • Prepare deployment config
          • Download & Extract Deployment Artifact
          • Prepare secrets
          • Prepare deploy template
        • Setup logs backup
        • Deploy infrastructure and application
          • Known Issues
        • Verify Deployment
        • Teardown infrastructure and application
    • Azure
      • Azure CSP Automated Setup Guide
        • Overview
          • Infrastructure Layout
          • Prerequisites
            • Login to the Azure portal
            • Service Principal Setup
              • App Registration Service Principal
              • Subscription Access grant to App Registration
            • Resource Group
            • Deployment State Storage
            • Base Domain
            • Certificates
              • Key Vault
              • Wildcard Certificate
            • Increase quota for GPU VM type
        • Prepare deployment config
          • Download & Extract Deployment Artifact
          • Prepare secrets
          • Prepare deploy template
        • Setup logs backup
        • Deploy infrastructure and application
          • Known Issues
        • Verify Deployment
        • Teardown infrastructure and application
    • GCP
      • GCP CSP Automated Setup Guide
        • Overview
          • Infrastructure Layout
          • Prerequisites
            • Login to the GCP Console
            • IAM & Admin Setup
              • Service Account
            • Deployment State Storage
            • Base Domain
        • Prepare deployment config
          • Download & Extract Deployment Artifact
          • Prepare secrets
          • Prepare deploy template
        • Setup logs backup
        • Deploy infrastructure and application
          • Known Issues
        • Verify Deployment
        • Teardown infrastructure and application
  • Tokkio Customization
    • Tokkio - Customization
      • Customizing a microservice within Tokkio
      • Advanced Customization: Adding a new microservice to Tokkio
      • Hosting microservice & artifact
      • Vision AI Customization
      • UI Customization
      • Menu Customization
      • Avatar and Avatar Scene Customization
    • Tokkio Avatar Customization
      • Update the UCS Application
  • Software Modules
    • Video Storage Toolkit
      • Video Storage Toolkit
        • Introduction
        • Features
        • VST Architecture
        • UI interaction
        • API Guide
        • VST Configuration
    • Vision and AI analytics
      • Disable Vision AI
        • UI frontend modifications
        • Tokkio backend modifications (optional)
          • Method 1: Using UCS Studio
          • Method 2: Manual
      • Face Detection and Tracking
        • Overview
        • Usage
      • Metropolis Occupancy Alerts
        • Overview
          • Design
            • Input
            • Output
        • Metropolis Occupancy Alerts functional modules
        • FOV
        • FOV Occupancy Alerts:
        • Role of Metropolis Occupancy Alerts in Tokkio App
          • Vision trigger to start and stop a conversation
        • How to configure Metropolis Occupancy Alerts for Tokkio
          • Configure FOV and Occupancy Alerts
            • Alert rule attributes
            • Helm installation time values overrides
          • Configure redis stream for Metropolis Occupancy Alerts
            • Redis Stream parameters
    • UI Server
      • UI Server
        • Connections Between Microservices and UI Server
          • REST API and gRPC
          • Redis Messages (Incoming and Outgoing)
        • UI and UI Server Websocket Connection and API Interaction
        • UI Server Source
        • UI Server APIs
        • Error Reporting
    • UI
      • UI
        • Introduction
        • Deployment Options
          • Deploy with production build
        • UI Flow
          • Configuration
            • Menu interactions
            • Cart interactions
            • View interactions
    • Ingress
      • Tokkio Ingress
        • Architecture
        • Usage
          • Params
          • Connections
        • Tokkio Ingress Source
        • Supported Platforms
    • Stream Distribution and Routing
      • Stream Distribution & Routing
        • Overview
        • Configuring the Agent
    • Menu
      • Menu Customization
        • Menu DB
          • Understanding Menu DB
          • Modifying Menu DB
            • Downloading DB
            • Adding an Item
            • Removing an Item
            • Modifying an item
              • Updating Menu Tag (Example)
          • Uploading Menu DB
          • Updating Menu Images
          • Updating Tokkio Deployment
            • When Menu DB is Updated
            • When Menu Images are Updated
          • Functionality Customization - Menu Schema, CM, Plugin Server and UI
      • Menu
        • API Guide
          • Path Table
          • Path Details
    • Cart Manager
      • Cart Manager
        • Cart Manager Source
        • API guide
    • Plugin Server
      • Plugin Server
        • Order-Food Plugin
        • Tokkio Food ordering bot
          • Colang
          • Order-Food - order_food.py
          • Intent Handlers
          • Access Endpoints
        • API Guide
    • UMIM Action Server
      • UMIM Action Server for Tokkio
  • Tokkio Troubleshooting
    • Troubleshooting Guide
• Kairos
  • Unreal Engine Integration
    • NVIDIA ACE 2.1 Plugin
      • Install the ACE Plugin
        • Prerequisites
        • Installing to a Packaged Engine
        • Installing to a Source Engine
        • Installing to a Source Project
        • Activate the Plugin
      • NVIDIA Audio2Face
        • Initial Editor Setup
        • Audio2Face Connection Setting
        • Add NVIDIA ACE Face Animation Support to MetaHumans
          • Bypass the Default MetaHuman MouthClose Animation
          • Update the ARKit Pose Mapping and Animation Sequence
        • Add ACE Audio Component to a Character
          • Spatial Audio (Optional)
          • Other ACE Audio Component Settings (Optional)
        • Import Speech Clips
        • Animating a Character from a Sound Wave Audio Clip
        • Animating a Character from a Local WAV File (Optional)
        • Adjusting Character Emotion (Optional)
        • Adjusting Audio2Face Parameters (Optional)
      • Troubleshooting
        • Failed to connect Error
        • Lip Corners Stretch Down Too Much in MetaHuman Facial Animations
        • Lips Don’t Open Wide Enough in MetaHuman Facial Animations
        • Project Settings Don’t Persist
      • Notes for Non-MetaHuman Characters
      • Upgrading from a Previous Plugin Version
      • Legacy Live Link Interface Documentation (Removal scheduled for a future release)
        • Basic Instructions in Unreal Engine
    • Kairos Unreal Sample Project
      • Quick Start Guide
      • MetaHuman
        • Adding New MetaHuman
        • [Optional] Audio Components
        • [Optional] Idle Animation
      • UI Overview
        • Actor Controls
          • Target Actor
          • Zoom To Character
          • Toggle Idle Animations
        • Audio Source
          • Example
          • Local File
          • Record
          • Play
      • Audio2Face
        • Settings
        • Emotion Controls
        • Tune Parameters

Microservices

• Animation Graph Microservice
• Omniverse Renderer Microservice
• Audio2Face
  • Overview
  • Quick start
    • Prerequisites
      • NVAIE access
      • NGC Personal Key
    • Local Container Quick Start
      • Dependencies
      • Running Docker Compose
  • Glossary of Terms
    • Common terminology
    • NVIDIA technology
    • 3rd Party Techonology
  • Migrating from EA3
    • Audio2Face Controller
      • Service Interface
      • Audio data
        • Audio header
        • Audio content
        • End of audio
      • Animation data
    • Audio2Face without controller
  • Architecture overview
    • Data flow
    • Default Ports
      • Docker Compose Default Configs
      • Kubernetes Default Configs
  • Audio2Face Microservice
    • Overview
    • Communication
      • Input
      • Output
      • Stream number
    • Configuration
  • Audio2Face Controller Microservice
    • Overview
    • Communication
      • ID management
    • Configuration
  • Sample application connecting to A2F Controller
    • Assumptions
    • Sending an audio clip
    • Results
  • Sample application sending audio to Audio2Face Microservice
    • Assumptions
    • Sending an audio clip
  • Unreal Engine Interaction
    • Using Unreal Engine with Audio2Face
  • gRPC through A2F Controller
    • Service definition
    • Service protobuf objects
    • Dependencies
  • gRPC directly with Audio2Face
    • Client side
      • Service
      • Protobuf data
    • Server side
      • Service
      • Protobuf data
  • Kubernetes Deployment
    • Quick deployment on microk8s
    • Microservice configs
  • Container Deployment
    • Prerequisites
    • Quick Deployment of Audio2Face and A2F Controller Microservices
    • Running Audio2Face Microservice
      • Microservice configs
      • Starting the container
      • Generating the models
      • Starting the service
    • Running A2F Controller Microservice
      • Microservice configs
      • Starting the container
      • Starting the service
  • Sharing Audio2Face Compute Resources
    • Sharing Audio2Face compute with a Renderer
  • Parameter Tuning Guide
    • Inference Parameters
      • Model default parameters
      • Deployment parameters
      • Runtime parameters
    • Emotion Post-processing Parameters
    • Notes
  • Network Customization and Accessibility
    • Audio2Face Docker-Compose Customization
    • Exposing Audio2Face Deployment from WSL
      • Opening a port in Windows Firewall
      • Rerouting Traffic to your WSL deployment
  • Notes
    • TRT Engine Generation
    • Resampling Audio Data
      • Resampling Guidelines
    • Known issues
    • Q&A
    • gRPC Response Status Code
  • Changelog
    • v1.0.0
      • SDK Versions
      • Features
• ACE Agent
  • Getting Started
    • Overview
      • Structure of this Document
    • Quick Start Guide
    • Release Notes
      • ACE Agent Release 4.0.0
        • Key Features and Enhancements
        • Compatibility
        • Breaking Changes
        • Limitations
        • Known Issues
      • ACE Agent Release 3.0.1 EA
        • Fixed Bugs
        • Known Issues
      • ACE Agent Release 3.0.0 EA
        • Key Features and Enhancements
        • Announcements
        • Limitations
        • Known Issues
  • Architecture
    • Architecture Introduction
      • Chat Engine Server Architecture
      • Plugin Server Architecture
      • ACE Agent Event Architecture
  • Deployment
    • Interface Introduction
      • CLI Interface
      • HTTP Server Interface
      • Event Interface
      • gRPC Interface
    • Docker Environment
      • CLI Interface
      • HTTP Server Interface
      • Event Interface
      • gRPC Interface
      • Event Interface with Speech
      • Model Deployment
    • Kubernetes Environment
      • Deploying a Bot via UCS Application
      • Building a Custom Helm Chart using ACE Agent Microservices
        • Building a Chatbot UCS Application
        • Building Speech Bot UCS Application
        • Building LangChain Bot UCS Application
        • Building Event Interface-Based Bot UCS Application
    • Python Environment
      • Chat Engine
        • CLI Interface
        • HTTP Server Interface
        • Event Interface
      • Plugin Server
      • Model Deployment
      • NLP Server
      • Clean Up
      • Python Native Application
    • Sample Clients
      • Web Sample Application
        • Text-Based Web Sample Application
        • Speech-based Web Sample Application
        • Launching the Web Sample Application with Certificates
      • gRPC Server Sample Client
      • Chat Python Sample Client
      • Event Sample Client
  • Tutorials
    • Building a Bot using Colang 1.0
      • Stock Bot
        • Creating a Bot
        • Specifying Bot Configurations
        • Specifying Colang Dialog Modeling
        • Testing the Bot
      • Adding a Plugin Service
      • Adding Speech to a Bot
    • Building LangChain-Based Bots
      • Creating the LangChain Plugin
      • Connecting the Plugin to the Chat Controller
      • Connecting the Plugin to the Chat Engine
        • Creating the Bot and Colang Configurations
        • Testing the Bot
    • Building a Bot using NVIDIA RAG Examples
      • Connecting Chat Controller to the RAG Plugin
      • Connecting the Plugin to the Chat Engine
        • Creating the Bot and Colang Configurations
        • Testing the Bot
    • Building a Bot using Colang 2.0 and Event Interface
      • Prerequisites
      • Step 1: Making the Greeting Multimodal
      • Step 2: Leveraging LLMs to Answer User Questions
      • Step 3: Making the Bot Proactive
      • Step 4: Interrupting the Bot
      • Step 5: Back-Channeling
    • Customizing a Bot
      • Selecting an LLM Model
      • Using an On-Premise LLM Model Deployed via NIM
      • Custom Actions
        • Creating a New Custom Action
        • Overriding a System Action
      • Using a Custom NLP Model
        • Deploying a Custom NLP Model with the NLP Server
          • Custom Model Integration using the @model_api Decorator
          • Custom Model Integration using the @pytriton Decorator
      • Customizing ASR Recognition with Word Boosting
      • Customizing TTS Pronunciation using IPA
      • Using 3rd Party Text-to-Speech (TTS) Solutions
    • Deploying Multiple Bots
  • Sample Bots
    • Sample Bots Introduction
    • Food Ordering Bot
    • Stock Market Bot
    • Gaming Non-Playing Character (NPC) Bot
    • Spanish Weather Bot
    • RAG Bot
    • RAG LangChain Bot
    • DuckDuckGo LangChain Bot
    • Colang 2.0 Bot
    • Plan and Execute (LangGraph) Bot
  • Configuration Guide
    • Bot Configurations Introduction
    • General Configurations
      • Bot Name and Version
      • Configuring the NLP and Plugin Server Endpoints
      • Configuring the NLP Server Model Parameters
        • Intent Slot Models
        • Named Entity Recognizer Models
        • Text Classifier Models
      • Configuring Bot Voices
      • Multilingual Configurations
      • LLM Model Configurations
        • LangChain Models
        • NVIDIA AI Endpoints
        • Custom LLM Models using the LangChain Wrapper
        • LLM Prompts
    • Slots Configurations
      • Defining Slots
      • Adding Lookup and Regex Rules
      • Adding Synonyms to Slots
      • Defining Slot Memory
      • Grouping Slots using Composite Slots
      • Correcting Slots using Slot Resolution
      • Linking Slots to Similar Slot Values
    • Plugin Configurations
      • Defining Plugin
      • Adding Custom Plugin
      • Importing Parameters into Plugins
    • Model Configurations
      • Triton Inference Server
      • Riva Skills Server
      • Custom Models
    • Chat Engine Configurations
    • Speech Configurations
      • BotRuntimeGrpc
      • SpeechPipelineManager
      • Riva ASR
      • Word Boosting
      • Chat Engine
      • Riva TTS
      • IPA Dictionary
      • Riva Logger
      • NvMsgBrokerC2DReceiver
      • NvMsgBrokerD2CTransmitter
      • Audio2FaceGrpc
      • HttpClient
      • BotControllerUMIM
  • User Guide
    • Using Colang
    • Integration with LangChain and LlamaIndex
    • NLP Server
      • API Server
      • Model Registry
      • Model Servers
      • Config Manager
      • Model Manager
      • Custom Model Client
        • @model_api Decorator
        • @pytriton Decorator
    • Speech AI
      • ASR Module
      • TTS Module
    • Training Models
    • Plugin Server
      • Using Plugin with Colang
      • Passing URL Parameters
      • Passing Context Information and Conversation History
      • Maintaining Plugin Memory
      • Accessing Custom Metadata
      • Parameter List
      • Logging and Monitoring
      • Adding Plugin Requirements
      • Interacting with the Plugin Server
    • Streaming in ACE Agent
      • Streaming Overview
      • Streaming Exceptions
      • TTS Streaming
  • API Guide
    • API Introduction
    • HTTP Interface
    • gRPC Interface
    • Plugin Server
    • NLP Server
  • Best Practices
    • Best Practices Introduction
    • Planning your Bot
      • Step 1: Choose your Bot’s Domain of Expertise
      • Step 2: Determine the Conversation Flow and Knowledge Bases
      • Step 3: Determine the Language Model
      • Step 4: Determine the Dialog Modeling Language
      • Step 5: Plan More Modalities
      • Step 6: Determine the Frontend Channel
      • Step 7: Determine the Deployment Infrastructure
    • Managing Conversation Context
      • Setting Conversation Context
      • Deleting Conversation Context
      • Saving Conversation Context
    • Local Development with Kubernetes Environment
    • Logging and Debugging Issues
      • Log Location
      • Log Naming Convention
        • Exposing Chat Engine Logs to the Console
      • Audio and Metadata Logger
    • Security Considerations
  • Reference
    • Basic Concepts
      • Bot
      • Chat Controller
      • Chat Engine
      • Colang
      • Context
      • Entity
      • Intent
      • Knowledge Base
      • NeMo Guardrails
      • NLP Server
      • Plugin Server
      • Prompt
      • RAG (Retrieval-Augmented Generation)
      • Slot
    • Support Matrix
      • Hardware
      • Software
    • Migrating from Bot Maker 3.0.0 to ACE Agent 4.0.0
    • Troubleshooting
    • End User License Agreement
    • Notices
      • Notice
      • Google
      • Trademarks
      • Copyright
• Riva ASR
• Riva TTS
• Riva Translation
• Live Portrait Microservice
  • Speech Live Portrait Microservice
• Voice Font Microservice
• Resource Downloader
  • NGC Resource Downloader
    • Prerequisites
    • Create a Resource in NGC for the Asset
    • Uploading a Version to the NGC Resource
  • Manual Resource Downloader
  • Create Your Own Resource Downloader

Microservices (Early Access)

• Audio2Face Authoring
  • Before you Begin
  • Overview
  • Quick start
    • Prerequisites
      • NGC ACE EA access
      • NGC Access and Cloud Function Run Key
      • NGC Setup
    • Trying Audio2Face Authoring Microservice
  • WSL Setup Guide
    • Prerequisites
    • Setup Docker without Docker Desktop
    • Install CUDA Toolkit
    • Install NVIDIA Container Toolkit
    • Install NGC CLI
  • Architecture overview
  • Audio2Face Authoring Microservice
    • Overview
    • Communication
      • Batch size
    • Configuration
  • Sample applications connecting to A2F Authoring Microservice
    • Setup
    • Using a local deployment
    • Using NVCF
  • gRPC directly with Audio2Face Authoring
    • Client side
      • Service
      • Protobuf data
  • Container Deployment
    • Dependencies
      • Using Docker Compose
      • Single container deployment
      • TRT Engine Generation
  • AudioClip Database
    • TTL
    • Maximum size
  • Security
    • Notes
      • Input
      • Logging & Auditing
      • AI models
      • Authentication & Authorization
      • Encryption
      • Data collection
  • Troubleshooting
    • NVCF hosting limitations
    • Known issues
  • Low-FPS and latency Troubleshooting
    • Latency with the MACE Plugin
    • Implementing your own client
      • In-sync
      • Async
    • Lowering latency of the A2F Authoring Microservice
      • Docker Compose
      • Running the docker container
    • Latency due to geolocation
  • Changelog
    • v0.2.8
      • SDK Versions
      • Features
  • Licenses
• Unreal Renderer Microservice

Interfaces

• Animation Data Format
• Colang (2.0.0-beta)
• UMIM (1.3.0)

Tools

• Avatar Customization
• Unified Cloud Services Tools (UCS Tools)
• Maya-ACE

Licenses

• EULA