NVIDIA ACE

NVIDIA ACE is a suite of real-time AI solutions for end-to-end development of interactive avatars and digital human applications at-scale. Its customizable microservices offer the fastest and most versatile solution for bringing avatars to life at-scale, based on NVIDIA’s Unified Compute Services, full-stack AI platform and RTX technology. Leverage ACE to seamlessly integrate NVIDIA AI into your application, including Riva for speech and translation AI, Voice Font for transferring your voice to a digital avatar, NeMo LLM for natural language understanding, Audio2Emotion, Animation Graph, Omniverse Renderer or A2F-2D or A2F-3D for AI powered 2D and 3D character animation. Or, further optimize for cloud scale with microservices and domain-specific AI workflows like NVIDIA Maxine, NVIDIA Tokkio, Animation Pipeline, and NVIDIA Kairos, to deliver advanced avatars across video conferencing customer service use cases and gaming.

Release Notes

• ACE Release Notes
  • 24.06 Release
    • NIMs for Digital Human and ACE Microservices
      • NVIDIA AI Enterprise Supported
        • Riva ASR 2.15.1
        • Riva TTS 2.15.1
        • Riva NMT 2.15.1
        • Audio2Face - 3D
        • Omniverse Renderer Microservice 1.0.1
        • Animation Graph Microservice 1.0.1
        • ACE Agent 4.0.0
      • Early Access Microservices
        • SpeechLivePortrait 0.1.0
        • Nemotron 4.5B SLM 0.1.0
        • VoiceFont 1.1.1
    • ACE Reference Workflows
      • Customer Service Workflow
        • New Reference Components
      • Game Character Workflow
        • New Reference Components
        • New Samples
    • Tools
      • UCS Tools 2.5
      • Avatar Configurator 1.0.1

Getting Started

• First Steps
  • Find Resources
  • Try Out ACE
  • Development Setup
  • Workflows
• Development Setup
  • Prerequisites
  • Install NVIDIA driver
  • Install NGC and set API key
  • Install UCS Tools
  • Install Docker
  • Login to the nvcr.io container registry
  • Install NVIDIA container toolkit

Workflows

• Animation Pipeline
• Tokkio
• Kairos
  • Unreal Engine Integration
    • NVIDIA ACE Unreal Plugin 2.3
    • Kairos Unreal Sample Project

Microservices

• Animation Graph Microservice
• Omniverse Renderer Microservice
• Audio2Face-3D
  • Overview
  • Getting Started
    • Prerequisites
      • NVAIE access
      • NGC Personal Key
      • Export the API key
      • Docker Login to NGC
    • Launching the Audio2Face-3D NIM
      • Supported Models
      • Running Inference
      • Model Caching
      • Stopping the container
  • Architecture overview
    • Audio2Face-3D NIM Data Flow
  • Audio2Face-3D Microservice
    • Bidirectional gRPC Communication
      • Input
      • Output
    • Configuration
  • Observability
    • Overview
    • Traces
    • Metrics
    • Quick start the NIM container and visualize metrics and traces
  • Sample application connecting to Audio2Face-3D
    • Assumptions
    • Setting up the sample application
    • Health checking
    • Interacting with Audio2Face-3D
    • Results
    • Resampling Audio Data
      • Resampling Guidelines
  • Performance
    • Performance evaluation
    • Modifying deployment configurations for benchmarking
    • Results
    • Performance benchmark
    • Performance troubleshooting
  • Fetch current configuration from Audio2Face-3D
    • Assumptions
    • Setting up the config fetcher application
    • Fetching configuration
    • Results
  • gRPC with Audio2Face-3D
    • Bidirectional Streaming gRPC
      • Service definition
      • Service protobuf objects
      • Dependencies
    • Config Fetch gRPC
      • Service definition
    • Unidirectional Streaming gRPC
  • Unreal Engine Interaction
    • Using Unreal Engine with Audio2Face-3D
  • A2F-3D NIM Manual Container Deployment and Configuration
    • Prerequisites
    • Configuration files
      • 1. The Stylization configuration file
        • Claire Config
        • James Config
        • Mark Config
      • 2. The Deployment configuration file
      • 3. The Advanced configuration file
    • How to use configuration files
    • Model caching
    • Starting the A2F-3D NIM with custom entrypoint
      • Generating the TRT engine
      • Starting the service
    • Changing Configuration - The Shortest Way
    • Changing Configuration - The Flexible Way
      • Example 1: Setting the Stylization config to use Mark
      • Example 2: Updating the type of Endpoint to Unidirectional
    • Advanced Stylization
    • Configuration files for Unreal Engine Metahuman
    • Parameter Tuning Guide
    • Environment variables
    • Volumes
    • Quick Deployment of Audio2Face-3D Microservices
  • Kubernetes Deployment
    • 1. Install microk8s, GPU addon, set local path provisioner and set secrets
    • 1. Download and install the helm chart
    • 2. Optional - Expose the pod IP publicly
    • 3. Check it is up and running
    • Uninstall the app
  • Support Matrix
    • Models
    • Optimized configurations
    • Non-optimized configurations
    • Software
  • Migrating from 1.0 to 1.2
    • Audio2Face Controller
      • Service Interface
    • Audio2Face-3D Microservice
      • Audio stream header
      • Blendshape parameters
      • Migrating Configuration files
      • Migrating Kubernetes deployment
  • Sharing Audio2Face-3D Compute Resources
    • Sharing Audio2Face-3D compute with a Renderer
  • Network Customization and Accessibility
    • Audio2Face-3D Docker-Compose Customization
    • Exposing Audio2Face-3D Deployment from WSL
      • Opening a port in Windows Firewall
      • Rerouting Traffic to your WSL deployment
  • Glossary of Terms
    • Common terminology
    • NVIDIA technology
    • 3rd Party Technology
  • Security
    • Notes
      • Input
      • Logging & Auditing
      • AI models
      • Authentication & Authorization
      • Encryption
      • Data collection
  • Release Notes
    • v1.2.0
      • SDK Versions
      • Features
    • v1.0.0
      • SDK Versions
      • Features
      • Key improvements
  • Troubleshooting
    • Known issues
    • Q&A
    • gRPC Response Status Code
    • Audio2Face-3D NIM Public API
    • GStreamer-WARNING
• ACE Agent
  • Getting Started
    • Overview
      • Structure of this Document
    • Quick Start Guide
    • Release Notes
      • ACE Agent Release 4.1.0
        • Key Features and Enhancements
        • Compatibility
        • Limitations
        • Deprecated and Removed Features
        • Known Issues
      • ACE Agent Release 4.0.0
        • Key Features and Enhancements
        • Compatibility
        • Breaking Changes
        • Limitations
        • Known Issues
      • ACE Agent Release 3.0.1 EA
        • Fixed Bugs
        • Known Issues
      • ACE Agent Release 3.0.0 EA
        • Key Features and Enhancements
        • Announcements
        • Limitations
        • Known Issues
  • Architecture
    • Architecture Introduction
      • ACE Agent Event Architecture
      • Chat Engine Server Architecture
      • Plugin Server Architecture
  • Deployment
    • Interface Introduction
      • CLI Interface
      • Event Interface
      • HTTP Server Interface
      • gRPC Interface
    • Docker Environment
      • CLI Interface
      • HTTP Server Interface
      • Event Interface
      • gRPC Interface with HTTP Interface
      • Event Interface with Speech
      • Model Deployment
    • Kubernetes Environment
      • Deploying a Bot via UCS Application
      • Building a Custom Helm Chart using ACE Agent Microservices
        • Building a Chatbot UCS Application
        • Building a Speech Bot UCS Application
        • Building a LangChain Bot UCS Application
      • Creating Kubernetes Secrets
      • Bot and UCS Applications Customization
        • Local Development with Kubernetes Environment
        • Using Local Bot Configurations with UCS Applications
        • Using 3rd Party Text-to-Speech (TTS) Solutions
        • Deploying Multilingual Bots with Kubernetes
    • Python Environment
      • Chat Engine
        • CLI Interface
        • HTTP Server Interface
        • Event Interface
      • Plugin Server
      • Model Deployment
      • NLP Server
      • Clean Up
      • Python Native Application
    • Sample Clients
      • Web Sample Application
        • Text-Based Web Sample Application
        • Speech-Based Web Sample Application
      • gRPC Server Sample Client
      • Chat Python Sample Client
      • Event Sample Client
  • Tutorials
    • Building a Bot using Colang 2.0 and Event Interface
      • Prerequisites
      • Step 1: Making the Greeting More Interesting
      • Step 2: Leveraging LLMs to Answer User Questions
      • Step 3: Making the Bot Proactive
      • Step 4: Interrupting the Bot
      • Step 5: Back-Channeling
      • Step 6: Running Python Code
      • Step 7: Integrating a Plugin Service
    • Building LangChain-Based Bots
      • Creating the LangChain Plugin
      • Connecting the Plugin to the Chat Controller
    • Building a Low Latency Speech-To-Speech RAG Bot
      • Creating the RAG Plugin
      • Connecting Chat Controller to the RAG Plugin
      • Connecting the Plugin to the Chat Engine
        • Creating the Bot and Colang Configurations
        • Testing the Bot
      • Improving the User Experience for Speech-To-Speech Conversations
        • Updating Bot Configurations
        • Testing the Bot
    • Customizing a Bot
      • Selecting an LLM Model
      • Using an On-Premise LLM Model Deployed via NIM
      • Creating a New Custom Action
      • Using a Custom NLP Model
        • Deploying a Custom NLP Model with the NLP Server
          • Custom Model Integration using the @model_api Decorator
          • Custom Model Integration using the @pytriton Decorator
      • Customizing ASR Recognition with Word Boosting
      • Customizing ASR Recognition for Long Pause Handling
      • Customizing TTS Pronunciation using IPA
      • Using 3rd Party Text-to-Speech (TTS) Solutions
  • Sample Bots
    • Sample Bots Introduction
    • Chitchat Bot
    • LLM Bot
    • RAG Bot
    • Stock Market Bot
    • Gaming Non-Playing Character (NPC) Bot
    • DuckDuckGo LangChain Bot
    • Spanish Weather Bot
      • Spanish NMT Bot
      • Spanish LLM Bot
    • Food Ordering Bot
    • Plan and Execute (LangGraph) Bot
    • Colang 1.0 Sample Bots
  • Configuration Guide
    • Bot Configurations Introduction
    • General Configurations
      • Bot Name and Version
      • Configuring the NLP and Plugin Server Endpoints
      • Multilingual Configurations
      • LLM Model Configurations
        • LangChain Models
        • NVIDIA AI Endpoints
        • Custom LLM Models using the LangChain Wrapper
        • LLM Prompts
    • Slots Configurations
      • Defining Slots
      • Adding Lookup and Regex Rules
      • Adding Synonyms to Slots
      • Defining Slot Memory
      • Grouping Slots using Composite Slots
      • Correcting Slots using Slot Resolution
      • Linking Slots to Similar Slot Values
    • Plugin Configurations
      • Defining Plugin
      • Adding Custom Plugin
      • Importing Parameters into Plugins
    • Model Configurations
      • Triton Inference Server
      • Riva Skills Server
      • Custom Models
    • Chat Engine Configurations
    • Speech Configurations
      • BotRuntimeGrpc
      • SpeechPipelineManager
      • Riva ASR
        • Word Boosting
      • Chat Engine
      • Riva TTS
        • IPA Dictionary
      • Riva Logger
      • NvMsgBrokerC2DReceiver
      • NvMsgBrokerD2CTransmitter
      • Audio2FaceGrpc
      • HttpClient
      • BotControllerUMIM
      • LivePortraitGrpc
  • User Guide
    • Using Colang
    • Integration with LangChain and LlamaIndex
    • NLP Server
      • API Server
      • Model Registry
      • Model Servers
      • Config Manager
      • Model Manager
      • Custom Model Client
        • @model_api Decorator
        • @pytriton Decorator
    • Speech AI
      • ASR Module
      • TTS Module
    • Training Models
    • Plugin Server
      • Using Plugin with Colang
      • Passing URL Parameters
      • Passing Context Information and Conversation History
      • Maintaining Plugin Memory
      • Accessing Custom Metadata
      • Parameter List
      • Logging and Monitoring
      • Adding Plugin Requirements
      • Interacting with the Plugin Server
    • Streaming in ACE Agent
      • Streaming in Colang 2.0 Bots
        • Text Streaming
      • Streaming in Colang 1.0 Bots
        • Text Streaming
        • Streaming Exceptions
        • TTS Streaming
  • API Guide
    • API Introduction
    • HTTP Interface
    • gRPC Interface
    • Plugin Server
    • NLP Server
  • Best Practices
    • Best Practices Introduction
    • Planning your Bot
      • Step 1: Choose your Bot’s Domain of Expertise
      • Step 2: Determine the Conversation Flow and Knowledge Bases
      • Step 3: Determine the Language Model
      • Step 4: Determine the Dialog Modeling Language
      • Step 5: Plan More Modalities
      • Step 6: Determine the Frontend Channel
      • Step 7: Determine the Deployment Infrastructure
    • Logging and Debugging Issues
      • Log Location
      • Log Naming Convention
        • Exposing Chat Engine Logs to the Console
      • Audio and Metadata Logger
    • Security Considerations
  • Reference
    • Basic Concepts
      • Bot
      • Chat Controller
      • Chat Engine
      • Colang
      • Context
      • Entity
      • Intent
      • Knowledge Base
      • NeMo Guardrails
      • NLP Server
      • Plugin Server
      • Prompt
      • RAG (Retrieval-Augmented Generation)
      • Slot
    • Support Matrix
      • Hardware
      • Software
    • Migrating from ACE Agent 4.0.0 to ACE Agent 4.1.0
    • Troubleshooting
    • End User License Agreement
    • Notices
      • Notice
      • Google
      • Trademarks
      • Copyright
• Riva ASR
• Riva TTS
• Riva Translation
• Audio2Face-2D Microservice
• Voice Font Microservice
• Resource Downloader
  • NGC Resource Downloader
    • Prerequisites
    • Create a Resource in NGC for the Asset
    • Uploading a Version to the NGC Resource
  • Manual Resource Downloader
  • Create Your Own Resource Downloader

Microservices (Early Access)

• Audio2Face-3D Authoring
  • Before you Begin
  • Overview
  • Quick start
    • Prerequisites
      • NGC ACE EA access
      • NGC Access and Cloud Function Run Key
      • Setup sample application
    • Trying Audio2Face-3D Authoring Microservice
  • WSL Setup Guide
    • Prerequisites
    • Setup Docker without Docker Desktop
    • Install CUDA Toolkit
    • Install NVIDIA Container Toolkit
    • Install NGC CLI
  • Architecture overview
  • Audio2Face-3D Authoring Microservice
    • Overview
    • Communication
      • Batch size
    • Configuration
  • Sample applications connecting to A2F-3D Authoring Microservice
    • Setup
    • Using a local deployment
    • Using NVCF
  • gRPC directly with Audio2Face-3D Authoring
    • Client side
      • Service
      • Protobuf data
  • Container Deployment
    • Dependencies
      • Using Docker Compose
      • Single container deployment
      • TRT Engine Generation
  • AudioClip Database
    • TTL
    • Maximum size
  • Security
    • Notes
      • Input
      • Logging & Auditing
      • AI models
      • Authentication & Authorization
      • Encryption
      • Data collection
  • Troubleshooting
    • NVCF hosting limitations
    • Startup failures
    • Known issues
  • Low-FPS and latency Troubleshooting
    • Latency with the MACE Plugin
    • Implementing your own client
      • In-sync
      • Async
    • Lowering latency of the A2F-3D Authoring Microservice
      • Docker Compose
      • Running the docker container
    • Latency due to geolocation
  • Changelog
    • v0.3.6
      • SDK Versions
      • Features
    • v0.2.8
      • SDK Versions
      • Features
  • Licenses
• Unreal Renderer Microservice

Interfaces

• Animation Data Format
• Colang (2.0.0-beta)
• UMIM (1.3.1)

Tools

• Avatar Customization
• Unified Cloud Services Tools (UCS Tools)
• Maya-ACE

Licenses

• EULA