Is this page helpful?

NVIGI CHATTERBOX TTS MODEL CARD#

Version: 1.0.0
Release Date: January 2026
Publisher: NVIDIA Corporation
Base Models: Chatterbox-Turbo and Chatterbox-Multilingual by Resemble AI

1. MODEL OVERVIEW#

1.1 NVIGI Chatterbox TTS Overview#

The NVIDIA In-Game Inferencing (NVIGI) Chatterbox TTS Plugin Pack provides optimized integration of Resemble AI’s state-of-the-art Chatterbox text-to-speech models — Chatterbox-Turbo (English) and Chatterbox-Multilingual (23 languages) — for PC applications and games. This pack brings professional-grade, zero-shot voice cloning capabilities to interactive applications through the NVIGI plugin architecture.

NVIGI Chatterbox TTS enables developers to:

Generate natural, human-like speech in real-time
Clone any voice from just 5 seconds of reference audio
Run inference on CUDA (NVIDIA GPUs), Vulkan (cross-platform), or D3D12 (Windows)
Efficiently share GPU resources between graphics and AI workloads
Build complete voice-to-voice AI pipelines with ASR and GPT integration

Key Features:

High-Quality Speech Synthesis: Natural prosody and rhythm using the Chatterbox Turbo model (English, 350M parameters) or Chatterbox Multilingual model (23 languages, 520M LLaMA backbone)
Multi-backend Support: CUDA, Vulkan, and D3D12 inference backends
Zero-Shot Voice Cloning: Create custom voices from minimal reference audio (the resulting speaker embedding is bound to either Turbo or Multilingual and is not interchangeable)
Long-Text Handling: Built-in text chunking for processing extended prompts (character-based fallback for Multilingual zh / ja)
Paralinguistic Tags (Turbo only): Support for expressive non-speech sounds like [laugh], [cough], [sigh], [gasp], [chuckle], [groan], [sniff], [shush], [clear throat]. Not supported by Chatterbox Multilingual.
CFG-Controlled Pacing (Multilingual only): Runtime cfg_weight parameter tunes speech pacing via Classifier-Free Guidance. Turbo runs without CFG.
Native Language Preprocessing (Multilingual only): Chinese Cangjie encoding, Japanese Kanji→Hiragana conversion, Korean Jamo decomposition, NFKD normalization, and Hebrew diacritization happen inside the plugin.
Hallucination Protection (Multilingual only): Alignment analyzer suppresses early EOS and forces termination on runaway generations.
CIG Support: CUDA In Graphics optimization for concurrent rendering and AI
Full Voice Pipeline: Optional GPT and ASR integration for complete conversational AI

See the Model Feature Comparison in the Programming Guide for the full per-feature breakdown.

1.2 Chatterbox-Turbo Base Model Overview#

Chatterbox-Turbo is the most efficient model in Resemble AI’s Chatterbox family of open-source text-to-speech models. Released under the MIT license, it has quickly become one of the leading open-source voice AI models, consistently outperforming commercial alternatives like ElevenLabs in blind evaluations.

Model Characteristics:

State-of-the-Art Quality: Natural prosody and rhythm with human-like speech output
Real-Time Performance: Faster-than-realtime inference, optimized for interactive applications
Zero-Shot Capability: Clone any voice without training, using just 5-10 seconds of reference audio
Native Paralinguistic Support: Built-in support for non-speech sounds and expressive tags
Efficient Architecture: 350M parameters optimized for lower compute and VRAM usage
Single-Step Decoder: Distilled speech-token-to-mel decoder (1 step vs. 10 steps in previous models)

Application Areas:

Low-latency voice agents and conversational AI
Gaming and interactive entertainment
Narration and creative workflows
Voice-enabled applications and assistants
Accessibility tools

2. MODEL ARCHITECTURE#

2.1 Chatterbox-Turbo Architecture#

Chatterbox-Turbo features a streamlined 350M parameter architecture designed for efficiency and real-time performance:

Model Components:

Text Encoder: Processes input text and converts to semantic tokens
Speech Token Generator: Transformer-based model producing speech tokens
Mel-Spectrogram Decoder: Single-step distilled decoder (reduced from 10 steps in standard Chatterbox)
Vocoder: HiFT-GAN-based neural vocoder for waveform synthesis
Speaker Embedding System: Zero-shot voice cloning via speaker embeddings extracted from reference audio

Key Architectural Improvements:

Parameter Efficiency: 350M parameters vs. 500M in standard Chatterbox
Decoder Distillation: 10x reduction in mel-spectrogram generation steps
Native Paralinguistic Tags: Built-in support without post-processing
Optimized for Latency: Designed for sub-200ms production inference

Technical Specifications:

Output Sample Rate: 24 kHz mono
Audio Format: 16-bit PCM WAV
Reference Audio: 5-10 seconds of clean speech (WAV format)
Speaker Embedding Size: 192 dimensions (extracted from reference audio)
Inference Backend: GGML/llama.cpp with CUDA, Vulkan, or D3D12 support

2.2 Chatterbox Multilingual Model#

The Chatterbox Multilingual model extends the base architecture with 23-language support:

Supported Languages: Arabic (ar), Danish (da), German (de), Greek (el), English (en), Spanish (es), Finnish (fi), French (fr), Hebrew (he), Hindi (hi), Italian (it), Japanese (ja), Korean (ko), Malay (ms), Dutch (nl), Norwegian (no), Polish (pl), Portuguese (pt), Russian (ru), Swedish (sv), Swahili (sw), Turkish (tr), Chinese (zh)

Key Differences from Turbo:

Text Vocabulary: 2454 tokens (vs. 704 for English-only), covering multilingual graphemes
Classifier-Free Guidance (CFG): Always on (weight 0.5) for better multilingual quality
Language Preprocessing: Automatic lowercase, NFKD normalization, and language-specific transforms (Korean Jamo decomposition, Chinese Cangjie encoding)
Alignment Analysis: Token-based EOS control to prevent hallucinations and premature stopping

Model Files:

multilingual_t3_text_emb.gguf - Text embeddings with multilingual tokenizer
multilingual_chatterbox_llama_backbone_tts_q4_0.gguf - LLaMA 520M backbone (Q4_0 quantized)
S3Gen-Multilingual-F16.gguf - Speech decoder
cangjie_mapping.json - Chinese-character → Cangjie encoding table (auto-discovered for Chinese)
kanji_readings.json - Kanji → Hiragana reading table (auto-discovered for Japanese)
dicta-bert-q8.gguf / dicta-bert-q8-heads.gguf - DictaBERT diacritization model (auto-discovered for Hebrew)

2.3 NVIGI Plugin Architecture#

The NVIGI SDK uses a plugin architecture where TTS functionality is provided through the nvigi.plugin.tts.chatterbox plugins:

Plugin Components:

nvigi.plugin.tts.chatterbox.cuda.dll - CUDA backend for NVIDIA GPUs
nvigi.plugin.tts.chatterbox.vk.dll - Vulkan backend for cross-platform
nvigi.plugin.tts.chatterbox.d3d12.dll - D3D12 backend for Windows (requires driver 580.61+)

Integration Features:

Standard NVIGI plugin API for easy application integration
Support for CUDA, Vulkan, and D3D12 inference backends
Can be combined with GPT and ASR plugins for voice-to-voice AI pipelines
Integration with D3D12 for concurrent graphics and AI workloads with shared context
CUDA In Graphics (CIG) support for efficient GPU sharing

Note: This NVIGI implementation includes both Chatterbox-Turbo and Chatterbox-Multilingual models. Select the model type during instance creation with TTSChatterboxCreationParameters::modelType.

3. HOW TO USE THE MODEL#

3.1 Prerequisites#

Hardware Requirements:

GPU: NVIDIA GPU with Compute Capability 7.0+ (RTX 20x0 series or newer)
Memory: 16 GB RAM minimum
VRAM: ~2.3 GB for Turbo TTS-only, ~3.5 GB for Multilingual TTS-only, ~8.5 GB+ for full pipeline (ASR + GPT + TTS with 3D sample)

Software Requirements:

Operating System: Windows 10/11 (64-bit)
Graphics Driver: NVIDIA graphics driver r580 or newer
Development Tools: Visual Studio 2022 with C++ tools (for rebuilding samples)

3.2 Quick Start Setup#

Step 1: Basic Setup (TTS Only)#

Run the setup script to download and install the NVIGI Core SDK:

.\setup_sample.bat

Or simply double-click setup_sample.bat in Windows Explorer.

Step 2: Full Setup (ASR + GPT + TTS - Optional)#

For the complete voice-to-voice AI pipeline:

Download the NVIGI Developer Full SDK from: https://developer.nvidia.com/rtx/in-game-inferencing
Extract to a local directory
Run setup with the SDK path:

.\setup_sample.bat "C:\path\to\nvigi_developer_full_sdk"

3.3 Running Your First TTS Sample#

Command-Line Sample#

Navigate to the release directory and run:

cd bin\x64\Release
.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\aaron_turbo.json --text "Hello from Chatterbox Turbo TTS." --output hello.wav --backend cuda

This will:

Initialize the Chatterbox TTS plugin with the Turbo model
Load the specified speaker embedding for voice characteristics
Generate speech from the input text
Save the output as a 24 kHz WAV file
Display timing statistics (inference time, audio length, RTF)

Try Different Voices:

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\lucy_turbo.json --text "This is Lucy speaking with Chatterbox Turbo." --output lucy_hello.wav --backend cuda

Interactive Mode (Windows):

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\aaron_turbo.json --interactive --backend cuda

Multilingual Mode:

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\french.json --multilingual --language fr --text "Bonjour, je suis un modèle multilingue." --output bonjour.wav --backend cuda

3.4 3D Interactive Sample#

Experience TTS in a full 3D application with GUI:

cd bin\x64\Release
.\nvigi.3d.exe

D3D12 Mode (default) - Full voice-to-voice pipeline with D3D12 backends (CUDA fallback):

Speak into your microphone (ASR transcribes speech)
GPT generates a response
Chatterbox TTS speaks the response aloud
Or type text directly and press Enter
Uses D3D12 backends for TTS, GPT, and ASR when available (requires driver 580.61+)
Automatically falls back to CUDA backends if D3D12 plugins are unavailable

Vulkan Mode - Full pipeline with Vulkan backends:

.\nvigi.3d.exe -vk

3.5 Voice Cloning: Create Custom Voices#

Setting Up Voice Cloning Environment#

Navigate to the voice_cloning directory:

cd voice_cloning

Run the setup script:

⚠️ Third-Party Software Notice: This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

.\setup_venv.ps1

This will:

Check for Python 3.11 (or install automatically)
Install all required dependencies (PyTorch, Chatterbox TTS, etc.)

Creating Speaker Embeddings#

Run setup_venv.ps1 once to create and activate the venv at voice_cloning\venv. The script activates the venv in the current PowerShell session, so subsequent python invocations resolve to the venv interpreter automatically. In a new shell, re-activate first with .\venv\Scripts\Activate.ps1.

Basic usage:

python get_voice_embeddings.py --wav path/to/reference_audio.wav --dump-json my_voice.json

With Hugging Face token (required for first-time model download):

python get_voice_embeddings.py --wav reference_audio.wav --dump-json my_voice.json --hf-token YOUR_HF_TOKEN

Or set environment variable:

$env:HF_TOKEN = "YOUR_HF_TOKEN"
python get_voice_embeddings.py --wav reference_audio.wav --dump-json my_voice.json

With custom emotion exaggeration:

python get_voice_embeddings.py --wav reference_audio.wav --dump-json my_voice.json --exaggeration 0.8

Voice Cloning Command-line Options#

Argument	Description
`--wav <path>`	Path to reference WAV file (5-10 seconds recommended)
`--model-type <turbo\|base\|multilingual>`	Model family the embedding targets (default: `turbo`). Must match the `modelType` used at inference.
`--dump-json <path>`	Output path for speaker embedding JSON file
`--exaggeration <float>`	Emotion exaggeration baked into the embedding (default: 0.5, range: 0.0-1.0). Affects Multilingual only; ignored by Turbo.
`--hf-token <token>`	Hugging Face token for model download

Tips for Best Voice Cloning Results#

Audio Quality: Use clean, high-quality audio with minimal background noise
Duration: 5-10 seconds of speech is optimal
Format: WAV format required (convert other formats using FFmpeg)
Content: Natural speech works best (avoid music, singing, or heavily processed audio)
Emotion: --exaggeration controls emotion capture (0.0 = neutral, 1.0 = highly expressive)

Using Custom Speaker Embeddings#

Copy your generated embedding to the speaker embeddings directory:

copy my_voice.json data\nvigi.test\spk_emb\

Use with CLI sample:

cd bin\x64\Release
.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\my_voice.json --text "Hello with my custom voice!" --output output.wav

The new voice will also appear in the speaker dropdown in the 3D sample.

3.6 Using Paralinguistic Tags (Turbo only)#

Chatterbox-Turbo supports expressive non-speech sounds. These tags are not supported by the Multilingual model — the multilingual tokenizer does not include them, so they will either be dropped or split into individual letters and read aloud verbatim. If you need an expressive cue with the Multilingual model, surround the line with text that conveys the emotion instead.

Supported Tags:

[laugh] - Natural laughter
[chuckle] - Light chuckling
[cough] - Coughing sound
[sigh] - Sighing
[gasp] - Sharp intake of breath
[groan] - Groaning sound
[sniff] - Sniffing
[shush] - Shushing sound
[clear throat] - Throat clearing

Example Usage:

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\aaron_turbo.json --text "Hi there [chuckle], have you got one minute to chat about this?" --output expressive.wav

Best Practices:

Use tags in appropriate context - surrounding text should match the emotion
Tags work best when they fit naturally into the speech flow
Avoid overusing tags - one or two per sentence maximum for natural results

3.7 Application Integration#

To integrate TTS into your own application:

Review the Programming Guide: See Programming Guide for detailed API documentation
Study Sample Source Code: Examine source/samples/ for implementation examples
Link Against NVIGI SDK: Include headers from include/ directory:
```
#include "nvigi_tts_chatterbox.h"
```
Load Plugin at Runtime: Load nvigi.plugin.tts.chatterbox.cuda.dll, nvigi.plugin.tts.chatterbox.vk.dll, or nvigi.plugin.tts.chatterbox.d3d12.dll dynamically

Key Integration Topics:

Loading plugins and models
Choosing model type (eTurbo vs eMultilingual) and a matching speaker JSON
Configuring speaker embeddings
Selecting backend (CUDA, Vulkan, or D3D12)
Handling text chunking for long prompts
D3D12 backend setup (requires D3D12 device and command queues)
Setting language_id and tuning cfg_weight (Multilingual only)
Using paralinguistic tags for expressive speech (Turbo only)
Error handling and performance optimization

3.8 Building Samples from Source (Optional)#

To modify or rebuild the included samples:

Open a VS2022 Developer Command Prompt
Navigate to the pack root directory
Run .\setup.bat to generate build files
Open _project/vs2022/nvigi.chatterbox.sln in Visual Studio 2022
Build the desired configuration (Debug/Release/Production)
Run .\copy_sdk_binaries.bat Release to deploy binaries

Note: Plugin DLLs are pre-built and cannot be rebuilt from this pack. Only sample applications can be rebuilt.

4. TRAINING DATA AND MODEL DEVELOPMENT#

4.1 Training Data Overview#

The Chatterbox-Turbo and Chatterbox-Multilingual models were trained by Resemble AI on curated datasets designed for high-quality speech synthesis:

Data Characteristics:

Language: Turbo — primarily English. Multilingual — speech across 23 supported languages.
Content: Diverse speech samples covering various speaking styles, emotions, and contexts
Quality: High-quality audio recordings with clean speech and minimal background noise
Diversity: Multiple speakers with varied vocal characteristics, ages, and speaking styles
Source: Data sourced from freely available content on the internet

Training Approach:

Zero-Shot Learning: Model trained to generalize to unseen voices using speaker embeddings
Paralinguistic Integration: Native training on expressive tags and non-speech sounds
Prosody Modeling: Emphasis on natural rhythm, intonation, and pacing
Emotion Capture: Training to preserve and reproduce emotional characteristics from reference audio

4.2 Model Capabilities and Limitations#

Supported Features:

Zero-shot voice cloning from 5-10 seconds of reference audio (both models)
Natural prosody and human-like speech patterns (both models)
Multilingual TTS in 23 languages (via eMultilingual model type)
Paralinguistic tags [laugh], [chuckle], [cough], [sigh], [gasp], [groan], [sniff], [shush], [clear throat] — Turbo only
Emotion exaggeration control (0.0-1.0 scale, baked into the speaker embedding by the voice cloning script) — Multilingual only; the value is ignored by Turbo
CFG-controlled speech pacing via cfg_weight — Multilingual only; the value is ignored by Turbo
Native language preprocessing (Korean Jamo, Chinese Cangjie, Japanese Kanji→Hiragana, NFKD normalization, Hebrew diacritization) — Multilingual only
Alignment-analyzer hallucination protection (EOS suppression / forced termination) — Multilingual only
Long-text handling via automatic chunking (default: 40 words per chunk; switches to ~30-character chunks for Multilingual zh / ja) — both models
Real-time, faster-than-realtime inference performance (Turbo is ~2× faster than Multilingual on the same backend)

See Model Feature Comparison in the Programming Guide for the complete per-feature breakdown.

Current Limitations:

Turbo model: English only. Use the Multilingual model for other languages. Does not use Classifier-Free Guidance — cfg_weight is ignored. Does not run the alignment analyzer used by the Multilingual model.
Multilingual model: Does not support paralinguistic tags — the multilingual tokenizer vocabulary does not contain [laugh], [sigh], [gasp], etc. Sending these tags will not produce the intended non-speech sound. Russian stress-mark prediction is not yet integrated. Hebrew diacritization, Chinese Cangjie encoding, and Japanese Kanji→Hiragana conversion require the auxiliary assets (dicta-bert-q8*.gguf, cangjie_mapping.json, kanji_readings.json) shipped alongside the model.
Speaker JSON binding: Speaker embeddings are tied to the model that produced them (model_type field set by get_voice_embeddings.py). Using a Turbo embedding with eMultilingual (or vice versa) is rejected at createInstance; legacy embeddings without the field are accepted with a warning.
First inference with a new speaker embedding may have higher latency due to warmup requirements
Paralinguistic tags work best when surrounding text matches the intended emotion (Turbo only — see above)
Voice cloning quality depends on reference audio quality (clean speech, minimal background noise)
Input text quality: Both models work best with grammatically complete sentences and standard punctuation. Short, incomplete inputs (e.g., “the”, “that”, “hundred”) may produce extra unintelligible audio, as the model does not always recognize them as complete utterances. Repeated or excessive punctuation (e.g., Hello!!!, Really?!) can also cause unexpected output. For best results, use well-formed sentences with single punctuation marks.

5. LICENSE INFORMATION#

5.1 NVIGI Chatterbox TTS Plugin Pack License#

License: LicenseRef-NvidiaProprietary
SPDX-License-Identifier: LicenseRef-NvidiaProprietary

Use of this software is governed by the NVIDIA Software License Agreement. If you do not have a signed agreement, you may not use, reproduce, distribute, or modify this software.

For license terms, see: https://www.nvidia.com/en-us/agreements/

5.2 Dependency Licenses#

NVIGI Core:

NVIGI Core (nvigi_core) is licensed separately
License: https://raw.githubusercontent.com/NVIDIA-RTX/NVIGI-Core/refs/heads/main/LICENSE.txt

Microsoft DirectX:

Microsoft DirectX DLL libraries are governed by the Microsoft Software License Terms for Microsoft DirectX
License: https://www.nuget.org/packages/Microsoft.Direct3D.D3D12/1.615.1/License

NVIDIA Models:

NVIDIA models used with this SDK are governed by the NVIDIA Community Models License
License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/

5.3 Chatterbox-Turbo Base Model License#

The Chatterbox-Turbo model by Resemble AI is licensed under the MIT License:

MIT License

Copyright (c) 2025 Resemble AI

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

5.4 Third-Party Acknowledgements#

Chatterbox-Turbo incorporates or was inspired by the following projects:

For complete third-party license information, see:

3rd-party-licenses.md (HTML documentation)
NOTICE.txt in the package root (additional notices)

5.5 Important Legal Notices#

Content Warning:

Some of the models you may download with the NVIDIA In-Game Inferencing (NVIGI) SDK have been designed for gaming or other applications that involve roleplay. As such, the outputs of these models may be violent, offensive or indecent. By using these models, you assume the risk of any harm caused by any response or output of them.

Responsible AI:

Users are responsible for ensuring that generated audio is used ethically and in compliance with all applicable laws and regulations. Do not use this model for malicious purposes, including but not limited to:

Creating deepfakes or impersonating individuals without consent
Generating misleading or deceptive content
Violating privacy or intellectual property rights
Harassing, threatening, or causing harm to others

Citation:

If you use Chatterbox-Turbo in your research or project, please cite:

@misc{chatterboxtts2025,
  author       = {{Resemble AI}},
  title        = {{Chatterbox-TTS}},
  year         = {2025},
  howpublished = {\url{https://github.com/resemble-ai/chatterbox}},
  note         = {GitHub repository}
}

6. ADDITIONAL RESOURCES#

Documentation:

Getting Started Guide
Programming Guide
Samples Documentation
README.md in the package root - Main getting started documentation

External Resources:

Support:

Review the Programming Guide for detailed API documentation
Check Samples Documentation for usage examples and troubleshooting
See the Troubleshooting section in Samples Documentation