Getting Started with Chatterbox TTS Plugin Pack

What is Chatterbox?

Chatterbox is an open-source, MIT-licensed text-to-speech model developed by Resemble AI. It has quickly become one of the leading open-source voice AI models, consistently outperforming commercial alternatives like ElevenLabs in blind evaluations.

Why Chatterbox?

• State-of-the-Art Quality: Natural prosody and rhythm with human-like speech output

• Zero-Shot Voice Cloning: Clone any voice with just 5 seconds of reference audio—no training required

• Real-Time Performance: Faster-than-realtime inference, perfect for interactive applications

• Emotion Control: Emotion exaggeration control for expressive speech (Multilingual only — Turbo ignores the bake-in)

• Paralinguistic Tags (Turbo only): Support for non-speech sounds like [clear throat], [sigh], [shush], [cough], [groan], [sniff], [gasp], [chuckle], [laugh]

What is NVIGI Chatterbox TTS?

The NVIDIA In-Game Inferencing (NVIGI) Chatterbox TTS Plugin Pack brings Chatterbox to PC applications and games through the NVIGI plugin architecture. This pack provides optimized integration with support for multiple inference backends (CUDA, Vulkan, D3D12) and seamless GPU sharing with graphics workloads.

Key Features

• High-Quality Speech Synthesis: Natural-sounding voice output using the Chatterbox Turbo (English) or Multilingual (23 languages) model

• Multi-backend Support: Run on CUDA (NVIDIA GPUs), Vulkan (any Vulkan-capable GPU), or D3D12 (Windows)

• Zero-Shot Voice Cloning: Create custom voices from just a few seconds of reference audio via speaker embeddings (each embedding is bound to either Turbo or Multilingual and is not interchangeable)

• Long-Text Handling: Built-in text chunking for processing longer prompts (word-based by default; character-based for Multilingual zh / ja)

• Paralinguistic Tags (Turbo only): Support for expressive tags like [clear throat], [sigh], [shush], [cough], [groan], [sniff], [gasp], [chuckle], [laugh]. The Multilingual model’s vocabulary does not include these tags.

• CFG-Controlled Speech Pacing (Multilingual only): Runtime cfg_weight for tunable pacing. Turbo runs without CFG.

• Language-Specific Preprocessing (Multilingual only): Chinese Cangjie, Japanese Kanji→Hiragana, Korean Jamo, NFKD normalization, and Hebrew diacritization happen inside the plugin

• CIG Support: CUDA In Graphics optimization for efficient GPU sharing with rendering

• Full Voice Pipeline: Optional integration with GPT and ASR for complete voice-to-voice AI interactions

For the full per-feature breakdown of what is exclusive to Turbo vs Multilingual, see Model Feature Comparison in the Programming Guide.

Architecture

The NVIGI SDK uses a plugin architecture where TTS functionality is provided through the nvigi.plugin.tts.chatterbox plugins. These plugins:

• Implement the standard NVIGI plugin API for easy integration

• Support CUDA, Vulkan, and D3D12 backends

• Can be combined with GPT and ASR plugins for voice-to-voice AI pipelines

• Integrate with D3D12/CUDA for concurrent graphics and AI workloads

Quick Start Guide

Prerequisites

Before getting started, ensure you have:

• Hardware:

  • GPU: NVIDIA GPU with Compute Capability 7.0+ (RTX 20x0 series or newer)

  • Memory: 16 GB RAM minimum

  • VRAM: ~2.3 GB for Turbo TTS-only, ~3.5 GB for Multilingual TTS-only, ~8.5 GB+ for full pipeline (ASR + GPT + TTS with 3D sample)

• Software:

  • Windows 10/11 (64-bit)

  • NVIDIA graphics driver r580 or newer

  • Visual Studio 2022 with C++ development tools (for rebuilding samples)

1. Set Up the Environment

Before running any samples, you need to set up the required dependencies using the provided setup script:

Basic Setup (TTS Only):

.\setup_sample.bat

Or simply double-click setup_sample.bat in Windows Explorer to run it.

This downloads and installs the NVIGI Core SDK automatically.

Important: The NVIGI core runtime DLLs (nvigi.core.framework.dll, nvigi.plugin.hwi.cuda.dll, nvigi.plugin.hwi.common.dll, nvigi.plugin.hwi.d3d12.dll, amd_ags_x64.dll) are not distributed with this pack — they are pulled in by setup_sample.bat from the NVIGI Core SDK via packman. Running any of the samples before executing setup_sample.bat will fail at DLL load with a missing nvigi.core.framework.dll.

Full Setup (ASR + GPT + TTS):

To enable the complete voice-to-voice AI pipeline with GPT and ASR support:

1. Download the NVIGI Developer Full SDK from the NVIDIA In-Game Inferencing SDK page

2. Extract the downloaded SDK to a local directory

3. Run the setup script with the path to the extracted SDK:

.\setup_sample.bat "C:\path\to\nvigi_developer_full_sdk"

This will copy the GPT and ASR plugin DLLs and download the required models for voice-to-voice AI interactions.

2. Run Your First TTS Sample (Command-Line)

The fastest way to experience TTS is with the pre-built command-line sample:

cd bin\x64\Release
.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\aaron_turbo.json --text "Hello from Chatterbox Turbo TTS." --output hello.wav --backend cuda

This will:

• Initialize the Chatterbox TTS plugin with the Turbo model

• Load the specified speaker embedding for voice characteristics

• Generate speech from the input text

• Save the output as a 24 kHz WAV file

• Display timing statistics (inference time, audio length, RTF)

Tip — short game dialogue: Use complete sentences (>5 words) with standard punctuation for best quality. Very short utterances common in games — e.g. "Yes", "Help!", "Over here!" — may produce audio artifacts because the model has too little context to anchor prosody. The simplest mitigation is to pad short lines ("Yes, I will." instead of "Yes.") or pre-render canned barks at build time. See Mitigations for short game dialogue for the full list of strategies.

Try different voices:

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\lucy_turbo.json --text "This is Lucy speaking with Chatterbox Turbo." --output lucy_hello.wav --backend cuda

Try interactive mode (Windows only):

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\aaron_turbo.json --interactive --backend cuda

Interactive mode allows you to enter text prompts continuously with real-time audio playback.

Try multilingual TTS (23 languages):

.\nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\french.json --multilingual --language fr --text "Bonjour le monde." --output bonjour.wav

The --multilingual flag selects the multilingual model, and --language specifies the target language code (e.g., fr, de, zh, ja, hi).

Batch mode (JSONL, one model load): For many utterances, pass --batch path/to/jobs.jsonl with --multilingual and your backend (--backend / -b). Each line is one JSON object with required string fields "out" (WAV path) and "text", and optional "lang" and "speaker" (defaults fall back to --language / --speaker). Use --batch_skip_existing true to skip lines whose output file already exists, and --batch_quiet true for less console output. The sample loads NVIGI once and reuses the same instance for every line.

For detailed CLI options, see CLI Sample documentation.

3. Try the 3D Interactive Sample

Experience TTS in a full 3D application with GUI:

cd bin\x64\Release
.\nvigi.3d.exe

In D3D12 mode (default) - Full voice-to-voice pipeline with D3D12 backends (CUDA fallback):

• Speak into your microphone (ASR transcribes your speech)

• GPT generates a response to your input

• Chatterbox TTS speaks the response aloud

• Or type text directly and press Enter to interact

• Uses D3D12 backends for TTS, GPT, and ASR when available (requires driver 580.61+)

• Automatically falls back to CUDA backends if D3D12 plugins are unavailable

Note: D3D12 backend performance characteristics:

• First inference may take approximately 6 seconds (one-time shader compilation cost).

• Changing speaker embeddings may add approximately 1 second delay to the first inference after the change.

Mitigation: Run a one-off dummy inference during your application’s loading screen — and once per speaker embedding you intend to use during gameplay — so the user-visible inference always runs at steady-state speed. See D3D12 Warmup Latency for the full recipe and additional strategies.

In Vulkan mode - Full voice-to-voice pipeline with Vulkan backends:

.\nvigi.3d.exe -vk

• Same full pipeline as D3D12 mode (ASR, GPT, TTS)

• Uses Vulkan backends for TTS, GPT, and ASR inference

Note: When attempting to share the Vulkan device between the graphics pipeline and the Chatterbox TTS plugin, the generated output may be incorrect. The plugin creates its own separate Vulkan device (without sharing) to ensure correct output.

For detailed 3D sample usage, see 3D Sample documentation.

4. Explore the Models and Speaker Embeddings

The pack includes Chatterbox TTS models and sample speaker embeddings:

• Chatterbox Turbo Model (English-only): Located in data/nvigi.models/nvigi.plugin.tts.chatterbox/{019BD494-0D97-7223-B9D5-C9286933B8B7}/

  • Fastest inference (no CFG, no alignment analysis, single-step Meanflow decoder)

  • Supports paralinguistic tags ([laugh], [sigh], etc.)

  • Ignores cfg_weight and language_id runtime parameters

  • Generates 24 kHz audio output

• Chatterbox Multilingual Model (23 languages): Located in data/nvigi.models/nvigi.plugin.tts.chatterbox/{A60EB5CF-9551-4B86-865B-CDC3CDBE61C4}/

  • CFG-controlled pacing (tunable via cfg_weight, default 0.5)

  • Built-in alignment analyzer suppresses early EOS / runaway generation

  • Language-specific preprocessing (Korean Jamo, NFKD normalization, lowercase) plus auto-discovered auxiliary assets for Chinese (cangjie_mapping.json), Japanese (kanji_readings.json), and Hebrew (dicta-bert-q8*.gguf)

  • language_id runtime parameter is required (ISO code such as "en", "fr", "zh")

  • Does not support paralinguistic tags

  • Generates 24 kHz audio output

See Model Feature Comparison for the full feature-by-model matrix.

• All models support CUDA, Vulkan, and D3D12 backends

• Speaker Embeddings: Located in data/nvigi.test/spk_emb/

  • Turbo voices: aaron_turbo.json, lucy_turbo.json

  • Multilingual per-language voices: arabic.json, chinese.json, danish.json, dutch.json, english.json, finnish.json, french.json, german.json, greek.json, hebrew.json, hindi.json, italian.json, japanese.json, korean.json, malay.json, norwegian.json, polish.json, portuguese.json, russian.json, spanish.json, swahili.json, swedish.json, turkish.json (23 total, one per supported language)

  • Additional multilingual voices: ethan_multilingual.json, meera_multilingual.json, speaker_base.json

  • Turbo and Multilingual embeddings are not interchangeable — use one that matches the selected model type

  • Add your own custom voices via voice cloning

For full model details, see Models and Test Data.

5. Create Custom Voices with Voice Cloning

The Chatterbox TTS plugin supports zero-shot voice cloning, allowing you to create custom speaker embeddings from just a few seconds of reference audio.

Voice Cloning Requirements

• Python 3.11 (automatically installed if not found)

• NVIDIA GPU with CUDA support (recommended) or CPU

• 5-10 seconds of clear reference audio (WAV format)

• Hugging Face account with access token (for downloading the Chatterbox model)

RTX 50-series (Blackwell) note: The packaged torch==2.6.0+cu124 (which matches chatterbox-tts 0.1.7’s pin) does not contain sm_120 kernels, so on RTX 50-series cards get_voice_embeddings.py automatically falls back to CPU and prints a clear [device] WARNING block at startup. Extraction still completes correctly and the resulting speaker JSON works at full GPU speed in the runtime TTS plugin — only this one-off extraction step is slower (~10–14 s per WAV vs ~1–2 s on a supported GPU). To opt into GPU extraction on Blackwell, see “Voice cloning runs on CPU even though the GPU is CUDA-capable” in docs/troubleshooting.md.

Setting Up the Voice Cloning Environment

1. Open PowerShell and navigate to the voice_cloning directory:

cd <PACK_ROOT>\voice_cloning

2. Run the setup script to create the Python virtual environment:

.\setup_venv.ps1

This script will:

• Check for Python 3.11 (or install it automatically if not found)

• Install all required dependencies (PyTorch, Chatterbox TTS, etc.)

Creating Speaker Embeddings

Make sure venv is activated. Use the get_voice_embeddings.py script to extract speaker embeddings from a reference audio file:

Basic usage:

python get_voice_embeddings.py --wav path/to/reference_audio.wav --dump-json my_voice.json

With Hugging Face token (required for first-time model download):

python get_voice_embeddings.py --wav reference_audio.wav --dump-json my_voice.json --hf-token YOUR_HF_TOKEN

Or set the environment variable:

$env:HF_TOKEN = "YOUR_HF_TOKEN"
python get_voice_embeddings.py --wav reference_audio.wav --dump-json my_voice.json

With custom emotion exaggeration:

python get_voice_embeddings.py --wav reference_audio.wav --dump-json my_voice.json --exaggeration 0.8

Voice Cloning Command-line Options

Argument	Description
--wav <path>	Path to the reference WAV file (5-10 seconds recommended)
`–model-type <turbo	base
--dump-json <path>	Output path for the speaker embedding JSON file
--exaggeration <float>	Emotion exaggeration value baked into the embedding (default: 0.5, range: 0.0-1.0). Affects Multilingual output only; ignored by Turbo.
--hf-token <token>	Hugging Face token for model download (or use HF_TOKEN env var)

Tips for Best Results

• Audio Quality: Use clean, high-quality audio with minimal background noise

• Duration: 5-10 seconds of speech is optimal. Less may reduce quality, more won’t significantly improve it

• Format: WAV format is required. Convert other formats using tools like FFmpeg

• Content: Natural speech works best. Avoid music, singing, or heavily processed audio

• Emotion: The --exaggeration parameter controls how much emotion variation is captured (0.0 = neutral, 1.0 = highly expressive)

Using Custom Speaker Embeddings

Once you’ve created a speaker embedding JSON file, copy it to the speaker embeddings directory:

copy my_voice.json <PACK_ROOT>\data\nvigi.test\spk_emb\

The new voice will appear in the speaker dropdown in the 3D sample, or can be used with the CLI sample:

nvigi.tts.chatterbox.exe --models ..\..\..\data\nvigi.models --speaker ..\..\..\data\nvigi.test\spk_emb\my_voice.json --text "Hello with my custom voice!" --output output.wav

Troubleshooting Voice Cloning

See docs/troubleshooting.md for a fuller list of common first-run failures.

Problem: ModuleNotFoundError: No module named 'chatterbox'
Solution: The venv was not activated in the current shell. Run .\venv\Scripts\Activate.ps1 from voice_cloning\ first, then python get_voice_embeddings.py .... setup_venv.ps1 activates the venv automatically only for the session that ran it.

Problem: Hugging Face authentication error
Solution: Provide a valid HF token via --hf-token or HF_TOKEN environment variable. Get your token from huggingface.co/settings/tokens

Problem: CUDA out of memory
Solution: The script will fall back to CPU if CUDA memory is insufficient. You can also close other GPU applications

Problem: Poor voice quality in output
Solution: Try different reference audio, ensure it’s 5-10 seconds of clear speech, and experiment with the --exaggeration parameter

6. Rebuild Samples from Source (Optional)

To modify samples or debug integration:

1. Open a VS2022 Developer Command Prompt

2. Navigate to the pack root directory

3. Run .\setup.bat to generate build files

4. Open _project/vs2022/nvigi.chatterbox.sln in Visual Studio 2022

5. Build the desired configuration (Debug/Release/Production)

6. Run .\copy_sdk_binaries.bat Release to deploy binaries to bin/x64/Release

For detailed rebuild instructions, see Building the Sample from Source.

7. Integrate into Your Application

Once familiar with the samples, integrating TTS into your own application follows a straightforward five-step pattern. Each step is covered in detail with code examples in the Programming Guide.

Integration Overview

1. Include headers and load the NVIGI core library — Add #include <nvigi.h>, #include <nvigi_ai.h>, and #include "nvigi_tts_chatterbox.h" to your project. At runtime, load nvigi.core.framework.dll and retrieve the API function pointers (nvigiInit, nvigiLoadInterface, etc.). On Windows, if your EXE lives outside bin\x64\<config>, also call SetDefaultDllDirectories(...) + AddDllDirectory(packageBinDir) before loading the core framework so the plugins can later resolve their CUDA / cuBLAS / ggml dependencies. Preferences.utf8PathsToPlugins covers NVIGI plugin discovery only — not the OS-level DLL search the plugins use for their own runtime deps. See Step 1: Initialize NVIGI.

2. Load the TTS plugin interface — Choose the CUDA, Vulkan, or D3D12 plugin and call nvigiGetInterfaceDynamic (passing the nvigiLoadInterface function pointer) to get an ITextToSpeech pointer. See Step 2: Load the TTS Interface.

3. Create a TTS instance with your configuration — Set the model GUID, backend preference, model type (eTurbo or eMultilingual), and VRAM budget. For D3D12, provide device and command queues via D3D12Parameters. Then call createInstance. See Step 3: Create a TTS Instance.

4. Provide text and receive audio — Set the input text via data slots, configure runtime parameters (speaker embedding, temperature, chunking), and call evaluate(). Receive generated PCM audio (24 kHz, 16-bit mono) through the callback. The plugin streams audio in chunks: handle both kInferenceExecutionStateDataPending (intermediate chunks) and kInferenceExecutionStateDone (final chunk), and copy each chunk’s buffer before the callback returns. Callbacks that only handle Done silently truncate multi-chunk utterances. See Step 4: Run Inference.

5. Clean up — Destroy the instance, unload the interface, and shut down NVIGI. See Step 5: Cleanup.

Where to Start in the Code

The two sample implementations serve as integration templates:

• source/samples/nvigi.tts.chatterbox/sample_tts_chatterbox.cpp — Minimal command-line integration showing the full lifecycle (init, create, infer, cleanup) without any graphics dependencies. Start here for the simplest integration path.

• source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp — Real-world integration in a D3D12/Vulkan 3D application with CIG scheduling, showing how TTS fits alongside a rendering loop with GPU scheduling modes.

What’s Included in This Pack

The Chatterbox TTS Plugin Pack provides everything needed to integrate text-to-speech into your applications:

Pre-Built Components

• TTS Plugins: nvigi.plugin.tts.chatterbox.cuda.dll, nvigi.plugin.tts.chatterbox.vk.dll, and nvigi.plugin.tts.chatterbox.d3d12.dll

• NVIGI Core: Runtime libraries for the NVIGI plugin framework (downloaded via setup script)

• TTS Models: Chatterbox Turbo (English) and Chatterbox Multilingual (23 languages) with all required components

• Sample Applications: Pre-built executables demonstrating TTS integration

• Headers & APIs: Complete SDK headers for plugin integration

Included Samples

The pack includes two samples with full source code:

• nvigi.tts.chatterbox - Command-line TTS demo

  • Text-to-speech with customizable parameters

  • Multiple backend support (CUDA, Vulkan, D3D12)

  • WAV file output

  • Located in: source/samples/nvigi.tts.chatterbox/

• nvigi.3d - 3D interactive sample with GUI

  • Full voice-to-voice AI pipeline (both D3D12 and Vulkan modes)

  • D3D12 backends (with CUDA fallback) or Vulkan backends for inference

  • Real-world integration example

  • Located in: source/samples/nvigi.3d/

All samples are pre-built in bin/x64/ and ready to run after setup.

Test Data

The pack includes speaker embeddings for testing in data/nvigi.test/spk_emb/:

• Turbo (English): aaron_turbo.json, lucy_turbo.json

• Multilingual per-language embeddings: arabic.json, chinese.json, danish.json, dutch.json, english.json, finnish.json, french.json, german.json, greek.json, hebrew.json, hindi.json, italian.json, japanese.json, korean.json, malay.json, norwegian.json, polish.json, portuguese.json, russian.json, spanish.json, swahili.json, swedish.json, turkish.json (23 total, one per supported language)

• Additional multilingual voices: ethan_multilingual.json, meera_multilingual.json, speaker_base.json

Known Issues and Important Notes

Current Release

General

• Voice Embedding Warmup: The user needs to manually trigger warmup each time they change voice embeddings. First inference with a new speaker embedding may have higher latency.

• Sample applications require the NVIGI core DLLs to be present in the bin/x64/ directory. Run setup_sample.bat to set this up.

TTS Plugin Specific

• CUDA Backend: Recommended for best performance on NVIDIA GPUs.

• Vulkan Backend: Alternative for broader GPU compatibility, but may have slightly lower performance.

  • Vulkan Device Sharing Limitation: When attempting to share the Vulkan device between the graphics pipeline and the Chatterbox TTS plugin, the generated output may be incorrect. The plugin creates its own separate Vulkan device (without sharing) to ensure correct output.

• D3D12 Backend: Windows-only backend using D3D12 compute shaders. Requires providing a D3D12 device and command queues via D3D12Parameters.

• Text Chunking: Long text is automatically split into chunks (default: 40 words per chunk). This improves quality but may add small pauses between chunks.

• Paralinguistic Tags (Turbo only): Tags like [clear throat], [sigh], [shush], [cough], [groan], [sniff], [gasp], [chuckle], [laugh] work best when used in appropriate context — surrounding text should match the emotion for natural results. Do not use these tags with the Multilingual model; its tokenizer does not contain them, so they will either be dropped or read out as individual letters.

• Short Inputs: For best quality, provide complete sentences (>5 words) with standard punctuation. Very short utterances common in games (1–2 words such as "Yes", "Help!", "Over here!") may produce extra unintelligible audio because the model has too little context to anchor prosody. For dialogue-heavy applications, pad short lines ("Yes, I will." instead of "Yes."), use a paralinguistic anchor ("[gasp] Help!"), or pre-render canned barks at build time. See Mitigations for short game dialogue for the full list of strategies.

Building from Source

• Plugin DLLs are pre-built and cannot be rebuilt from this pack.

• Sample applications can be rebuilt from source using the provided Visual Studio solution.

• After building, run .\copy_sdk_binaries.bat to deploy binaries to bin/x64/.

Performance Considerations

• VRAM Usage: Turbo TTS-only requires ~2.3 GB VRAM, Multilingual TTS-only requires ~3.5 GB VRAM. Full pipeline (ASR + GPT + TTS) with 3D sample requires ~8.5 GB VRAM.

• First Inference: Initial inference may be slower due to model warmup. On the D3D12 backend this is approximately 6 seconds for one-time shader compilation; CUDA and Vulkan see a smaller cold-start cost. Subsequent inferences run at steady-state speed.

  • Mitigation: Trigger a dummy inference (any short text, output discarded) on the loading screen so the user never observes this delay during gameplay. See D3D12 Warmup Latency for the recommended pattern.

• Voice Changes: Switching speaker embeddings adds approximately 1 second to the first inference after the switch on the D3D12 backend.

  • Mitigation: Pre-warm each speaker embedding you intend to use during the loading screen, group dialogue by speaker in latency-sensitive sections to amortise the per-switch cost, and avoid swapping voices on hot paths.

Getting Help

For issues, questions, or feedback:

• Review the Programming Guide for detailed API documentation

• Check Samples Documentation for usage examples and troubleshooting

• See the Troubleshooting section in samples.md

Next Steps

Now that you’re familiar with the basics:

1. Run the Samples - Start with the CLI sample for quick testing, then explore the 3D sample for full pipeline experience

2. Read the Programming Guide - Deep dive into the Chatterbox TTS API

3. Experiment with Voices - Try different speaker embeddings and explore voice cloning

4. Try Paralinguistic Tags (Turbo model only) - Add expressiveness with tags like [clear throat], [sigh], [gasp], [laugh]

5. Integrate into Your App - Use the samples as templates for your own application integration

For the best development experience, start with the command-line sample to understand TTS basics, then move to the 3D sample to see TTS in a complete voice-to-voice AI context.