TTS Chatterbox Programming Guide

Overview

The NVIDIA In-Game Inferencing (NVIGI) Chatterbox TTS plugin provides high-quality text-to-speech using GGML/llama.cpp. The plugin supports multiple backends (CUDA, Vulkan, D3D12) and integrates with NVIGI’s plugin architecture for easy application integration.

IMPORTANT: This guide might contain pseudo code, for the up to date implementation and source code which can be copy pasted please see the SDK’s Basic command line sample. For a modern C++ example (RAII, std::expected, builder patterns), see basic_tts.cpp, which uses the C++ wrapper in tts.hpp. For the low-level C API integrated into an ASR->GPT->TTS pipeline, see basic.cpp, and for graphics integration see NVIGIContext.cpp.

NOTE: It is best to use one GPU inference API (CUDA, Vulkan or D3D12) per application run/process and not switch between them at runtime for optimal performance and stability.

Key features:

Multi-backend support: CUDA, Vulkan or D3D12
Two model types: Chatterbox Turbo (English, faster, no CFG) and Chatterbox Multilingual (23 languages, CFG-controlled pacing). Some features are exclusive to one model — see Model Feature Comparison.
Speaker-embedding voices: Voice conditioning is supplied as a speaker embedding JSON; sample embeddings ship in the pack test data, and you can create your own with the bundled voice-cloning toolkit.
Long-text handling: Built-in text chunking (word-based by default; character-based for zh/ja in the Multilingual model)
CIG support: CUDA In Graphics for efficient GPU sharing for CUDA backend.
Paralinguistic tags (Turbo only): Support for [clear throat], [sigh], [shush], [cough], [groan], [sniff], [gasp], [chuckle], [laugh]. The Multilingual model’s tokenizer does not include these tags — they are silently dropped or split into letters and will not produce the intended non-speech sound.
Languages supported: English (Turbo); Arabic, Danish, German, Greek, English, Spanish, Finnish, French, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Swahili, Turkish, Chinese (Multilingual)

Model Feature Comparison

Many features behave differently — or only exist — for one of the two model types. The table below is the single source of truth for the rest of this document; per-feature sections below repeat the same restrictions where they apply.

Feature	Turbo (`eTurbo`)	Multilingual (`eMultilingual`)
Languages	English only	23 languages (ar, da, de, el, en, es, fi, fr, he, hi, it, ja, ko, ms, nl, no, pl, pt, ru, sv, sw, tr, zh)
`language_id` runtime parameter	Ignored (always English)	Required — the model uses a `[xx]` language token prepended to the text
`cfg_weight` (Classifier-Free Guidance / speech pacing)	Not used — value is ignored, internally forced to 0	Always on, default 0.5; tune at runtime per utterance
Paralinguistic tags `[laugh]` `[sigh]` `[gasp]` `[cough]` `[chuckle]` `[groan]` `[sniff]` `[shush]` `[clear throat]`	Supported — emitted in the trained voice	Not supported — tags are not in the multilingual vocabulary; do not use
Speaker embedding JSON	Turbo speaker JSON (e.g. the shipped `*_turbo.json`)	Multilingual speaker JSON. Not interchangeable with Turbo (rejected at `createInstance`).
Language-specific preprocessing (lowercase, NFKD, Korean Jamo, Chinese Cangjie, Japanese Kanji->Hiragana, Hebrew diacritization)	N/A	Applied automatically based on `language_id`. Chinese, Japanese and Hebrew also auto-load auxiliary assets (`cangjie_mapping.json`, `kanji_readings.json`, `dicta-bert-q8*.gguf`) from the model directory.
Text chunking (`maxWordsPerChunk`)	Word-based, default 40	Word-based by default; switches to character-based (30 chars per chunk) for `zh` and `ja`
Hallucination protection (EOS-rank alignment analyzer)	Not used	Active — suppresses early EOS and forces termination on degenerate / runaway outputs
S3Gen decoder	Single-step Meanflow (1 ODE step)	Standard 5-step ODE
LLaMA backbone	GPT-2 (`turbo_gpt2_backbone_v2.gguf`, ~250 MB F32)	LLaMA 520M (`multilingual_chatterbox_llama_backbone_tts_q4_0.gguf`, Q4_0)
Text tokenizer	GPT-2 byte-level BPE (50,276 vocab)	Standard BPE with explicit `[SPACE]` tokens + per-language `[xx]` tokens (2,454 vocab)
Model GUID	`{019BD494-0D97-7223-B9D5-C9286933B8B7}`	`{A60EB5CF-9551-4B86-865B-CDC3CDBE61C4}`
Required model files	`t3_turbo_text_emb.gguf`, `turbo_gpt2_backbone_v2.gguf`, `S3Gen-Meanflow-266M-F32.gguf`	`multilingual_t3_text_emb.gguf`, `multilingual_chatterbox_llama_backbone_tts_q4_0.gguf`, `S3Gen-Multilingual-F16.gguf` (+ auxiliary JSON/GGUF for zh / ja / he)
Recommended `vramBudgetMB`	>= 2,048 MB	>= 3,584 MB
Output audio	24 kHz, 16-bit PCM mono	24 kHz, 16-bit PCM mono

Available Plugins

Plugin	Backend	Plugin ID	Use Case
`nvigi.plugin.tts.chatterbox-ggml.cuda`	GGML CUDA	`nvigi::plugin::tts::chatterbox::ggml::cuda::kId`	NVIDIA GPUs with CUDA, best performance
`nvigi.plugin.tts.chatterbox-ggml.vulkan`	GGML Vulkan	`nvigi::plugin::tts::chatterbox::ggml::vulkan::kId`	Any Vulkan-capable GPU
`nvigi.plugin.tts.chatterbox-ggml.d3d12`	GGML D3D12	`nvigi::plugin::tts::chatterbox::ggml::d3d12::kId`	Windows D3D12-capable GPU

Key Concepts

Output Audio Format

The plugin generates 24 KHz, 16-bit PCM mono output audio.

Tips for Good Generation

Paralinguistic tags work for the Turbo model only. When using eTurbo, include tags such as [laugh], [sigh] or [gasp] in surrounding text that conveys the corresponding emotion (e.g. surround [gasp] with text that conveys surprise or shock) for natural results. Do not use paralinguistic tags with the Multilingual model — they are not in its tokenizer vocabulary and will either be dropped or split into individual letters that read aloud as nonsense.
For the Multilingual model, always set language_id to one of the 23 supported ISO codes; an empty / null value falls back to no language token, which can produce wrong-language pronunciation.

Known Limitations: Input Text Quality

The Chatterbox model produces the best results with grammatically complete sentences and standard punctuation. The following input patterns may result in reduced quality or garbled audio:

Incomplete words or sentence fragments such as “the”, “that”, “hundred”, or “a” on their own. The model may not recognize these as complete utterances and can generate extra, unintelligible audio after the spoken word.
Very short utterances of fewer than ~5 words such as "Yes", "Help!", or "Over here!" — common in game dialogue, but more likely to produce audio artifacts because the model has too little context to anchor prosody and end-of-utterance detection. See Mitigations for short game dialogue below.
Repeated or excessive punctuation such as Hello!!!, Really?!, or Wait... may produce incorrect audio.

Recommendations:

Always provide complete, grammatically correct sentences with proper punctuation.
Use single punctuation marks at the end of sentences (e.g., "Hello!" instead of "Hello!!!").
For best results, provide sentences of moderate to long length (>5 words). Very short inputs (1–2 words) are more likely to produce artifacts.

Mitigations for short game dialogue

Action lines like "Yes", "Help!", or "Over here!" are unavoidable in interactive dialogue. To keep them reliable:

Pad with a complete frame. Send "Yes, I will." or "Yes, sir." instead of bare "Yes." — the surrounding words give the model enough context to terminate cleanly. The extra word is cheap and rarely audible as filler in game audio.
Use a paralinguistic anchor. Tags like [laugh], [sigh], or [gasp] placed before or after a short line stabilise the prosody, e.g. "[gasp] Help!".
Pre-render canned lines. For a fixed roster of short barks ("Reloading.", "Take cover!", etc.), generate the WAVs once at build time or first-launch and play them back as ordinary audio assets — short utterances don’t need to be re-synthesised every time the line plays.
Concatenate consecutive short lines. If two short lines from the same speaker fire back-to-back ("Yes." "On it."), send them as one utterance ("Yes, on it.") so the model has more material to work with.

Data Slots

Data slots are the named bindings that NVIGI uses to pass inputs and outputs to an inference call. For Chatterbox TTS, you provide the prompt under the "text" slot and receive audio from the "audio" slot.

Slot Name	Constant	Direction	Type	Description
Prompt	`nvigi::kTTSDataSlotPrompt`	Input	`InferenceDataText`	Input text to speak
Generated Audio	`nvigi::kTTSDataSlotGeneratedAudio`	Output	`InferenceDataAudio`	Output PCM audio (24 kHz)

NOTE: The Chatterbox output slot is InferenceDataAudio (not InferenceDataByteArray as with some older TTS plugins). Read the PCM samples from audio->audio (a CpuData).

Model

Two model types are available. See Model Feature Comparison for the full per-feature breakdown.

Turbo (eTurbo): English-only, ~2× faster than Multilingual; supports paralinguistic tags; ignores cfg_weight and language_id.
- Files: t3_turbo_text_emb.gguf, S3Gen-Meanflow-266M-F32.gguf, turbo_gpt2_backbone_v2.gguf
- GUID: {019BD494-0D97-7223-B9D5-C9286933B8B7}
Multilingual (eMultilingual): 23-language model with always-on CFG, language-specific preprocessing, and a built-in alignment analyzer that suppresses early EOS / runaway generation.
- Files: multilingual_t3_text_emb.gguf, S3Gen-Multilingual-F16.gguf, multilingual_chatterbox_llama_backbone_tts_q4_0.gguf
- Auxiliary preprocessing assets auto-discovered from the model directory: cangjie_mapping.json (Chinese), kanji_readings.json (Japanese), dicta-bert-q8.gguf + dicta-bert-q8-heads.gguf (Hebrew diacritization). The plugin probes for these files at createInstance time; missing files only disable the corresponding language’s preprocessing — they do not fail instance creation.
- Supported languages: ar, da, de, el, en, es, fi, fr, he, hi, it, ja, ko, ms, nl, no, pl, pt, ru, sv, sw, tr, zh
- language_id in runtime parameters is required (e.g. "en", "fr", "zh"). When set, the plugin prepends a [xx] language token, applies lowercase + NFKD normalization, and runs language-specific transforms (Korean Jamo decomposition for ko, Cangjie encoding for zh, Kanji->Hiragana conversion for ja, DictaBERT diacritization for he). Other languages pass through with only the lowercase/NFKD normalization.
- Does not support paralinguistic tags — the multilingual tokenizer does not contain [laugh], [sigh], etc. Sending such tags produces no expressive effect.

Select the model type during instance creation using TTSChatterboxCreationParameters::modelType.

Speaker Embeddings

The runtime requires a speaker embedding JSON file that contains speaker conditioning data. Provide it through:

TTSChatterboxRuntimeParameters::speaker_json_path

Sample embeddings are included in data/nvigi.test/nvigi.tts/chatterbox/spk_emb/:

Turbo (English): aaron_turbo.json, lucy_turbo.json
Multilingual (per-language): arabic.json, chinese.json, dutch.json, english.json, finnish.json, french.json, german.json, greek.json, hebrew.json, hindi.json, italian.json, japanese.json, korean.json, polish.json, portuguese.json, russian.json, spanish.json, turkish.json
Additional multilingual voices: ethan_multilingual.json, meera_multilingual.json, speaker_base.json

Turbo and Multilingual speaker JSONs are not interchangeable: a multilingual voice must be used with the Multilingual model (and a Turbo voice with the Turbo model), otherwise instance creation is rejected.

Creating Custom Speaker Embeddings (Voice Cloning)

To clone a new voice, extract a speaker embedding JSON from a short reference recording using the bundled Python toolkit at data/nvigi.test/nvigi.tts/chatterbox/voice_cloning/.

One-time setup (Windows, PowerShell). setup_venv.ps1 is located in data/nvigi.test/nvigi.tts/chatterbox/voice_cloning/. It downloads an embedded Python 3.11 if needed, creates a local venv/, and installs the requirements:

cd data\nvigi.test\nvigi.tts\chatterbox\voice_cloning
.\setup_venv.ps1
# In a new shell later, re-activate first: .\venv\Scripts\Activate.ps1

Extract an embedding. The toolkit downloads the upstream Chatterbox weights from Hugging Face, so a Hugging Face token is required (pass --hf-token, or set the HF_TOKEN / HUGGING_FACE_HUB_TOKEN environment variable). Use a 5–10 second, clean mono .wav of the target speaker. Match --model-type to the model you will use at runtime:

# For the Turbo model
python get_voice_embeddings.py --wav reference.wav --model-type turbo --dump-json my_voice_turbo.json --hf-token hf_...

# For the Multilingual model
python get_voice_embeddings.py --wav reference.wav --model-type multilingual --dump-json my_voice_multilingual.json --hf-token hf_...

Key points:

The output JSON contains a model_type tag that the C++ plugin validates at createInstance time. Loading a turbo JSON with eMultilingual (or vice versa) is rejected with kResultInvalidParameter rather than producing garbled audio.
For Multilingual voices, the optional --default-language fr argument stores an ISO language preference in the output JSON. It does not restrict the voice to that language; callers can override language_id at runtime.
--exaggeration (default 0.5) controls emotion intensity baked into the embedding. It is applied for Multilingual but is a no-op for Turbo.
For best quality, use 5–10 seconds of clean reference audio.
Point TTSChatterboxRuntimeParameters::speaker_json_path (or the sample --speaker flag) at the generated JSON.

NOTE: The venv/ and python311/ directories the setup creates are local, git-ignored, and are not shipped in the pack.

Speech Pacing (CFG Weight) — Multilingual only

The cfg_weight parameter controls speech pacing for the Multilingual model via Classifier-Free Guidance:

Lower values (0.3–0.5): Slower, more natural speech with pauses and prosody
Higher values (1.0–3.0): Faster, more text-aligned speech
Default: 0.5 (matches PyTorch reference)
Only applies to the Multilingual model (eMultilingual). The Turbo model does not use CFG: the plugin forces cfg_weight to 0 internally, and any value passed in is silently ignored.

Set via TTSChatterboxRuntimeParameters::cfg_weight.

Text Chunking

Long text is split into chunks for smoother output:

Controlled by TTSChatterboxRuntimeParameters::maxWordsPerChunk
Default: 40
Set to 0 to disable chunking

Chunking uses punctuation-aware splitting and adds small fade-in trims to reduce audio artifacts when stitching.

For the Multilingual model with language_id set to "zh" or "ja", the plugin automatically switches to character-based chunking (~30 characters per chunk) because CJK text has no whitespace between words. The maxWordsPerChunk value still gates whether chunking is enabled (> 0) but its numeric magnitude is not used for these languages.

Hallucination Protection — Multilingual only

The Multilingual model includes an alignment analyzer that monitors the EOS token’s logit rank during generation and steers the sampler to prevent two failure modes that arise from the larger, more permissive vocabulary:

Premature stop: EOS is suppressed for at least 8 generation steps so very short prompts cannot collapse to silence.
Runaway generation: when EOS is consistently in the top-5 logits for 3 sustained steps, EOS is forced so the model terminates instead of looping.

There is no API knob for this — it is always on when modelType == eMultilingual and inactive when modelType == eTurbo. The Turbo model uses simple eos_token_id-based termination because its smaller GPT-2 vocabulary makes runaway generation less likely.

Backends and Graphics Integration

Backend options: cuda, vulkan, d3d12

CUDA backend: Supports CUDA In Graphics (CIG) for efficient GPU sharing with graphics.
Vulkan backend: Uses GGML’s Vulkan context for GPU inference.
- Note: When attempting to share the Vulkan device between the graphics pipeline and the Chatterbox TTS plugin, the generated output may be incorrect. The plugin creates its own separate Vulkan device (without sharing) to ensure correct output.
D3D12 backend: Uses GGML’s D3D12 compute shaders for GPU inference on Windows. Requires providing a D3D12 device, command queues, and memory allocation callbacks via D3D12Parameters.
- Note: First inference may take approximately 6 seconds. Changing speaker embeddings may add approximately 1 second delay to the first inference after the change. Application developers should handle these delays appropriately.

Scheduling Mode Options

When sharing the GPU between graphics and TTS inference, choose a scheduling mode based on your application’s priorities:

Mode	When to Use	Trade-off
Prioritize Graphics	Frame rate is critical and higher TTS latency can be tolerated (e.g., visually intensive scenes with background narration)	Smooth rendering, slower TTS response
Prioritize Inference	Low TTS latency is critical and some frame drops can be tolerated (e.g., real-time conversational AI where responsiveness matters most)	Fast TTS response, potential frame drops
Balanced	Good default for most applications	Reasonable frame rates and TTS latency without heavily favoring either workload

Getting Started

Important Note: The code examples in this guide are illustrative and may not run as-is. They are designed to show the API structure and usage patterns. For complete, working implementations, please refer to:

Modern C++ wrapper CLI: source/samples/nvigi.basic.cxx/tts/basic_tts.cpp (uses tts.hpp)

Low-level C API pipeline (ASR->GPT->TTS): source/samples/nvigi.basic/basic.cpp

Graphics Integration: source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp

Step 1: Initialize NVIGI

Please read the Programming Guide located in the NVIGI Core package to learn more about initializing and shutting down the NVIGI SDK. The modern C++ wrapper (core.hpp) simplifies this considerably; see basic_tts.cpp.

#include <nvigi.h>
#include <nvigi_ai.h>                  // ITextToSpeech, InferenceExecutionContext, ...
#include "nvigi_tts_chatterbox.h"

nvigi::Preferences pref{};
const char* pluginPaths[] = { sdkPath };
pref.logLevel = nvigi::LogLevel::eDefault;
pref.numPathsToPlugins = 1;
pref.utf8PathsToPlugins = pluginPaths;
pref.utf8PathToLogsAndData = sdkPath;

nvigi::Result result{};
if (NVIGI_FAILED(result, nvigiInit(pref, nullptr, nvigi::kSDKVersion)))
{
    // Handle error
}

Step 2: Load the TTS Interface

Choose a backend plugin ID:

nvigi::PluginID pluginId = nvigi::plugin::tts::chatterbox::ggml::cuda::kId;   // CUDA
// nvigi::PluginID pluginId = nvigi::plugin::tts::chatterbox::ggml::vulkan::kId; // Vulkan
// nvigi::PluginID pluginId = nvigi::plugin::tts::chatterbox::ggml::d3d12::kId;  // D3D12 (Windows only)

nvigi::ITextToSpeech* itts{};
if (NVIGI_FAILED(result, nvigiGetInterfaceDynamic(pluginId, &itts, nvigiLoadInterface)))
{
    // Handle error
}

Step 3: Create a TTS Instance

nvigi::CommonCreationParameters common{};
common.utf8PathToModels = "C:\\path\\to\\models";
// Turbo:        "{019BD494-0D97-7223-B9D5-C9286933B8B7}"
// Multilingual: "{A60EB5CF-9551-4B86-865B-CDC3CDBE61C4}"
common.modelGUID = "{019BD494-0D97-7223-B9D5-C9286933B8B7}";
common.vramBudgetMB = 2048; // Turbo needs >= 2048; Multilingual needs >= 3584.
                            // The model is filtered out (and createInstance fails) if the
                            // model config's declared VRAM exceeds this budget.

nvigi::TTSChatterboxCreationParameters ttsParams{};
ttsParams.modelType = nvigi::TTSChatterboxModelType::eTurbo; // or eMultilingual
ttsParams.chain(common);


nvigi::InferenceInstance* instance{};
if (NVIGI_FAILED(result, itts->createInstance(ttsParams, &instance)))
{
    // Handle error
}

IMPORTANT: The vramBudgetMB acts as a filter. The Multilingual model declares a 3072 MB requirement in its nvigi.model.config.json, so a budget below that will cause the model to be filtered out and createInstance to fail. Set vramBudgetMB to at least 3584 for Multilingual.

D3D12 Backend Setup

When using the D3D12 backend, you must create a D3D12 device and command queues, then pass them via D3D12Parameters chained to the creation parameters. You also need to provide memory allocation callbacks and export the D3D12 Agility SDK version:

#include <nvigi_d3d12.h>
#include <d3d12.h>
#include <dxgi1_6.h>
#include <wrl/client.h>

// D3D12 Agility SDK exports (required at global scope)
extern "C" __declspec(dllexport) UINT         D3D12SDKVersion = 615;
extern "C" __declspec(dllexport) const char* D3D12SDKPath = ".\\D3D12\\";

// Create D3D12 device and queues
Microsoft::WRL::ComPtr<ID3D12Device> d3d12Device;
Microsoft::WRL::ComPtr<ID3D12CommandQueue> directQueue, computeQueue, copyQueue;
// ... (create device via D3D12CreateDevice, create queues via CreateCommandQueue) ...

// Memory allocation callbacks
ID3D12Resource* myCreateCommittedResource(
    ID3D12Device* device, const D3D12_HEAP_PROPERTIES* pHeapProperties,
    D3D12_HEAP_FLAGS HeapFlags, const D3D12_RESOURCE_DESC* pDesc,
    D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE* pOptimizedClearValue,
    void* userContext)
{
    ID3D12Resource* resource = nullptr;
    device->CreateCommittedResource(pHeapProperties, HeapFlags, pDesc,
        InitialResourceState, pOptimizedClearValue, IID_PPV_ARGS(&resource));
    return resource;
}

void myDestroyResource(ID3D12Resource* pResource, void* userContext)
{
    if (pResource) pResource->Release();
}

// Chain D3D12Parameters into creation parameters
nvigi::D3D12Parameters d3d12Params{};
d3d12Params.device = d3d12Device.Get();
d3d12Params.queue = directQueue.Get();         // Direct (graphics) queue
d3d12Params.queueCompute = computeQueue.Get(); // Compute queue
d3d12Params.queueCopy = copyQueue.Get();       // Copy queue
d3d12Params.createCommittedResourceCallback = myCreateCommittedResource;
d3d12Params.destroyResourceCallback = myDestroyResource;
ttsParams.chain(d3d12Params);

// Then call createInstance as usual
itts->createInstance(ttsParams, &instance);

Note: The D3D12 device and queues must remain valid for the lifetime of the TTS instance. See source/samples/nvigi.basic/basic.cpp for a complete working example.

D3D12 Backend Performance Notes:

First inference may take approximately 6 seconds. Application developers should handle this delay appropriately.

Changing speaker embeddings may add approximately 1 second delay to the first inference after the change. Application developers should handle this delay appropriately.

Step 4: Run Inference

// Input text
nvigi::CpuData textBuffer{};
const char* text = "Hello from Chatterbox Turbo TTS.";
textBuffer.buffer = (void*)text;
textBuffer.sizeInBytes = (uint64_t)strlen(text) + 1;
nvigi::InferenceDataText prompt{textBuffer};

std::vector<nvigi::InferenceDataSlot> inputSlots = {
    { nvigi::kTTSDataSlotPrompt, prompt }
};
nvigi::InferenceDataSlotArray inputs{ inputSlots.size(), inputSlots.data() };

// Runtime parameters
nvigi::TTSChatterboxRuntimeParameters runtime{};
runtime.speaker_json_path = "C:\\path\\to\\speaker.json";
runtime.max_new_tokens = 768;
runtime.temperature = 0.8f;
runtime.cfg_weight = 0.5f; // Speech pacing: 0.3-0.5 = natural, 1.0-3.0 = faster (multilingual only)
// runtime.modelType = nvigi::TTSChatterboxModelType::eMultilingual; // must match creation params
// runtime.language_id = "fr";                                       // required for multilingual

// Streaming callback to receive results.
//
// The TTS plugin emits audio in chunks. For multi-chunk utterances you must
// drain `kInferenceExecutionStateDataPending` callbacks AS THEY ARRIVE -- the
// buffer pointed to by `audio->audio` is reused for the next chunk, so
// `Done` alone will only give you the final chunk's audio. Apps that only
// handle `Done` work for very short inputs that fit in a single chunk but
// silently truncate longer ones.
auto callback = [](const nvigi::InferenceExecutionContext* ctx,
                   nvigi::InferenceExecutionState state,
                   void* userData)
{
    // Process audio on both DataPending (intermediate chunk) and Done (last chunk).
    if ((state == nvigi::kInferenceExecutionStateDataPending ||
         state == nvigi::kInferenceExecutionStateDone) && ctx && ctx->outputs)
    {
        const nvigi::InferenceDataAudio* audio{};
        ctx->outputs->findAndValidateSlot(nvigi::kTTSDataSlotGeneratedAudio, &audio);
        if (audio)
        {
            // Copy audio->audio (CpuData) to your own buffer / queue NOW;
            // do NOT retain the pointer across the callback return -- the
            // plugin may reuse the storage for the next chunk.
            const bool isLast = (state == nvigi::kInferenceExecutionStateDone);
            // appendPcm(audio->audio.buffer, audio->audio.sizeInBytes, isLast);
        }
    }
    return state;
};

nvigi::InferenceExecutionContext execCtx{};
execCtx.instance = instance;
execCtx.inputs = &inputs;
execCtx.outputs = nullptr; // Let the plugin provide output slots for the callback
execCtx.runtimeParameters = runtime;
execCtx.callback = callback;
execCtx.callbackUserData = nullptr;

if (NVIGI_FAILED(result, instance->evaluate(&execCtx)))
{
    // Handle error
}

IMPORTANT: Input and output data slots provided within the execution context are only valid during the callback execution. The host application must copy any data it needs before returning from the callback.

NOTE: To cancel TTS inference, return nvigi::kInferenceExecutionStateCancel from the callback.

Async mode

The asynchronous mode (evaluateAsync) lets you submit input prompts while processing continues in the background — useful when TTS should begin before a GPT model has finished responding. When using evaluateAsync, you must send END_PROMPT_ASYNC as the final input to signal that no more text is coming. END_PROMPT_ASYNC is defined in nvigi_tts_chatterbox.h.

NOTE: The nvigi.3d sample drives per-chunk TTS on a dedicated worker thread using the blocking evaluate() (one chunk at a time, in order). This avoids the async job-lifecycle races that can drop or reorder speech when many short chunks arrive back-to-back from a streaming GPT. See NVIGIContext.cpp.

Step 5: Cleanup

itts->destroyInstance(instance);
nvigiUnloadInterface(pluginId, itts);
nvigiShutdown();

API Reference

TTSChatterboxCreationParameters

Field	Type	Default	Description
`modelType`	`TTSChatterboxModelType`	`eTurbo`	`eTurbo` (English, fast) or `eMultilingual` (23 languages)

TTSChatterboxRuntimeParameters

Field	Type	Default	Description
`max_new_tokens`	`uint32_t`	`768`	Max tokens generated by the model
`gpu_device`	`int`	`0`	GPU device index
`n_gpu_layers`	`int`	`99`	GPU layers for GGML/llama.cpp
`temperature`	`float`	`0.8f`	Sampling temperature
`speaker_json_path`	`const char*`	`nullptr`	Path to speaker embeddings JSON (required). Must match the runtime `modelType` — see Model Feature Comparison.
`maxWordsPerChunk`	`int32_t`	`40`	Words per chunk, `0` disables chunking. For Multilingual `zh`/`ja`, chunking switches to character-based (~30 chars).
`modelType`	`TTSChatterboxModelType`	`eTurbo`	Model type (must match creation params)
`language_id`	`const char*`	`nullptr`	Multilingual: required. ISO language code (e.g. `"en"`, `"fr"`, `"zh"`). Turbo: ignored (always English).
`cfg_weight`	`float`	`0.5f`	CFG weight for speech pacing (v2). Lower = slower/natural, higher = faster. Multilingual only — ignored by Turbo.

TTSChatterboxCapabilitiesAndRequirements

Field	Type	Description
`common`	`CommonCapabilitiesAndRequirements*`	Standard model list and budgets
`supportedLanguages`	`const char**`	Language strings from model configs
`modelTypes`	`const TTSChatterboxModelType*`	Model types from configs
`n_languages`	`uint32_t`	Number of entries in supportedLanguages array
`n_modelTypes`	`uint32_t`	Number of entries in modelTypes array

Error Handling

Always check return values with NVIGI_FAILED. Common errors:

Invalid model path or GUID: Verify utf8PathToModels and modelGUID
VRAM budget too small: Increase vramBudgetMB (Multilingual needs >= 3584; a low budget filters the model out and createInstance fails)
Missing speaker JSON: speaker_json_path is required
Speaker / model mismatch: A Turbo speaker JSON with eMultilingual (or vice versa) is rejected with kResultInvalidParameter
Backend mismatch: Ensure CUDA/Vulkan/D3D12 plugin matches your hardware
Missing D3D12 parameters: D3D12 backend requires D3D12Parameters with a valid device and queues

Best Practices

Set model type at creation: Use TTSChatterboxCreationParameters::modelType
Reuse instances: Avoid recreating per request
Provide a matching speaker JSON: Turbo speakers (e.g. aaron_turbo.json) only with eTurbo; multilingual speakers only with eMultilingual. The plugin rejects mismatches with kResultInvalidParameter at createInstance time when the JSON has a model_type tag, or warns at runtime if the tag is missing.
Chunk long text: Keep maxWordsPerChunk around 40 for longer prompts. For Multilingual zh/ja, character-based chunking is used automatically.
Use CUDA when available: Best performance on NVIDIA GPUs
Tune cfg_weight for your use case (Multilingual only): Use 0.3–0.5 for natural narration, 1.0+ for fast UI prompts. Can be changed per-utterance without reinitializing. Has no effect on Turbo.
Use paralinguistic tags only on Turbo: Tags like [laugh], [sigh], [gasp] are part of the Turbo vocabulary and produce in-voice non-speech sounds. The Multilingual model’s vocabulary does not contain them.
Always set language_id on Multilingual: Pass an explicit ISO code ("en", "fr", "zh", etc.). An empty / null value skips the [xx] language token and language-specific preprocessing, which can produce wrong-language pronunciation.

Performance Benchmarks

Benchmarking Methodology

The benchmarks below use the Chatterbox Turbo model with a representative set of 31 sentences ranging from 14 to 50 words, covering a variety of conversational styles and lengths. All tests were conducted on:

GPU: NVIDIA GeForce RTX 4090
Driver: 591.86

Key metrics:

RTF (Real-Time Factor): Ratio of inference time to generated audio duration. RTF < 1 means faster than real-time. Lower is better.
Speed-up: Equal to 1/RTF, showing how many times faster than real-time the system generates audio.
TTFA (Time to First Audio): Latency from inference start until the first audio chunk is available for playback (3D sample only).

Backend Comparison

3D Sample Performance

Inference running alongside a 3D rendering workload:

Backend	Avg Inference Time	Avg RTF	Speed-up
CUDA	1,281 ms	0.135	7.4x
D3D12	1,414 ms	0.147	6.8x
Vulkan	1,653 ms	0.175	5.7x

D3D12 Warmup Latency

The D3D12 backend has additional first-run overhead:

Event	Additional Latency
First inference after plugin load	~6 seconds (shader compilation)
First inference after changing speaker embeddings	~1 second

Subsequent inferences run at steady-state speeds. These costs cannot be eliminated, but they can be moved off the user-visible critical path by running a one-off dummy inference (any short text, output discarded) during a loading screen after the instance is created. If your application uses multiple speaker embeddings during gameplay, pre-warm each one on the loading screen as well.

Working Code Examples

These examples use the following path placeholders, which refer to locations that differ between a binary developer pack and a GitHub source tree:

<SDK_PLATFORM>: the target CPU architecture. See Platform Support for information on supported platforms.
<Configuration>: the build configuration (Release, Debug, or Production).

For production-ready integration examples, see:

source/samples/nvigi.basic.cxx/tts/basic_tts.cpp — modern C++ wrapper CLI (sync + async)
source/samples/nvigi.basic/basic.cpp — low-level C API in an ASR->GPT->TTS pipeline
source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp — TTS integration and callbacks in a 3D application

Example CLI usage (from bin\<SDK_PLATFORM>\<Configuration>):

# Turbo (default budget is sufficient)
.\nvigi.basic.tts.cxx.exe --sdk . --models ..\..\..\..\data\nvigi.models --backend cuda `
  --guid "{019BD494-0D97-7223-B9D5-C9286933B8B7}" `
  --speaker ..\..\..\..\data\nvigi.test\nvigi.tts\chatterbox\spk_emb\aaron_turbo.json `
  --text "Hello from Chatterbox Turbo." --output out_turbo.wav --play

# Multilingual (needs a larger VRAM budget than the 2048 default)
.\nvigi.basic.tts.cxx.exe --sdk . --models ..\..\..\..\data\nvigi.models --backend d3d12 `
  --guid "{A60EB5CF-9551-4B86-865B-CDC3CDBE61C4}" --multilingual --language fr --vram 4096 `
  --speaker ..\..\..\..\data\nvigi.test\nvigi.tts\chatterbox\spk_emb\french.json `
  --text "Bonjour, ceci est un test." --output out_ml.wav --play

NOTE: In PowerShell, always quote the model GUID (e.g. "{019BD494-...}"); an unquoted {...} is parsed as a script block.

Troubleshooting

This section covers the most common failures and their fixes. The Error Handling section above lists createInstance / evaluate return codes; the cases below are the runtime and voice-cloning issues that are not obvious from a return code alone.

Runtime

Speaker / model mismatch (silent or garbled audio)

Symptom — createInstance fails with a message like “speaker JSON was produced for the turbo model but the runtime is multilingual”.

Cause — The speaker embedding JSON carries a model_type field (turbo / multilingual) that must match the runtime model (eTurbo / eMultilingual). Turbo and Multilingual embeddings have identical shapes but are not interchangeable; a mismatched JSON yields silent or garbled audio.

Fix — Use the matching JSON (Turbo speakers such as aaron_turbo.json only with eTurbo; multilingual speakers only with eMultilingual), or re-extract with the correct --model-type (see voice cloning). Legacy JSONs without a model_type field are accepted with a warning rather than rejected.

`createInstance failed: 0x...`

Cause — Most commonly the model files for the requested GUID are missing, or the backend plugin DLL was not staged next to the executable.

Fix

Verify the GUID directory exists under data/nvigi.models/nvigi.plugin.tts.chatterbox-ggml/ (pull it via project.data.xml if absent).
Verify nvigi.plugin.tts.chatterbox-ggml.<backend>.dll (cuda, vk, or d3d12) is present alongside the sample executable.
Confirm vramBudgetMB is large enough (Multilingual needs >= 3584; a low budget filters the model out).

Voice cloning

The voice-cloning toolkit lives at data/nvigi.test/nvigi.tts/chatterbox/voice_cloning/; setup_venv.ps1 creates a local, git-ignored venv/. These issues apply to that Python step only — they do not affect runtime TTS, which uses the C++ plugin.

`ModuleNotFoundError: No module named 'chatterbox'`

Cause — The venv is not activated in the current shell, so python resolves to the system interpreter. setup_venv.ps1 activates the venv only for the shell that ran it.

Fix

cd data\nvigi.test\nvigi.tts\chatterbox\voice_cloning
.\venv\Scripts\Activate.ps1
python get_voice_embeddings.py ...

Extraction runs on CPU / is slow (~10 s per WAV)

Cause — Torch installed without CUDA support. Confirm with:

.\venv\Scripts\python.exe -c "import torch; print(torch.cuda.is_available(), torch.__version__)"

If the first value is False or the version lacks a +cuXXX suffix, torch is CPU-only. Recreate the venv (requirements.txt pins a CUDA wheel).

Extraction falls back to CPU on Blackwell (RTX 50-series, `sm_120`)

Cause — The pinned torch==2.6.0+cu124 ships kernels up to sm_90 (Hopper). Blackwell is compute capability 12.0, incompatible with those kernels, so the script probes at startup and falls back to CPU instead of crashing. CPU extraction produces bit-identical speaker JSONs — most users can ignore the warning.

Opt-in GPU on Blackwell (unsupported; breaks the chatterbox-tts==0.1.7 torch==2.6.0 pin):

cd data\nvigi.test\nvigi.tts\chatterbox\voice_cloning
.\venv\Scripts\Activate.ps1
pip install --pre torch torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
pip install --no-deps chatterbox-tts==0.1.7

Hugging Face authentication errors

Cause — Missing/invalid token, or a stale cached token at ~/.cache/huggingface/token used silently.

Fix — get_voice_embeddings.py prints the token source on startup; for automated/CI runs always pass an explicit token (--hf-token $env:HF_TOKEN). To clear a stale cached token:

Remove-Item "$env:USERPROFILE\.cache\huggingface\token" -ErrorAction SilentlyContinue

Support

For issues or feedback, refer to the NVIGI Developer Pack documentation or contact NVIDIA Developer Support.

TTS Chatterbox Programming Guide

Overview

Model Feature Comparison

Available Plugins

Key Concepts

Output Audio Format

Tips for Good Generation

Known Limitations: Input Text Quality

Mitigations for short game dialogue

Data Slots

Model

Speaker Embeddings

Creating Custom Speaker Embeddings (Voice Cloning)

Speech Pacing (CFG Weight) — Multilingual only

Text Chunking

Hallucination Protection — Multilingual only

Backends and Graphics Integration

Scheduling Mode Options

Getting Started

Step 1: Initialize NVIGI

Step 2: Load the TTS Interface

Step 3: Create a TTS Instance

D3D12 Backend Setup

Step 4: Run Inference

Async mode

Step 5: Cleanup

API Reference

TTSChatterboxCreationParameters

TTSChatterboxRuntimeParameters

TTSChatterboxCapabilitiesAndRequirements

Error Handling

Best Practices

Performance Benchmarks

Benchmarking Methodology

Backend Comparison

3D Sample Performance

D3D12 Warmup Latency

Working Code Examples

Troubleshooting

Runtime

Speaker / model mismatch (silent or garbled audio)

createInstance failed: 0x...

Voice cloning

ModuleNotFoundError: No module named 'chatterbox'

Extraction runs on CPU / is slow (~10 s per WAV)

Extraction falls back to CPU on Blackwell (RTX 50-series, sm_120)

Hugging Face authentication errors

Support

`createInstance failed: 0x...`

`ModuleNotFoundError: No module named 'chatterbox'`

Extraction falls back to CPU on Blackwell (RTX 50-series, `sm_120`)