Is this page helpful?

Programming Guide: Text-to-Speech with Chatterbox TTS using GGML/llama.cpp.#

Overview#

The NVIDIA In-Game Inferencing (NVIGI) Chatterbox TTS plugin provides high-quality text-to-speech using GGML/llama.cpp. The plugin supports multiple backends (CUDA, Vulkan, D3D12) and integrates with NVIGI’s plugin architecture for easy application integration.

Key features:

  • Multi-backend support: CUDA, Vulkan or D3D12

  • Two model types: Chatterbox Turbo (English, faster, no CFG) and Chatterbox Multilingual (23 languages, CFG-controlled pacing). Some features are exclusive to one model — see Model Feature Comparison.

  • Voice cloning: Voice cloning script to save speaker embeddings as JSON via a separate script.

  • Long-text handling: Built-in text chunking (word-based by default; character-based for zh/ja in the Multilingual model)

  • CIG support: CUDA In Graphics for efficient GPU sharing for CUDA backend.

  • Paralinguistic tags (Turbo only): Support for [clear throat], [sigh], [shush], [cough], [groan], [sniff], [gasp], [chuckle], [laugh]. The Multilingual model’s tokenizer does not include these tags — they are silently dropped or split into letters and will not produce the intended non-speech sound.

  • Languages supported: English (Turbo); Arabic, Danish, German, Greek, English, Spanish, Finnish, French, Hebrew, Hindi, Italian, Japanese, Korean, Malay, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Swahili, Turkish, Chinese (Multilingual)

Model Feature Comparison#

Many features behave differently — or only exist — for one of the two model types. The table below is the single source of truth for the rest of this document; per-feature sections below repeat the same restrictions where they apply.

Feature

Turbo (eTurbo)

Multilingual (eMultilingual)

Languages

English only

23 languages (ar, da, de, el, en, es, fi, fr, he, hi, it, ja, ko, ms, nl, no, pl, pt, ru, sv, sw, tr, zh)

language_id runtime parameter

Ignored (always English)

Required — the model uses a [xx] language token prepended to the text

cfg_weight (Classifier-Free Guidance / speech pacing)

Not used — value is ignored, internally forced to 0

Always on, default 0.5; tune at runtime per utterance

Paralinguistic tags [laugh] [sigh] [gasp] [cough] [chuckle] [groan] [sniff] [shush] [clear throat]

Supported — emitted in the trained voice

Not supported — tags are not in the multilingual vocabulary; do not use

Emotion --exaggeration (voice cloning)

Ignored — bake-in is no-op for the Turbo speaker JSON

Applied — captured in the speaker embedding

Speaker embedding JSON

Must be extracted with --model-type turbo

Must be extracted with --model-type multilingual. Not interchangeable with Turbo (rejected at createInstance).

Language-specific preprocessing (lowercase, NFKD, Korean Jamo, Chinese Cangjie, Japanese Kanji→Hiragana, Hebrew diacritization)

N/A

Applied automatically based on language_id. Chinese, Japanese and Hebrew also auto-load auxiliary assets (cangjie_mapping.json, kanji_readings.json, dicta-bert-q8*.gguf) from the model directory.

Text chunking (maxWordsPerChunk)

Word-based, default 40

Word-based by default; switches to character-based (30 chars per chunk) for zh and ja

Hallucination protection (EOS-rank alignment analyzer)

Not used

Active — suppresses early EOS and forces termination on degenerate / runaway outputs

S3Gen decoder

Single-step Meanflow (1 ODE step)

Standard 5-step ODE

LLaMA backbone

GPT-2 (turbo_gpt2_backbone_v2.gguf, ~250 MB F32)

LLaMA 520M (multilingual_chatterbox_llama_backbone_tts_q4_0.gguf, Q4_0)

Text tokenizer

GPT-2 byte-level BPE (50,276 vocab)

Standard BPE with explicit [SPACE] tokens + per-language [xx] tokens (2,454 vocab)

Model GUID

{019BD494-0D97-7223-B9D5-C9286933B8B7}

{A60EB5CF-9551-4B86-865B-CDC3CDBE61C4}

Required model files

t3_turbo_text_emb.gguf, turbo_gpt2_backbone_v2.gguf, S3Gen-Meanflow-266M-F32.gguf

multilingual_t3_text_emb.gguf, multilingual_chatterbox_llama_backbone_tts_q4_0.gguf, S3Gen-Multilingual-F16.gguf (+ auxiliary JSON/GGUF for zh / ja / he)

Recommended vramBudgetMB

≥ 2,048 MB

≥ 3,584 MB

Output audio

24 kHz, 16-bit PCM mono

24 kHz, 16-bit PCM mono

Available Plugins#

Plugin

Backend

Plugin ID

Use Case

nvigi.plugin.tts.chatterbox.cuda

GGML CUDA

nvigi::plugin::tts::chatterbox::cuda::kId

NVIDIA GPUs with CUDA, best performance

nvigi.plugin.tts.chatterbox.vulkan

GGML Vulkan

nvigi::plugin::tts::chatterbox::vulkan::kId

Any Vulkan-capable GPU

nvigi.plugin.tts.chatterbox.d3d12

GGML D3D12

nvigi::plugin::tts::chatterbox::d3d12::kId

Windows D3D12-capable GPU

Key Concepts#

Output Audio Format#

The plugin generates 24 KHz output audio.

Tips for Good Generation#

  • Paralinguistic tags work for the Turbo model only. When using eTurbo, include tags such as [laugh], [sigh] or [gasp] in surrounding text that conveys the corresponding emotion (e.g. surround [gasp] with text that conveys surprise or shock) for natural results. Do not use paralinguistic tags with the Multilingual model — they are not in its tokenizer vocabulary and will either be dropped or split into individual letters that read aloud as nonsense.

  • For voice cloning, we recommend at least 5 seconds of audio for optimal performance, and no more than 10 seconds as .wav file.

  • For the Multilingual model, always set language_id to one of the 23 supported ISO codes; an empty / null value falls back to no language token, which can produce wrong-language pronunciation.

Known Limitations: Input Text Quality#

The Chatterbox model produces the best results with grammatically complete sentences and standard punctuation. The following input patterns may result in reduced quality or garbled audio:

  • Incomplete words or sentence fragments such as “the”, “that”, “hundred”, or “a” on their own. The model may not recognize these as complete utterances and can generate extra, unintelligible audio after the spoken word.

  • Very short utterances of fewer than ~5 words such as "Yes", "Help!", or "Over here!" — common in game dialogue, but more likely to produce audio artifacts because the model has too little context to anchor prosody and end-of-utterance detection. See Mitigations for short game dialogue below.

  • Repeated or excessive punctuation such as Hello!!!, Really?!, or Wait... may produce incorrect audio.

Recommendations:

  1. Always provide complete, grammatically correct sentences with proper punctuation.

  2. Use single punctuation marks at the end of sentences (e.g., "Hello!" instead of "Hello!!!").

  3. For best results, provide sentences of moderate to long length (>5 words). Very short inputs (1–2 words) are more likely to produce artifacts.

Mitigations for short game dialogue#

Action lines like "Yes", "Help!", or "Over here!" are unavoidable in interactive dialogue. To keep them reliable:

  • Pad with a complete frame. Send "Yes, I will." or "Yes, sir." instead of bare "Yes." — the surrounding words give the model enough context to terminate cleanly. The extra word is cheap and rarely audible as filler in game audio.

  • Use a paralinguistic anchor. Tags like [laugh], [sigh], or [gasp] placed before or after a short line stabilise the prosody, e.g. "[gasp] Help!".

  • Pre-render canned lines. For a fixed roster of short barks ("Reloading.", "Take cover!", etc.), generate the WAVs once at build time or first-launch and play them back as ordinary audio assets — short utterances don’t need to be re-synthesised every time the line plays.

  • Concatenate consecutive short lines. If two short lines from the same speaker fire back-to-back ("Yes." "On it."), send them as one utterance ("Yes, on it.") so the model has more material to work with.

Data Slots#

Data slots are the named bindings that NVIGI uses to pass inputs and outputs to an inference call.
For Chatterbox TTS, you provide the prompt under the "text" slot and receive audio from the "audio" slot.

Slot Name

Constant

Direction

Type

Description

Prompt

nvigi::kTTSDataSlotPrompt

Input

InferenceDataText

Input text to speak

Generated Audio

nvigi::kTTSDataSlotGeneratedAudio

Output

InferenceDataAudio

Output PCM audio (24 kHz)

Model#

Two model types are available. See Model Feature Comparison for the full per-feature breakdown.

  • Turbo (eTurbo): English-only, ~2× faster than Multilingual; supports paralinguistic tags; ignores cfg_weight and language_id.

    • Files: t3_turbo_text_emb.gguf, S3Gen-Meanflow-266M-F32.gguf, turbo_gpt2_backbone_v2.gguf

    • GUID: {019BD494-0D97-7223-B9D5-C9286933B8B7}

  • Multilingual (eMultilingual): 23-language model with always-on CFG, language-specific preprocessing, and a built-in alignment analyzer that suppresses early EOS / runaway generation.

    • Files: multilingual_t3_text_emb.gguf, S3Gen-Multilingual-F16.gguf, multilingual_chatterbox_llama_backbone_tts_q4_0.gguf

    • Auxiliary preprocessing assets auto-discovered from the model directory: cangjie_mapping.json (Chinese), kanji_readings.json (Japanese), dicta-bert-q8.gguf + dicta-bert-q8-heads.gguf (Hebrew diacritization). The plugin probes for these files at createInstance time; missing files only disable the corresponding language’s preprocessing — they do not fail instance creation.

    • Supported languages: ar, da, de, el, en, es, fi, fr, he, hi, it, ja, ko, ms, nl, no, pl, pt, ru, sv, sw, tr, zh

    • language_id in runtime parameters is required (e.g. "en", "fr", "zh"). When set, the plugin prepends a [xx] language token, applies lowercase + NFKD normalization, and runs language-specific transforms (Korean Jamo decomposition for ko, Cangjie encoding for zh, Kanji→Hiragana conversion for ja, DictaBERT diacritization for he). Other languages pass through with only the lowercase/NFKD normalization.

    • Does not support paralinguistic tags — the multilingual tokenizer does not contain [laugh], [sigh], etc. Sending such tags produces no expressive effect.

Select the model type during instance creation using TTSChatterboxCreationParameters::modelType.

Speaker Embeddings#

The runtime requires a speaker embedding JSON file that contains speaker conditioning data. Provide it through:

TTSChatterboxRuntimeParameters::speaker_json_path

Sample embeddings are included in data/nvigi.test/spk_emb/:

  • Turbo (English): aaron_turbo.json, lucy_turbo.json

  • Multilingual (per-language): arabic.json, chinese.json, dutch.json, english.json, finnish.json, french.json, german.json, greek.json, hebrew.json, hindi.json, italian.json, japanese.json, korean.json, polish.json, portuguese.json, russian.json, spanish.json, turkish.json

  • Additional multilingual voices: ethan_multilingual.json, meera_multilingual.json, speaker_base.json

They can also be created via voice cloning script for other speakers. Multilingual embeddings must be produced with --model-type multilingual so the stored cond_emb matches what the multilingual T3 consumes.

Speech Pacing (CFG Weight) — Multilingual only#

The cfg_weight parameter controls speech pacing for the Multilingual model via Classifier-Free Guidance:

  • Lower values (0.3–0.5): Slower, more natural speech with pauses and prosody

  • Higher values (1.0–3.0): Faster, more text-aligned speech

  • Default: 0.5 (matches PyTorch reference)

  • Only applies to the Multilingual model (eMultilingual). The Turbo model does not use CFG: the plugin forces cfg_weight to 0 internally, and any value passed in is silently ignored.

Set via TTSChatterboxRuntimeParameters::cfg_weight.

Text Chunking#

Long text is split into chunks for smoother output:

  • Controlled by TTSChatterboxRuntimeParameters::maxWordsPerChunk

  • Default: 40

  • Set to 0 to disable chunking

Chunking uses punctuation-aware splitting and adds small fade-in trims to reduce audio artifacts when stitching.

For the Multilingual model with language_id set to "zh" or "ja", the plugin automatically switches to character-based chunking (~30 characters per chunk) because CJK text has no whitespace between words. The maxWordsPerChunk value still gates whether chunking is enabled (> 0) but its numeric magnitude is not used for these languages.

Hallucination Protection — Multilingual only#

The Multilingual model includes an alignment analyzer that monitors the EOS token’s logit rank during generation and steers the sampler to prevent two failure modes that arise from the larger, more permissive vocabulary:

  • Premature stop: EOS is suppressed for at least 8 generation steps so very short prompts cannot collapse to silence.

  • Runaway generation: when EOS is consistently in the top-5 logits for 3 sustained steps, EOS is forced so the model terminates instead of looping.

There is no API knob for this — it is always on when modelType == eMultilingual and inactive when modelType == eTurbo. The Turbo model uses simple eos_token_id-based termination because its smaller GPT-2 vocabulary makes runaway generation less likely.

Backends and Graphics Integration#

Backend options: cuda, vulkan, d3d12

  • CUDA backend: Supports CUDA In Graphics (CIG) for efficient GPU sharing with graphics.

  • Vulkan backend: Uses GGML’s Vulkan context for GPU inference.

    • Note: When attempting to share the Vulkan device between the graphics pipeline and the Chatterbox TTS plugin, the generated output may be incorrect. The plugin creates its own separate Vulkan device (without sharing) to ensure correct output.

  • D3D12 backend: Uses GGML’s D3D12 compute shaders for GPU inference on Windows. Requires providing a D3D12 device, command queues, and memory allocation callbacks via D3D12Parameters.

    • Note: First inference may take approximately 6 seconds. Changing speaker embeddings may add approximately 1 second delay to the first inference after the change. Application developers should handle these delays appropriately.

Scheduling Mode Options#

When sharing the GPU between graphics and TTS inference, choose a scheduling mode based on your application’s priorities:

Mode

When to Use

Trade-off

Prioritize Graphics

Frame rate is critical and higher TTS latency can be tolerated (e.g., visually intensive scenes with background narration)

Smooth rendering, slower TTS response

Prioritize Inference

Low TTS latency is critical and some frame drops can be tolerated (e.g., real-time conversational AI where responsiveness matters most)

Fast TTS response, potential frame drops

Balanced

Good default for most applications

Reasonable frame rates and TTS latency without heavily favoring either workload

Getting Started#

Important Note: The code examples in this guide are illustrative and may not run as-is. They are designed to show the API structure and usage patterns. For complete, working implementations, please refer to:

  • Standalone CLI TTS: source/samples/nvigi.tts.chatterbox/sample_tts_chatterbox.cpp

  • Graphics Integration: source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp

Step 1: Initialize NVIGI#

#include <nvigi.h>
#include <nvigi_ai.h>                  // ITextToSpeech, InferenceExecutionContext, kInferenceFlagDataPending, ...
#include "nvigi_tts_chatterbox.h"

#ifndef NVIGI_WINDOWS
#include <dlfcn.h>
#define LoadLibraryA(lib) dlopen(lib, RTLD_LAZY)
#define GetProcAddress dlsym
#define FreeLibrary dlclose
#endif

const char* sdkPath = "C:\\path\\to\\sdk";  // Path to SDK root directory (= bin\x64\<config> in the pack)

#ifdef NVIGI_WINDOWS
// One-time, process-wide DLL search-path setup for the host application.
//
// This is REQUIRED whenever the host EXE lives outside the package's
// bin\x64\<config> directory. Without it, NVIGI's own plugin discovery via
// `Preferences.utf8PathsToPlugins` (below) succeeds, but the plugin DLLs in
// turn dynamically load CUDA/cuBLAS/cuDNN/ggml at first inference and those
// loads use the *process-wide* DLL search order -- not the special flags we
// pass to LoadLibraryExA for the initial core framework load. If the EXE's
// directory and PATH don't already contain cublas64_12.dll, cublasLt64_12.dll,
// etc., the first inference crashes with `STATUS_DLL_NOT_FOUND`.
//
// Three equivalent options; pick one. (1) is recommended for shipping apps:
//
//   (1) Programmatic, no env changes:
//       Call SetDefaultDllDirectories(...) + AddDllDirectory(packageBinDir)
//       once at startup, BEFORE any LoadLibrary* on NVIGI or its plugins.
//   (2) Launch the EXE from inside bin\x64\<config> so it inherits a working
//       current directory -- fragile, especially for double-click launches.
//   (3) Prepend bin\x64\<config> to %PATH% in the parent shell -- works but
//       leaks into the user environment and is brittle across configs.
SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS |
                         LOAD_LIBRARY_SEARCH_USER_DIRS);
std::wstring packageBinDirW(/* widened sdkPath */);
AddDllDirectory(packageBinDirW.c_str());
#endif

// Load the NVIGI core library from SDK path.
// LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR is required so Windows resolves the core
// library's own dependencies (CUDA, HWI, amd_ags_x64.dll) from the same
// directory as nvigi.core.framework.dll rather than only the PATH.
std::string coreLibPath = std::string(sdkPath) + "\\nvigi.core.framework.dll";
HMODULE coreLib = LoadLibraryExA(
    coreLibPath.c_str(), NULL,
    LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);

// Get function pointers
auto nvigiInit = (PFun_nvigiInit*)GetProcAddress(coreLib, "nvigiInit");
auto nvigiShutdown = (PFun_nvigiShutdown*)GetProcAddress(coreLib, "nvigiShutdown");
auto nvigiLoadInterface = (PFun_nvigiLoadInterface*)GetProcAddress(coreLib, "nvigiLoadInterface");
auto nvigiUnloadInterface = (PFun_nvigiUnloadInterface*)GetProcAddress(coreLib, "nvigiUnloadInterface");

// Set up preferences
const char* pluginPaths[] = { sdkPath };
nvigi::Preferences pref{};
pref.logLevel = nvigi::LogLevel::eDefault;  // Options: eOff, eDefault, eVerbose, eCount
pref.showConsole = false;
pref.numPathsToPlugins = 1;
pref.utf8PathsToPlugins = pluginPaths;
pref.utf8PathToLogsAndData = sdkPath;

// Initialize
nvigi::Result result{};
if (NVIGI_FAILED(result, nvigiInit(pref, nullptr, nvigi::kSDKVersion)))
{
    // Handle error
}

Step 2: Load the TTS Interface#

Choose a backend plugin ID:

nvigi::PluginID pluginId = nvigi::plugin::tts::chatterbox::cuda::kId;   // CUDA
// nvigi::PluginID pluginId = nvigi::plugin::tts::chatterbox::vulkan::kId; // Vulkan
// nvigi::PluginID pluginId = nvigi::plugin::tts::chatterbox::d3d12::kId;  // D3D12 (Windows only)

nvigi::ITextToSpeech* itts{};
if (NVIGI_FAILED(result, nvigiGetInterfaceDynamic(pluginId, &itts, nvigiLoadInterface)))
{
    // Handle error
}

Step 3: Create a TTS Instance#

nvigi::CommonCreationParameters common{};
common.utf8PathToModels = "C:\\path\\to\\models";
// Turbo:        "{019BD494-0D97-7223-B9D5-C9286933B8B7}"
// Multilingual: "{A60EB5CF-9551-4B86-865B-CDC3CDBE61C4}"
common.modelGUID = "{019BD494-0D97-7223-B9D5-C9286933B8B7}";
common.vramBudgetMB = 2048; // Maximum VRAM the plugin can use (will error if plugin needs more)

nvigi::TTSChatterboxCreationParameters ttsParams{};
ttsParams.modelType = nvigi::TTSChatterboxModelType::eTurbo; // or eMultilingual
ttsParams.chain(common);


nvigi::InferenceInstance* instance{};
if (NVIGI_FAILED(result, itts->createInstance(ttsParams, &instance)))
{
    // Handle error
}

D3D12 Backend Setup#

When using the D3D12 backend, you must create a D3D12 device and command queues, then pass them via D3D12Parameters chained to the creation parameters. You also need to provide memory allocation callbacks and export the D3D12 Agility SDK version:

#include <nvigi_d3d12.h>
#include <d3d12.h>
#include <dxgi1_6.h>
#include <wrl/client.h>

// D3D12 Agility SDK exports (required at global scope)
extern "C" __declspec(dllexport) UINT         D3D12SDKVersion = 615;
extern "C" __declspec(dllexport) const char* D3D12SDKPath = ".\\D3D12\\";

// Create D3D12 device and queues
Microsoft::WRL::ComPtr<ID3D12Device> d3d12Device;
Microsoft::WRL::ComPtr<ID3D12CommandQueue> directQueue, computeQueue, copyQueue;
// ... (create device via D3D12CreateDevice, create queues via CreateCommandQueue) ...

// Memory allocation callbacks
ID3D12Resource* myCreateCommittedResource(
    ID3D12Device* device, const D3D12_HEAP_PROPERTIES* pHeapProperties,
    D3D12_HEAP_FLAGS HeapFlags, const D3D12_RESOURCE_DESC* pDesc,
    D3D12_RESOURCE_STATES InitialResourceState, const D3D12_CLEAR_VALUE* pOptimizedClearValue,
    void* userContext)
{
    ID3D12Resource* resource = nullptr;
    device->CreateCommittedResource(pHeapProperties, HeapFlags, pDesc,
        InitialResourceState, pOptimizedClearValue, IID_PPV_ARGS(&resource));
    return resource;
}

void myDestroyResource(ID3D12Resource* pResource, void* userContext)
{
    if (pResource) pResource->Release();
}

// Chain D3D12Parameters into creation parameters
nvigi::D3D12Parameters d3d12Params{};
d3d12Params.device = d3d12Device.Get();
d3d12Params.queue = directQueue.Get();         // Direct (graphics) queue
d3d12Params.queueCompute = computeQueue.Get(); // Compute queue
d3d12Params.queueCopy = copyQueue.Get();       // Copy queue
d3d12Params.createCommittedResourceCallback = myCreateCommittedResource;
d3d12Params.destroyResourceCallback = myDestroyResource;
ttsParams.chain(d3d12Params);

// Then call createInstance as usual
itts->createInstance(ttsParams, &instance);

Note: The D3D12 device and queues must remain valid for the lifetime of the TTS instance. See source/samples/nvigi.tts.chatterbox/sample_tts_chatterbox.cpp for a complete working example.

D3D12 Backend Performance Notes:

  • First inference may take approximately 6 seconds. Application developers should handle this delay appropriately.

  • Changing speaker embeddings may add approximately 1 second delay to the first inference after the change. Application developers should handle this delay appropriately.

Step 4: Run Inference#

// Input text
nvigi::CpuData textBuffer{};
const char* text = "Hello from Chatterbox Turbo TTS.";
textBuffer.buffer = (void*)text;
textBuffer.sizeInBytes = (uint64_t)strlen(text) + 1;
nvigi::InferenceDataText prompt{textBuffer};

std::vector<nvigi::InferenceDataSlot> inputSlots = {
    { nvigi::kTTSDataSlotPrompt, prompt }
};
nvigi::InferenceDataSlotArray inputs{ inputSlots.size(), inputSlots.data() };

// Runtime parameters
nvigi::TTSChatterboxRuntimeParameters runtime{};
runtime.speaker_json_path = "C:\\path\\to\\speaker.json";
runtime.max_new_tokens = 768;
runtime.temperature = 0.8f;
runtime.cfg_weight = 0.5f; // Speech pacing: 0.3-0.5 = natural, 1.0-3.0 = faster (multilingual only)

// Streaming callback to receive results.
//
// The TTS plugin emits audio in chunks. For multi-chunk utterances you must
// drain `kInferenceExecutionStateDataPending` callbacks AS THEY ARRIVE -- the
// buffer pointed to by `audio->audio` is reused for the next chunk, so
// `Done` alone will only give you the final chunk's audio. Apps that only
// handle `Done` (the original snippet style) work for very short inputs
// that fit in a single chunk but silently truncate longer ones.
auto callback = [](const nvigi::InferenceExecutionContext* ctx,
                   nvigi::InferenceExecutionState state,
                   void* userData)
{
    // Process audio on both DataPending (intermediate chunk) and Done (last chunk).
    if ((state == nvigi::kInferenceExecutionStateDataPending ||
         state == nvigi::kInferenceExecutionStateDone) && ctx && ctx->outputs)
    {
        const nvigi::InferenceDataAudio* audio{};
        ctx->outputs->findAndValidateSlot(nvigi::kTTSDataSlotGeneratedAudio, &audio);
        if (audio)
        {
            // Copy audio->audio (CpuData) to your own buffer / queue NOW;
            // do NOT retain the pointer across the callback return -- the
            // plugin may reuse the storage for the next chunk.
            const bool isLast = (state == nvigi::kInferenceExecutionStateDone);
            // appendPcm(audio->audio.buffer, audio->audio.sizeInBytes, isLast);
        }
    }
    return state;
};

nvigi::InferenceExecutionContext execCtx{};
execCtx.instance = instance;
execCtx.inputs = &inputs;
execCtx.outputs = nullptr; // Let the plugin provide output slots for the callback
execCtx.runtimeParameters = runtime;
execCtx.callback = callback;
execCtx.callbackUserData = nullptr;

if (NVIGI_FAILED(result, instance->evaluate(&execCtx)))
{
    // Handle error
}

Step 5: Cleanup#

itts->destroyInstance(instance);
nvigiUnloadInterface(pluginId, itts);
nvigiShutdown();
FreeLibrary(coreLib);

API Reference#

TTSChatterboxCreationParameters#

Field

Type

Default

Description

modelType

TTSChatterboxModelType

eTurbo

eTurbo (English, fast) or eMultilingual (23 languages)

TTSChatterboxRuntimeParameters#

Field

Type

Default

Description

max_new_tokens

uint32_t

768

Max tokens generated by the model

gpu_device

int

0

GPU device index

n_gpu_layers

int

99

GPU layers for GGML/llama.cpp

temperature

float

0.8f

Sampling temperature

speaker_json_path

const char*

nullptr

Path to speaker embeddings JSON (required). Must match the runtime modelType — see Model Feature Comparison.

maxWordsPerChunk

int32_t

40

Words per chunk, 0 disables chunking. For Multilingual zh/ja, chunking switches to character-based (~30 chars).

modelType

TTSChatterboxModelType

eTurbo

Model type (must match creation params)

language_id

const char*

nullptr

Multilingual: required. ISO language code (e.g. "en", "fr", "zh"). Turbo: ignored (always English).

cfg_weight

float

0.5f

CFG weight for speech pacing (v2). Lower = slower/natural, higher = faster. Multilingual only — ignored by Turbo.

TTSChatterboxCapabilitiesAndRequirements#

Field

Type

Description

common

CommonCapabilitiesAndRequirements*

Standard model list and budgets

supportedLanguages

const char**

Language strings from model configs

modelTypes

const TTSChatterboxModelType*

Model types from configs

n_languages

uint32_t

Number of entries in supportedLanguages array

n_modelTypes

uint32_t

Number of entries in modelTypes array

Error Handling#

Always check return values with NVIGI_FAILED. Common errors:

  • Invalid model path or GUID: Verify utf8PathToModels and modelGUID

  • VRAM budget too small: Increase vramBudgetMB

  • Missing speaker JSON: speaker_json_path is required

  • Backend mismatch: Ensure CUDA/Vulkan/D3D12 plugin matches your hardware

  • Missing D3D12 parameters: D3D12 backend requires D3D12Parameters with a valid device and queues

Best Practices#

  1. Set model type at creation: Use TTSChatterboxCreationParameters::modelType

  2. Reuse instances: Avoid recreating per request

  3. Provide a matching speaker JSON: Turbo speakers (e.g. aaron_turbo.json) only with eTurbo; multilingual speakers only with eMultilingual. The plugin rejects mismatches with kResultInvalidParameter at createInstance time when the JSON has a model_type tag, or warns at runtime if the tag is missing.

  4. Chunk long text: Keep maxWordsPerChunk around 40 for longer prompts. For Multilingual zh/ja, character-based chunking is used automatically.

  5. Use CUDA when available: Best performance on NVIDIA GPUs

  6. Tune cfg_weight for your use case (Multilingual only): Use 0.3–0.5 for natural narration, 1.0+ for fast UI prompts. Can be changed per-utterance without reinitializing. Has no effect on Turbo.

  7. Use paralinguistic tags only on Turbo: Tags like [laugh], [sigh], [gasp] are part of the Turbo vocabulary and produce in-voice non-speech sounds. The Multilingual model’s vocabulary does not contain them — sending such tags produces no expressive effect and may degrade prosody.

  8. Always set language_id on Multilingual: Pass an explicit ISO code ("en", "fr", "zh", etc.). An empty / null value skips the [xx] language token and language-specific preprocessing, which can produce wrong-language pronunciation.

Performance Benchmarks#

Benchmarking Methodology#

The benchmarks below use the Chatterbox Turbo model with a representative set of 31 sentences ranging from 14 to 50 words, covering a variety of conversational styles and lengths. All tests were conducted on:

  • GPU: NVIDIA GeForce RTX 4090

  • Driver: 591.86

Key metrics:

  • RTF (Real-Time Factor): Ratio of inference time to generated audio duration. RTF < 1 means faster than real-time. Lower is better.

  • Speed-up: Equal to 1/RTF, showing how many times faster than real-time the system generates audio.

  • TTFA (Time to First Audio): Latency from inference start until the first audio chunk is available for playback (3D sample only).

Backend Comparison#

CLI Sample Performance#

Standalone inference without any graphics workload:

Backend

Avg Inference Time

Avg Audio Duration

Avg RTF

Speed-up

CUDA

564 ms

9.5 s

0.061

16.4x

Vulkan

626 ms

9.5 s

0.066

15.2x

D3D12

1,525 ms

9.5 s

0.164

6.1x

3D Sample Performance#

Inference running alongside a 3D rendering workload:

Backend

Avg Inference Time

Avg RTF

Speed-up

CUDA

1,281 ms

0.135

7.4x

D3D12

1,414 ms

0.147

6.8x

Vulkan

1,653 ms

0.175

5.7x

Note: 3D sample RTFs are higher than CLI because the GPU is shared between rendering and inference. The CUDA backend shows the largest difference between CLI and 3D modes because CIG scheduling introduces overhead to balance both workloads. D3D12 shows the smallest gap as it natively shares the GPU via D3D12 command queues.

Latency by Audio Duration (CLI)#

Average inference time grouped by generated audio duration:

Audio Duration

CUDA (ms)

Vulkan (ms)

D3D12 (ms)

< 7 s

426

411

951

7–9 s

522

556

1,362

9–11 s

597

688

1,664

11–13 s

613

698

1,716

13 s+

615

686

1,685

CUDA and Vulkan inference times remain relatively flat as audio duration increases, producing better (lower) RTF for longer outputs. D3D12 shows more variation but also benefits from longer outputs.

Time to First Audio (3D Sample)#

TTFA measures how quickly the first audio chunk is available for playback, which directly affects perceived responsiveness in interactive applications:

Backend

TTFA (ms)

CUDA

~970

D3D12

~1,200

Vulkan

~1,140

Measured during sequential 3D sample inference with sentences of 41–50 words.

VRAM Usage#

Per-backend peak VRAM recorded during sequential cli sample inference of 9–25 s of audio:

Backend

VRAM Usage

CUDA

1,938 MB

D3D12

1,691 MB

Vulkan

1,519 MB

Set CommonCreationParameters::vramBudgetMB to at least 2,048 MB for Turbo and at least 3,584 MB for Multilingual. If the plugin requires more VRAM than the budget allows, instance creation will fail.

D3D12 Warmup Latency#

The D3D12 backend has additional first-run overhead:

Event

Additional Latency

First inference after plugin load

~6 seconds (shader compilation)

First inference after changing speaker embeddings

~1 second

Subsequent inferences run at steady-state speeds. These costs cannot be eliminated, but they can be moved off the user-visible critical path so the first piece of in-game dialogue still meets your latency target.

Mitigation: pre-warm during loading screens#

Run a one-off dummy inference (any short text, output discarded) after the plugin instance is created but before any user-facing audio is requested. The ~6 s shader-compilation cost is then paid on the loading screen instead of on the first in-game line.

If your application uses multiple speaker embeddings during gameplay, pre-warm each one on the loading screen as well to amortise the additional ~1 s per-speaker cost.

Other strategies for latency-sensitive applications#

  • Group dialogue by speaker. In interactive cutscenes or scripted exchanges, schedule consecutive lines from the same speaker together so the per-switch ~1 s cost is paid less often.

  • Use one fixed voice per ambient context. Reserve dynamic speaker switching for scripted scenes that already absorb a small load time; let ambient NPCs share a single pre-warmed voice.

  • Maintain a small pool of pre-warmed instances. For highly dynamic dialogue with many concurrent voices, create N TTS instances at level-load time, each pre-warmed with one of the expected speakers, and dispatch lines to whichever instance is idle.

These strategies are additive — for most applications, the loading-screen warmup alone is sufficient.

Performance Recommendations#

  1. CUDA is recommended for the best inference performance on NVIDIA GPUs (16.4x real-time in CLI)

  2. Vulkan provides comparable CLI performance (15.2x) and is a good alternative for broader GPU compatibility

  3. D3D12 is slower for standalone inference (6.1x) but shows the smallest performance gap in the 3D sample, making it a reasonable choice for D3D12-based applications

  4. For interactive applications, target sentences of moderate length (20–40 words) for a good balance between natural-sounding output and consistent latency (~500–700 ms on CUDA)

  5. Warmup: Run a dummy inference during application startup to avoid cold-start latency affecting the user experience

Working Code Examples#

For production-ready integration examples, see:

  • source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp (TTS integration and callbacks)

  • source/samples/nvigi.tts.chatterbox/sample_tts_chatterbox.cpp (standalone CLI sample)

Example CLI usage:

nvigi.tts.chatterbox.exe --models C:\path\to\models --speaker C:\path\to\speaker.json --text "Hello from Chatterbox Turbo." --output output.wav --backend cuda --max_new_tokens 768 --gpu_device 0 --n_gpu_layers 99 --temperature 0.8 --cfg_weight 0.5 --max_words_per_chunk 40

Interactive mode (Windows only):

nvigi.tts.chatterbox.exe --models C:\path\to\models --speaker C:\path\to\speaker.json --interactive --backend cuda

Support#

For issues or feedback, refer to the NVIGI Developer Pack documentation or contact NVIDIA Developer Support.