Is this page helpful?

Programming Guide: Automatic Speech Recognition with Riva ASR using ONNX Runtime#

1. Overview#

The NVIDIA In-Game Inferencing (NVIGI) ASR (Automatic Speech Recognition) Riva ONNX Runtime plugins provide high-performance speech-to-text capabilities for applications and games. These plugins leverage ONNX Runtime with optimized backends to deliver accurate transcription of audio input in real-time.

The Riva ASR plugins support:

  • Multilingual recognition: 8 languages – English, French, German, Italian, Spanish, Korean, Japanese, and Mandarin Chinese

  • Multiple backend options: CPU, TensorRT-RTX GPU, and DirectML (DML) for broad hardware compatibility

  • Streaming mode: Real-time transcription with partial results during recording

  • Offline mode: Process pre-recorded audio files

  • Word boosting: Emphasize or suppress specific words in recognition

  • Silence detection: Detect silence periods in streaming audio

  • Auto punctuation: Optional automatic punctuation insertion and capitalization

  • Flexible decoding: Greedy or beam search strategies

  • CIG support: CUDA In Graphics integration for efficient GPU sharing (D3D12/Vulkan, TensorRT-RTX only)

2. Available Plugins#

The SDK provides two ASR Riva ORT plugins:

Plugin

Backend

Plugin ID

Use Case

nvigi.plugin.asr.riva-ort.cpu

ONNX Runtime (CPU)

nvigi::plugin::asr::riva_ort::cpu::kId

CPU-only systems, development

nvigi.plugin.asr.riva-ort.gpu

ONNX Runtime (TensorRT-RTX/DirectML)

nvigi::plugin::asr::riva_ort::gpu::kId

GPU-accelerated (TensorRT-RTX on NVIDIA GPUs, falls back to DirectML otherwise)

3. Key Concepts#

3.1 Audio Requirements#

The Riva ASR plugins expect audio in the following format:

  • Sample rate: 16,000 Hz (16 kHz)

  • Bit depth: 16-bit PCM

  • Channels: Mono (single channel)

If your audio is in a different format, you must convert it before passing it to the plugin.

3.2 Operating Modes#

Offline Mode (Non-Streaming)#

Process entire audio files at once. Best for:

  • Post-processing of recorded audio

  • Batch transcription

  • Non-interactive applications

Streaming Mode#

Process audio in real-time as it arrives. Best for:

  • Live microphone input

  • Interactive conversations

  • Real-time captioning

Streaming mode provides:

  • Partial results: Incremental transcription as speech progresses

  • Lower latency: Results available before complete utterance finishes

  • Accumulated results: Complete transcription built up over time

3.3 Data Slots#

The ASR plugin uses named data slots for input and output:

Slot Name

Constant

Direction

Type

Description

Audio

nvigi::kASRDataSlotAudio

Input

InferenceDataAudio

Raw audio data (16kHz, 16-bit, mono)

Transcribed Text

nvigi::kASRDataSlotTranscribedText

Output

InferenceDataText

Accumulated transcription with enough context for accuracy (offline: final result, streaming: accumulated so far)

Partial Text

nvigi::kASRDataSlotPartialText

Output

InferenceDataText

Intermediate transcription of recent audio without enough context (streaming only, less accurate)

Understanding Partial vs Transcribed Text in Streaming:

  • Transcribed Text (Accumulated): Uses enough past and future context for maximum accuracy. This is the high-quality transcription built up over time as the model receives sufficient context.

  • Partial Text: Real-time transcription of the most recent audio without enough context. Useful for immediate feedback but may be less accurate and can change as more context becomes available.

3.4 Benchmarking Methodology#

Throughout the programming guide, wherever possible, we have provided metrics that we hope are useful to the developer in making specific choices about the runtime and creation time parameters. For this purpose, we’ve chosen the Librispeech test-clean split dataset from https://www.openslr.org/12. We converted the .flac audio files to .wav files before running various tests. The clean test-split had 2620 audio samples.

The metrics below establish baseline performance characteristics across key dimensions (accuracy, and speed). These metrics are intended to inform your parameter selection and trade-off analysis. We acknowledge that specific applications may benefit from additional or alternative metrics tailored to their unique requirements.

Avg. WER

Corpus WER

Perfect WER %

High WER %

Avg RTF

Speed up

  • WER (Word Error Rate): The percentage of words incorrectly transcribed, calculated as (insertions + deletions + substitutions) / total_reference_words.

  • Avg. WER: The average of per-utterance WER values. Each audio sample’s WER is calculated independently, then averaged across all samples. This gives equal weight to each utterance regardless of length.

  • Corpus WER: The WER calculated over the entire corpus by summing all errors and dividing by total words across all samples. This is weighted by utterance length, so longer utterances have more impact on the final score. This is the standard metric used in ASR research papers.

  • Perfect WER %: Percentage of test instances where WER was exactly 0.0 (perfect transcription).

  • High WER % (>=50%): Percentage of test instances where WER was 50% or higher (significantly degraded transcription).

  • RTF (Real Time Factor): Ratio of processing time to audio length. RTF < 1 means faster than real-time. Lower is better.

  • Speed up: Equal to 1/RTF, showing how many times faster than real-time the system performs.

The GPU tests have been carried out on NVIDIA RTX 5090 Blackwell Generation GPU. CPU tests were conducted on Intel Core i9-14900K processor.

4. Getting Started#

Important Note: The code examples in this guide are illustrative and may not run as-is. They are designed to show the API structure and usage patterns. For complete, working implementations, please refer to:

  • Streaming ASR: source/samples/nvigi.asr.sample/sample_asr.cpp

  • Graphics Integration: source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp

4.1 Initialize NVIGI#

Before using any ASR plugin, initialize the NVIGI framework:

#include <nvigi.h>
#include "nvigi_asr_riva.h"

#ifndef NVIGI_WINDOWS
#include <dlfcn.h>
#define LoadLibraryA(lib) dlopen(lib, RTLD_LAZY)
#define GetProcAddress dlsym
#define FreeLibrary dlclose
#endif

// Load the NVIGI core library from SDK path
const char* sdkPath = "C:\\path\\to\\sdk";  // Path to SDK root directory
std::string coreLibPath = std::string(sdkPath) + "\\nvigi.core.framework.dll";
HMODULE coreLib = LoadLibraryA(coreLibPath.c_str());

// Get function pointers
auto nvigiInit = (PFun_nvigiInit*)GetProcAddress(coreLib, "nvigiInit");
auto nvigiShutdown = (PFun_nvigiShutdown*)GetProcAddress(coreLib, "nvigiShutdown");
auto nvigiLoadInterface = (PFun_nvigiLoadInterface*)GetProcAddress(coreLib, "nvigiLoadInterface");
auto nvigiUnloadInterface = (PFun_nvigiUnloadInterface*)GetProcAddress(coreLib, "nvigiUnloadInterface");

// Set up preferences
const char* pluginPaths[] = { sdkPath };
nvigi::Preferences pref{};
pref.logLevel = nvigi::LogLevel::eOff;  // Options: eOff, eDefault, eVerbose, eCount
pref.showConsole = false;
pref.numPathsToPlugins = 1;
pref.utf8PathsToPlugins = pluginPaths;
pref.utf8PathToLogsAndData = sdkPath;

// Initialize
if (NVIGI_FAILED(result, nvigiInit(pref, nullptr, nvigi::kSDKVersion)))
{
    // Handle error
}

4.2 Load the ASR Interface#

Choose the appropriate backend (CPU or GPU):

// Select plugin ID based on your backend preference
nvigi::PluginID pluginId = nvigi::plugin::asr::riva_ort::cpu::kId;  // CPU backend
// OR
nvigi::PluginID pluginId = nvigi::plugin::asr::riva_ort::gpu::kId;  // GPU backend (TensorRT-RTX or DirectML)

// Load the interface
nvigi::IAutoSpeechRecognition* iasr{};
if (NVIGI_FAILED(result, nvigiGetInterfaceDynamic(pluginId, &iasr, nvigiLoadInterface)))
{
    // Handle error
}

4.3 Create an ASR Instance#

Configure and create an instance with your desired settings:

// Common parameters
nvigi::CommonCreationParameters common{};
common.utf8PathToModels = "C:\\path\\to\\models";
common.modelGUID = "{B5D4F28E-3DB9-4FDA-A78B-A569AA22FCD0}";  // English Conformer-CTC (see Multilingual Support for other languages)
common.vramBudgetMB = 2048;  // Maximum VRAM the plugin can use (will error if plugin needs more)

// ASR-specific parameters
nvigi::ASRRivaORTCreationParameters asrParams{};
asrParams.chain(common);

// Configure for offline mode (default)
asrParams.enableStreamingMode = false;
asrParams.enableAutoPunctuation = false;

// GPU backend options
asrParams.preferDML = false;  // false = TensorRT-RTX if possible (default on NVIDIA GPUs), true = DirectML

// Create instance
nvigi::InferenceInstance* instance{};
if (NVIGI_FAILED(result, iasr->createInstance(asrParams, &instance)))
{
    // Handle error
}

4.4 Run Inference (Offline Mode)#

Process a complete audio file:

// Define a callback to receive results
struct TranscriptionContext {
    std::string transcribedText;
    std::atomic<bool> audioPending{true};
};

nvigi::InferenceExecutionState transcriptionCallback(
    const nvigi::InferenceExecutionContext* ctx, 
    nvigi::InferenceExecutionState state, 
    void* userData)
{
    auto* myCtx = static_cast<TranscriptionContext*>(userData);
    
    if (ctx && ctx->outputs)
    {
        // Get transcribed text from output slot
        const nvigi::InferenceDataText* textOutput{};
        if (ctx->outputs->findAndValidateSlot(nvigi::kASRDataSlotTranscribedText, &textOutput))
        {
            myCtx->transcribedText = textOutput->getUTF8Text();
        }
    }
    
    if (state == nvigi::kInferenceExecutionStateDone)
    {
        myCtx->audioPending = false;
    }
    
    return state;
}

// Load your audio data (must be 16kHz, 16-bit, mono)
std::vector<int16_t> audioSamples = loadAudioFile("audio.wav");

// Set up audio data
nvigi::CpuData audioData{};
audioData.buffer = reinterpret_cast<uint8_t*>(audioSamples.data());
audioData.sizeInBytes = audioSamples.size() * sizeof(int16_t);

nvigi::InferenceDataAudio audio{audioData};
audio.bitsPerSample = 16;
audio.samplingRate = 16000;
audio.channels = 1;

// Create input slot
std::vector<nvigi::InferenceDataSlot> slots = {
    {nvigi::kASRDataSlotAudio, audio}
};
nvigi::InferenceDataSlotArray inputs{slots.size(), slots.data()};

// Set up runtime parameters
nvigi::ASRRivaORTRuntimeParameters runtime{};
runtime.sampling = nvigi::ASRRivaSamplingStrategy::eBeamSearch;  // or eGreedy for offline mode
runtime.streamingMode = false;

// Create callback context
TranscriptionContext ctx{};

// Create execution context
nvigi::InferenceExecutionContext execCtx{};
execCtx.instance = instance;
execCtx.inputs = &inputs;
execCtx.runtimeParameters = runtime;
execCtx.callback = transcriptionCallback;
execCtx.callbackUserData = &ctx;

// Run inference
if (NVIGI_FAILED(result, instance->evaluate(&execCtx)))
{
    // Handle error
}

// Wait for completion
while (ctx.audioPending)
{
    std::this_thread::sleep_for(std::chrono::milliseconds(10));
}

// Use the transcription result
std::cout << "Transcription: " << ctx.transcribedText << std::endl;

4.5 Cleanup#

Always clean up resources:

// Destroy instance
iasr->destroyInstance(instance);

// Unload interface
nvigiUnloadInterface(pluginId, iasr);

// Shutdown NVIGI
nvigiShutdown();

// Free library
FreeLibrary(coreLib);

5. Streaming Mode#

Streaming mode enables real-time transcription with partial results. This is ideal for live microphone input or interactive applications.

5.1 Configuring Streaming Mode#

Enable streaming when creating the instance:

nvigi::ASRRivaORTCreationParameters asrParams{};
asrParams.chain(common);

// Enable streaming mode
asrParams.enableStreamingMode = true;

// Streaming configuration (optional, these are defaults)
// Chunk size: Recommanded 380ms for Conformer models (default), Recommanded to 900ms for Parakeet models (needs more context)
asrParams.streamingChunkSizeMs = 380;     // Process audio in chunks
asrParams.streamingBeamSize = 32;         // Beam size for decoder
asrParams.streamingLmWeight = 0.8f;       // Language model weight
asrParams.streamingWordScore = 1.0f;      // Word insertion score
asrParams.streamingContextSizeLeftRightMs = 2500;  // Context padding in milliseconds. Higher values delay accumulated text but partial text remains available.

5.2 Streaming Parameters Explained#

  • streamingChunkSizeMs: Size of audio chunks to process internally (milliseconds). Default: 380ms, 900ms recommended for Parakeet models (needs more context for optimal accuracy).

  • streamingBeamSize: Beam search width. Larger values improve accuracy but increase computation. Default: 32.

  • streamingLmWeight: Weight for language model scores. Range [0, 1]. Default: 0.8.

  • streamingWordScore: Word insertion score. Helps control output length. Default: 1.0.

  • streamingContextSizeLeftRightMs: Context padding size (left and right) in milliseconds for providing context to the acoustic model. Larger values provide more historical context but increase memory usage. Default: 2500ms.

    • Important: Higher values delay when accumulated transcription results are finalized and sent, as the model waits for more context. However, partial transcription results (if enablePartialDecoding is enabled) will still be available in real-time during this delay, allowing users to see preliminary results while the accumulated text is being processed.

5.3 Streaming Workflow#

Streaming uses a callback-based approach with three signals: START, DATA, and STOP.

1. Define a Callback Function#

// Context structure to track streaming results
struct StreamingContext {
    std::string transcribedText;      // Final accumulated text
    std::string currentAccumulated;   // Current accumulated text during streaming
    std::string mostRecentPartial;    // Most recent partial transcription
    std::atomic<bool> audioPending{true};
    int updateCount = 0;              // Count of updates received
};

nvigi::InferenceExecutionState streamingCallback (
    const nvigi::InferenceExecutionContext* ctx, 
    nvigi::InferenceExecutionState state, 
    void* userData)
{
    auto* myCtx = static_cast<StreamingContext*>(userData);
    
    if (ctx && ctx->outputs)
    {
        // Get accumulated text (high-quality, uses enough context)
        const nvigi::InferenceDataText* accumulatedText{};
        ctx->outputs->findAndValidateSlot(nvigi::kASRDataSlotTranscribedText, &accumulatedText);
        std::string accumulated = accumulatedText ? accumulatedText->getUTF8Text() : "";
        
        // Get partial text (real-time, recent audio only, less accurate)
        const nvigi::InferenceDataText* partialText{};
        bool hasPartial = ctx->outputs->findAndValidateSlot(nvigi::kASRDataSlotPartialText, &partialText);
        std::string partial = hasPartial && partialText ? partialText->getUTF8Text() : "";
        
        if (state == nvigi::kInferenceExecutionStateDataPartial)
        {
            // Partial result during streaming
            // - 'accumulated': high-quality transcription with enough past/future context
            // - 'partial': real-time preview of recent audio, may change with more context
            myCtx->currentAccumulated = accumulated;
            myCtx->mostRecentPartial = partial;
            myCtx->updateCount++;
            
            std::cout << "Update " << myCtx->updateCount 
                      << " - Accumulated: '" << accumulated << "'";
            if (!partial.empty())
            {
                std::cout << ", Partial: '" << partial << "'";
            }
            std::cout << std::endl;
        }
        
        if (state == nvigi::kInferenceExecutionStateDataPending || 
            state == nvigi::kInferenceExecutionStateDone)
        {
            // Final or pending result
            myCtx->transcribedText = accumulated;
            myCtx->currentAccumulated = accumulated;
        }
    }
    
    if (state == nvigi::kInferenceExecutionStateDone)
    {
        std::cout << "Final: " << myCtx->transcribedText << std::endl;
        myCtx->audioPending = false;
    }
    
    return state;
}

2. Initialize Streaming#

// Create context
StreamingContext streamCtx{};

// Set up audio data structure
nvigi::CpuData audioData{};
nvigi::InferenceDataAudio audio{audioData};
audio.bitsPerSample = 16;
audio.samplingRate = 16000;
audio.channels = 1;

std::vector<nvigi::InferenceDataSlot> slots = {{nvigi::kASRDataSlotAudio, audio}};
nvigi::InferenceDataSlotArray inputs{slots.size(), slots.data()};

// Set up runtime parameters
nvigi::ASRRivaORTRuntimeParameters runtime{};
nvigi::StreamingParameters stream{};
runtime.streamingMode = true;
runtime.sampling = nvigi::ASRRivaSamplingStrategy::eBeamSearch;
stream.mode = nvigi::StreamingMode::eStreamingModeInputOutput;  // Only mode available (parameter not used currently)
stream.signal = nvigi::StreamSignal::eStreamSignalStart;
runtime.chain(stream);

// Create execution context
nvigi::InferenceExecutionContext execCtx{};
execCtx.instance = instance;
execCtx.inputs = &inputs;
execCtx.runtimeParameters = runtime;
execCtx.callback = streamingCallback;
execCtx.callbackUserData = &streamCtx;

// Send START signal with empty audio
audioData.buffer = nullptr;
audioData.sizeInBytes = 0;
instance->evaluateAsync(&execCtx);

3. Send Audio DATA#

// Load your audio samples (16kHz, 16-bit, mono)
std::vector<int16_t> audioSamples = loadAudioFile("speech.wav");

// Stream audio in chunks (e.g., 5KB chunks)
const uint64_t chunkSizeBytes = 5 * 1024;  // 5KB chunks
uint64_t offset = 0;
const uint64_t totalBytes = audioSamples.size() * sizeof(int16_t);

// After sending START, switch to DATA signal
stream.signal = nvigi::StreamSignal::eStreamSignalData;

while (offset < totalBytes)
{
    // Calculate chunk size for this iteration
    uint64_t sizeToSend = std::min(chunkSizeBytes, totalBytes - offset);
    
    // Update audio buffer pointer and size
    audioData.buffer = reinterpret_cast<uint8_t*>(audioSamples.data()) + offset;
    audioData.sizeInBytes = sizeToSend;
    
    // Send audio chunk
    instance->evaluateAsync(&execCtx);
    
    offset += sizeToSend;
    
    // Optional: add delay to simulate real-time streaming
    if (offset < totalBytes)
    {
        std::this_thread::sleep_for(std::chrono::milliseconds(50));
    }
}

4. Send STOP Signal and Wait for Completion#

// Signal end of stream
stream.signal = nvigi::StreamSignal::eStreamSignalStop;
audioData.buffer = nullptr;
audioData.sizeInBytes = 0;

instance->evaluateAsync(&execCtx);

// Wait for final callback with state == kInferenceExecutionStateDone
while (streamCtx.audioPending)
{
    std::this_thread::sleep_for(std::chrono::milliseconds(10));
}

// Use final transcription
std::cout << "Complete transcription: " << streamCtx.transcribedText << std::endl;

6. Features#

6.1 Auto Punctuation#

Enable automatic punctuation insertion and capitalization:

asrParams.enableAutoPunctuation = true;

When enabled, the ASR will:

  • Automatically insert periods, commas, question marks, and other punctuation

  • Capitalize proper nouns and sentence beginnings

  • Improve overall readability of transcriptions

Note: Auto punctuation is currently available for English models only. Enabling it with non-English models will have no effect.

6.2 Multilingual Support#

The ASR plugin supports speech recognition in 8 languages. Each language has a dedicated Conformer-CTC model identified by a unique GUID. To use a specific language, set the modelGUID to the corresponding value when creating the ASR instance:

Language

Model Name

GUID

English (en-US)

Conformer-CTC

{B5D4F28E-3DB9-4FDA-A78B-A569AA22FCD0}

English (en-US)

Parakeet-0.6B

{53211BAC-8B34-4AE5-83C3-9B4CA0B2AE6D}

French (fr-FR)

Conformer-CTC-French

{7B9ED4D0-D15C-450F-BCC8-8FA7B460BFC0}

German (de-DE)

Conformer-CTC-German

{632AA9FC-5C76-460D-B915-791E41F40028}

Italian (it-IT)

Conformer-CTC-Italian

{33A53EA0-AD25-4CF1-B4FB-9B1513FDA24F}

Spanish (es-ES)

Conformer-CTC-Spanish

{20AD4848-5CB4-49CA-B48F-AF1AA61B4EB2}

Korean (ko-KR)

Conformer-CTC-Korean

{14BA55B2-0357-4B7B-B021-B9F1D8DFB77A}

Japanese (ja-JP)

Conformer-CTC-Japanese

{29A8FAFA-A9BB-485C-AEEF-BFF4512DFE91}

Mandarin (zh-CN)

Conformer-CTC-Mandarin

{0DDAD855-F9C3-42B4-AEED-18C9D0440923}

Selecting a Language#

To switch languages, change the modelGUID in your creation parameters:

// French ASR
common.modelGUID = "{7B9ED4D0-D15C-450F-BCC8-8FA7B460BFC0}";

// Japanese ASR
common.modelGUID = "{29A8FAFA-A9BB-485C-AEEF-BFF4512DFE91}";

// Spanish ASR
common.modelGUID = "{20AD4848-5CB4-49CA-B48F-AF1AA61B4EB2}";

All multilingual models use the same plugin (nvigi.plugin.asr.riva-ort) and the same API – only the GUID changes. Streaming mode, offline mode, backend selection, word boosting, and silence detection all work identically across languages.

Multilingual Considerations#

  • Audio format: All languages require the same 16kHz, 16-bit, mono PCM audio input.

  • Auto punctuation: Currently available for English models only.

  • Word boosting: Works across all languages. Provide boosted words in the target language’s script (e.g., Kanji/Hiragana for Japanese, Hangul for Korean, Hanzi for Mandarin).

  • Switching languages: Requires destroying the current instance and creating a new one with the desired language’s GUID. Language cannot be changed at runtime without re-creating the instance.

6.3 Word Boosting#

Improve recognition of specific words by providing boost scores. Words with higher scores are more likely to be recognized, while negative scores suppress words.

// Define word boosting entries
static const nvigi::WordBoostingEntry wordBoostingEntries[] = {
    {"fugatto", 10.0f},    // Boost recognition of "fugatto"
    {"nvidia", 5.0f},      // Boost recognition of "nvidia"
    {"um", -100.0f}        // Suppress filler word "um"
};

// Apply word boosting
asrParams.wordBoostingEntries = wordBoostingEntries;
asrParams.wordBoostingEntriesCount = sizeof(wordBoostingEntries) / sizeof(wordBoostingEntries[0]);

Use cases for word boosting:

  • Domain-specific vocabulary (technical terms, product names)

  • User-specific vocabulary (usernames, place names)

  • Suppressing common misrecognitions

Score recommendations:

  • Keep boost scores between -50 and +50 for optimal results

  • Start with moderate scores (±10) and adjust based on results

  • Scores outside this range may reduce overall accuracy

Troubleshooting word boosting:

If word boosting does not seem to work for a specific word, it is most likely because the tokens predicted by the model differ from the tokens obtained when tokenizing the boosted word. The ASR model uses subword tokenization, so a word may be split into multiple tokens that don’t match the boosted word exactly.

Solution: Try different orthographic variations of the word:

  • Phonetic spellings (e.g., “fugatto” vs “fugato” vs “fougatto”)

  • Common misspellings that the model might predict Runtime Updates:

Word boosting can be updated at runtime by providing updated wordBoostingEntries in the runtime parameters when calling instance->evaluate() (offline) or instance->evaluateAsync() (streaming). When word boosting is updated at runtime, expect approximately 400ms additional latency for the first inference after the update. This is because the system needs to rebuild internal data structures (trie and decoders) to reflect the new boosting configuration.

To avoid this latency cost during critical inference calls, consider performing a warmup inference immediately after updating word boosting:

// Update word boosting in runtime parameters
runtime.wordBoostingEntries = newWordBoostingEntries;
runtime.wordBoostingEntriesCount = newWordBoostingEntriesCount;

// Perform warmup inference with minimal audio to absorb the rebuild cost
// This ensures subsequent inferences don't incur the 400ms penalty
instance->evaluate(&execCtx);  // or evaluateAsync() for streaming

// Now subsequent inferences will have normal latency
instance->evaluate(&execCtx);

6.4 Silence Detection (Streaming Only)#

In streaming mode, you can enable silence detection to identify pauses in speech:

// Enable silence detection with 1000ms window
runtime.silenceDetectionTimeMs = 1000;  // Detect 1-second silences
runtime.silenceSensitivityThreshold = 0.85f;  // Sensitivity (0.0 - 1.0, default 0.85)

When silence is detected, the accumulated text will include <Silence> markers. This is useful for:

  • Detecting natural breaks in conversation

  • Identifying when a user has stopped speaking

Important: The <Silence> marker is temporary and will disappear from the accumulated text as soon as the speaker starts talking again. This allows you to detect silence periods in real-time during streaming.

Note: Silence detection uses a rolling average window, so the exact number of <Silence> markers may vary slightly based on speech patterns.

6.5 Sampling Strategies#

Two sampling strategies are available:

// Greedy sampling (faster, may be less accurate) - OFFLINE MODE ONLY
runtime.sampling = nvigi::ASRRivaSamplingStrategy::eGreedy;

// Beam search (slower, more accurate) - Available for both offline and streaming
runtime.sampling = nvigi::ASRRivaSamplingStrategy::eBeamSearch;

Important Notes:

  • Streaming mode: Only beam search is supported. Use the streamingBeamSize parameter to control beam width.

  • Offline mode: Both greedy and beam search are available.

    • Use beam search for production quality transcriptions

    • Use greedy for faster processing when accuracy is less critical

The following benchmarks were conducted on NVIDIA RTX 5090 GPU using the LibriSpeech test-clean dataset (2620 samples). See Benchmarking Methodology for metric definitions.

Conformer Model:

Backend

Sampling Strategy

Avg. WER

Corpus WER

Perfect WER %

High WER %

Avg RTF

Speed up

GPU

eBeamSearch (offline)

2.39%

2.16%

74.5

0.34

0.00253

395.2x

GPU

eGreedy (offline)

2.60%

2.33%

72.1

0.42

0.00211

474.6x

Parakeet Model:

Backend

Sampling Strategy

Avg. WER

Corpus WER

Perfect WER %

High WER %

Avg RTF

Speed up

GPU

eBeamSearch (offline)

2.23%

1.95%

76.0

0.31

0.00284

352.2x

GPU

eGreedy (offline)

2.27%

1.94%

75.8

0.34

0.00259

386.5x

Note: Greedy sampling is only available in offline mode. Beam search is always used for streaming mode. Word boosting is not available with Greedy decoding.

7. Performance Optimization#

7.1 Backend Selection#

The SDK provides two plugins with different backend support:

  • CPU Backend (nvigi::plugin::asr::riva_ort::cpu::kId):

    • Uses ONNX Runtime CPU execution provider

    • Supports all modern x64 CPUs

    • Best for: CPU-only systems, limited amount of VRAM

  • GPU Backend (nvigi::plugin::asr::riva_ort::gpu::kId):

    • Supports both TensorRT-RTX and DirectML execution providers

    • TensorRT-RTX (default on NVIDIA GPUs): Best performance for NVIDIA GPUS

    • DirectML (via preferDML = true or fallback): Compatible with any DirectX 12 GPU (NVIDIA, AMD, Intel)

    • Automatically falls back to DirectML if TensorRT-RTX is unavailable

// Use GPU backend with TensorRT-RTX (default on NVIDIA GPUs)
nvigi::PluginID pluginId = nvigi::plugin::asr::riva_ort::gpu::kId;
asrParams.preferDML = false;  // Use TensorRT-RTX (falls back to DirectML if unavailable)

// Use GPU backend with DirectML (broader GPU compatibility)
nvigi::PluginID pluginId = nvigi::plugin::asr::riva_ort::gpu::kId;
asrParams.preferDML = true;  // Force DirectML

Preventing Backend Fallback#

By default, the GPU backend will automatically fall back to a slower backend if the preferred one fails (TRT-RTX → DML → CPU). For testing or production scenarios where you want to ensure a specific backend is used, you can enable crash-on-fallback:

// Fail instead of falling back to a slower backend
asrParams.crashOnFallback = true;

When crashOnFallback is enabled, instance creation will fail with an error if the expected backend cannot be used.

TensorRT-RTX Engine Compilation: When using TensorRT-RTX for the first time, the engine will take approximately 30 seconds to compile. This compiled engine is then cached in the same folder specified by utf8PathToLogsAndData, so subsequent runs will not experience this overhead. The cache is reused across application restarts, ensuring optimal performance after the initial compilation.

CPU Thread Configuration#

For CPU backend, you can configure the number of threads used by the ONNX Runtime execution provider:

// Configure CPU thread count (default: 16)
asrParams.cpuNumThreads = 8;  // Capped at hardware concurrency

The thread count is automatically capped at the system’s hardware concurrency (number of logical CPU cores). A global thread pool is used across all ONNX Runtime sessions for efficient resource sharing.

Offline Performance#

Backend Comparison (Conformer, librispeech test-clean):

Backend

Avg RTF

Speed up

TensorRT-RTX

0.00245

408.8x

DirectML

0.00511

195.8x

CPU

0.0198

50.4x

Backend Comparison (Parakeet, librispeech test-clean):

Backend

Avg RTF

Speed up

TensorRT-RTX

0.00290

345.4x

DirectML

0.00642

155.7x

CPU

0.0255

39.2x

Latency by Audio Duration (Conformer, Offline):

Audio Duration

TensorRT-RTX (ms)

DirectML (ms)

CPU (ms)

1-3s

10.53

19.60

73.05

3-6s

13.79

27.98

95.97

6-9s

17.98

37.67

138.55

9-12s

22.41

47.56

184.73

12-15s

27.75

60.27

238.69

15-18s

31.43

68.91

292.27

18-21s

36.02

79.30

361.80

21-24s

39.77

86.09

435.79

24s+

47.31

106.70

626.61

Latency by Audio Duration (Parakeet, Offline):

Audio Duration

TensorRT-RTX (ms)

DirectML (ms)

CPU (ms)

1-3s

12.16

23.82

109.19

3-6s

16.26

34.74

141.16

6-9s

21.27

47.05

185.40

9-12s

26.51

59.96

229.06

12-15s

33.10

76.36

280.23

15-18s

37.64

88.69

334.21

18-21s

43.08

102.93

398.53

21-24s

47.28

111.48

446.10

24s+

57.72

140.34

595.87

Streaming Performance#

Backend Comparison (Conformer, librispeech test-clean):

Backend

Avg RTF

Speed up

TensorRT-RTX

0.0164

61.1x

DirectML

0.0380

26.3x

CPU

0.196

5.1x

Backend Comparison (Parakeet, librispeech test-clean):

Backend

Avg RTF

Speed up

TensorRT-RTX

0.0179

55.9x

DirectML

0.0493

20.3x

CPU

0.326

3.1x

7.2 Memory Requirements#

Specify the maximum VRAM budget for GPU backends:

common.vramBudgetMB = 2048;  // Maximum VRAM the plugin can use (in MB)

Actual VRAM usage can range from 500 MB to 1.6 GB depending on the model selected, user-defined parameters, and the graphics card in use.

Important: This parameter specifies the maximum amount of VRAM that the plugin can use. If the plugin requires more VRAM than this budget allows, it will raise an error. Consider:

  • The model being used (e.g., Parakeet-0.6B requires more VRAM than Conformer-CTC)

  • The graphics card and its available VRAM

  • Other GPU workloads running simultaneously

  • Setting a sufficient budget to avoid runtime errors

7.3 Streaming Chunk Size#

The internal chunk size affects processing characteristics. Larger chunk sizes provide more acoustic context, improving accuracy (lower WER), and improves RTF because there is lesser workload that runs on GPU. So the metrics are highly scenario specific.

// Recommended chunk sizes based on model type:
asrParams.streamingChunkSizeMs = 380;  // For Conformer models (default)
// OR
asrParams.streamingChunkSizeMs = 900;  // For Parakeet models (needs more context)

Model-Specific Recommendations:

  • Conformer models: 380ms (default) - Efficient with less context, balanced latency and accuracy

  • Parakeet models: 900ms - Requires more context for optimal accuracy due to architectural differences

  • Note: You can send audio chunks of any size to evaluateAsync() regardless of this setting

Parakeet Model Streaming Performance (librispeech test-clean):

Chunk size (ms)

Avg. WER

Corpus WER

Perfect WER %

High WER %

Avg RTF

Speed up

380

3.33%

2.72%

67.2

0.84

0.0180

55.5x

900

3.08%

2.48%

69.8

0.80

0.0146

68.5x

Streaming Performance on Challenging Audio (librispeech test-other):

Model

Chunk (ms)

Avg. WER

Corpus WER

Perfect WER %

High WER %

Avg RTF

Speed up

Conformer

380

6.66%

5.53%

52.8

1.84

0.0181

55.2x

Conformer

900

6.37%

5.18%

54.4

1.77

0.0162

61.9x

Parakeet

380

6.91%

5.31%

54.1

2.35

0.0190

52.6x

Parakeet

700

6.45%

4.97%

55.3

2.14

0.0166

60.2x

Parakeet

900

6.37%

4.82%

56.9

2.21

0.0162

61.7x

The test-other dataset contains more challenging audio with accents, background noise, and varied acoustic conditions.

7.4 Streaming Context size#

This parameter is available through the streamingContextSizeLeftRightMs parameter for the streaming use case. Larger values provide more historical context but turns out to be a bit slower. Default: 2500 ms

Context Size (ms)

Avg. WER

Corpus WER

Perfect WER %

High WER %

Avg RTF

Speed up

1920

3.19%

2.89%

66.7

0.50

0.0166

60.3x

2500

2.87%

2.58%

69.6

0.50

0.0169

59.1x

4000

2.68%

2.34%

71.7

0.46

0.0167

59.8x

8. Graphics Integration#

For game applications, the ASR plugin supports graphics API integration through D3D12 and Vulkan parameters:

  • With TensorRT-RTX: Enables CUDA In Graphics (CIG) for efficient GPU resource sharing between AI inference and rendering

  • With DirectML: Uses your game’s D3D12 device to create command queues for proper integration

Note: CIG is only available with TensorRT-RTX backend.

8.1 D3D12Parameters Structure#

The D3D12Parameters structure provides the necessary D3D12 context for both CUDA In Graphics (CIG) and DirectML backends.

Field

Type

Required

Description

device

ID3D12Device*

Yes

Your game’s D3D12 device

queue

ID3D12CommandQueue*

Yes (for CIG)

Used by CIG to create a shared CUDA context

queueCompute

ID3D12CommandQueue*

Optional

Asynchronous compute queue (D3D12_COMMAND_LIST_TYPE_COMPUTE). Used by DirectML for inference operations

createCommittedResourceCallback

Function pointer

Optional

Custom callback for creating D3D12 committed resources

destroyResourceCallback

Function pointer

Optional

Custom callback for destroying D3D12 resources

createCommitResourceUserContext

void*

Optional

User context passed to createCommittedResourceCallback

destroyResourceUserContext

void*

Optional

User context passed to destroyResourceCallback

flags

D3D12ParametersFlags

Optional

Flags controlling D3D12 behavior (see below)

D3D12ParametersFlags:

  • eNone: No flags (default)

  • eDisableReBAR: Disable use of ReBAR (D3D12_HEAP_TYPE_GPU_UPLOAD) - may degrade performance

  • eComputeQueueSharedWithFrame: Indicates the provided compute queue is also used for main rendering

8.2 D3D12 Integration with CIG (TensorRT-RTX)#

For CUDA In Graphics integration, provide your D3D12 device and queue:

Important: The compute queue provided to CIG must be part of the same TSG (Timeslice Group) as the graphics queue of the game. This ensures proper synchronization and resource sharing between AI inference and rendering operations.

Recommendation: For CIG, it is recommended to provide only a compute queue in the queue parameter for optimal performance and resource scheduling.

#ifdef NVIGI_WINDOWS
// Initialize D3D12 parameters for CIG
nvigi::D3D12Parameters d3d12Params{};
d3d12Params.device = myD3D12Device;  // Your ID3D12Device*
d3d12Params.queue = myComputeQueue;  // Recommended: Use compute queue (D3D12_COMMAND_LIST_TYPE_COMPUTE) for CIG

// Chain D3D12 parameters with ASR parameters
nvigi::ASRRivaORTCreationParameters asrParams{};
asrParams.chain(common);
asrParams.chain(d3d12Params);  // Chain D3D12 params for CIG
asrParams.preferDML = false;   // Use TensorRT-RTX (default)

// Create instance
nvigi::InferenceInstance* instance{};
iasr->createInstance(asrParams, &instance);
#endif

8.3 D3D12 Integration with DirectML#

For DirectML, provide at minimum the device and optionally a compute queue:

#ifdef NVIGI_WINDOWS
// Initialize D3D12 parameters for DirectML
nvigi::D3D12Parameters d3d12Params{};
d3d12Params.device = myD3D12Device;  // Your ID3D12Device*

// Option 1: Use your game's queues
d3d12Params.queue = myGraphicsQueue;         // Optional for DML, but recommended
d3d12Params.queueCompute = myComputeQueue;   // Used for DirectML inference operations

// Option 2: Create a dedicated compute queue for AI
D3D12_COMMAND_QUEUE_DESC queueDesc = {};
queueDesc.Type = D3D12_COMMAND_LIST_TYPE_COMPUTE;
queueDesc.Priority = D3D12_COMMAND_QUEUE_PRIORITY_NORMAL;
queueDesc.Flags = D3D12_COMMAND_QUEUE_FLAG_NONE;
ID3D12CommandQueue* aiComputeQueue = nullptr;
myD3D12Device->CreateCommandQueue(&queueDesc, IID_PPV_ARGS(&aiComputeQueue));
d3d12Params.queueCompute = aiComputeQueue;

// Chain D3D12 parameters with ASR parameters
nvigi::ASRRivaORTCreationParameters asrParams{};
asrParams.chain(common);
asrParams.chain(d3d12Params);  // Chain D3D12 params for DirectML
asrParams.preferDML = true;    // Use DirectML

// Create instance
nvigi::InferenceInstance* instance{};
iasr->createInstance(asrParams, &instance);
#endif

8.4 Custom Resource Allocation Callbacks#

For games that need fine-grained control over D3D12 resource allocation, you can provide custom callbacks:

// Custom resource creation callback
ID3D12Resource* createCommittedResource(
    ID3D12Device* device, 
    const D3D12_HEAP_PROPERTIES* pHeapProperties,
    D3D12_HEAP_FLAGS HeapFlags, 
    const D3D12_RESOURCE_DESC* pDesc,
    D3D12_RESOURCE_STATES InitialResourceState, 
    const D3D12_CLEAR_VALUE* pOptimizedClearValue, 
    void* userContext)
{
    ID3D12Resource* resource = nullptr;
    HRESULT hr = device->CreateCommittedResource(
        pHeapProperties, HeapFlags, pDesc, 
        InitialResourceState, pOptimizedClearValue, 
        IID_PPV_ARGS(&resource));
    
    if (FAILED(hr))
    {
        // Handle error - log, track allocation failure, etc.
        return nullptr;
    }
    
    // Optional: Track allocation in your resource manager
    // MyResourceManager* mgr = static_cast<MyResourceManager*>(userContext);
    // mgr->TrackAllocation(resource, pDesc->Width);
    
    return resource;
}

// Custom resource destruction callback
void destroyResource(ID3D12Resource* pResource, void* userContext)
{
    // Optional: Untrack allocation
    // MyResourceManager* mgr = static_cast<MyResourceManager*>(userContext);
    // mgr->UntrackAllocation(pResource);
    
    pResource->Release();
}

// Setup D3D12 parameters with callbacks
nvigi::D3D12Parameters d3d12Params{};
d3d12Params.device = myD3D12Device;
d3d12Params.queue = myGraphicsQueue;
d3d12Params.createCommittedResourceCallback = createCommittedResource;
d3d12Params.destroyResourceCallback = destroyResource;
d3d12Params.createCommitResourceUserContext = &myResourceManager;  // Optional
d3d12Params.destroyResourceUserContext = &myResourceManager;       // Optional

8.5 VulkanParameters Structure#

The VulkanParameters structure provides the necessary Vulkan context for CUDA In Graphics (CIG) integration.

Field

Type

Required

Description

physicalDevice

VkPhysicalDevice

Yes

Your game’s Vulkan physical device

device

VkDevice

Yes

Your game’s Vulkan logical device

instance

VkInstance

Yes

Your game’s Vulkan instance

queue

VkQueue

Yes (for CIG)

Graphics queue (VK_QUEUE_GRAPHICS_BIT)

queueCompute

VkQueue

Optional

Compute queue (VK_QUEUE_COMPUTE_BIT, no VK_QUEUE_GRAPHICS_BIT)

allocateMemoryCallback

Function pointer

Optional

Custom callback for allocating Vulkan memory

freeMemoryCallback

Function pointer

Optional

Custom callback for freeing Vulkan memory

allocateMemoryCallbackUserContext

void*

Optional

User context passed to allocateMemoryCallback

freeMemoryCallbackUserContext

void*

Optional

User context passed to freeMemoryCallback

Note: All VulkanParameters members are optional. If not provided, NVIGI will create internal device, instance, and queues. However, for proper CIG integration with your game, you should provide your game’s Vulkan context.

8.6 Vulkan Integration#

#ifdef NVIGI_WINDOWS
// Initialize Vulkan parameters with your game's Vulkan context
nvigi::VulkanParameters vulkanParams{};
vulkanParams.physicalDevice = myVkPhysicalDevice;  // Your VkPhysicalDevice
vulkanParams.device = myVkDevice;                   // Your VkDevice
vulkanParams.instance = myVkInstance;               // Your VkInstance
vulkanParams.queue = myGraphicsQueue;               // Graphics queue (VK_QUEUE_GRAPHICS_BIT)

// Optionally provide compute queue for better scheduling
vulkanParams.queueCompute = myComputeQueue;         // VK_QUEUE_COMPUTE_BIT

// Chain Vulkan parameters with ASR parameters
nvigi::ASRRivaORTCreationParameters asrParams{};
asrParams.chain(common);
asrParams.chain(vulkanParams);  // Chain Vulkan params for CIG
asrParams.preferDML = false;    // Use TensorRT-RTX

// Create instance
nvigi::InferenceInstance* instance{};
iasr->createInstance(asrParams, &instance);
#endif

Important for Vulkan games: Provide VulkanParameters with your game’s Vulkan device and queue for CUDA In Graphics (CIG) integration with TensorRT-RTX backend.

8.7 When to Chain Graphics Parameters#

Backend

Use Case

Chain D3D12Parameters?

Chain VulkanParameters?

TensorRT-RTX

D3D12 game with CIG

Yes - Required for CIG

No

TensorRT-RTX

Vulkan game with CIG

No

Yes - Required for CIG

TensorRT-RTX

No graphics (standalone)

Optional

Optional

DirectML

D3D12 game

Yes - Recommended for proper queue integration

N/A (DML is D3D12 only)

DirectML

No graphics (standalone)

Optional (NVIGI creates internal queues)

N/A

CPU

Any

No (ignored)

No (ignored)

8.8 GPU Scheduling Modes#

When using CIG, you can control GPU scheduling priorities:

#ifdef NVIGI_WINDOWS
// Get HWI (Hardware Interface) for scheduling control
nvigi::IHWICommon* ihwiCommon{};
nvigiGetInterfaceDynamic(plugin::hwi::common::kId, &ihwiCommon, nvigiLoadInterface);

// Set scheduling mode
ihwiCommon->SetGpuInferenceSchedulingMode(SchedulingMode::kBalance);

// ... Run inference ...

// Clean up
nvigiUnloadInterface(plugin::hwi::common::kId, ihwiCommon);
#endif

Scheduling Mode Options:

Mode

Description

Use When

kPrioritizeGraphics

Gives GPU priority to 3D rendering over inference

Frame rate is critical and you can tolerate higher ASR latency

kPrioritizeCompute

Gives GPU priority to inference over rendering

Low ASR latency is critical (e.g., real-time voice commands) and you can tolerate some frame drops

kBalance

Balanced split between graphics and compute

Good default for most applications; provides reasonable frame rates and ASR latency without tuning

Note: For games with low graphics intensity (e.g., 2D games, simple 3D scenes, or applications running well below GPU capacity), the difference between scheduling modes will be minimal since the GPU has enough headroom to handle both workloads without contention.

Note: CUDA In Graphics (CIG) is only available with the TensorRT-RTX backend on NVIDIA GPUs with driver version 575 or higher. CIG is not available when using DirectML.

9. Error Handling#

Always check return values using the NVIGI_FAILED macro:

nvigi::Result result;
if (NVIGI_FAILED(result, iasr->createInstance(asrParams, &instance)))
{
    // Log error code and message
    std::cerr << "Error code: " << result.error << std::endl;
    // Handle error appropriately
}

Common error scenarios:

  • Model not found: Check utf8PathToModels and model GUID

  • VRAM budget exceeded: Increase vramBudgetMB, ensure GPU has enough memory, or use CPU backend

  • Invalid audio format: Ensure 16kHz, 16-bit, mono

  • Plugin not found: Check plugin path and DLL availability

10. Complete Examples#

Note: These examples are illustrative to demonstrate API usage patterns. For complete, working code, see the Working Code Examples section below.

10.1 Example 1: Offline Processing with Word Boosting#

#include <nvigi.h>
#include "nvigi_asr_riva.h"
#include <iostream>
#include <vector>
#include <atomic>

// Callback context
struct TranscriptionContext {
    std::string transcribedText;
    std::atomic<bool> audioPending{true};
};

// Callback function
nvigi::InferenceExecutionState transcriptionCallback(
    const nvigi::InferenceExecutionContext* ctx, 
    nvigi::InferenceExecutionState state, 
    void* userData)
{
    auto* myCtx = static_cast<TranscriptionContext*>(userData);
    
    if (ctx && ctx->outputs)
    {
        const nvigi::InferenceDataText* textOutput{};
        if (ctx->outputs->findAndValidateSlot(nvigi::kASRDataSlotTranscribedText, &textOutput))
        {
            myCtx->transcribedText = textOutput->getUTF8Text();
        }
    }
    
    if (state == nvigi::kInferenceExecutionStateDone)
    {
        myCtx->audioPending = false;
    }
    
    return state;
}

int main()
{
    // Initialize NVIGI (see Step 1)
    // Load interface (see Step 2)
    nvigi::IAutoSpeechRecognition* iasr{};
    // ... (initialization code)
    
    // Create instance for offline mode with word boosting
    nvigi::CommonCreationParameters common{};
    common.utf8PathToModels = "C:\\models";
    common.modelGUID = "{B5D4F28E-3DB9-4FDA-A78B-A569AA22FCD0}";
    
    // Define word boosting
    static const nvigi::WordBoostingEntry wordBoostingEntries[] = {
        {"nvidia", 10.0f},
        {"fugatto", 10.0f}
    };
    
    nvigi::ASRRivaORTCreationParameters asrParams{};
    asrParams.chain(common);
    asrParams.enableStreamingMode = false;
    asrParams.enableAutoPunctuation = true;
    asrParams.wordBoostingEntries = wordBoostingEntries;
    asrParams.wordBoostingEntriesCount = 2;
    
    nvigi::InferenceInstance* instance{};
    iasr->createInstance(asrParams, &instance);
    
    // Load audio (must be 16kHz, 16-bit, mono)
    std::vector<int16_t> audioSamples = loadAudioFile("speech.wav");
    
    // Set up audio data
    nvigi::CpuData audioData{};
    audioData.buffer = reinterpret_cast<uint8_t*>(audioSamples.data());
    audioData.sizeInBytes = audioSamples.size() * sizeof(int16_t);
    
    nvigi::InferenceDataAudio audio{audioData};
    audio.bitsPerSample = 16;
    audio.samplingRate = 16000;
    audio.channels = 1;
    
    std::vector<nvigi::InferenceDataSlot> slots = {{nvigi::kASRDataSlotAudio, audio}};
    nvigi::InferenceDataSlotArray inputs{slots.size(), slots.data()};
    
    // Set up runtime parameters
    nvigi::ASRRivaORTRuntimeParameters runtime{};
    runtime.sampling = nvigi::ASRRivaSamplingStrategy::eBeamSearch;
    runtime.streamingMode = false;
    
    // Create callback context
    TranscriptionContext ctx{};
    
    // Set up execution context
    nvigi::InferenceExecutionContext execCtx{};
    execCtx.instance = instance;
    execCtx.inputs = &inputs;
    execCtx.runtimeParameters = runtime;
    execCtx.callback = transcriptionCallback;
    execCtx.callbackUserData = &ctx;
    
    // Run inference
    instance->evaluate(&execCtx);
    
    // Wait for completion
    while (ctx.audioPending)
    {
        std::this_thread::sleep_for(std::chrono::milliseconds(10));
    }
    
    // Use result
    std::cout << "Transcription: " << ctx.transcribedText << std::endl;
    
    // Cleanup
    iasr->destroyInstance(instance);
    // Shutdown (see Step 5)
    
    return 0;
}

10.2 Example 2: Real-Time Streaming with Silence Detection#

#include <nvigi.h>
#include "nvigi_asr_riva.h"
#include <iostream>
#include <vector>
#include <atomic>

// Streaming context
struct StreamingContext {
    std::string transcribedText;
    std::string currentAccumulated;
    std::string mostRecentPartial;
    std::atomic<bool> audioPending{true};
    int updateCount = 0;
    bool lastHadSilence = false;
    int silenceCount = 0;
};

// Streaming callback
nvigi::InferenceExecutionState streamingCallback(
    const nvigi::InferenceExecutionContext* ctx, 
    nvigi::InferenceExecutionState state, 
    void* userData)
{
    auto* myCtx = static_cast<StreamingContext*>(userData);
    
    if (ctx && ctx->outputs)
    {
        // Get accumulated text (high-quality, uses enough context)
        const nvigi::InferenceDataText* accumulatedText{};
        ctx->outputs->findAndValidateSlot(nvigi::kASRDataSlotTranscribedText, &accumulatedText);
        std::string accumulated = accumulatedText ? accumulatedText->getUTF8Text() : "";
        
        // Get partial text (real-time, recent audio only, less accurate)
        const nvigi::InferenceDataText* partialText{};
        bool hasPartial = ctx->outputs->findAndValidateSlot(nvigi::kASRDataSlotPartialText, &partialText);
        std::string partial = hasPartial && partialText ? partialText->getUTF8Text() : "";
        
        if (state == nvigi::kInferenceExecutionStateDataPartial)
        {
            // Check for silence detection in accumulated text
            bool currentHasSilence = (accumulated.find("<Silence>") != std::string::npos);
            if (!myCtx->lastHadSilence && currentHasSilence)
            {
                myCtx->silenceCount++;
                std::cout << "Silence detected (count: " << myCtx->silenceCount << ")" << std::endl;
            }
            myCtx->lastHadSilence = currentHasSilence;
            
            myCtx->currentAccumulated = accumulated;
            myCtx->mostRecentPartial = partial;
            myCtx->updateCount++;
            
            std::cout << "Update " << myCtx->updateCount << " - Accumulated: '" << accumulated << "'";
            if (!partial.empty())
            {
                std::cout << ", Partial: '" << partial << "'";
            }
            std::cout << std::endl;
        }
        
        if (state == nvigi::kInferenceExecutionStateDataPending || 
            state == nvigi::kInferenceExecutionStateDone)
        {
            myCtx->transcribedText = accumulated;
            myCtx->currentAccumulated = accumulated;
        }
    }
    
    if (state == nvigi::kInferenceExecutionStateDone)
    {
        std::cout << "Final: " << myCtx->transcribedText << std::endl;
        myCtx->audioPending = false;
    }
    
    return state;
}

int main()
{
    // Initialize NVIGI and load interface
    nvigi::IAutoSpeechRecognition* iasr{};
    // ... (initialization code)
    
    // Create instance for streaming mode
    nvigi::CommonCreationParameters common{};
    common.utf8PathToModels = "C:\\models";
    common.modelGUID = "{B5D4F28E-3DB9-4FDA-A78B-A569AA22FCD0}";
    
    nvigi::ASRRivaORTCreationParameters asrParams{};
    asrParams.chain(common);
    asrParams.enableStreamingMode = true;
    asrParams.streamingChunkSizeMs = 380;  // 380ms for Conformer, 900ms for Parakeet
    asrParams.streamingBeamSize = 32;
    
    nvigi::InferenceInstance* instance{};
    iasr->createInstance(asrParams, &instance);
    
    // Load audio
    std::vector<int16_t> audioSamples = loadAudioFile("speech.wav");
    
    // Set up audio data
    nvigi::CpuData audioData{};
    nvigi::InferenceDataAudio audio{audioData};
    audio.bitsPerSample = 16;
    audio.samplingRate = 16000;
    audio.channels = 1;
    
    std::vector<nvigi::InferenceDataSlot> slots = {{nvigi::kASRDataSlotAudio, audio}};
    nvigi::InferenceDataSlotArray inputs{slots.size(), slots.data()};
    
    // Set up runtime parameters with silence detection
    nvigi::ASRRivaORTRuntimeParameters runtime{};
    nvigi::StreamingParameters stream{};
    runtime.streamingMode = true;
    runtime.sampling = nvigi::ASRRivaSamplingStrategy::eBeamSearch;
    runtime.silenceDetectionTimeMs = 1000;  // Detect 1-second silences
    runtime.silenceSensitivityThreshold = 0.85f;
    stream.mode = nvigi::StreamingMode::eStreamingModeInputOutput;  // Only mode available (parameter not used currently)
    stream.signal = nvigi::StreamSignal::eStreamSignalStart;
    runtime.chain(stream);
    
    // Create context
    StreamingContext streamCtx{};
    
    // Set up execution context
    nvigi::InferenceExecutionContext execCtx{};
    execCtx.instance = instance;
    execCtx.inputs = &inputs;
    execCtx.runtimeParameters = runtime;
    execCtx.callback = streamingCallback;
    execCtx.callbackUserData = &streamCtx;
    
    // Send START signal
    audioData.buffer = nullptr;
    audioData.sizeInBytes = 0;
    instance->evaluateAsync(&execCtx);
    
    // Send audio data in chunks
    const uint64_t chunkSizeBytes = 5 * 1024;
    uint64_t offset = 0;
    const uint64_t totalBytes = audioSamples.size() * sizeof(int16_t);
    
    stream.signal = nvigi::StreamSignal::eStreamSignalData;
    
    while (offset < totalBytes)
    {
        uint64_t sizeToSend = std::min(chunkSizeBytes, totalBytes - offset);
        audioData.buffer = reinterpret_cast<uint8_t*>(audioSamples.data()) + offset;
        audioData.sizeInBytes = sizeToSend;
        
        instance->evaluateAsync(&execCtx);
        offset += sizeToSend;
        
        if (offset < totalBytes)
        {
            std::this_thread::sleep_for(std::chrono::milliseconds(50));
        }
    }
    
    // Send STOP signal
    stream.signal = nvigi::StreamSignal::eStreamSignalStop;
    audioData.buffer = nullptr;
    audioData.sizeInBytes = 0;
    instance->evaluateAsync(&execCtx);
    
    // Wait for completion
    while (streamCtx.audioPending)
    {
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
    }
    
    std::cout << "Complete transcription: " << streamCtx.transcribedText << std::endl;
    std::cout << "Total silences detected: " << streamCtx.silenceCount << std::endl;
    
    // Cleanup
    iasr->destroyInstance(instance);
    // Shutdown
    
    return 0;
}

11. Working Code Examples#

For complete, production-ready implementations, refer to these sample files:

  • Streaming ASR Implementation: source/samples/nvigi.asr.sample/sample_asr.cpp

    • Complete working example of streaming mode with callbacks, audio chunking, and proper initialization

  • Graphics Integration (D3D12/Vulkan): source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp

    • Real-world implementation showing graphics API integration for games

These samples demonstrate proper error handling, resource management, and integration patterns that can be used in production code.

12. Troubleshooting#

12.1 Audio Format Issues#

Problem: Poor transcription quality or errors
Solution: Ensure audio is exactly 16kHz, 16-bit, mono. Convert if necessary:

// Convert stereo to mono
if (numChannels == 2)
{
    std::vector<int16_t> monoData(wavData.size() / 2);
    for (size_t i = 0; i < monoData.size(); i++)
    {
        int32_t left = wavData[2 * i];
        int32_t right = wavData[2 * i + 1];
        monoData[i] = static_cast<int16_t>((left + right) / 2);
    }
    wavData = std::move(monoData);
}

12.2 Streaming Timeouts#

Problem: No final callback received
Solution:

  • Ensure STOP signal is sent after all audio

  • Check that callback function is properly set

  • Increase timeout duration for very long audio

12.3 Memory Issues#

Problem: Instance creation or inference fails due to VRAM budget exceeded
Solution:

  • Increase vramBudgetMB to provide more VRAM budget for the plugin

  • Ensure your GPU has sufficient VRAM available (check GPU specifications)

  • Use CPU backend instead of GPU backend if VRAM is limited

12.4 Poor Accuracy#

Problem: Incorrect transcriptions
Solution:

  • Use eBeamSearch sampling instead of eGreedy

  • Increase streamingBeamSize for streaming mode

  • Enable enableAutoPunctuation

  • Ensure audio is clear and noise-free

  • Use word boosting for domain-specific vocabulary

13. API Reference#

13.1 ASRRivaORTCreationParameters#

Field

Type

Default

Description

wordBoostingEntries

const WordBoostingEntry*

nullptr

Array of word boosting entries

wordBoostingEntriesCount

uint32_t

0

Number of word boosting entries

enableAutoPunctuation

bool

false

Enable automatic punctuation and capitalization

enableStreamingMode

bool

false

Enable streaming mode

preferDML

bool

false

Prefer DirectML over TensorRT-RTX (GPU backend only)

streamingChunkSizeMs

int32_t

380

Internal chunk size in milliseconds (Recommanded to set it to 900ms for Parakeet)

streamingBeamSize

int32_t

32

Beam search width for streaming

streamingLmWeight

float

0.8f

Language model weight (0.0 - 1.0)

streamingWordScore

float

1.0f

Word insertion score

streamingContextSizeLeftRightMs

int32_t

2500

Context padding size (left and right) in milliseconds

crashOnFallback

bool

false

If true, fail instead of falling back to a slower backend (useful for testing)

cpuNumThreads

int32_t

16

Number of threads for CPU execution provider (capped at hardware concurrency)

13.2 WordBoostingEntry#

Field

Type

Description

word

const char*

Word to boost or suppress

score

float

Boost score (positive = emphasize, negative = suppress)

13.3 ASRRivaORTRuntimeParameters#

Field

Type

Default

Description

sampling

ASRRivaSamplingStrategy

eBeamSearch

Sampling strategy (eGreedy for offline only, eBeamSearch for both)

streamingMode

bool

false

Enable streaming mode for this evaluation

silenceDetectionTimeMs

int32_t

-1

Silence detection window in ms (streaming only, -1 = disabled)

silenceSensitivityThreshold

float

0.85f

Silence sensitivity threshold (0.0 - 1.0)

13.4 StreamingParameters#

Field

Type

Description

mode

StreamingMode

Streaming mode - only eStreamingModeInputOutput is available (parameter not currently used)

signal

StreamSignal

Stream signal: eStreamSignalStart, eStreamSignalData, eStreamSignalStop

13.5 ASRRivaSamplingStrategy#

Value

Description

eGreedy

Greedy decoding (faster, less accurate, offline mode only)

eBeamSearch

Beam search decoding (slower, more accurate, both offline and streaming)

13.6 InferenceExecutionState#

Value

Description

kInferenceExecutionStateDataPartial

Partial result available (streaming)

kInferenceExecutionStateDataPending

Data is pending processing

kInferenceExecutionStateDone

Processing complete, final result available

13.7 Data Slots#

Slot Constant

Direction

Type

Description

nvigi::kASRDataSlotAudio

Input

InferenceDataAudio

Raw audio data (16kHz, 16-bit, mono)

nvigi::kASRDataSlotTranscribedText

Output

InferenceDataText

Accumulated transcription with enough past/future context (high accuracy)

nvigi::kASRDataSlotPartialText

Output

InferenceDataText

Real-time transcription of recent audio without enough context (streaming only, less accurate)

14. Best Practices#

  1. Initialize once: Call nvigiInit once at application startup, not for each inference

  2. Reuse instances: Create instances once and reuse for multiple inferences to avoid overhead

  3. Match audio format: Always provide 16kHz, 16-bit, mono PCM audio

  4. Use beam search: Prefer eBeamSearch for production quality transcriptions

  5. Enable auto punctuation: Improves readability and capitalizes proper nouns

  6. Choose appropriate backend:

    • Use GPU backend with TensorRT-RTX when possible for best performance on NVIDIA GPUs (default)

    • Use CPU backend for limited VRAM scenarios, development, and testing

  7. Use callbacks: Always implement callbacks to receive results asynchronously

  8. Handle errors: Always check return values with NVIGI_FAILED

  9. Clean up: Destroy instances and shutdown NVIGI on exit

  10. Thread safety: Each instance should be used by only one thread at a time

  11. Streaming signals: Always send START, DATA, and STOP in order

  12. Word boosting: Use positive scores to emphasize domain-specific vocabulary, negative scores to suppress

  13. Silence detection: Enable in streaming mode to detect natural pauses (1000-2000ms windows work well)

  14. Multilingual: Select the correct model GUID for your target language; all other API usage remains the same across languages

15. Additional Resources#

16. Support#

For issues, questions, or feedback, please contact NVIDIA Developer Support or refer to the NVIGI Developer Pack documentation.

On this page