Is this page helpful?

Getting Started with RIVA ASR Plugin Pack#

1. What is NVIGI RIVA ASR?#

The NVIDIA In-Game Inferencing (NVIGI) RIVA ASR Plugin Pack provides automatic speech recognition (ASR) capabilities for PC applications and games. Built on the NVIGI plugin architecture, it enables seamless integration of real-time and offline speech-to-text functionality with support for multiple inference backends (CPU, DirectML, TensorRT-RTX).

1.1 Key Features#

  • Streaming & Offline Recognition: Real-time streaming ASR with partial results, or batch offline transcription

  • Multi-backend Support: Run on CPU (ONNX Runtime), DirectML (GPU), or TensorRT-RTX (NVIDIA RTX GPUs)

  • Advanced Decoding: Greedy or beam search with KenLM language model support

  • Automatic Punctuation: Optional BERT-based punctuation and capitalization

  • Word Boosting: Custom vocabulary emphasis for domain-specific terms

  • Silence Detection: Automatic speech/pause detection in streaming mode

  • Multilingual Support: Speech recognition in 8 languages – English, French, German, Italian, Spanish, Korean, Japanese, and Mandarin Chinese

  • Production-Ready Models: Pre-configured Conformer-CTC models per language and English Parakeet-0.6B

1.2 Architecture#

The NVIGI SDK uses a plugin architecture where ASR functionality is provided through the nvigi.plugin.asr.riva-ort plugin. This plugin:

  • Implements the standard NVIGI plugin API for easy integration

  • Supports both synchronous and asynchronous inference

  • Can be swapped with other ASR plugins with minimal code changes

  • Integrates with D3D12/CUDA for concurrent graphics and AI workloads

2. Quick Start Guide#

2.1 Prerequisites#

Before getting started, ensure you have:

  • Hardware: NVIDIA RTX 20x0 series or newer (RTX 30x0/40x0 recommended). ASR VRAM usage ranges from 500 MB to 1.6 GB depending on the model, parameters, and graphics card.

  • OS: Windows 10 20H1 (build 19041) or newer

  • Driver: NVIDIA graphics driver r555.85 or newer

  • Development (for rebuilding samples): Visual Studio 2019/2022 with Windows SDK 10.0.19041+

2.2 Run Your First ASR Sample (Command-Line)#

The fastest way to experience ASR is with the pre-built command-line sample:

cd bin\x64
.\nvigi.asr.sample.exe -m ..\..\data\nvigi.models -b gpu

This will:

  • Initialize the ASR plugin with the Conformer-CTC model

  • Capture audio from your default microphone (when no -w option is provided)

  • Display real-time transcriptions as you speak

Alternative: Test with a WAV file in offline mode (recommended for better accuracy):

.\nvigi.asr.sample.exe -m ..\..\data\nvigi.models -b gpu -w ..\..\data\nvigi.test\nvigi.asr\jfk.wav --mode offline

Streaming vs. Offline Mode#

The sample supports two processing modes:

  • Offline mode (--mode offline): Recommended for most use cases. More accurate and faster. Transcription is available after recording completes. Ideal for voice commands, push-to-talk, and batch processing.

  • Streaming mode (--mode streaming, default): Use only when real-time transcription during recording is required (e.g., live captioning, seeing text while still speaking).

For detailed usage and options, see nvigi.asr.sample documentation.

2.3 Try the 3D Interactive Sample#

Experience ASR in a 3D application with GUI:

cd bin\x64
.\nvigi.3d.exe

  • Use the UI to enable microphone input

  • Speak and see transcriptions appear in real-time

  • Explore different ASR settings and backends

For detailed 3D sample usage, see 3D Sample documentation.

2.4 Explore the Models#

The pack includes production-ready ASR models:

English Models:

Model

Language

GUID

Conformer-CTC

English (en-US)

{B5D4F28E-3DB9-4FDA-A78B-A569AA22FCD0}

Parakeet-0.6B

English (en-US)

{53211BAC-8B34-4AE5-83C3-9B4CA0B2AE6D}

Multilingual Models (Conformer-CTC):

Language

GUID

French (fr-FR)

{7B9ED4D0-D15C-450F-BCC8-8FA7B460BFC0}

German (de-DE)

{632AA9FC-5C76-460D-B915-791E41F40028}

Italian (it-IT)

{33A53EA0-AD25-4CF1-B4FB-9B1513FDA24F}

Spanish (es-ES)

{20AD4848-5CB4-49CA-B48F-AF1AA61B4EB2}

Korean (ko-KR)

{14BA55B2-0357-4B7B-B021-B9F1D8DFB77A}

Japanese (ja-JP)

{29A8FAFA-A9BB-485C-AEEF-BFF4512DFE91}

Mandarin (zh-CN)

{0DDAD855-F9C3-42B4-AEED-18C9D0440923}

All models include: acoustic model, preprocessing, and KenLM language model. Auto-punctuation is available for English models.

Note

Only the English Conformer-CTC model is bundled with the pack. Parakeet-0.6B and all multilingual models must be downloaded separately.

To download all available models: download_data.bat

To download specific models only: download_data.bat parakeet french mandarin

Available model names: parakeet, french, german, italian, spanish, korean, japanese, mandarin

Models are located under data/nvigi.models/nvigi.plugin.asr.riva-ort/{GUID}/. To use a specific language, provide the corresponding GUID as the modelGUID when creating the ASR instance.

For full model details, see ASR Models.

2.5 Rebuild Samples from Source (Optional)#

To modify samples or debug integration:

  1. Open a VS2022 Developer Command Prompt

  2. Navigate to the pack root directory

  3. Run .\setup.bat to generate build files

  4. Open _project/vs2022/nvigi.rivaASR.sln in Visual Studio 2022

  5. Build the desired configuration (Debug/Release)

  6. Run .\copy_sdk_binaries.bat <config> to deploy binaries

For detailed rebuild instructions, see Rebuilding Samples.

2.6 Integrate into Your Application#

Once familiar with the samples, integrating ASR into your own application follows a straightforward five-step pattern. Each step is covered in detail with code examples in the Programming Guide.

Integration Overview#

  1. Include headers and load the NVIGI core library – Add #include <nvigi.h> and #include "nvigi_asr_riva.h" to your project. At runtime, load nvigi.core.framework.dll and retrieve the API function pointers (nvigiInit, nvigiLoadInterface, etc.). See Initialize NVIGI.

  2. Load the ASR plugin interface – Choose the CPU or GPU plugin and call nvigiLoadInterface to get an IAutoSpeechRecognition pointer. See Load the ASR Interface.

  3. Create an ASR instance with your configuration – Set the model GUID (language), backend preference, streaming vs. offline mode, and optional features like auto punctuation or word boosting. Then call createInstance. See Create an ASR Instance.

  4. Feed audio and receive transcriptions – Provide 16kHz 16-bit mono PCM audio via data slots. For offline mode, call evaluate() with the complete audio. For streaming mode, send START/DATA/STOP signals via evaluateAsync() and receive results through a callback. See Run Inference and Streaming Mode.

  5. Clean up – Destroy the instance, unload the interface, and shut down NVIGI. See Cleanup.

Where to Start in the Code#

The two sample implementations serve as integration templates:

  • source/samples/nvigi.asr.sample/sample_asr.cpp – Minimal command-line integration showing the full lifecycle (init, create, stream/offline, cleanup) without any graphics dependencies. Start here for the simplest integration path.

  • source/samples/nvigi.3d/src/nvigi/NVIGIContext.cpp – Real-world integration in a D3D12/Vulkan 3D application with CIG scheduling, showing how ASR fits alongside a rendering loop.

3. What’s Included in This Pack#

The RIVA ASR Plugin Pack provides everything needed to integrate automatic speech recognition into your applications:

3.1 Pre-Built Components#

  • ASR Plugin: nvigi.plugin.asr.riva-ort.dll - Production-ready speech recognition plugin

  • NVIGI Core: Runtime libraries for the NVIGI plugin framework

  • ASR Models: Conformer-CTC models for 8 languages (English, French, German, Italian, Spanish, Korean, Japanese, Mandarin) and Parakeet-0.6B for English

  • Sample Applications: Pre-built executables demonstrating ASR integration

  • Headers & APIs: Complete SDK headers for plugin integration

3.2 Included Samples#

The pack includes two samples with full source code:

  • nvigi.asr.sample - Command-line ASR demo

    • Supports both streaming (real-time) and offline (batch) modes

    • Microphone input or WAV file processing

    • Configurable decoding strategies and parameters

    • Located in: source/samples/nvigi.asr.sample/

  • nvigi.3d - 3D interactive sample with GUI

    • Voice-controlled 3D application

    • Visual transcription display

    • Real-world integration example

    • Located in: source/samples/nvigi.3d/

All samples are pre-built in bin/x64/ and ready to run immediately.

4. Known Issues and Important Notes#

4.1 Release Notes#

General#

  • On some system configurations, the 3D sample may crash on first model load. If this occurs, reboot and rerun the sample.

  • Sample applications require the NVIGI core DLLs to be present in the bin/x64/ directory or accessible via system PATH.

ASR Plugin Specific#

  • DirectML Backend: First inference may have higher latency as DirectML compiles shaders. Subsequent inferences will be faster.

  • TensorRT-RTX Backend: First time the backend is run on a machine, it requires approximately 30 seconds to build the engine. The engine is then cached locally and reused for subsequent runs. The engine will need to be rebuilt only if the TensorRT-RTX version is updated.

  • Streaming Mode: Very short audio chunks (<100ms) may result in no output. Recommended chunk size:

    • Conformer models: 380ms (default) - efficient with less context

    • Parakeet models: 900ms - needs more context for optimal accuracy

  • Silence Detection: Default parameters work well for most scenarios, but may need tuning for specific audio conditions (e.g., noisy environments).

  • Punctuation Model: Optional punctuation adds ~10-15% overhead. Disable if not needed via ASRRivaORTCreationParameters::enableAutoPunctuation = false.

Building from Source#

  • Plugin DLLs (nvigi.plugin.asr.riva-ort.gpu.dll and nvigi.plugin.asr.riva-ort.cpu.dll) are pre-built and cannot be rebuilt from this pack.

  • Sample applications can be rebuilt from source using the provided Visual Studio solution.

  • After building, run .\copy_sdk_binaries.bat <config> to deploy binaries to bin/x64/.

Performance Considerations#

  • CPU Backend: Slower than GPU but no VRAM usage. Suitable for low-end systems or when GPU is busy with graphics.

  • DirectML Backend: Good balance of performance and compatibility. Works on any DirectX 12 capable GPU.

  • TensorRT-RTX Backend: Best performance on NVIDIA RTX GPUs. Requires CUDA and TensorRT installation.

4.2 Getting Help#

For issues, questions, or feedback:

5. Next Steps#

Now that you’re familiar with the basics:

  1. Explore the Samples - Run and modify the included samples to understand ASR integration patterns

  2. Read the Programming Guide - Deep dive into the RIVA ASR API

  3. Experiment with Parameters - Try different decoding strategies, backends, and streaming parameters

  4. Integrate into Your App - Use the samples as templates for your own application integration

For the best development experience, start with the 3D sample to see ASR in a realistic application context, then explore the command-line ASR sample for detailed parameter tuning and integration patterns.