Is this page helpful?

RIVA ASR Developer Pack Overview#

1. What’s Included#

The RIVA ASR Plugin Pack provides automatic speech recognition (ASR) capabilities for NVIDIA In-Game Inferencing (NVIGI) applications. This release is built on GGML as the inference engine and ships two Parakeet-TDT models. The previous ONNX-Runtime-based plugins (nvigi.plugin.asr.riva-ort.*) have been removed.

The pack includes:

  • Four pre-built ASR plugin DLLs — one per backend (CUDA, Vulkan, D3D12, CPU)

  • Two Parakeet-TDT models in GGUF format (v2 English, v3 multilingual)

  • Command-line (nvigi.asr.sample) and 3D GUI (nvigi.3d) sample applications

  • Complete source code for samples and full NVIGI headers for integration

  • Comprehensive programming guides and documentation

2. Automatic Speech Recognition Models#

2.1 Included Models#

Two Parakeet-TDT-0.6B models ship with the pack. Both use the Token-and-Duration Transducer (TDT) decoder and are shipped as mixed-precision GGUF files (encoder matmul weights Q4_K, decoder matrices F16, everything else F32) for an on-disk size of ~380 MB (v2) / ~390 MB (v3). See Section 2.3 for the per-component precision breakdown. Peak runtime VRAM is ~1.2 GB per instance (weights + GGML compute-graph scratch).

Plugin

Model Name

GUID

Languages

On-disk

Status

nvigi.plugin.asr.riva-ggml.*

Parakeet-TDT-0.6B-v2

{163BD6DC-248D-4238-BC9B-B76D2AB1C3DD}

English (en-US)

~380 MB

Included

nvigi.plugin.asr.riva-ggml.*

Parakeet-TDT-0.6B-v3

{9739C15F-9A65-484D-B091-FFA4B9DDD0EE}

25 European languages (multilingual)

~390 MB

Included

The same model files are used by every backend — there is no per-backend conversion step. Swap backends by loading a different plugin DLL; keep the model directory the same.

Languages supported by Parakeet-TDT-0.6B-v3: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Russian, Ukrainian. The acoustic model auto-detects the language of the input audio; no language tag is required at runtime.

2.2 Model Files#

Each model ships as a single GGUF file alongside its nvigi.model.config.json metadata. No preprocessor, punctuation, language-model, or lexicon files are required — all three stages (mel spectrogram → FastConformer encoder → TDT decoder) live in one GGUF.

data/nvigi.models/nvigi.plugin.asr.riva-ggml/
├── {163BD6DC-248D-4238-BC9B-B76D2AB1C3DD}/         # Parakeet-TDT v2
│   ├── model-q4k.gguf
│   └── nvigi.model.config.json
└── {9739C15F-9A65-484D-B091-FFA4B9DDD0EE}/         # Parakeet-TDT v3
    ├── model-q4k.gguf
    └── nvigi.model.config.json

2.3 Quantization Recipe#

The shipped model-q4k.gguf is not uniformly quantized. Only the encoder’s large matmul weights are Q4_K; the decoder’s weight matrices are F16; the preprocessor, joint network, and all biases/norms remain F32. This split keeps the dominant compute in Q4_K while protecting numerically sensitive, low-volume tensors.

Component

Precision

Rationale

Preprocessor (mel filterbank + Hann window)

F32

2 small static tensors; runs on CPU

Encoder — attention / FFN / pointwise-conv weights (264 matrices)

Q4_K

~7× compression on the dominant cost

Encoder — biases, norms, subsampling conv, depthwise conv

F32

Too small to benefit from quantization

Decoder — embedding + LSTM weight matrices (5 matrices)

F16

2× compression; preserves autoregressive quality

Decoder — LSTM biases

F32

Tiny; keep at full precision

Joint network

F32

~4 MB total; not worth quantizing

Tensor-count summary (identical for v2 and v3, 653 tensors total):

Component

F32

F16

Q4_K

Total

Preprocessor

2

0

0

2

Encoder

372

0

264

636

Decoder

4

5

0

9

Joint network

6

0

0

6

2.4 Model Capabilities#

  • Offline transcription: Process complete audio buffers in one call (synchronous evaluate() and non-blocking evaluateAsync())

  • Multi-backend: CUDA (NVIDIA GPUs), Vulkan (cross-vendor), D3D12 (cross-vendor), CPU

  • Decoding: TDT greedy decoding (only)

  • Built-in warmup: Five short warmup transcriptions at instance creation stabilize CUDA-graph / D3D12-shader specialization so the first user-facing inference is not slowed down

  • CIG (CUDA-In-Graphics): Efficient GPU sharing with a D3D12 or Vulkan rendering loop when using the CUDA backend on NVIDIA GPUs

  • D3D12 device interop: The D3D12 backend can reuse the application’s ID3D12Device, queues, and resource-allocator callbacks

  • Vulkan device interop: The Vulkan backend can reuse the application’s VkInstance, VkPhysicalDevice, VkDevice, queues, and memory-allocator callbacks

  • GPU scheduling modes: Prioritize Graphics / Prioritize Inference / Balance via IHWICuda on the CUDA backend

3. Accuracy and Performance#

3.1 Methodology#

  • Dataset: LibriSpeech test-clean (2620 utterances) and test-other (2939 utterances), obtained from https://www.openslr.org/12. .flac inputs were converted to 16 kHz mono 16-bit PCM .wav prior to inference.

  • Decoding: Greedy TDT (the only decoding mode exposed by this plugin).

  • WER normalization: Whisper-style — case-folding and punctuation stripping — applied to reference and hypothesis before scoring.

  • GPU hardware: NVIDIA GeForce RTX 4090 (Ada Lovelace). Driver 596.21.

  • CPU hardware: Intel Core i9-14900K, 32 logical threads (8 P-cores × 2 SMT + 16 E-cores). CommonCreationParameters::numThreads set to std::thread::hardware_concurrency().

  • Warmup: Numbers reflect post-warmup steady-state. The five warmup transcriptions the plugin runs internally at createInstance are excluded from timing.

3.2 Metric definitions#

  • Avg WER: arithmetic mean of per-utterance WER values.

  • Corpus WER: length-weighted WER computed over the entire corpus. Standard ASR research metric; less noisy than Avg WER on short utterances.

  • Perfect %: share of utterances with WER == 0.

  • High WER %: share of utterances with WER ≥ 50% (typically audio-capture failures or severe domain mismatch).

  • RTF: processing time divided by audio duration. RTF < 1 is faster than realtime; 1 / RTF is the speed-up factor.

3.3 Accuracy on LibriSpeech test-clean#

Backend

Avg WER

Corpus WER

Perfect %

High WER %

CUDA

2.04%

1.69%

76.37%

0.34%

Vulkan

2.07%

1.69%

76.53%

0.38%

D3D12

2.07%

1.70%

76.41%

0.38%

CPU

2.06%

1.68%

76.53%

0.38%

Backend

Avg WER

Corpus WER

Perfect %

High WER %

CUDA

2.52%

1.98%

72.75%

0.42%

Vulkan

2.56%

2.00%

72.63%

0.50%

D3D12

2.53%

2.00%

72.71%

0.42%

CPU

2.52%

1.98%

72.67%

0.38%

3.4 Accuracy on LibriSpeech test-other#

Backend

Avg WER

Corpus WER

Perfect %

High WER %

CUDA

3.99%

3.14%

64.68%

1.02%

Vulkan

4.01%

3.14%

64.65%

1.05%

D3D12

3.99%

3.13%

64.68%

1.05%

CPU

4.04%

3.18%

64.48%

1.09%

Backend

Avg WER

Corpus WER

Perfect %

High WER %

CUDA

4.60%

3.72%

61.28%

1.19%

Vulkan

4.64%

3.72%

61.11%

1.22%

D3D12

4.60%

3.70%

61.35%

1.16%

CPU

4.62%

3.74%

61.08%

1.22%

3.5 Speed (RTF)#

Real-Time Factor (processing time / audio duration). Lower is faster; the multiplier in parentheses is 1 / RTF — how much faster than realtime each backend is. RTF is driven by model architecture and backend, not by dataset — v2 and v3 are both 0.6B-parameter TDT models with the same q4k quantization, so numbers barely move across the four dataset/model combinations.

Backend

v2 test-clean

v2 test-other

v3 test-clean

v3 test-other

CUDA

0.00283 (353× realtime)

0.00301 (332×)

0.00279 (358×)

0.00308 (324×)

Vulkan

0.00505 (198×)

0.00548 (183×)

0.00534 (187×)

0.00555 (180×)

D3D12

0.01445 (69×)

0.01551 (65×)

0.01464 (68×)

0.01544 (65×)

CPU

0.02192 (46×)

0.02201 (45×)

0.02263 (44×)

0.02236 (45×)

Backend ordering is identical in every column: Vulkan ~1.8× slower than CUDA, D3D12 ~5× slower, CPU ~7.7× slower.

3.6 Latency by audio length#

Average per-utterance processing time in milliseconds, broken down by audio duration bucket. Shows how each backend scales with input length.

Audio Duration

Sample count

CUDA (ms)

Vulkan (ms)

D3D12 (ms)

CPU (ms)

1–3 s

391

15.8

17.8

64.8

62.1

3–6 s

966

17.9

26.2

74.6

100.4

6–9 s

546

20.9

41.4

131.0

160.5

9–12 s

286

24.3

49.2

141.3

221.7

12–15 s

192

27.0

57.4

151.9

282.8

15–18 s

114

30.5

66.2

166.0

346.8

18–21 s

60

33.8

78.4

181.7

419.0

21–24 s

30

37.1

87.2

184.7

472.3

24 s+

35

41.5

100.9

206.3

599.1

Audio Duration

Sample count

CUDA (ms)

Vulkan (ms)

D3D12 (ms)

CPU (ms)

1–3 s

391

15.5

18.6

62.6

64.7

3–6 s

966

17.6

27.4

75.8

103.5

6–9 s

546

20.7

43.6

132.9

165.9

9–12 s

286

24.0

52.2

143.9

228.9

12–15 s

192

26.5

61.2

155.0

293.2

15–18 s

114

30.0

71.4

169.7

355.5

18–21 s

60

33.6

82.8

184.7

429.9

21–24 s

30

36.6

91.5

189.0

483.0

24 s+

35

41.5

111.5

214.2

619.9

4. Important Documentation in the Developer Pack#

  • Samples

    • The best starting doc — the samples ship pre-compiled and ready-to-run. Covers running, rebuilding, and debugging both samples.

  • RIVA ASR Plugin Programming Guide

    • Detailed API documentation for the RIVA ASR GGML plugin: backend selection, graphics interop, CIG, runtime parameters, and migration notes from the ORT plugin.

  • Getting Started

    • Prerequisites, first sample run, and high-level integration checklist.

5. Contents of the Developer Pack#

The pack contains three things: the four per-backend plugin DLLs, the two Parakeet-TDT models, and two rebuildable sample apps (CLI + 3D). Everything else is headers, build machinery, or documentation.

5.1 Top-level layout#

Six top-level folders, each with a distinct purpose:

Folder

What’s in it

bin/x64/

Pre-built runtime binaries — plugin DLLs, NVIGI core, CUDA / DXC / Agility SDK runtimes, sample EXEs, compiled shaders

nvigi_core/

NVIGI core headers, import libraries, pre-built core DLLs per build config, PDBs

include/

Public plugin header to integrate against (nvigi_asr_riva.h)

source/

Rebuildable sample source (nvigi.asr.sample, nvigi.3d)

data/

Shipped data — GGUF models under nvigi.models/, 3D-sample assets (glTF, fonts, logo) under nvigi.test/nvigi.3d/, and the ASR sample audio (jfk.wav) under nvigi.test/nvigi.asr/

docs/

Sphinx-generated HTML documentation (this document)

Root-level files you’ll interact with: setup.bat, build.bat, copy_sdk_binaries.bat, premake.lua, project.xml. Everything under premake/ and tools/ is build-system machinery you only touch when rebuilding samples.

5.2 Where to find things#

I want…

Look at…

The plugin DLL for my backend

bin/x64/nvigi.plugin.asr.riva-ggml.<backend>.dll

The NVIGI loader

bin/x64/nvigi.core.framework.dll

The two shipped models

data/nvigi.models/nvigi.plugin.asr.riva-ggml/{GUID}/model-q4k.gguf

Headers for my integration

include/nvigi_asr_riva.h + nvigi_core/include/nvigi.h + nvigi_core/include/nvigi_ai.h

The CLI sample binary

bin/x64/nvigi.asr.sample.exe

The 3D sample binary

bin/x64/nvigi.3d.exe

Source for the CLI sample

source/samples/nvigi.asr.sample/

Source for the 3D sample

source/samples/nvigi.3d/

D3D12 Agility SDK runtime

bin/x64/D3D12/

Pre-compiled 3D-sample shaders

bin/x64/shaders/framework/{dxbc,dxil,spirv}/

Core DLLs per build config (Release / Debug / Production)

nvigi_core/bin/{Release_x64, Debug_x64, Production_x64}/

PDBs for core DLLs

nvigi_core/symbols/

The data/nvigi.models/ directory is backend-independent. All four backends read the same GGUF file. Plugin DLLs are shipped pre-built; sample rebuild instructions live in Samples. For the literal on-disk tree, see Appendix A.

6. Shipping Checklist#

What to copy next to your game executable.

6.1 Always required (regardless of backend)#

File

Source

nvigi.core.framework.dll

bin/x64/

nvigi.core.framework.json

bin/x64/ — plugin-discovery config consumed by the loader; ship alongside the DLL

nvigi.plugin.hwi.common.dll

bin/x64/

data/nvigi.models/nvigi.plugin.asr.riva-ggml/<GUID>/model-q4k.gguf

data/ (one or both models)

VC++ 2015–2022 redistributable (MSVCP140.dll, VCRUNTIME140.dll, VCRUNTIME140_1.dll, VCOMP140.DLL)

Installed on the target machine

6.2 Per-backend#

Backend

Ship from bin/x64/

CPU

nvigi.plugin.asr.riva-ggml.cpu.dll

CUDA

nvigi.plugin.asr.riva-ggml.cuda.dll, cudart64_12.dll, cublas64_12.dll, cublasLt64_12.dll

CUDA + CIG

CUDA row + nvigi.plugin.hwi.cuda.dll, cig_scheduler_settings.dll, cupti64_2025.1.0.dll

Vulkan

nvigi.plugin.asr.riva-ggml.vk.dll

D3D12

nvigi.plugin.asr.riva-ggml.d3d12.dll, dxcompiler.dll, dxil.dll, D3D12/D3D12Core.dll

D3D12 + device interop

D3D12 row + nvigi.plugin.hwi.d3d12.dll

System / driver DLLs (nvcuda.dll, vulkan-1.dll, d3d12.dll, dxgi.dll, …) are not shipped in the pack — they come from the GPU driver or Windows 10+. Do not redistribute them.

No in-process fallback between backends. If the backend DLL you ship fails to load (missing CUDA runtime, unsupported driver, etc.), that backend is unavailable for the rest of the process. Ship more than one backend DLL and pick at startup if you need to support multiple hardware configurations — the 3D sample’s SelectAutoPlugin is a working reference for such selection logic.

Appendix A. Full directory listing#

The literal on-disk layout of the pack, for reference:

riva_asr_pack/
├── bin/x64/                                           # Pre-built binaries
│   ├── nvigi.plugin.asr.riva-ggml.cuda.dll            # ASR plugin (CUDA)
│   ├── nvigi.plugin.asr.riva-ggml.vk.dll              # ASR plugin (Vulkan)
│   ├── nvigi.plugin.asr.riva-ggml.d3d12.dll           # ASR plugin (D3D12)
│   ├── nvigi.plugin.asr.riva-ggml.cpu.dll             # ASR plugin (CPU)
│   ├── nvigi.core.framework.dll                       # NVIGI plugin loader
│   ├── nvigi.core.framework.json                      # Plugin-discovery config (required by loader)
│   ├── nvigi.plugin.hwi.common.dll                    # Common HWI utilities
│   ├── nvigi.plugin.hwi.cuda.dll                      # CUDA HWI (CIG, scheduling)
│   ├── nvigi.plugin.hwi.d3d12.dll                     # D3D12 HWI
│   ├── cudart64_12.dll                                # CUDA 12 runtime
│   ├── cublas64_12.dll                                # cuBLAS
│   ├── cublasLt64_12.dll                              # cuBLASLt
│   ├── cupti64_2025.1.0.dll                           # CUDA Profiling Tools Interface
│   ├── cig_scheduler_settings.dll                     # CIG scheduler
│   ├── amd_ags_x64.dll                                # AMD GPU Services
│   ├── dxcompiler.dll, dxil.dll                       # DXC (used by nvigi.3d)
│   ├── nvigi.asr.sample.exe                           # Command-line sample
│   ├── nvigi.3d.exe                                   # 3D GUI sample
│   ├── D3D12/                                         # D3D12 Agility SDK runtime
│   │   ├── D3D12Core.dll
│   │   └── d3d12SDKLayers.dll
│   └── shaders/                                       # Pre-compiled shaders for nvigi.3d
│       └── framework/{dxbc,dxil,spirv}/...
│
├── include/                                           # ASR plugin header
│   └── nvigi_asr_riva.h
│
├── nvigi_core/                                        # NVIGI core headers + framework libs
│   ├── include/                                       # nvigi.h, nvigi_ai.h, nvigi_d3d12.h, ...
│   ├── bin/{Debug_x64,Release_x64,Production_x64}/    # Pre-built core DLLs per config
│   ├── lib/{Debug_x64,Release_x64,Production_x64}/    # Core import libraries
│   └── symbols/                                       # PDBs for core DLLs
│
├── source/                                            # Rebuildable sample source
│   └── samples/
│       ├── nvigi.asr.sample/                          # Command-line sample
│       └── nvigi.3d/                                  # 3D GUI sample (Donut-based)
│
├── data/
│   ├── nvigi.models/                                  # ASR models (shared across all backends)
│   │   └── nvigi.plugin.asr.riva-ggml/
│   │       ├── {163BD6DC-248D-4238-BC9B-B76D2AB1C3DD}/      # Parakeet-TDT v2
│   │       │   ├── model-q4k.gguf
│   │       │   └── nvigi.model.config.json
│   │       └── {9739C15F-9A65-484D-B091-FFA4B9DDD0EE}/      # Parakeet-TDT v3
│   │           ├── model-q4k.gguf
│   │           └── nvigi.model.config.json
│   └── nvigi.test/
│       ├── nvigi.3d/                                  # 3D sample assets (glTF scenes, fonts, logo)
│       └── nvigi.asr/
│           └── jfk.wav                                # Ready-to-run sample audio for nvigi.asr.sample (16 kHz, 16-bit, mono)
│
├── premake/, tools/                                   # Build system (premake5 + packman)
├── docs/                                              # Sphinx-generated HTML + source markdown
├── build.bat, setup.bat, copy_sdk_binaries.bat        # Build helpers
├── LICENSE.txt, NOTICE.txt
└── premake.lua, project.xml                           # Solution generator

Appendix B. Migration from the ORT Plugin#

Only relevant if you previously integrated nvigi.plugin.asr.riva-ort.*. Skip if this is your first time using Riva ASR.

v1 shipped Conformer / Parakeet CTC models on ONNX Runtime. This release ships Parakeet TDT v2 / v3 on GGML. Capability delta:

Feature

ORT plugin (v1)

GGML plugin (this release)

Inference engine

ONNX Runtime

GGML

Model format

ONNX

GGUF

Acoustic models

Conformer-CTC, Parakeet-CTC

Parakeet-TDT-0.6B-v2, Parakeet-TDT-0.6B-v3

Languages

English (en-US)

English (v2), 25 European languages (v3)

Backends

CPU, TensorRT-RTX, DirectML

CUDA, Vulkan, D3D12, CPU

Offline mode

✓

✓

Streaming mode

✓ (chunked, partial + accumulated text)

—

Auto punctuation

Separate BERT punc model

Emitted natively by v2 and v3

Word boosting

✓

—

Silence detection

✓ (streaming only)

—

Sampling

Greedy, Beam search with KenLM

TDT greedy

Unsupported features do not have stub APIs — the corresponding fields are absent from ASRRivaGGMLCreationParameters / ASRRivaGGMLRuntimeParameters entirely.