Is this page helpful?

Convert a NeMo ASR Model for the Nemotron ASR Plugin#

This workflow converts a supported NVIDIA NeMo ASR .nemo archive into the GGUF format consumed by the Nemotron ASR plugin. It quantizes eligible model weights and verifies that the converted model can be loaded by a selected packaged backend.

Use this workflow for supported base or fine-tuned NeMo ASR models. It does not train or fine-tune a model. Load verification confirms runtime compatibility; evaluate the converted model on representative audio to validate transcription accuracy.

The end-to-end workflow is:

  1. Start with a supported NeMo ASR .nemo archive.

  2. Select the matching model family.

  3. Convert the NeMo archive to GGUF.

  4. Quantize the eligible model weights.

  5. Verify that the GGUF loads with a selected packaged backend.

  6. Install the verified GGUF under a new model GUID.

  7. Evaluate transcription accuracy on representative audio.

The converter supports these NeMo ASR model families:

parakeet-tdt-v2
parakeet-tdt-v3
nemotron-streaming-en-rnnt
nemotron-3.5-streaming-rnnt

1. Compatibility contract#

A fine-tuned archive must preserve more than a marketing family name. It must retain the supported model topology, tensor layout/names, tokenizer and vocabulary relationships, decoder/joint contract, and—for streaming models—the cache-aware encoder geometry and cache metadata expected by the runtime. Verification can reject structural or backend load incompatibility, but only evaluation on representative audio can qualify accuracy.

The internal/convert_parakeet_tdt.py and internal/convert_streaming_rnnt.py files are implementation components of the supported wrapper. Their direct CLIs are not the customer contract.

2. Isolated Python environment#

Python 3 is required. This workflow has been tested with Python 3.12. Use an isolated virtual environment and install:

py -3 -m venv .venv-nemotron-asr
.\.venv-nemotron-asr\Scripts\python.exe -m pip install -r tools\nemo-quantization\requirements-quantization.txt

The shipped requirements install gguf, librosa, numpy, pyyaml, sentencepiece, and torch.

Keep the source .nemo local. The wrapper does not download models. Process only trusted .nemo archives and perform conversion in an isolated workspace: the streaming converter extracts the archive contents as part of conversion.

3. Convert, quantize, and verify#

nemo_asr_quantize.py is the supported end-to-end entry point. It converts the .nemo archive, quantizes the GGUF, and runs mandatory backend-load verification.

Run from the pack root:

.\.venv-nemotron-asr\Scripts\python.exe tools\nemo-quantization\nemo_asr_quantize.py `
  --nemo C:\models\fine-tuned.nemo `
  --output C:\models\converted\model-q4k.gguf `
  --model-family nemotron-3.5-streaming-rnnt `
  --quantization q4_k `
  --verification-backend cpu `
  --sdk .\bin\x64

Option

Values/meaning

--nemo

Required existing local .nemo file

--output

Required output GGUF

--model-family

One of the four families listed above

--quantization

f16 (FP16), q8_0 (8-bit/INT8), q5_k or q5_0 (5-bit), q4_k or q4_0 (4-bit), q3_k (3-bit), or q2_k (2-bit); default q4_k

--decoder-weight-type

Streaming decoder/joint type: f32 (FP32), bf16 (BF16), fp16 (FP16), or q8_0 (8-bit/INT8); default q8_0

--quantizer

Explicit local path to asr_quantize.exe

--verification-backend

cpu, cuda, d3d12, or vk; default cpu

--verifier

Explicit local path to nvigi.asr.model_verify.exe

--sdk

Explicit package bin/x64 runtime directory

The wrapper locates its internal converter, emits an intermediate GGUF, calls the quantizer, and then verifies the output. If mandatory verification fails, it removes the output.

3.1 Family precision behavior#

Family

Eligible encoder weights

Decoder/joint from wrapper conversion

Streaming RNNT

--quantization through post-quantization

--decoder-weight-type, default Q8_0

Parakeet TDT

--quantization through post-quantization

Converter leaves decoder/joint F32

This table describes wrapper output, not the exact release recipe of the four shipped models.

4. Standalone asr_quantize.exe#

asr_quantize.exe <input.gguf> <output.gguf> <type>

Accepted types are f16, q8_0, q5_k, q4_k, q4_0, q5_0, q3_k, and q2_k. Only eligible encoder linear and pointwise-convolution weights change. Biases, normalization tensors, depthwise/stem convolutions, decoder, joint, embedding, and preprocessor tensors remain as-is. Always write to a separate output and verify it.

5. Verify a model directory#

The Python wrapper accepts a directory containing exactly one ASR GGUF and an optional nvigi.model.config.json:

.\.venv-nemotron-asr\Scripts\python.exe tools\nemo-quantization\verify_asr_model.py `
  --model-dir C:\models\converted `
  --backend cpu `
  --verifier .\bin\x64\nvigi.asr.model_verify.exe `
  --sdk .\bin\x64

Option

Meaning

--model-dir

Required directory containing one GGUF and optional config

--backend

One selected backend: cpu, cuda, d3d12, or vk; default cpu

--verifier

Path to the native verifier when auto-discovery is unsuitable

--sdk

Runtime bin/x64 containing nvigi.core.framework.dll

The direct native form is:

.\bin\x64\nvigi.asr.model_verify.exe --sdk .\bin\x64 --model-dir C:\models\converted --backend cpu

For one invocation, the verifier:

  1. requires exactly one .gguf in the input directory;

  2. creates a temporary nvigi.plugin.asr.nemotron-ggml repository with a dummy GUID;

  3. hard-links the GGUF when possible and copies it otherwise;

  4. creates a temporary config if the input directory does not supply one;

  5. loads the selected backend plugin, creates and warms the model instance, destroys it, and unloads cleanly.

It verifies only the selected backend. Repeat the command for every intended backend. It does not transcribe audio, compute WER, qualify accuracy, or permanently register the model.

6. Install the converted model#

Generate a new GUID for the customer model; do not reuse a shipped model GUID. Use the same brace-wrapped GUID for the directory name and the sample’s --model value. The opening and closing braces are required. Create:

data/nvigi.models/nvigi.plugin.asr.nemotron-ggml/{YOUR-GUID}/
  model-q4k.gguf
  nvigi.model.config.json

A minimal config is:

{
  "name": "My adapted Nemotron ASR model",
  "vram": 4096,
  "language": "en-US"
}

Choose vram from measured peak requirements across the backends and workloads you intend to support, with release-appropriate margin; do not copy 4096 blindly. Set language to truthful model metadata such as a qualified locale or multi. Preserve the GGUF and config together in the GUID directory.

Run an inference smoke test with the packaged audio. This example uses CPU:

.\bin\x64\nvigi.asr.sample.exe `
  --model-dir .\data\nvigi.models `
  --model "{YOUR-GUID}" `
  --backend cpu `
  --wav-file .\data\nvigi.test\nvigi.asr\jfk.wav `
  --streaming

Success requires exit code 0 and a non-empty transcript. Replace cpu with another backend to test it. Use --streaming for a converted streaming RNNT family; omit it for a Parakeet TDT family.

Verification’s temporary dummy repository is not installation; normal discovery requires the permanent directory above. After the smoke test, evaluate transcription accuracy on representative audio; backend load verification does not qualify accuracy.

7. Troubleshooting#

Failure

Action

Converter rejects topology/tensors

Confirm exact supported family, tokenizer/vocabulary, and cache geometry were preserved

Quantizer not found

Pass --quantizer <pack>\bin\x64\asr_quantize.exe

Verifier/runtime not found

Pass both --verifier and --sdk explicitly

Directory contains zero/multiple GGUFs

Stage exactly one GGUF for a verification invocation

One backend passes, another fails

Verification is backend-specific; inspect that backend’s runtime/dependencies

Model loads but text quality regresses

Load verification does not measure WER; run a representative accuracy suite

See Developer Pack for the shipped model inventory and Programming Guide for offline versus streaming integration.