Inference

This page covers how to load ASR models and run inference in NeMo.

Loading Checkpoints

From a local file:

import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.restore_from("path/to/checkpoint.nemo")

From HuggingFace:

model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")

Exporting a Transformer Encoder to ONNX

Export an offline Transformer encoder with NeMo’s existing API:

model.eval()
encoder = model.encoder
encoder.set_max_audio_length(5000)
encoder.export("encoder.onnx", onnx_opset_version=17)

Set the maximum length to cover the longest feature sequence you need; the exported positional cache cannot grow later. Batch size and sequence length may vary within that limit.

Inputs are audio_signal of shape (batch, features, frames) and int64 length of shape (batch,). Outputs are outputs of shape (batch, channels, encoded_frames) and int64 encoded_lengths.

Export uses scaled dot-product attention with the encoder’s weights, positional encodings and masks. Ordinary training and inference retain FlexAttention. Check outputs, lengths and transcripts with your intended runtime and precision, since rounding can affect recognition.

Preprocessing, decoding, streaming caches and bypass_pre_encode are outside this interface. For RNNT/TDT, model.export("model.onnx") exports the encoder and combined prediction/joint network separately.

Building a TensorRT Engine for a Transformer Encoder

To run the offline encoder with TensorRT, compile it directly into a standalone plan with Torch-TensorRT, without an ONNX file:

python examples/asr/export/build_torch_trt_encoder.py \
    --model model.nemo --engine encoder.plan

The script captures the encoder with torch.export and converts the whole graph in FP32 with TF32 disabled. Unsupported operations fail the conversion; the plan has no PyTorch fallback. The plan has the same inputs and outputs as the ONNX export above.

Batch size and padded frame count are dynamic within the limits given by --min-frames, --opt-frames, --max-frames, --opt-batch and --max-batch. Building requires a CUDA device and matching PyTorch, Torch-TensorRT and TensorRT versions, and running the plan requires TensorRT only. The script refuses to overwrite an existing --engine file. Build the plan for the GPU and TensorRT version that will run it, and compare its outputs with the ONNX or PyTorch results at the same precision. The example README lists the validated versions, the limits of the conversion and the minimum frame count with feature stacking.

Basic Transcription

Python API:

outputs = model.transcribe(audio=["file1.wav", "file2.wav"], batch_size=2)
print(outputs[0].text)

The audio argument accepts file paths (strings), lists of paths, numpy arrays, or PyTorch tensors. Audio must be 16 kHz mono-channel.

Numpy/Tensor inputs:

import soundfile as sf
audio, sr = sf.read("audio.wav", dtype='float32')
outputs = model.transcribe([audio], batch_size=1)

Command line:

python examples/asr/transcribe_speech.py \
    pretrained_name="nvidia/parakeet-tdt-0.6b-v2" \
    audio_dir=<path_to_audio_dir>

Batch generator (for incremental processing):

model.transcribe() already handles large file lists internally via batching. Use transcribe_generator only when you need to process results incrementally (e.g., writing to disk batch-by-batch to avoid holding all outputs in memory):

config = model.get_transcribe_config()
config.batch_size = 32
for batch_outputs in model.transcribe_generator(audio_files, override_config=config):
    # write batch results to disk immediately
    ...

TranscribeConfig fields:

Field	Default	Description
batch_size	4	Batch size for inference. Larger = better throughput but more memory.
return_hypotheses	False	If True, return Hypothesis objects (with timestamps, scores) instead of plain strings.
use_lhotse	True	Use Lhotse dataloading for inference.
num_workers	None	Number of DataLoader workers.
channel_selector	None	Select channel(s) from multi-channel audio. 'average' to average across channels.
timestamps	None	If True, return word/segment timestamps (requires return_hypotheses=True).
augmentor	None	Optional augmentation config to apply during transcription.
verbose	True	Show progress bar.

For multi-task models (Canary), MultiTaskTranscriptionConfig extends this with prompt, text_field, lang_field, and enable_chunking.

Alignments:

hyps = model.transcribe(audio=["file.wav"], return_hypotheses=True)
alignments = hyps[0].alignments

Timestamps

Obtain word, segment, or character timestamps with Parakeet models (CTC/RNNT/TDT):

Simple usage:

hypotheses = model.transcribe(["audio.wav"], timestamps=True)

for stamp in hypotheses[0].timestamp['word']:
    print(f"{stamp['start']}s - {stamp['end']}s : {stamp['word']}")

for stamp in hypotheses[0].timestamp['segment']:
    print(f"{stamp['start']}s - {stamp['end']}s : {stamp['segment']}")

Advanced configuration:

For Transducer decoding strategies (greedy, beam, TSD, ALSD, mAES) and their parameters, see the Transducer Decoding section in Configs. For CTC and AED decoding classes, see the API reference. For decoding customization (confidence, CUDA graphs, language models, word boosting), see ASR Language Modeling and Customization.

from omegaconf import open_dict

decoding_cfg = model.cfg.decoding
with open_dict(decoding_cfg):
    decoding_cfg.preserve_alignments = True
    decoding_cfg.compute_timestamps = True
    decoding_cfg.segment_seperators = [".", "?", "!"]
    decoding_cfg.word_seperator = " "
    model.change_decoding_strategy(decoding_cfg)

hypotheses = model.transcribe(["audio.wav"], return_hypotheses=True)
timestamp_dict = hypotheses[0].timestamp

time_stride = 8 * model.cfg.preprocessor.window_stride
for stamp in timestamp_dict['word']:
    start = stamp['start_offset'] * time_stride
    end = stamp['end_offset'] * time_stride
    word = stamp['char'] if 'char' in stamp else stamp['word']
    print(f"{start:0.2f} - {end:0.2f} : {word}")

Long Audio Inference

For audio longer than what fits in memory (especially with Conformer’s quadratic attention):

Buffered / chunked inference:

Divide audio into overlapping chunks and merge outputs. Scripts are in examples/asr/asr_chunked_inference.

Local attention (recommended for Fast Conformer):

Switch to Longformer-style local+global attention for linear-cost inference on audio >1 hour:

model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-ctc-1.1b")
model.change_attention_model(
    self_attention_model="rel_pos_local_attn",
    att_context_size=[128, 128]
)

Or via CLI:

python examples/asr/speech_to_text_eval.py \
    (...other parameters...) \
    ++model_change.conformer.self_attention_model="rel_pos_local_attn" \
    ++model_change.conformer.att_context_size=[128, 128]

Subsampling memory optimization:

For very long files where even the subsampling module runs out of memory:

model.change_subsampling_conv_chunking_factor(1)  # auto-chunk subsampling

Multi-task Inference (Canary)

Canary models use prompt slots to control transcription behavior.

Via manifest:

from nemo.collections.asr.models import EncDecMultiTaskModel

canary = EncDecMultiTaskModel.from_pretrained("nvidia/canary-1b-v2")
decode_cfg = canary.cfg.decoding
decode_cfg.beam.beam_size = 1
canary.change_decoding_strategy(decode_cfg)

results = canary.transcribe("manifest.json", batch_size=16)

For the manifest format required by Canary models, see Canary Manifest Format.

Via direct parameters:

results = canary.transcribe(
    audio=["audio.wav"],
    batch_size=4,
    source_lang="en",
    target_lang="en",
    pnc=True,
)

Enforcing a Single Language

For multilingual Canary models, you can enforce a specific output language by explicitly setting source_lang and target_lang. When both are set to the same language, the model will transcribe in that language only:

results = canary.transcribe(
    audio=["audio.wav"],
    source_lang="de",
    target_lang="de",
)

This prevents phonetic drift where the model may switch languages mid-utterance.

Streaming Inference

NeMo provides a unified streaming-first Pipeline API for real-time ASR under nemo.collections.asr.inference. It supports buffered CTC/RNNT/TDT pipelines (overlapping chunks with any offline model) and cache-aware CTC/RNNT pipelines (processes each frame once using cached activations).

See the Streaming ASR Pipelines tutorial for a comprehensive walkthrough covering buffered and cache-aware pipelines, per-stream options, EoU detection, word timestamps, per-stream biasing, ITN, and speech translation.

See Cache-aware Streaming Conformer for model architecture details.

Prompt-conditioned Streaming (Multilingual)

Prompt-conditioned cache-aware streaming models (EncDecRNNTBPEModelWithPrompt and EncDecHybridRNNTCTCBPEModelWithPrompt) select a language prompt at inference time via target_lang. The cache-aware simulation script accepts the flag directly:

python examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
    model_path=<path_to_nemo_checkpoint> \
    dataset_manifest=<path_to_manifest> \
    target_lang=en-US \
    decoder_type=rnnt \
    strip_lang_tags=true

target_lang must be a key of the model’s prompt_dictionary. A bare language code such as en is also accepted when it matches exactly one locale key (for example en-US). An unsupported language raises an error. target_lang=auto (the default when target_lang is unset) selects the language-agnostic prompt; quality on languages the model was not trained on is not guaranteed. This script applies one prompt to every sample. For per-utterance languages, use model.transcribe(), which reads each manifest entry’s lang field when target_lang is unset and uses auto for entries without one. Setting strip_lang_tags=true removes <xx-XX> language tags from the decoded text (pattern is customizable via lang_tag_pattern).

In Python, call set_inference_prompt once before decoding:

model = nemo_asr.models.EncDecRNNTBPEModelWithPrompt.restore_from("path/to/model.nemo")
model.set_inference_prompt("en-US")
# ... subsequent conformer_stream_step / streaming pipeline calls use this prompt ...

See RNN-T with Prompt Conditioning Configuration for the full model config reference.

Apple MPS Support

Inference on Apple M-Series GPUs is supported with PyTorch 2.0+:

PYTORCH_ENABLE_MPS_FALLBACK=1 python examples/asr/speech_to_text_eval.py \
    (...other parameters...) \
    allow_mps=true

Execution Flow

When writing custom inference scripts, follow the execution flow diagram at the ASR examples README.