ASR Language Modeling and Customization#

NeMo supports decoding-time customization techniques such as language modeling and word boosting, which improve transcription accuracy by incorporating external knowledge or domain-specific vocabulary—without retraining the model.

Decoder Types#

NeMo ASR models use different decoder architectures. The table below summarizes them:

Decoder	Type	Description	Models
CTC	Non-autoregressive	Connectionist Temporal Classification. Fast inference, supports LM fusion and word boosting.	Parakeet-CTC, FastConformer-CTC
RNN-T	Autoregressive	Recurrent Neural Network Transducer. Strong accuracy, streaming-friendly.	Parakeet-RNNT, FastConformer-Transducer
TDT	Autoregressive	Token-and-Duration Transducer. Extends RNN-T with duration prediction for better timestamps.	Parakeet-TDT
AED	Autoregressive	Attention Encoder-Decoder. Multi-task capable (ASR + AST), prompt-based language control.	Canary-1B, Canary-1B-V2, Canary-1B-Flash
Hybrid	Both	Joint RNN-T + CTC training. Use either decoder at inference time.	FastConformer Hybrid models

Language Modeling#

In NeMo two approaches of external language modeling are supported:

Language Model Fusion:
Language model (LM) fusion integrates scores from an external statistical n-gram model into the ASR decoder. This helps guide decoding toward more likely word sequences based on text corpora.

NeMo provides two approaches for language model shallow fusion with ASR systems:
1. NGPU-LM (Recommended for Production)
GPU-accelerated LM fusion for all major model types: CTC, RNN-T, TDT, and AED models.

Customization during both greedy and beam decoding.

Fast beam decoding for all major model types, offering only 20% RTFx difference between beam and greedy decoding.

Integration with NGPU-LM GPU-based ngram LM.

For details, please refer to NGPU-LM (GPU-based N-gram Language Model) Language Model Fusion
2. KenLM (Traditional CPU-based)
CPU-based LM fusion using the KenLM library.

Note

These approaches, especially beam decoding, can be extremely slow and are retained in the repository primarily for backward compatibility. If possible, we recommend using NGPU-LM for improved performance.

For details, please refer to N-gram Language Model Fusion
Neural Rescoring:
When using the neural rescoring approach, a neural network is used to score candidates. A candidate is the text transcript predicted by the ASR model’s decoder. The top K candidates produced by beam search decoding (with a beam width of K) are given to a neural language model for ranking. The language model assigns a score to each candidate, which is usually combined with the scores from beam search decoding to produce the final scores and rankings.

For details, please refer to Neural Rescoring.

Word Boosting#

Word boosting increases the likelihood of specific words or phrases during decoding by applying a positive bias, helping the model better recognize names, uncommon terms, and custom vocabulary.

GPU-PB (preferred): GPU-accelerated phrase-boosting for CTC, RNN-T/TDT, and AED (Canary) models supporting greedy and beam search decoding.
Flashlight-based Word Boosting: Word-boosting method for CTC models with external n-gram LM.
CTC-WS: Context-biasing (Word Boosting) without External LM: Word-boosting method for hybrid (Transducer-CTC) models without LM.

For details, please refer to: Word Boosting.

LM Training#

NeMo provides tools for training n-gram language models that can be used for language model fusion or word-boosting. For details, please refer to: Scripts for building and merging N-gram Language Models.

CUDA Graphs#

CUDA graphs accelerate decoding by capturing and replaying GPU operations, eliminating kernel launch overhead. Support varies by decoder strategy:

Strategy	Config Parameter	Default	Notes
`greedy_batch` (RNN-T, TDT)	`use_cuda_graph_decoder`	`true`	Requires `loop_labels=True` and `blank_as_pad=True`
`maes_batch`, `malsd_batch` (beam)	`allow_cuda_graphs`	`true`	Batched beam search strategies
Non-batched `greedy` / `beam`	N/A	N/A	Not supported; standard decoding used

To disable CUDA graphs (e.g. for debugging or when preserving alignments with frame-looping):

Via Python (at runtime):

model.disable_cuda_graphs()

Greedy decoding — use use_cuda_graph_decoder=true/false:

python examples/asr/speech_to_text_eval.py \
   pretrained_name="nvidia/parakeet-rnnt-1.1b" \
   dataset_manifest=<dataset_manifest> \
   batch_size=32 \
   output_filename=decoded.jsonl \
   rnnt_decoding.strategy="greedy_batch" \
   rnnt_decoding.greedy.use_cuda_graph_decoder=true

Beam decoding — use allow_cuda_graphs=true/false:

python examples/asr/speech_to_text_eval.py \
   pretrained_name="nvidia/parakeet-rnnt-1.1b" \
   dataset_manifest=<dataset_manifest> \
   batch_size=32 \
   output_filename=decoded.jsonl \
   rnnt_decoding.strategy="malsd_batch" \
   rnnt_decoding.beam.max_symbols_per_step=10 \
   rnnt_decoding.beam.beam_size=12 \
   rnnt_decoding.beam.allow_cuda_graphs=true

When unsupported, NeMo falls back to standard decoding automatically.

Confidence Estimation#

NeMo supports per-frame, per-token, and per-word confidence scores during decoding. Confidence estimation helps applications decide when to trust ASR output and when to request human review.

decoding:
  confidence_cfg:
    preserve_frame_confidence: false
    preserve_token_confidence: false
    preserve_word_confidence: false
    exclude_blank: true
    aggregation: "mean"       # mean, min, max, prod
    method_cfg:
      name: "entropy"         # max_prob or entropy
      entropy_type: "tsallis" # gibbs, tsallis, renyi
      alpha: 0.33
      entropy_norm: "exp"     # lin or exp

Confidence methods:

max_prob: Maximum token probability as confidence. Simple and fast.
entropy: Normalized entropy of the log-likelihood vector (default). Entropy types:
- gibbs: Standard Gibbs entropy
- tsallis: Tsallis entropy (default, recommended)
- renyi: Renyi entropy

Aggregation combines frame-level scores into token/word scores: mean, min, max, or prod.

For TDT models, set tdt_include_duration_confidence: true to include duration prediction confidence.