Is this page helpful?

Word Boosting#

GPU-PB#

GPU-PB is a GPU-accelerated Phrase-Boosting method supported for CTC, RNN-T/TDT, and AED (Canary) models based on NGPU-LM infrastructure. The method supports greedy and beam search decoding, including CUDA graphs mode. GPU-PB is compatible with NGPU-LM at the same decoding run.

GPU-PB is applied only at the decoding step in shallow fusion mode. You do not need to retrain the ASR model. During greedy or beam search decoding, GPU-PB rescales ASR model scores with a boosting tree at the token level. The boosting tree is built from a context phrases list, which is provided by the user.

NOTE: for ASR models that support capitalization by default (e.g., Canary or parakeet-tdt-0.6b-v2), you need to capitalize all the key phrases in advance (and capitalize the full word for abbreviations). You can use LLM for this task.

More details about GPU-PB method can be found in the original paper.

Usage#

We support three ways to pass the context phrases into the decoding script:

  1. Build a boosting tree for a specific ASR model (step 0.0) and use it for all the decoding evaluation by boosting_tree.model_path (step 1.1-3.1).

  2. Provide a file with context phrases boosting_tree.key_phrases_file - one phrase per line (step 1.1-3.1).

  3. Provide a python list of context phrases boosting_tree.key_phrases_list (step 1.1-3.1).

The use of the Phrase-Boosting tree is controlled by boosting_tree config (BoostingTreeModelConfig) for all the models. For prepared boosting tree use boosting_tree.model_path=${PATH_TO_BTREE}. We recommend to provide the list of context phrases directly into speech_to_text_eval.py by boosting_tree.key_phrases_file=${KEY_WORDS_LIST}.

List of the most important parameters:

  • strategy - The strategy to use for decoding depending on the model type (CTC - greedy_batch or beam_batch; RNN-T/TDT - greedy_batch or malsd_batch; AED - beam).

  • model_path, key_phrases_file, key_phrases_list - The way to pass the context phrases into the decoding script.

  • context_score - The score for each arc transition in the context graph (1.0 is recommended).

  • depth_scaling - The scaling factor for the depth of the context graph (2.0 is recommended for CTC, RNN-T and TDT, 1.0 for Canary).

  • boosting_tree_alpha - Weight of the GPU-PB boosting tree during shallow fusion decoding (tune it according to your data).

0.0. [Optional] Build the boosting tree for a specific ASR model:

python scripts/asr_context_biasing/build_gpu_boosting_tree.py \
        asr_model_path=${ASR_NEMO_MODEL_FILE} \
        key_phrases_file=${CONTEXT_BIASING_LIST} \
        save_to=${PATH_TO_SAVE_BTREE} \
        context_score=${CONTEXT_SCORE} \
        depth_scaling=${DEPTH_SCALING} \
        use_triton=True

1.1. CTC greedy batch decoding:

python examples/asr/speech_to_text_eval.py \
    model_path=${MODEL_NAME} \
    dataset_manifest=${EVAL_MANIFEST} \
    batch_size=${BATCH_SIZE} \
    output_filename=${OUT_MANIFEST} \
    ctc_decoding.strategy="greedy_batch" \
    ctc_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
    ctc_decoding.greedy.boosting_tree.context_score=1.0 \
    ctc_decoding.greedy.boosting_tree.depth_scaling=2.0 \
    ctc_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}

1.2. CTC beam batch decoding:

python examples/asr/speech_to_text_eval.py \
    model_path=${MODEL_NAME} \
    dataset_manifest=${EVAL_MANIFEST} \
    batch_size=${BATCH_SIZE} \
    output_filename=${OUT_MANIFEST} \
    ctc_decoding.strategy="beam_batch" \
    ctc_decoding.beam.beam_size=${BEAM_SIZE} \
    ctc_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
    ctc_decoding.beam.boosting_tree.context_score=1.0 \
    ctc_decoding.beam.boosting_tree.depth_scaling=2.0 \
    ctc_decoding.beam.boosting_tree_alpha=${BT_ALPHA}

2.1. RNN-T/TDT greedy batch decoding:

python examples/asr/speech_to_text_eval.py \
    model_path=${MODEL_NAME} \
    dataset_manifest=${EVAL_MANIFEST} \
    batch_size=${BATCH_SIZE} \
    output_filename=${OUT_MANIFEST} \
    rnnt_decoding.strategy="greedy_batch" \
    rnnt_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
    rnnt_decoding.greedy.boosting_tree.context_score=1.0 \
    rnnt_decoding.greedy.boosting_tree.depth_scaling=2.0 \
    rnnt_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}

2.2. RNN-T/TDT beam (malsd_batch) decoding:

python examples/asr/speech_to_text_eval.py \
    model_path=${MODEL_NAME} \
    dataset_manifest=${EVAL_MANIFEST} \
    batch_size=${BATCH_SIZE} \
    output_filename=${OUT_MANIFEST} \
    rnnt_decoding.strategy="malsd_batch" \
    rnnt_decoding.beam.beam_size=${BEAM_SIZE} \
    rnnt_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
    rnnt_decoding.beam.boosting_tree.context_score=1.0 \
    rnnt_decoding.beam.boosting_tree.depth_scaling=2.0 \
    rnnt_decoding.beam.boosting_tree_alpha=${BT_ALPHA}

3.1. AED (Canary) greedy (beam_size=1) or beam (beam_size>1) decoding:

python examples/asr/speech_to_text_eval.py \
    model_path=${MODEL_NAME} \
    dataset_manifest=${EVAL_MANIFEST} \
    batch_size=${BATCH_SIZE} \
    output_filename=${OUT_MANIFEST} \
    multitask_decoding.strategy="beam" \
    multitask_decoding.beam.beam_size=${BEAM_SIZE} \
    multitask_decoding.beam.boosting_tree.key_phrases_file=${CONTEXT_BIASING_LIST} \
    multitask_decoding.beam.boosting_tree.context_score=1.0 \
    multitask_decoding.beam.boosting_tree.depth_scaling=1.0 \
    multitask_decoding.beam.boosting_tree_alpha=${BT_ALPHA} \
    gt_lang_attr_name="target_lang" \
    gt_text_attr_name="text"

Results evaluation#

You can compute the F-score for the list of context phrases directly from the decoding manifest.

python scripts/asr_context_biasing/compute_key_words_fscore.py \
        --input_manifest=${DECODING_MANIFEST} \
        --key_words_file=${CONTEXT_PHRASES_LIST}

Per-Stream Phrase Boosting#

Per-stream (per-utterance) phrase boosting extends GPU-PB to allow specifying different key phrases for each audio stream or utterance in a batch. This is useful when different utterances require different context biasing (e.g., different speaker names, product terms, or domain vocabulary per audio).

Per-stream boosting is currently supported for greedy label-looping decoding with Transducers (RNN-T, TDT), including cache-aware streaming models.

Manifest-based Usage#

Specify per-utterance key phrases in your manifest using the biasing_request field:

{"audio_filepath": "/data/file1.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["one phrase"]}}}
{"audio_filepath": "/data/file2.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["other phrases", "and this one"]}}}

Use the streaming inference script with use_per_stream_biasing=true:

python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
    --config-path="../conf/asr_streaming_inference/" \
    --config-name=cache_aware_rnnt.yaml \
    audio_file="<manifest_with_boosting_requests>" \
    output_filename="result.jsonl" \
    asr.model_name="nvidia/parakeet-rnnt-1.1b" \
    asr.decoding.greedy.enable_per_stream_biasing=True

Python API Usage#

from omegaconf import open_dict
from nemo.collections.asr.models import EncDecRNNTBPEModel
from nemo.collections.asr.parts.context_biasing.biasing_multi_model import BiasingRequestItemConfig
from nemo.collections.asr.parts.context_biasing.boosting_graph_batched import BoostingTreeModelConfig
from nemo.collections.asr.parts.utils.rnnt_utils import Hypothesis

asr_model = EncDecRNNTBPEModel.from_pretrained("nvidia/parakeet-rnnt-1.1b")
asr_model.to("cuda")

with open_dict(asr_model.cfg.decoding):
    asr_model.cfg.decoding.strategy = "greedy_batch"
    asr_model.cfg.decoding.greedy.loop_labels = True
    asr_model.cfg.decoding.greedy.enable_per_stream_biasing = True
asr_model.change_decoding_strategy(asr_model.cfg.decoding)

biasing_requests = [
    BiasingRequestItemConfig(
        boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["one phrase"]),
        boosting_model_alpha=2.0,
    ),
    None,  # no biasing for this utterance
    BiasingRequestItemConfig(
        boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["other phrases"]),
        boosting_model_alpha=1.0,
    ),
]

results = asr_model.transcribe(
    audio=["file1.wav", "file2.wav", "file3.wav"],
    partial_hypothesis=[
        Hypothesis.empty_with_biasing_cfg(biasing_cfg=req) if req else None
        for req in biasing_requests
    ],
    return_hypotheses=True,
)

Caching#

Building a boosting model from a phrase list has some overhead. NeMo provides caching mechanisms to speed up repeated use of the same phrases:

Strategy

Description

Recommended For

Memory

Set cache_key on BiasingRequestItemConfig to cache compiled models in memory by a string key.

Repeated phrase sets

Disk

Set model_path on BoostingTreeModelConfig to save/load compiled models from disk.

Persistent caching

Decoder

Set auto_manage_multi_model=False and manually manage models in the decoder’s multi-model.

Advanced use cases

With memory caching, per-stream boosting achieves near-zero overhead compared to global (shared) boosting.

Flashlight-based Word Boosting#

The Flashlight decoder supports word boosting during CTC decoding using a KenLM binary and corresponding lexicon. Word boosting only works in lexicon-decoding mode and does not function in lexicon-free mode. It allows you to bias the decoder for certain words by manually increasing or decreasing the probability of emitting specific words. This can be very helpful if you have uncommon or industry-specific terms that you want to ensure are transcribed correctly.

For more information, go to word boosting

To use word boosting in NeMo, create a simple tab-separated text file. Each line should contain a word to be boosted, followed by a tab, and then the boosted score for that word.

For example:

nvidia      40
geforce     50
riva        80
turing      30
badword     -100

Positive scores boost words higher in the LM decoding step so they show up more frequently, whereas negative scores squelch words so they show up less frequently. The recommended range for the boost score is +/- 20 to 100.

The boost file handles both in-vocabulary words and OOV words just fine, so you can specify both IV and OOV words with corresponding scores.

You can then pass this file to your Flashlight config object during decoding:

# Lexicon-based decoding
python eval_beamsearch_ngram_ctc.py ... \
       decoding_strategy="flashlight" \
       decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
       decoding.beam.flashlight_cfg.boost_path='/path/to/my_boost_file.boost' \
       decoding.beam.flashlight_cfg.beam_size_token = 32 \
       decoding.beam.flashlight_cfg.beam_threshold = 25.0

CTC-WS: Context-biasing (Word Boosting) without External LM#

NeMo toolkit supports a fast context-biasing method for CTC and Transducer (RNN-T) ASR models with CTC-based Word Spotter. The method involves decoding CTC log probabilities with a context graph built for words and phrases from the context-biasing list. The spotted context-biasing candidates (with their scores and time intervals) are compared by scores with words from the greedy CTC decoding results to improve recognition accuracy and prevent false accepts of context-biasing.

A Hybrid Transducer-CTC model (a shared encoder trained together with CTC and Transducer output heads) enables the use of the CTC-WS method for the Transducer model. Context-biasing candidates obtained by CTC-WS are also filtered by the scores with greedy CTC predictions and then merged with greedy Transducer results.

Scheme of the CTC-WS method:

CTC-WS scheme

High-level overview of the context-biasing words replacement with CTC-WS method:

CTC-WS high level overview

More details about CTC-WS context-biasing can be found in the tutorial.

To use CTC-WS context-biasing, you need to create a context-biasing text file that contains words/phrases to be boosted, with its transcriptions (spellings) separated by underscore. Multiple transcriptions can be useful for abbreviations (“gpu” -> “g p u”), compound words (“nvlink” -> “nv link”), or words with common mistakes in the case of our ASR model (“nvidia” -> “n video”).

Example of the context-biasing file:

nvidia_nvidia
omniverse_omniverse
gpu_gpu_g p u
dgx_dgx_d g x_d gx
nvlink_nvlink_nv link
ray tracing_ray tracing

The main script for CTC-WS context-biasing in NeMo is:

{NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py

Context-biasing is managed by apply_context_biasing parameter [true or false]. Other important context-biasing parameters are:

  • beam_threshold - threshold for CTC-WS beam pruning.

  • context_score - per token weight for context biasing.

  • ctc_ali_token_weight - per token weight for CTC alignment (prevents false acceptances of context-biasing words).

All the context-biasing parameters are selected according to the default values in the script. You can tune them according to your data and ASR model (list all the values in the [] separated by commas) for example: beam_threshold=[7.0,8.0,9.0], context_score=[3.0,4.0,5.0], ctc_ali_token_weight=[0.5,0.6,0.7]. The script will run the recognition with all the combinations of the parameters and will select the best one based on WER value.

# Context-biasing with the CTC-WS method for CTC ASR model
python {NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py \
        nemo_model_file={ctc_model_name} \
        input_manifest={test_nemo_manifest} \
        preds_output_folder={exp_dir} \
        decoder_type="ctc" \
        acoustic_batch_size=64 \
        apply_context_biasing=true \
        context_file={cb_list_file_modified} \
        beam_threshold=[7.0] \
        context_score=[3.0] \
        ctc_ali_token_weight=[0.5]

To use Transducer head of the Hybrid Transducer-CTC model, you need to set decoder_type=rnnt.