Building and Deploying ASR Pipelines

Understanding riva-build#

The riva-build command is the first step in deploying models to Riva. It takes trained models and various configuration parameters to create a Riva Model Intermediate Representation (RMIR) file that can be deployed to Triton Inference Server.

Input Requirements#

riva-build accepts the following inputs:

1. Model Files (Required)

.riva files: Pre-trained or custom models in Riva format, available from NGC Catalog
.nemo files: Models trained with NVIDIA NeMo, which are automatically converted to Riva format using the integrated nemo2riva utility during the build process

2. Additional Model Artifacts (Optional, depending on pipeline configuration)

Language Models (LM): N-gram or neural language models for improved accuracy with CTC-based acoustic models
Inverse Text Normalization (ITN): WFST-based models that convert spoken-form text to written form (e.g., “twenty three” → “23”)

3. Configuration Parameters

Pipeline-specific parameters: Decoder type, chunk size, batch size, feature extraction settings, etc.
Hydra configuration: Configuration files that define default settings for streaming/offline modes
Model names and paths: User-defined names for components and paths to output files

Output#

riva-build generates a Riva Model Intermediate Representation (RMIR) file that contains:

Serialized model artifacts
Pipeline configuration parameters
Metadata for deployment

This RMIR file is:

Platform-portable: Can be transferred between systems with the same architecture
Hardware-independent: Not yet optimized for specific GPU hardware
Ready for deployment: Can be passed to riva-deploy for GPU-specific optimization

Workflow Overview#

                ┌────────────────────────────────────┐
                │             Input Files            │
                ├────────────────────────────────────┤
                │  • Acoustic Model (.riva or .nemo) │
                │  • Language Model (optional)       │
                │  • ITN/PnC Models (optional)       │
                │  • Configuration Parameters        │
                └────────────────┬───────────────────┘
                                 │
                                 ▼
                         ┌───────────────┐
                         │  riva-build   │  ← Converts .nemo to .riva (if needed)
                         │               │  ← Packages all artifacts
                         └───────┬───────┘  ← Applies configuration
                                 │
                                 ▼
                      ┌─────────────────────┐
                      │   RMIR File Output  │  ← Architecture-portable
                      │       (.rmir)       │  ← Hardware-independent
                      └──────────┬──────────┘  ← Contains all pipeline info
                                 │
                                 ▼
                     [ Ready for riva-deploy ]

Note: Each ASR pipeline (e.g., streaming en-US with CTC) requires one RMIR file. Punctuation and Capitalization models require separate RMIR files built with different configuration settings.

Building ASR Pipelines: Practical Examples#

Now that you understand what riva-build does, let’s look at practical examples of building ASR pipelines.

Hydra Configuration System#

The riva-build command uses Hydra for configuration management. You can specify configurations using:

--config-path: Path to the configuration directory (can be pkg://servicemaker.configs.asr or an absolute path like /path/to/riva-speech/python/servicemaker/servicemaker/configs/asr)
--config-name: Name of the configuration file (e.g., streaming or offline)

Note: By default, riva-build uses the streaming ASR configuration (--config-path=pkg://servicemaker.configs.asr --config-name=streaming), so you can omit these parameters for streaming configurations. For offline configurations, you can use just --config-name=offline or the full specification.

Basic Streaming ASR Pipeline#

In the simplest use case, you can build an ASR pipeline for the StreamingRecognize API call (refer to protobuf-docs-asr) without any language model as follows:

riva-build \
    --config-path=pkg://servicemaker.configs.asr \
    --config-name=streaming \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name> \
    wfst_tokenizer_model=<wfst_tokenizer_model> \
    wfst_verbalizer_model=<wfst_verbalizer_model> \
    decoder=greedy

Since the streaming configuration is the default, you can also omit the --config-path and --config-name parameters:

riva-build \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name> \
    wfst_tokenizer_model=<wfst_tokenizer_model> \
    wfst_verbalizer_model=<wfst_verbalizer_model> \
    decoder=greedy

where:

output_path specifies the path to the Riva rmir file that will be generated
source_path specifies a list of .riva or .nemo files to use as input (see below for .nemo file usage)
<rmir_filename> is the name of the RMIR file that will be generated
<riva_filename> is the name of the riva file to use as input
<encryption_key> is the key used to encrypt the files. The encryption key for the pre-trained Riva models uploaded on NGC is tlt_encode.
<name>,<acoustic_model_name> and <featurizer_name> are optional user-defined names for the components in the model repository.
<wfst_tokenizer_model> is the name of the WFST tokenizer model file to use for inverse text normalization of ASR transcripts. Refer to inverse-text-normalization for more details.
<wfst_verbalizer_model> is the name of the WFST verbalizer model file to use for inverse text normalization of ASR transcripts. Refer to inverse-text-normalization for more details.
decoder_type is the type of decoder to use. Valid values are flashlight, os2s, greedy and pass_through. We recommend using flashlight for all CTC models. Refer to Decoder Hyper-Parameters for more details.

Upon successful completion of this command, a file named <rmir_filename> is created in the /servicemaker-dev/ folder. Since no language model is specified, the Riva greedy decoder is used to predict the transcript based on the output of the acoustic model. If your .riva archives are encrypted you need to include :<encryption_key> at the end of the RMIR filename and Riva filename. Otherwise, this is unnecessary.

Note

Advanced Configuration Customization: You can directly modify configuration files in riva-speech/python/servicemaker/servicemaker/configs/asr/ and pass the directory path using --config-path=/absolute/path/to/configs/asr. This allows you to customize default parameters in the config files and eliminate the need for command-line arguments. Simply modify the YAML files and reference them with --config-path and --config-name.

Using NeMo Models Directly#

Riva supports using .nemo files directly with riva-build through integrated nemo2riva conversion. The conversion happens automatically during the build process, eliminating the need for a separate conversion step.

Key Benefits:

No manual conversion required
Conversion parameters specified inline with the build command
Support for multiple export formats (ONNX, TorchScript, etc.)
Can mix .nemo and .riva files in the same pipeline

To use a .nemo file, specify it in the source_path with conversion parameters:

riva-build \
    --config-path=pkg://servicemaker.configs.asr \
    --config-name=streaming \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    'source_path=[{path: /servicemaker-dev/<nemo_filename>, nemo2riva: {format: onnx, onnx_opset: 19, key: tlt_encode}}]' \
    name=<pipeline_name> \
    decoder=greedy

where:

path specifies the path to the .nemo file
nemo2riva is a structured config containing conversion parameters:
- format: Export format (onnx, ts, ckpt, nemo, state, or pytorch)
- onnx_opset: ONNX opset version (typically 19)
- key: Encryption key for the converted model
- Additional nemo2riva parameters can be specified (e.g., max_batch, max_dim, etc.)

You can mix .nemo and .riva files in the same command:

riva-build \
    --config-path=pkg://servicemaker.configs.asr \
    --config-name=streaming \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    'source_path=[{path: model1.nemo, nemo2riva: {format: onnx}}, model2.riva:key]' \
    name=<pipeline_name>

For more details on nemo2riva parameters, refer to the Notes in the Summary of riva-build Commands section below.

Note: Extract CTC/RNNT head from Hybrid ASR using this script before running riva-build.

For embedded platforms, using a batch size of 1 is recommended since it achieves the lowest memory footprint. To use a batch size of 1, refer to the riva-build-optional-parameters section and set the various min_batch_size, max_batch_size, opt_batch_size, and max_execution_batch_size parameters to 1 while executing the riva-build command.

Model Artifacts#

To build complete ASR pipelines, you’ll need various model artifacts depending on your requirements. The following table shows where to find pre-trained models for different components:

Model Components:

Acoustic Model: The core ASR model (required) - converts audio to text
Language Model (LM): Optional enhancement for CTC models - improves accuracy using linguistic context
Inverse Text Normalization (ITN): Optional - converts spoken form to written form (e.g., “twenty three” → “23”)
Punctuation and Capitalization (PnC): Optional - adds punctuation and capitalization to raw transcripts

Available Pre-trained Models:

Acoustic Model	Language Model (LM)	Inverse Text Normalization (ITN)	Punctuation and Capitalization (PnC)
Parakeet CTC	4gram LM	en-US ITN	en-US PnC
Parakeet RNNT	N/A	en-US ITN	en-US PnC
Parakeet TDT	N/A	Not needed	Not needed
Canary	N/A	Not needed	Not needed

Note: The Parakeet TDT and Canary models can directly produce punctuated and capitalized text, and apply inverse text normalization (ITN) natively. Therefore, when using these models, you do not need to supply separate Punctuation & Capitalization (PnC) or ITN models in the pipeline.

Downloading Models from Hugging Face#

To download these models from Hugging Face, you can use the Hugging Face CLI:

Install the Hugging Face CLI:

pip install huggingface_hub

Download a model:

# Download models using HF-cli
huggingface-cli download <HF model key> --local-dir <path to download models>

Building Punctuation and Capitalization (PnC) Models#

Important: PnC models are built separately from ASR pipelines and require their own RMIR files.

Why separate?

PnC uses different model architecture (typically BERT-based NLP models)
Uses different Hydra configuration (pkg://servicemaker.configs.punctuation)
One PnC model can be shared across multiple ASR pipelines with the same language
Enables/disables dynamically via enable_automatic_punctuation flag in ASR requests

Building a PnC RMIR:

riva-build \
    --config-path=pkg://servicemaker.configs.punctuation \
    --config-name=base \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    language_code=en-US \
    name=riva-punctuation-en-US

where:

language_code specifies the BCP-47 language code (e.g., en-US). When ASR requests have enable_automatic_punctuation set to true, Riva will use the PnC model matching the requested language code.
name is an optional user-defined name for the pipeline (recommended: riva-punctuation-<language_code>)

Deployment Note: When deploying, pass all RMIR files to riva-deploy:

# Deploy ASR pipeline with PnC support
riva-deploy /servicemaker-dev/asr-model.rmir:tlt_encode /data/models/
riva-deploy /servicemaker-dev/pnc-en-us.rmir:tlt_encode /data/models/

The PnC model will be automatically used when ASR requests include enable_automatic_punctuation=true.

Summary of riva-build Commands#

The following summary lists the riva-build commands used to generate the RMIR files from the Quick Start scripts for different models, modes, and their limitations:

For details about the parameters passed to riva-build to customize the ASR pipeline, run:

riva-build --help

Streaming/Offline Recognition#

The Riva ASR pipeline can be configured for both streaming and offline recognition use cases. When using the StreamingRecognize API call (refer to protobuf-docs-asr), we recommend the following riva-build parameters for low-latency streaming recognition with the Conformer acoustic model:

riva-build \
    --config-path=pkg://servicemaker.configs.asr \
    --config-name=streaming \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name> \
    wfst_tokenizer_model=<wfst_tokenizer_model> \
    wfst_verbalizer_model=<wfst_verbalizer_model> \
    decoder=greedy \
    chunk_size=0.16 \
    padding_size=1.92 \
    ms_per_timestep=40 \
    nn.fp16_needs_obey_precision_pass=True \
    greedy_decoder.asr_model_delay=-1 \
    endpointing.residue_blanks_at_start=-2 \
    featurizer.use_utterance_norm_params=False \
    featurizer.precalc_norm_time_steps=0 \
    featurizer.precalc_norm_params=False

Or simply (since streaming is the default):

riva-build \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name> \
    wfst_tokenizer_model=<wfst_tokenizer_model> \
    wfst_verbalizer_model=<wfst_verbalizer_model> \
    decoder=greedy \
    chunk_size=0.16 \
    padding_size=1.92 \
    ms_per_timestep=40 \
    nn.fp16_needs_obey_precision_pass=True \
    greedy_decoder.asr_model_delay=-1 \
    endpointing.residue_blanks_at_start=-2 \
    featurizer.use_utterance_norm_params=False \
    featurizer.precalc_norm_time_steps=0 \
    featurizer.precalc_norm_params=False

For high throughput streaming recognition with the StreamingRecognize API call, chunk_size and padding_size can be set as follows:

    chunk_size=0.8 \
    padding_size=1.6

Finally, to configure the ASR pipeline for offline recognition with the Recognize API call (refer to protobuf-docs-asr), you can use the offline configuration with the Conformer acoustic model. You can specify this using just --config-name=offline:

riva-build --config-name=offline \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name>

Or use the full specification:

riva-build \
    --config-path=pkg://servicemaker.configs.asr \
    --config-name=offline \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name>

Note

When deploying the offline ASR models with riva-deploy, TensorRT warnings indicating that memory requirements of format conversion cannot be satisfied might appear in the logs. These warnings should not affect functionality and can be ignored.

Language Models#

Riva ASR supports decoding with an n-gram language model. The n-gram language model can be provided in a few different ways.

A .arpa format file.
A KenLM binary format file.

For more information on building language models, refer to the training-language-models section.

ARPA Format Language Model#

To configure the Riva ASR pipeline to use an n-gram language model stored in arpa format, replace:

    decoder=greedy

with

    decoder=flashlight \
    decoding_language_model_arpa=<arpa_filename> \
    decoding_vocab=<decoder_vocab_file>

KenLM Binary Language Model#

To generate the Riva RMIR file when using a KenLM binary file to specify the language model, replace:

    decoder=greedy

with

    decoder=flashlight \
    decoding_language_model_binary=<KENLM_binary_filename> \
    decoding_vocab=<decoder_vocab_file>

Decoder Hyper-Parameters#

The decoder language model hyper-parameters can also be specified from the riva-build command.

You can specify the Flashlight decoder hyper-parameters beam_size, beam_size_token, beam_threshold, lm_weight and word_insertion_score by specifying

    decoder=flashlight \
    decoding_language_model_binary=<arpa_filename> \
    decoding_vocab=<decoder_vocab_file> \
    flashlight_decoder.beam_size=<beam_size> \
    flashlight_decoder.beam_size_token=<beam_size_token> \
    flashlight_decoder.beam_threshold=<beam_threshold> \
    flashlight_decoder.lm_weight=<lm_weight> \
    flashlight_decoder.word_insertion_score=<word_insertion_score>

Where:

beam_size is the maximum number of hypothesis the decoder holds at each step
beam_size_token is the maximum number of tokens the decoder considers at each step
beam_threshold is the threshold to prune hypothesis
lm_weight is the weight of the language model used when scoring hypothesis
word_insertion_score is the word insertion score used when scoring hypothesis

For advanced users, additional decoder hyper-parameters can also be specified. Refer to Riva-build Optional Parameters for a list of those parameters and their description.

Flashlight Decoder Lexicon#

The Flashlight decoder used in Riva is a lexicon-based decoder and only emits words that are present in the decoder vocabulary file passed to the riva-build command. The decoder vocabulary file used to generate the ASR pipelines in the Quick Start scripts include words that cover a wide range of domains and should provide accurate transcripts for most applications.

It is also possible to build an ASR pipeline using your own decoder vocabulary file by using the parameter decoding_vocab of the riva-build command. For example, you could start with the riva-build commands used to generate the ASR pipelines in our Quick Start scripts from section Building and Deploying ASR Pipelines and provide your own lexicon decoder vocabulary file. You will need to ensure that words of interest are in the decoder vocabulary file. The Riva ServiceMaker automatically tokenizes the words in the decoder vocabulary file. The number of tokenization for each word in the decoder vocabulary file can be controlled with the --flashlight_decoder.num_tokenization parameter.

(Advanced) Manually Adding Additional Tokenizations of Words in Lexicon#

It is also possible to manually add additional tokenizations for the words in the decoder vocabulary by performing the following steps:

The riva-build and riva-deploy commands provided in the previous section store the lexicon in the /data/models/citrinet-1024-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/lexicon.txt file of the Triton model repository.

To add additional tokenizations to the lexicon, copy the lexicon file:

cp /data/models/citrinet-1024-en-US-asr-streaming-ctc-decoder-cpu-streaming/1/lexicon.txt decoding_lexicon.txt

and add the SentencePiece tokenization for the word of interest. For example, you could add:

manu ▁ma n u
manu ▁man n n ew
manu ▁man n ew

to the decoding_lexicon.txt file so that the word manu is generated in the transcript if the acoustic model predicts those tokens. You will need to ensure that the new lines follow the indentation/space pattern like the rest of the file and that the tokens used are part of the tokenizer model. After this is done, regenerate the model repository using the new decoding lexicon by passing decoding_lexicon=decoding_lexicon.txt to riva-build instead of decoding_vocab=decoding_vocab.txt.

Flashlight Decoder Lexicon Free#

The Flashlight decoder can also be used without a lexicon. Lexicon free decoding is performed with a character based language model. Lexicon free decoding with flashlight can be enabled by adding flashlight_decoder.use_lexicon_free_decoding=True to riva-build and specifying a character based language model via decoding_language_model_binary=<path/to/charlm>.

OpenSeq2Seq Decoder#

Riva uses the OpenSeq2Seq decoder for beam-search decoding with a language model. For example:

riva-build \
   output_path=<rmir_filename>:<key> \
   source_path=[<riva_filename>:<key>] \
   name=citrinet-1024-zh-CN-asr-streaming \
   ms_per_timestep=80 \
   featurizer.use_utterance_norm_params=False \
   featurizer.precalc_norm_time_steps=0 \
   featurizer.precalc_norm_params=False \
   endpointing.residue_blanks_at_start=-2 \
   chunk_size=0.16 \
   left_padding_size=1.92 \
   right_padding_size=1.92 \
   decoder=os2s \
   os2s_decoder.language_model_alpha=0.5 \
   os2s_decoder.language_model_beta=1.0 \
   os2s_decoder.beam_search_width=128 \
   language_code=zh-CN

Where:

os2s_decoder.language_model_alpha is the weight given to the language model during the beam search.
os2s_decoder.language_model_beta is the word insertion score.
os2s_decoder.beam_search_width is the number of partial hypotheses to keep at each step of the beam search.

All of these parameters effect performance. Latency increases as these parameters increase in value. The suggested ranges are listed below.

Parameter	Minimum	Maximum
`os2s_decoder.beam_search_width`	16	64
`os2s_decoder.language_model_alpha`	0.5	1.5
`os2s_decoder.language_model_beta`	1.0	3.0

Beginning/End of Utterance Detection#

Riva ASR uses an algorithm that detects the beginning and end of utterances. This algorithm is used to reset the ASR decoder state, and to trigger a call to the punctuator model. By default, the beginning of an utterance is flagged when 20% of the frames in a 300ms window has nonblank characters. The end of an utterance is flagged when 98% of the frames in a 800ms window are blank characters. You can tune those values for their particular use case by using the following riva-build parameters:

  endpointing.start_history=300 \
  endpointing.start_th=0.2 \
  endpointing.stop_history=800 \
  endpointing.stop_th=0.98

Additionally, it is possible to disable the beginning/end of utterance detection by passing endpointing_type=none to riva-build.

Note that in this case, the decoder state resets after the full audio signal has been sent by the client. Similarly, the punctuator model is only called once.

Streaming Speaker Diarization#

Riva currently supports speaker diarization in streaming mode via the Sortformer Diarizer model. For more details on Sortformer speaker diarization, refer to the Streaming Speaker Diarization section in the ASR Overview.

Sortformer#

To enable Sortformer speaker diarization in the ASR pipeline, pass the following additional parameters to riva-build when building a streaming ASR model:

riva-build \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>,<sortformer_diarizer_riva_filename>:<encryption_key>] \
    sortformer=enabled \
    diarizer_type=sortformer

where:

<sortformer_diarizer_riva_filename> is the .riva Sortformer model to use. For example, you can use the Sortformer Diarizer Riva model available on NGC.
<encryption_key> is the key used to encrypt the file. The encryption key for the pre-trained Riva models uploaded on NGC is tlt_encode.

Note: Sortformer currently supports up to maximum of 4 speakers.

Neural-Based Voice Activity Detection#

It is possible to use a neural-based Voice Activity Detection (VAD) algorithm in Riva ASR. This can help to filter out noise in the audio, and can help reduce spurious words from appearing in the ASR transcripts. To use the neural-based VAD algorithm in the ASR pipeline, pass the following additional parameters to riva-build:

Silero VAD#

riva-build \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>,<silero_vad_riva_filename>:<encryption_key>] \
    vad=enabled \
    vad_type=silero \
    neural_vad_nn.optimization_graph_level=-1 \
    neural_vad.filter_speech_first=false \
    neural_vad.onset=0.85 \
    neural_vad.offset=0.3 \
    neural_vad.min_duration_on=0.2 \
    neural_vad.min_duration_off=0.5 \
    neural_vad.pad_offset=0.08 \
    neural_vad.pad_onset=0.3 \
    neural_vad.features_mask_value=-16.635

where:

<silero_vad_riva_filename> is the .riva silero VAD model to use. For example, you can use the Silero VAD Riva model available on NGC.
<encryption_key> is the key used to encrypt the file. The encryption key for the pre-trained Riva models uploaded on NGC is tlt_encode.
neural_vad.onset is the minimum probability threshold for detecting the start of a speech segment.
neural_vad.offset is the minimum probability threshold for detecting the end of a speech segment.
neural_vad.min_duration_on is the minimum duration of a speech segment to be considered as a speech segment.
neural_vad.min_duration_off is the minimum duration of a non-speech segment to be considered as a non-speech segment.
neural_vad.pad_onset is the duration of audio (in seconds) to pad the onset of a speech segment.
neural_vad.pad_offset is the duration of audio (in seconds) to pad the offset of a speech segment.
neural_vad.features_mask_value is the value to use to mask the features of a non-speech segment.

Several of these parameters can be configured at runtime using the custom_configuration parameter. The configurable parameters are:

onset
offset
min_duration_on
min_duration_off
pad_onset
pad_offset

Example of runtime configuration:

--custom_configuration="neural_vad.onset:0.9,neural_vad.offset:0.4,neural_vad.min_duration_on:0.3,neural_vad.min_duration_off:0.6"

MarbleNet VAD#

riva-build \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>,<marblenet_vad_riva_filename>:<encryption_key>] \
    vad=enabled \
    vad_type=neural \
    neural_vad_nn.optimization_graph_level=-1

where:

<marblenet_vad_riva_filename> is the .riva marblenet VAD model to use. For example, you can use the MarbleNet VAD Riva model available on NGC.
<encryption_key> is the key used to encrypt the file. The encryption key for the pre-trained Riva models uploaded on NGC is tlt_encode.

Note that using a neural VAD component in the ASR pipeline will have an impact on latency and throughput of the deployed Riva ASR server.

Generating Multiple Transcript Hypotheses#

By default, the Riva ASR pipeline is configured to only generate the best transcript hypothesis for each utterance. It is possible to generate multiple transcript hypotheses by passing the parameter max_supported_transcripts=N to the riva-build command, where N is the maximum number of hypotheses to generate. With these changes, the client application can retrieve the multiple hypotheses by setting the max_alternatives field of RecognitionConfig to values greater than 1.

Impact of Chunk Size and Padding Size on Performance and Accuracy (Advanced)#

The chunk_size and padding_size parameters used to configure Riva ASR can have a significant impact on accuracy and performance. A brief description of those parameters can be found in section Riva-build Optional Parameters. Riva provides pre-configured ASR pipelines, with preset values of chunk_size and padding_size: a low-latency streaming configuration, a high throughput streaming configuration, and an offline configuration. Those configurations should suit most deployment scenarios. The chunk_size and padding_size values used for those configurations can be found in a table in section Building and Deploying ASR Pipelines.

The chunk_size parameter is the duration of the audio chunk in seconds processed by the Riva server for every streaming request. Hence, in streaming mode, Riva returns one response for every chunk_size seconds of audio. A lower value of chunk_size will therefore reduce the user-perceived latency as the transcript will get updated more frequently.

The padding_size parameter is the duration in seconds of the padding prepended and appended to the chunk_size. The Riva acoustic model processes an input tensor corresponding to an audio duration of 2*(padding_size) + chunk_size for every new chunk of audio it receives. Increasing padding_size or chunk_size typically helps to improve accuracy of the transcripts since the acoustic model has access to more context. However, increasing padding_size reduces the maximum number of concurrent streams supported by Riva ASR, since it will increase the size of the input tensor fed to the acoustic model for every new chunk.

Sharing Acoustic and Feature Extractor Models Across Multiple ASR Pipelines (Advanced)#

It is possible to configure the Riva ASR service such that multiple ASR pipelines share the same feature extractor and acoustic models, thus allowing to reduce GPU memory usage. This option can be used, for example, to deploy multiple ASR pipelines where each pipeline uses a different language model, but share the same acoustic model and feature extractor. This can be achieved by specifying the parameters acoustic_model_name and featurizer_name in the riva-build command:

riva-build \
    output_path=/servicemaker-dev/<rmir_filename>:<encryption_key> \
    source_path=[/servicemaker-dev/<riva_filename>:<encryption_key>] \
    name=<pipeline_name> \
    acoustic_model_name=<acoustic_model_name> \
    featurizer_name=<featurizer_name> \
    wfst_tokenizer_model=<wfst_tokenizer_model> \
    wfst_verbalizer_model=<wfst_verbalizer_model> \
    decoder=greedy

where:

<acoustic_model_name> is the user-defined name for the acoustic model component of the ASR pipeline
<featurizer_name> is the user-defined name for the feature extractor component of the ASR pipeline

If multiple ASR pipelines are built, each with a different name, but with the same acoustic_model_name and featurizer_name, they will share the same acoustic and feature extractor models.

When running the riva-deploy command, you must pass the -f option to ensure that all the ASR pipelines that share the acoustic model and feature extractor are initialized properly.

Note

<acoustic_model_name> and <featurizer_name> are global and can conflict across model pipelines. Override this only in cases when you know what other models will be deployed and you want to share the featurizer and/or acoustic models across different ASR pipelines. When specifying <acoustic_model_name> you should make sure that there will not be any incompatibilities in acoustic model weights or input shapes. Similarly, when specifying <featurizer_name>, you should make sure that that all ASR pipelines with the same <featurizer_name> use the same feature extractor parameters.

Riva-build Optional Parameters#

For details about the parameters passed to riva-build to customize the ASR pipeline, issue:

riva-build --help

Riva Deploy#

The riva-deploy command takes the RMIR file generated by riva-build and deploys it to create a Triton-compatible model repository. During deployment, models are optimized for the specific GPU hardware on the target machine using TensorRT and other NVIDIA acceleration libraries.

Basic Usage#

The basic syntax for riva-deploy is:

riva-deploy [-f] <rmir_filename>:<key> <output_dir>

where:

<rmir_filename> is the path to the RMIR file generated by riva-build
<key> is the encryption key used when building the RMIR file (e.g., tlt_encode for NGC models)
<output_dir> is the directory where the Triton model repository will be created
-f (optional) forces redeployment even if the model repository already exists

Example Commands#

Deploy an ASR model to create a Triton model repository:

riva-deploy /servicemaker-dev/conformer-en-US-asr-streaming.rmir:tlt_encode /data/models/

GPU Architecture Considerations#

The model repository generated by riva-deploy is optimized for the specific GPU architecture of the deployment machine. This means:

Models optimized on an A100 GPU may not work on a V100 GPU
Models optimized on Ampere architecture may not work on Hopper architecture
Always run riva-deploy on the target machine where the Riva server will run

If you need to deploy to multiple machines with different GPU architectures, you must run riva-deploy separately on each machine using the same RMIR file.

TensorRT Optimization#

During deployment, riva-deploy uses TensorRT to optimize neural network models. This process:

Analyzes the model architecture
Selects optimal kernels for the target GPU
Fuses layers and operations where possible
Calibrates precision (FP32, FP16, FP8) based on the configuration

This optimization can take several minutes depending on model size and complexity.

Overriding Configuration During Deployment#

The riva-deploy command supports the --override_config parameter, which allows you to override specific configuration settings at deployment time without regenerating the RMIR file. This is particularly useful for:

Enabling advanced features like FP8 precision (e.g., nn.use_trt_fp8:True).
Tuning performance parameters
Adjusting batch sizes or memory settings

Syntax#

riva-deploy --override_config="module.key:value" <rmir_filename>:<key> <output_dir>

For multiple overrides, separate them with commas:

riva-deploy --override_config="module1.key1:value1,module2.key2:value2" <rmir_filename>:<key> <output_dir>

Note

FP8 precision support requires NVIDIA ADA GPUs or newer architectures. Using FP8 can significantly improve throughput while maintaining accuracy for most ASR models.

Override Format:

Format: module.key:value
Module: Configuration module name (e.g., nn, featurizer, decoder)
Key: Configuration parameter name
Value: New value for the parameter.

Deployment Directory Structure#

After successful deployment, the <output_dir> will contain a Triton model repository with the following structure:

<output_dir>/
├── <pipeline_name>/
│   ├── config.pbtxt
│   └── 1/
│       └── model files
├── <acoustic_model_name>/
│   ├── config.pbtxt
│   └── 1/
│       └── model.plan

Each component (pipeline, acoustic model) has its own directory with:

config.pbtxt: Triton configuration file
1/: Version directory containing the optimized model files

NVIDIA Riva

Building and Deploying ASR Pipelines

Contents

Building and Deploying ASR Pipelines#

Understanding riva-build#

Input Requirements#

Output#

Workflow Overview#

Building ASR Pipelines: Practical Examples#

Hydra Configuration System#

Basic Streaming ASR Pipeline#

Using NeMo Models Directly#

Model Artifacts#

Downloading Models from Hugging Face#

Building Punctuation and Capitalization (PnC) Models#

Summary of riva-build Commands#

Streaming/Offline Recognition#

Language Models#

ARPA Format Language Model#

KenLM Binary Language Model#

Decoder Hyper-Parameters#

Flashlight Decoder Lexicon#

(Advanced) Manually Adding Additional Tokenizations of Words in Lexicon#

Flashlight Decoder Lexicon Free#

OpenSeq2Seq Decoder#

Beginning/End of Utterance Detection#

Streaming Speaker Diarization#

Sortformer#

Neural-Based Voice Activity Detection#

Silero VAD#

MarbleNet VAD#

Generating Multiple Transcript Hypotheses#

Impact of Chunk Size and Padding Size on Performance and Accuracy (Advanced)#

Sharing Acoustic and Feature Extractor Models Across Multiple ASR Pipelines (Advanced)#

Riva-build Optional Parameters#

Riva Deploy#

Basic Usage#

Example Commands#

GPU Architecture Considerations#

TensorRT Optimization#

Overriding Configuration During Deployment#

Syntax#

Deployment Directory Structure#