NeMo ASR Collection API

NVIDIA Docs Hub NVIDIA NeMo Framework User Guide NeMo ASR Collection API

Model Classes

class nemo.collections.asr.models.EncDecCTCModel(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.models.asr_model.ASRModel, nemo.collections.asr.models.asr_model.ExportableEncDecModel, nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin, nemo.collections.asr.parts.mixins.interctc_mixin.InterCTCMixin, nemo.collections.asr.parts.mixins.transcription.ASRTranscriptionMixin

Base class for encoder decoder CTC-based models.

change_vocabulary(new_vocabulary: List[str], decoding_cfg: Optional[omegaconf.DictConfig] = None)

Changes vocabulary used during CTC decoding process. Use this method when fine-tuning on from pre-trained model. This method changes only decoder and leaves encoder and pre-processing modules unchanged. For example, you would use it if you want to use pretrained encoder when fine-tuning on a data in another language, or when you’d need model to learn capitalization, punctuation and/or special characters.

If new_vocabulary == self.decoder.vocabulary then nothing will be changed.

Parameters

Returns: None

register_artifact(config_path: str, src: str, verify_src_exists: bool = True)

Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to(“mymodel.nemo”) is called.

How it works:

It always returns existing absolute path which can be used during Model constructor call
EXCEPTION: src is None or “” in which case nothing will be done and src will be returned

It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts

Copy
Copied!

            
            If "src" is local existing path:
    then it will be returned in absolute path form.
elif "src" starts with "nemo_file:unique_artifact_name":
    .nemo will be untarred to a temporary folder location and an actual existing path will be returned
else:
    an error will be raised.

WARNING: use .register_artifact calls in your models’ constructors. The returned path is not guaranteed to exist after you have exited your model’s constructor.

Parameters
Returns
Return type

setup_optimization(optim_config: Optional[Union[omegaconf.DictConfig, Dict]] = None, optim_kwargs: Optional[Dict[str, Any]] = None)

Prepares an optimizer from a string name and its optional config parameters.

Parameters

setup_test_data(test_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the test data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_training_data(train_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the training data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_validation_data(val_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the validation data loader via a Dict-like object.

Parameters

Supported Datasets:

transcribe(audio: Union[str, List[str], torch.Tensor, numpy.ndarray], batch_size: int = 4, return_hypotheses: bool = False, num_workers: int = 0, channel_selector: Optional[Union[int, Iterable[int], str]] = None, augmentor: omegaconf.DictConfig = None, verbose: bool = True, override_config: Optional[nemo.collections.asr.parts.mixins.transcription.TranscribeConfig] = None) → Union[List[str], List[Hypothesis], Tuple[List[str]], Tuple[List[Hypothesis]]]

If modify this function, please remember update transcribe_partial_audio() in nemo/collections/asr/parts/utils/trancribe_utils.py

Uses greedy decoding to transcribe audio files. Use this method for debugging and prototyping.

Parameters
Returns

class nemo.collections.asr.models.EncDecCTCModelBPE(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.models.ctc_models.EncDecCTCModel, nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin

Encoder decoder CTC-based models with Byte Pair Encoding.

change_vocabulary(new_tokenizer_dir: Union[str, omegaconf.DictConfig], new_tokenizer_type: str, decoding_cfg: Optional[omegaconf.DictConfig] = None)

Changes vocabulary of the tokenizer used during CTC decoding process. Use this method when fine-tuning on from pre-trained model. This method changes only decoder and leaves encoder and pre-processing modules unchanged. For example, you would use it if you want to use pretrained encoder when fine-tuning on a data in another language, or when you’d need model to learn capitalization, punctuation and/or special characters.

Parameters

Returns: None

register_artifact(config_path: str, src: str, verify_src_exists: bool = True)

Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to(“mymodel.nemo”) is called.

How it works:

It always returns existing absolute path which can be used during Model constructor call
EXCEPTION: src is None or “” in which case nothing will be done and src will be returned

It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts

Copy
Copied!

            
            If "src" is local existing path:
    then it will be returned in absolute path form.
elif "src" starts with "nemo_file:unique_artifact_name":
    .nemo will be untarred to a temporary folder location and an actual existing path will be returned
else:
    an error will be raised.

WARNING: use .register_artifact calls in your models’ constructors. The returned path is not guaranteed to exist after you have exited your model’s constructor.

Parameters
Returns
Return type

setup_optimization(optim_config: Optional[Union[omegaconf.DictConfig, Dict]] = None, optim_kwargs: Optional[Dict[str, Any]] = None)

Prepares an optimizer from a string name and its optional config parameters.

Parameters

setup_test_data(test_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the test data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_training_data(train_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the training data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_validation_data(val_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the validation data loader via a Dict-like object.

Parameters

Supported Datasets:

If modify this function, please remember update transcribe_partial_audio() in nemo/collections/asr/parts/utils/trancribe_utils.py

Uses greedy decoding to transcribe audio files. Use this method for debugging and prototyping.

Parameters
Returns

class nemo.collections.asr.models.EncDecRNNTModel(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.models.asr_model.ASRModel, nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin, nemo.collections.asr.models.asr_model.ExportableEncDecModel, nemo.collections.asr.parts.mixins.transcription.ASRTranscriptionMixin

Base class for encoder decoder RNNT-based models.

change_vocabulary(new_vocabulary: List[str], decoding_cfg: Optional[omegaconf.DictConfig] = None)

Changes vocabulary used during RNNT decoding process. Use this method when fine-tuning a pre-trained model. This method changes only decoder and leaves encoder and pre-processing modules unchanged. For example, you would use it if you want to use pretrained encoder when fine-tuning on data in another language, or when you’d need model to learn capitalization, punctuation and/or special characters.

Parameters

Returns: None

register_artifact(config_path: str, src: str, verify_src_exists: bool = True)

Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to(“mymodel.nemo”) is called.

How it works:

It always returns existing absolute path which can be used during Model constructor call
EXCEPTION: src is None or “” in which case nothing will be done and src will be returned

It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts

Copy
Copied!

            
            If "src" is local existing path:
    then it will be returned in absolute path form.
elif "src" starts with "nemo_file:unique_artifact_name":
    .nemo will be untarred to a temporary folder location and an actual existing path will be returned
else:
    an error will be raised.

WARNING: use .register_artifact calls in your models’ constructors. The returned path is not guaranteed to exist after you have exited your model’s constructor.

Parameters
Returns
Return type

setup_optimization(optim_config: Optional[Union[omegaconf.DictConfig, Dict]] = None, optim_kwargs: Optional[Dict[str, Any]] = None)

Prepares an optimizer from a string name and its optional config parameters.

Parameters

setup_test_data(test_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the test data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_training_data(train_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the training data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_validation_data(val_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the validation data loader via a Dict-like object.

Parameters

Supported Datasets:

transcribe(audio: List[str], batch_size: int = 4, return_hypotheses: bool = False, partial_hypothesis: Optional[List[Hypothesis]] = None, num_workers: int = 0, channel_selector: Optional[Union[int, Iterable[int], str]] = None, augmentor: omegaconf.DictConfig = None, verbose: bool = True, override_config: Optional[nemo.collections.asr.parts.mixins.transcription.TranscribeConfig] = None) → Union[List[str], List[Hypothesis], Tuple[List[str]], Tuple[List[Hypothesis]]]

Uses greedy decoding to transcribe audio files. Use this method for debugging and prototyping.

Parameters
Returns

class nemo.collections.asr.models.EncDecRNNTBPEModel(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.models.rnnt_models.EncDecRNNTModel, nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin

Base class for encoder decoder RNNT-based models with subword tokenization.

change_vocabulary(new_tokenizer_dir: Union[str, omegaconf.DictConfig], new_tokenizer_type: str, decoding_cfg: Optional[omegaconf.DictConfig] = None)

Changes vocabulary used during RNNT decoding process. Use this method when fine-tuning on from pre-trained model. This method changes only decoder and leaves encoder and pre-processing modules unchanged. For example, you would use it if you want to use pretrained encoder when fine-tuning on data in another language, or when you’d need model to learn capitalization, punctuation and/or special characters.

Parameters

Returns: None

register_artifact(config_path: str, src: str, verify_src_exists: bool = True)

Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to(“mymodel.nemo”) is called.

How it works:

It always returns existing absolute path which can be used during Model constructor call
EXCEPTION: src is None or “” in which case nothing will be done and src will be returned

It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts

Copy
Copied!

            
            If "src" is local existing path:
    then it will be returned in absolute path form.
elif "src" starts with "nemo_file:unique_artifact_name":
    .nemo will be untarred to a temporary folder location and an actual existing path will be returned
else:
    an error will be raised.

WARNING: use .register_artifact calls in your models’ constructors. The returned path is not guaranteed to exist after you have exited your model’s constructor.

Parameters
Returns
Return type

setup_optimization(optim_config: Optional[Union[omegaconf.DictConfig, Dict]] = None, optim_kwargs: Optional[Dict[str, Any]] = None)

Prepares an optimizer from a string name and its optional config parameters.

Parameters

setup_test_data(test_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the test data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_training_data(train_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the training data loader via a Dict-like object.

Parameters

Supported Datasets:

setup_validation_data(val_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Sets up the validation data loader via a Dict-like object.

Parameters

Supported Datasets:

Uses greedy decoding to transcribe audio files. Use this method for debugging and prototyping.

Parameters
Returns

class nemo.collections.asr.models.EncDecClassificationModel(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.models.classification_models._EncDecBaseModel

Encoder decoder Classification models.

register_artifact(config_path: str, src: str, verify_src_exists: bool = True)

Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to(“mymodel.nemo”) is called.

How it works:

It always returns existing absolute path which can be used during Model constructor call
EXCEPTION: src is None or “” in which case nothing will be done and src will be returned

It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts

Copy
Copied!

            
            If "src" is local existing path:
    then it will be returned in absolute path form.
elif "src" starts with "nemo_file:unique_artifact_name":
    .nemo will be untarred to a temporary folder location and an actual existing path will be returned
else:
    an error will be raised.

WARNING: use .register_artifact calls in your models’ constructors. The returned path is not guaranteed to exist after you have exited your model’s constructor.

Parameters
Returns
Return type

setup_optimization(optim_config: Optional[Union[omegaconf.DictConfig, Dict]] = None, optim_kwargs: Optional[Dict[str, Any]] = None)

Prepares an optimizer from a string name and its optional config parameters.

Parameters

setup_test_data(test_data_config: Optional[Union[omegaconf.DictConfig, Dict]], use_feat: bool = False)

(Optionally) Setups data loader to be used in test

Parameters

Returns:

setup_training_data(train_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

Setups data loader to be used in training

Parameters

Returns:

setup_validation_data(val_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

class nemo.collections.asr.models.EncDecSpeakerLabelModel(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.modelPT.ModelPT, nemo.collections.asr.models.asr_model.ExportableEncDecModel

Encoder decoder class for speaker label models. Model class creates training, validation methods for setting up data performing model forward pass. Expects config dict for

preprocessor
Jasper/Quartznet Encoder
Speaker Decoder

register_artifact(config_path: str, src: str, verify_src_exists: bool = True)

Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to(“mymodel.nemo”) is called.

How it works:

It always returns existing absolute path which can be used during Model constructor call
EXCEPTION: src is None or “” in which case nothing will be done and src will be returned

It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts

Copy
Copied!

            
            If "src" is local existing path:
    then it will be returned in absolute path form.
elif "src" starts with "nemo_file:unique_artifact_name":
    .nemo will be untarred to a temporary folder location and an actual existing path will be returned
else:
    an error will be raised.

WARNING: use .register_artifact calls in your models’ constructors. The returned path is not guaranteed to exist after you have exited your model’s constructor.

Parameters
Returns
Return type

setup_optimization(optim_config: Optional[Union[omegaconf.DictConfig, Dict]] = None, optim_kwargs: Optional[Dict[str, Any]] = None)

Prepares an optimizer from a string name and its optional config parameters.

Parameters

setup_test_data(test_data_layer_params: Optional[Union[omegaconf.DictConfig, Dict]])

(Optionally) Setups data loader to be used in test

Parameters

Returns:

setup_training_data(train_data_layer_config: Optional[Union[omegaconf.DictConfig, Dict]])

Setups data loader to be used in training

Parameters

Returns:

setup_validation_data(val_data_layer_config: Optional[Union[omegaconf.DictConfig, Dict]])

class nemo.collections.asr.models.hybrid_asr_tts_models.ASRWithTTSModel(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.models.asr_model.ASRModel

Hybrid ASR-TTS model: a transparent wrapper for ASR model with frozen text-to-spectrogram pretrained model, which allows to use text-only data for training/finetuning Text-only data can be mixed with audio-text pairs

classmethod from_asr_config(asr_cfg: omegaconf.DictConfig, asr_model_type: Union[str, nemo.collections.asr.models.hybrid_asr_tts_models.ASRWithTTSModel.ASRModelTypes], tts_model_path: Union[str, pathlib.Path], enhancer_model_path: Optional[Union[pathlib.Path, str]] = None, trainer: Optional[pytorch_lightning.Trainer] = None)

classmethod from_pretrained_models(asr_model_path: Union[str, pathlib.Path], tts_model_path: Union[str, pathlib.Path], enhancer_model_path: Optional[Union[pathlib.Path, str]] = None, asr_model_fuse_bn: bool = False, cfg: Optional[omegaconf.DictConfig] = None, trainer: Optional[pytorch_lightning.Trainer] = None)

Load model from pretrained ASR and TTS models :param asr_model_path: path to .nemo ASR model checkpoint :param tts_model_path: path to .nemo TTS model checkpoint :param enhancer_model_path: path to .nemo enhancer model checkpoint :param asr_model_fuse_bn: automatically fuse batchnorm layers in ASR model :param cfg: optional config for hybrid model :param trainer: Pytorch-Lightning trainer

Returns

save_asr_model_to(save_path: str)

setup_training_data(train_data_config: Optional[Union[omegaconf.DictConfig, Dict]])

class nemo.collections.asr.models.confidence_ensemble.ConfidenceEnsembleModel(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.modelPT.ModelPT

Implementation of the confidence ensemble model.

See https://arxiv.org/abs/2306.15824 for details.

Note

Currently this class only support transcribe method as it requires full-utterance confidence scores to operate.

transcribe(paths2audio_files: List[str], batch_size: int = 4, return_hypotheses: bool = False, num_workers: int = 0, channel_selector: Optional[Union[int, Iterable[int], str]] = None, augmentor: Optional[omegaconf.DictConfig] = None, verbose: bool = True, **kwargs) → List[str]

Modules

class nemo.collections.asr.modules.ConvASREncoder(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.core.classes.exportable.Exportable, nemo.core.classes.mixins.access_mixins.AccessMixin

Convolutional encoder for ASR models. With this class you can implement JasperNet and QuartzNet models.

Based on these papers:

input_example(max_batch=1, max_dim=8192)

property input_types

property output_types

class nemo.collections.asr.modules.ConvASRDecoder(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.core.classes.exportable.Exportable, nemo.core.classes.mixins.adapter_mixins.AdapterModuleMixin

Simple ASR Decoder for use with CTC-based models such as JasperNet and QuartzNet

Based on these papers:

add_adapter(name: str, cfg: omegaconf.DictConfig)

Add an Adapter module to this module.

Parameters

input_example(max_batch=1, max_dim=256)

property input_types

property output_types

class nemo.collections.asr.modules.ConvASRDecoderClassification(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.core.classes.exportable.Exportable

Simple ASR Decoder for use with classification models such as JasperNet and QuartzNet

Based on these papers:

input_example(max_batch=1, max_dim=256)

property input_types

property output_types

class nemo.collections.asr.modules.SpeakerDecoder(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.core.classes.exportable.Exportable

Speaker Decoder creates the final neural layers that maps from the outputs of Jasper Encoder to the embedding layer followed by speaker based softmax loss.

Parameters

input_example(max_batch=1, max_dim=256)

property input_types

property output_types

class nemo.collections.asr.modules.ConformerEncoder(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.collections.asr.parts.mixins.streaming.StreamingEncoder, nemo.core.classes.exportable.Exportable, nemo.core.classes.mixins.access_mixins.AccessMixin

The encoder for ASR model of Conformer. Based on this paper: ‘Conformer: Convolution-augmented Transformer for Speech Recognition’ by Anmol Gulati et al. https://arxiv.org/abs/2005.08100

Parameters

feat_in (int) – the size of feature channels
n_layers (int) – number of layers of ConformerBlock
d_model (int) – the hidden size of the model
feat_out (int) – the size of the output features Defaults to -1 (means feat_out is d_model)
subsampling (str) – the method of subsampling, choices=[‘vggnet’, ‘striding’, ‘dw-striding’, ‘stacking’, ‘stacking_norm’] Defaults to striding.
subsampling_factor (int) – the subsampling factor which should be power of 2 Defaults to 4.
subsampling_conv_chunking_factor (int) – optionally, force chunk inputs (helpful for large inputs) Should be power of 2, 1 (auto-chunking, default), or -1 (no chunking)
subsampling_conv_channels (int) – the size of the convolutions in the subsampling module Defaults to -1 which would set it to d_model.
reduction (str, Optional) – the method of reduction, choices=[‘pooling’, ‘striding’]. If no value is passed, then no reduction is performed and the models runs with the original 4x subsampling.
reduction_position (int, Optional) – the index of the layer to apply reduction. If -1, apply reduction at the end.
reduction_factor (int) – the reduction factor which should be either 1 or a power of 2 Defaults to 1.
ff_expansion_factor (int) – the expansion factor in feed forward layers Defaults to 4.
self_attention_model (str) –
type of the attention layer and positional encoding

’rel_pos’:
relative positional embedding and Transformer-XL

’rel_pos_local_attn’:
relative positional embedding and Transformer-XL with local attention using overlapping chunks. Attention context is determined by att_context_size parameter.

’abs_pos’:
absolute positional embedding and Transformer

Default is rel_pos.
pos_emb_max_len (int) – the maximum length of positional embeddings Defaults to 5000
n_heads (int) – number of heads in multi-headed attention layers Defaults to 4.
att_context_size (List[Union[List[int],int]]) – specifies the context sizes on each side. Each context size should be a list of two integers like [100,100]. A list of context sizes like [[100,100],[100,50]] can also be passed. -1 means unlimited context. Defaults to [-1,-1]
att_context_probs (List[float]) – a list of probabilities of each one of the att_context_size when a list of them is passed. If not specified, uniform distribution is being used. Defaults to None
att_context_style (str) – ‘regular’ or ‘chunked_limited’. Defaults to ‘regular’
xscaling (bool) – enables scaling the inputs to the multi-headed attention layers by sqrt(d_model) Defaults to True.
untie_biases (bool) – whether to not share (untie) the bias weights between layers of Transformer-XL Defaults to True.
conv_kernel_size (int) – the size of the convolutions in the convolutional modules Defaults to 31.
conv_norm_type (str) – the type of the normalization in the convolutional modules Defaults to ‘batch_norm’.
conv_context_size (list) – it can be”causal” or a list of two integers while conv_context_size[0]+conv_context_size[1]+1==conv_kernel_size. None means [(conv_kernel_size-1)//2, (conv_kernel_size-1)//2], and ‘causal’ means [(conv_kernel_size-1), 0]. Defaults to None.
conv_dual_mode (bool) – specifies if convolution should be dual mode when dual_offline mode is being used. When enables, the left half of the convolution kernel would get masked in streaming cases. Defaults to False
dropout (float) – the dropout rate used in all layers except the attention layers Defaults to 0.1.
dropout_pre_encoder (float) – the dropout rate used before the encoder Defaults to 0.1.
dropout_emb (float) – the dropout rate used for the positional embeddings Defaults to 0.1.
dropout_att (float) – the dropout rate used for the attention layer Defaults to 0.0.
stochastic_depth_drop_prob (float) – if non-zero, will randomly drop layers during training. The higher this value, the more often layers are dropped. Defaults to 0.0.
stochastic_depth_mode (str) – can be either “linear” or “uniform”. If set to “uniform”, all layers have the same probability of drop. If set to “linear”, the drop probability grows linearly from 0 for the first layer to the desired value for the final layer. Defaults to “linear”.
stochastic_depth_start_layer (int) – starting layer for stochastic depth. All layers before this will never be dropped. Note that drop probability will be adjusted accordingly if mode is “linear” when start layer is > 1. Defaults to 1.
global_tokens (int) – number of tokens to be used for global attention. Only relevant if self_attention_model is ‘rel_pos_local_attn’. Defaults to 0.
global_tokens_spacing (int) – how far apart the global tokens are Defaults to 1.
global_attn_separate (bool) – whether the q, k, v layers used for global tokens should be separate. Defaults to False.

change_attention_model(self_attention_model: Optional[str] = None, att_context_size: Optional[List[int]] = None, update_config: bool = True, device: Optional[torch.device] = None)

Update the self_attention_model which changes the positional encoding and attention layers.

Parameters

self_attention_model (str) –
type of the attention layer and positional encoding

’rel_pos’:
relative positional embedding and Transformer-XL

’rel_pos_local_attn’:
relative positional embedding and Transformer-XL with local attention using overlapping windows. Attention context is determined by att_context_size parameter.

’abs_pos’:
absolute positional embedding and Transformer

If None is provided, the self_attention_model isn’t changed. Defaults to None.
att_context_size (List[int]) – List of 2 ints corresponding to left and right attention context sizes, or None to keep as it is. Defaults to None.
update_config (bool) – Whether to update the config or not with the new attention model. Defaults to True.
device (torch.device) – If provided, new layers will be moved to the device. Defaults to None.

change_subsampling_conv_chunking_factor(subsampling_conv_chunking_factor: int)

Update the conv_chunking_factor (int) Default is 1 (auto) Set it to -1 (disabled) or to a specific value (power of 2) if you OOM in the conv subsampling layers

Parameters

property disabled_deployment_input_names

property disabled_deployment_output_names

input_example(max_batch=1, max_dim=256)

property input_types

property input_types_for_export

property output_types

property output_types_for_export

set_max_audio_length(max_audio_length)

setup_streaming_params(chunk_size: Optional[int] = None, shift_size: Optional[int] = None, left_chunks: Optional[int] = None, att_context_size: Optional[list] = None, max_context: int = 10000)

This function sets the needed values and parameters to perform streaming. The configuration would be stored in self.streaming_cfg. The streaming configuration is needed to simulate streaming inference.

Parameters

class nemo.collections.asr.modules.SqueezeformerEncoder(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.core.classes.exportable.Exportable, nemo.core.classes.mixins.access_mixins.AccessMixin

The encoder for ASR model of Squeezeformer. Based on this paper: ‘Squeezeformer: An Efficient Transformer for Automatic Speech Recognition’ by Sehoon Kim et al. https://arxiv.org/abs/2206.00888

Parameters

input_example(max_batch=1, max_dim=256)

property input_types

make_pad_mask(max_audio_length, seq_lens)

property output_types

set_max_audio_length(max_audio_length)

class nemo.collections.asr.modules.RNNEncoder(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule, nemo.core.classes.exportable.Exportable

The RNN-based encoder for ASR models. Followed the architecture suggested in the following paper: ‘STREAMING END-TO-END SPEECH RECOGNITION FOR MOBILE DEVICES’ by Yanzhang He et al. https://arxiv.org/pdf/1811.06621.pdf

Parameters

input_example()

property input_types

property output_types

class nemo.collections.asr.modules.RNNTDecoder(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTDecoder, nemo.core.classes.exportable.Exportable, nemo.core.classes.mixins.adapter_mixins.AdapterModuleMixin

A Recurrent Neural Network Transducer Decoder / Prediction Network (RNN-T Prediction Network). An RNN-T Decoder/Prediction network, comprised of a stateful LSTM model.

Parameters

prednet –
A dict-like object which contains the following key-value pairs.

pred_hidden:
int specifying the hidden dimension of the prediction net.

pred_rnn_layers:
int specifying the number of rnn layers.

Optionally, it may also contain the following:

forget_gate_bias:
float, set by default to 1.0, which constructs a forget gate initialized to 1.0. Reference: [An Empirical Exploration of Recurrent Network Architectures](http://proceedings.mlr.press/v37/jozefowicz15.pdf)

t_max:
int value, set to None by default. If an int is specified, performs Chrono Initialization of the LSTM network, based on the maximum number of timesteps t_max expected during the course of training. Reference: [Can recurrent neural networks warp time?](https://openreview.net/forum?id=SJcKhk-Ab)

weights_init_scale:
Float scale of the weights after initialization. Setting to lower than one sometimes helps reduce variance between runs.

hidden_hidden_bias_scale:
Float scale for the hidden-to-hidden bias scale. Set to 0.0 for the default behaviour.

dropout:
float, set to 0.0 by default. Optional dropout applied at the end of the final LSTM RNN layer.
vocab_size – int, specifying the vocabulary size of the embedding layer of the Prediction network, excluding the RNNT blank token.
normalization_mode – Can be either None, ‘batch’ or ‘layer’. By default, is set to None. Defines the type of normalization applied to the RNN layer.
random_state_sampling – bool, set to False by default. When set, provides normal-distribution sampled state tensors instead of zero tensors during training. Reference: [Recognizing long-form speech using streaming end-to-end models](https://arxiv.org/abs/1910.11455)
blank_as_pad –
bool, set to True by default. When set, will add a token to the Embedding layer of this prediction network, and will treat this token as a pad token. In essence, the RNNT pad token will be treated as a pad token, and the embedding layer will return a zero tensor for this token.

It is set by default as it enables various batch optimizations required for batched beam search. Therefore, it is not recommended to disable this flag.

add_adapter(name: str, cfg: omegaconf.DictConfig)

Add an Adapter module to this module.

Parameters

batch_concat_states(batch_states: List[List[torch.Tensor]]) → List[torch.Tensor]

Concatenate a batch of decoder state to a packed state.

Parameters

batch_states (list) – batch of decoder states B x ([L x (H)], [L x (H)])

Returns

decoder states

Return type

(tuple)

batch_copy_states(old_states: List[torch.Tensor], new_states: List[torch.Tensor], ids: List[int], value: Optional[float] = None) → List[torch.Tensor]

Copy states from new state to old state at certain indices.

Parameters

old_states (list) – packed decoder states (L x B x H, L x B x H)
new_states – packed decoder states (L x B x H, L x B x H)
ids (list) – List of indices to copy states at.
value (optional float) – If a value should be copied instead of a state slice, a float should be provided

Returns

batch of decoder states with partial copy at ids (or a specific value).

batch_initialize_states(batch_states: List[torch.Tensor], decoder_states: List[List[torch.Tensor]])

Create batch of decoder states.

Parameters

batch_states (list) – batch of decoder states ([L x (B, H)], [L x (B, H)])
decoder_states (list of list) – list of decoder states [B x ([L x (1, H)], [L x (1, H)])]

Returns

batch of decoder states

Return type

batch_states (tuple)

classmethod batch_replace_states_all(src_states: Tuple[torch.Tensor, torch.Tensor], dst_states: Tuple[torch.Tensor, torch.Tensor])

classmethod batch_replace_states_mask(src_states: Tuple[torch.Tensor, torch.Tensor], dst_states: Tuple[torch.Tensor, torch.Tensor], mask: torch.Tensor)

batch_score_hypothesis(hypotheses: List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis], cache: Dict[Tuple[int], Any], batch_states: List[torch.Tensor]) → Tuple[torch.Tensor, List[torch.Tensor], torch.Tensor]

Used for batched beam search algorithms. Similar to score_hypothesis method.

Parameters
Returns
Return type

batch_select_state(batch_states: List[torch.Tensor], idx: int) → List[List[torch.Tensor]]

Get decoder state from batch of states, for given id.

Parameters

batch_states (list) – batch of decoder states ([L x (B, H)], [L x (B, H)])
idx (int) – index to extract state from batch of states

Returns

decoder states for given id

Return type

(tuple)

batch_split_states(batch_states: Tuple[torch.Tensor, torch.Tensor]) → list[Tuple[torch.Tensor, torch.Tensor]]

initialize_state(y: torch.Tensor) → Tuple[torch.Tensor, torch.Tensor]

Initialize the state of the LSTM layers, with same dtype and device as input y. LSTM accepts a tuple of 2 tensors as a state.

Parameters
Returns

input_example(max_batch=1, max_dim=1)

property input_types

mask_select_states(states: Tuple[torch.Tensor, torch.Tensor], mask: torch.Tensor) → Tuple[torch.Tensor, torch.Tensor]

Return states by mask selection :param states: states for the batch :param mask: boolean mask for selecting states; batch dimension should be the same as for states

Returns

property output_types

predict(y: Optional[torch.Tensor] = None, state: Optional[List[torch.Tensor]] = None, add_sos: bool = True, batch_size: Optional[int] = None) → Tuple[torch.Tensor, List[torch.Tensor]]

Stateful prediction of scores and state for a (possibly null) tokenset. This method takes various cases into consideration : - No token, no state - used for priming the RNN - No token, state provided - used for blank token scoring - Given token, states - used for scores + new states

Here: B - batch size U - label length H - Hidden dimension size of RNN L - Number of RNN layers

Parameters

y – Optional torch tensor of shape [B, U] of dtype long which will be passed to the Embedding. If None, creates a zero tensor of shape [B, 1, H] which mimics output of pad-token on EmbeddiNg.
state – An optional list of states for the RNN. Eg: For LSTM, it is the state list length is 2. Each state must be a tensor of shape [L, B, H]. If None, and during training mode and random_state_sampling is set, will sample a normal distribution tensor of the above shape. Otherwise, None will be passed to the RNN.
add_sos – bool flag, whether a zero vector describing a “start of signal” token should be prepended to the above “y” tensor. When set, output size is (B, U + 1, H).
batch_size – An optional int, specifying the batch size of the y tensor. Can be infered if y and state is None. But if both are None, then batch_size cannot be None.

Returns

A tuple (g, hid) such that -

If add_sos is False:

g:
hid:

If add_sos is True:

g:
hid:

score_hypothesis(hypothesis: nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis, cache: Dict[Tuple[int], Any]) → Tuple[torch.Tensor, List[torch.Tensor], torch.Tensor]

Similar to the predict() method, instead this method scores a Hypothesis during beam search. Hypothesis is a dataclass representing one hypothesis in a Beam Search.

Parameters
Returns
Return type

class nemo.collections.asr.modules.StatelessTransducerDecoder(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTDecoder, nemo.core.classes.exportable.Exportable

A Stateless Neural Network Transducer Decoder / Prediction Network. An RNN-T Decoder/Prediction stateless network that simply takes concatenation of embeddings of the history tokens as the output.

Parameters

batch_concat_states(batch_states: List[List[torch.Tensor]]) → List[torch.Tensor]

Concatenate a batch of decoder state to a packed state.

Parameters

batch_states (list) – batch of decoder states B x ([(C)]

Returns

decoder states

Return type

(tuple)

batch_copy_states(old_states: List[torch.Tensor], new_states: List[torch.Tensor], ids: List[int], value: Optional[float] = None) → List[torch.Tensor]

Copy states from new state to old state at certain indices.

Parameters
Returns

batch_initialize_states(batch_states: List[torch.Tensor], decoder_states: List[List[torch.Tensor]])

Create batch of decoder states.

Parameters

batch_states (list) – batch of decoder states ([(B, H)])
decoder_states (list of list) – list of decoder states [B x ([(1, C)]]

Returns

batch of decoder states

Return type

batch_states (tuple)

classmethod batch_replace_states_all(src_states: list[torch.Tensor], dst_states: list[torch.Tensor])

classmethod batch_replace_states_mask(src_states: list[torch.Tensor], dst_states: list[torch.Tensor], mask: torch.Tensor)

Used for batched beam search algorithms. Similar to score_hypothesis method.

Parameters
Returns
Return type

batch_select_state(batch_states: List[torch.Tensor], idx: int) → List[List[torch.Tensor]]

Get decoder state from batch of states, for given id.

Parameters

batch_states (list) – batch of decoder states [(B, C)]
idx (int) – index to extract state from batch of states

Returns

decoder states for given id

Return type

(tuple)

batch_split_states(batch_states: list[torch.Tensor]) → list[list[torch.Tensor]]

initialize_state(y: torch.Tensor) → List[torch.Tensor]

Initialize the state of the RNN layers, with same dtype and device as input y.

Parameters

y – A torch.Tensor whose device the generated states will be placed on.

Returns

List of torch.Tensor, each of shape [L, B, H], where

input_example(max_batch=1, max_dim=1)

property input_types

mask_select_states(states: Optional[List[torch.Tensor]], mask: torch.Tensor) → Optional[List[torch.Tensor]]

Return states by mask selection :param states: states for the batch :param mask: boolean mask for selecting states; batch dimension should be the same as for states

Returns

property output_types

predict(y: Optional[torch.Tensor] = None, state: Optional[torch.Tensor] = None, add_sos: bool = True, batch_size: Optional[int] = None) → Tuple[torch.Tensor, List[torch.Tensor]]

Stateful prediction of scores and state for a tokenset.

Here: B - batch size U - label length C - context size for stateless decoder D - total embedding size

Parameters

y – Optional torch tensor of shape [B, U] of dtype long which will be passed to the Embedding. If None, creates a zero tensor of shape [B, 1, D] which mimics output of pad-token on Embedding.
state – An optional one-element list of one tensor. The tensor is used to store previous context labels. The tensor uses type long and is of shape [B, C].
add_sos – bool flag, whether a zero vector describing a “start of signal” token should be prepended to the above “y” tensor. When set, output size is (B, U + 1, D).
batch_size – An optional int, specifying the batch size of the y tensor. Can be infered if y and state is None. But if both are None, then batch_size cannot be None.

Returns

A tuple (g, state) such that -

If add_sos is False:

g:
state:

If add_sos is True:

g:
state:

score_hypothesis(hypothesis: nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis, cache: Dict[Tuple[int], Any]) → Tuple[torch.Tensor, List[torch.Tensor], torch.Tensor]

Similar to the predict() method, instead this method scores a Hypothesis during beam search. Hypothesis is a dataclass representing one hypothesis in a Beam Search.

Parameters
Returns
Return type

class nemo.collections.asr.modules.RNNTJoint(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTJoint, nemo.core.classes.exportable.Exportable, nemo.core.classes.mixins.adapter_mixins.AdapterModuleMixin

A Recurrent Neural Network Transducer Joint Network (RNN-T Joint Network). An RNN-T Joint network, comprised of a feedforward model.

Parameters

jointnet –
A dict-like object which contains the following key-value pairs. encoder_hidden: int specifying the hidden dimension of the encoder net. pred_hidden: int specifying the hidden dimension of the prediction net. joint_hidden: int specifying the hidden dimension of the joint net activation: Activation function used in the joint step. Can be one of [‘relu’, ‘tanh’, ‘sigmoid’].

Optionally, it may also contain the following: dropout: float, set to 0.0 by default. Optional dropout applied at the end of the joint net.
num_classes – int, specifying the vocabulary size that the joint network must predict, excluding the RNNT blank token.
vocabulary – Optional list of strings/tokens that comprise the vocabulary of the joint network. Unused and kept only for easy access for character based encoding RNNT models.
log_softmax – Optional bool, set to None by default. If set as None, will compute the log_softmax() based on the value provided.
preserve_memory –
Optional bool, set to False by default. If the model crashes due to the memory intensive joint step, one might try this flag to empty the tensor cache in pytorch.

Warning: This will make the forward-backward pass much slower than normal. It also might not fix the OOM if the GPU simply does not have enough memory to compute the joint.
fuse_loss_wer –
Optional bool, set to False by default.

Fuses the joint forward, loss forward and wer forward steps. In doing so, it trades of speed for memory conservation by creating sub-batches of the provided batch of inputs, and performs Joint forward, loss forward and wer forward (optional), all on sub-batches, then collates results to be exactly equal to results from the entire batch.

When this flag is set, prior to calling forward, the fields loss and wer (either one) must be set using the RNNTJoint.set_loss() or RNNTJoint.set_wer() methods.

Further, when this flag is set, the following argument fused_batch_size must be provided as a non negative integer. This value refers to the size of the sub-batch.

When the flag is set, the input and output signature of forward() of this method changes. Input - in addition to encoder_outputs (mandatory argument), the following arguments can be provided.
- decoder_outputs (optional). Required if loss computation is required.
- encoder_lengths (required)
- transcripts (optional). Required for wer calculation.
- transcript_lengths (optional). Required for wer calculation.
- compute_wer (bool, default false). Whether to compute WER or not for the fused batch.
Output - instead of the usual joint log prob tensor, the following results can be returned.
- loss (optional). Returned if decoder_outputs, transcripts and transript_lengths are not None.
- wer_numerator + wer_denominator (optional). Returned if transcripts, transcripts_lengths are provided
  and compute_wer is set.
fused_batch_size – Optional int, required if fuse_loss_wer flag is set. Determines the size of the sub-batches. Should be any value below the actual batch size per GPU.

add_adapter(name: str, cfg: omegaconf.DictConfig)

Add an Adapter module to this module.

Parameters

property disabled_deployment_input_names

input_example(max_batch=1, max_dim=8192)

property input_types

joint_after_projection(f: torch.Tensor, g: torch.Tensor) → torch.Tensor

Compute the joint step of the network after projection.

Here, B = Batch size T = Acoustic model timesteps U = Target sequence length H1, H2 = Hidden dimensions of the Encoder / Decoder respectively H = Hidden dimension of the Joint hidden step. V = Vocabulary size of the Decoder (excluding the RNNT blank token).

Note

The implementation of this model is slightly modified from the original paper. The original paper proposes the following steps : (enc, dec) -> Expand + Concat + Sum [B, T, U, H1+H2] -> Forward through joint hidden [B, T, U, H] – *1 *1 -> Forward through joint final [B, T, U, V + 1].

Parameters
Returns

property output_types

project_encoder(encoder_output: torch.Tensor) → torch.Tensor

Project the encoder output to the joint hidden dimension.

Parameters
Returns

project_prednet(prednet_output: torch.Tensor) → torch.Tensor

Project the Prediction Network (Decoder) output to the joint hidden dimension.

Parameters
Returns

class nemo.collections.asr.modules.SampledRNNTJoint(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.modules.rnnt.RNNTJoint

A Sampled Recurrent Neural Network Transducer Joint Network (RNN-T Joint Network). An RNN-T Joint network, comprised of a feedforward model, where the vocab size will be sampled instead of computing the full vocabulary joint.

Parameters

jointnet –
A dict-like object which contains the following key-value pairs. encoder_hidden: int specifying the hidden dimension of the encoder net. pred_hidden: int specifying the hidden dimension of the prediction net. joint_hidden: int specifying the hidden dimension of the joint net activation: Activation function used in the joint step. Can be one of [‘relu’, ‘tanh’, ‘sigmoid’].

Optionally, it may also contain the following: dropout: float, set to 0.0 by default. Optional dropout applied at the end of the joint net.
num_classes – int, specifying the vocabulary size that the joint network must predict, excluding the RNNT blank token.
n_samples – int, specifies the number of tokens to sample from the vocabulary space, excluding the RNNT blank token. If a given value is larger than the entire vocabulary size, then the full vocabulary will be used.
vocabulary – Optional list of strings/tokens that comprise the vocabulary of the joint network. Unused and kept only for easy access for character based encoding RNNT models.
log_softmax – Optional bool, set to None by default. If set as None, will compute the log_softmax() based on the value provided.
preserve_memory –
Optional bool, set to False by default. If the model crashes due to the memory intensive joint step, one might try this flag to empty the tensor cache in pytorch.

Warning: This will make the forward-backward pass much slower than normal. It also might not fix the OOM if the GPU simply does not have enough memory to compute the joint.
fuse_loss_wer –
Optional bool, set to False by default.

Fuses the joint forward, loss forward and wer forward steps. In doing so, it trades of speed for memory conservation by creating sub-batches of the provided batch of inputs, and performs Joint forward, loss forward and wer forward (optional), all on sub-batches, then collates results to be exactly equal to results from the entire batch.

When this flag is set, prior to calling forward, the fields loss and wer (either one) must be set using the RNNTJoint.set_loss() or RNNTJoint.set_wer() methods.

Further, when this flag is set, the following argument fused_batch_size must be provided as a non negative integer. This value refers to the size of the sub-batch.

When the flag is set, the input and output signature of forward() of this method changes. Input - in addition to encoder_outputs (mandatory argument), the following arguments can be provided.
- decoder_outputs (optional). Required if loss computation is required.
- encoder_lengths (required)
- transcripts (optional). Required for wer calculation.
- transcript_lengths (optional). Required for wer calculation.
- compute_wer (bool, default false). Whether to compute WER or not for the fused batch.
Output - instead of the usual joint log prob tensor, the following results can be returned.
- loss (optional). Returned if decoder_outputs, transcripts and transript_lengths are not None.
- wer_numerator + wer_denominator (optional). Returned if transcripts, transcripts_lengths are provided
  and compute_wer is set.
fused_batch_size – Optional int, required if fuse_loss_wer flag is set. Determines the size of the sub-batches. Should be any value below the actual batch size per GPU.

sampled_joint(f: torch.Tensor, g: torch.Tensor, transcript: torch.Tensor, transcript_lengths: torch.Tensor) → torch.Tensor

Compute the sampled joint step of the network.

# Reference - [Memory-Efficient Training of RNN-Transducer with Sampled Softmax](https://arxiv.org/abs/2203.16868)

Note

The implementation of this joint model is slightly modified from the original paper. The original paper proposes the following steps : (enc, dec) -> Expand + Concat + Sum [B, T, U, H1+H2] -> Forward through joint hidden [B, T, U, H] – *1 *1 -> Forward through joint final [B, T, U, V + 1].

We instead split the joint hidden into joint_hidden_enc and joint_hidden_dec and act as follows: enc -> Forward through joint_hidden_enc -> Expand [B, T, 1, H] – *1 dec -> Forward through joint_hidden_dec -> Expand [B, 1, U, H] – *2 (*1, *2) -> Sum [B, T, U, H] -> Sample Vocab V_Pos (for target tokens) and V_Neg -> (V_Neg is sampled not uniformly by as a rand permutation of all vocab tokens, then eliminate all Intersection(V_Pos, V_Neg) common tokens to avoid duplication of loss) -> Concat new Vocab V_Sampled = Union(V_Pos, V_Neg) -> Forward partially through the joint final to create [B, T, U, V_Sampled]

Parameters
Returns

Parts

class nemo.collections.asr.parts.submodules.jasper.JasperBlock(*args: Any, **kwargs: Any)

Bases: torch.nn.Module, nemo.core.classes.mixins.adapter_mixins.AdapterModuleMixin, nemo.core.classes.mixins.access_mixins.AccessMixin

Constructs a single “Jasper” block. With modified parameters, also constructs other blocks for models such as QuartzNet and Citrinet.

For Jasper : separable flag should be False
For QuartzNet : separable flag should be True
For Citrinet : separable flag and se flag should be True

Note that above are general distinctions, each model has intricate differences that expand over multiple such blocks.

For further information about the differences between models which use JasperBlock, please review the configs for ASR models found in the ASR examples directory.

Parameters

forward(input_: Tuple[List[torch.Tensor], Optional[torch.Tensor]]) → Tuple[List[torch.Tensor], Optional[torch.Tensor]]

Forward pass of the module.

Parameters
Returns

Mixins

class nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin

Bases: abc.ABC

ASR BPE Mixin class that sets up a Tokenizer via a config

This mixin class adds the method _setup_tokenizer(…), which can be used by ASR models which depend on subword tokenization.

The setup_tokenizer method adds the following parameters to the class -

In addition to these variables, the method will also instantiate and preserve a tokenizer (subclass of TokenizerSpec) if successful, and assign it to self.tokenizer.

The mixin also supports aggregate tokenizers, which consist of ordinary, monolingual tokenizers. If a conversion between a monolongual and an aggregate tokenizer (or vice versa) is detected, all registered artifacts will be cleaned up.

save_tokenizers(directory: str)

Save the model tokenizer(s) to the specified directory.

Parameters

class nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin

Bases: nemo.collections.asr.parts.mixins.asr_adapter_mixins.ASRAdapterModelMixin

ASRModuleMixin is a mixin class added to ASR models in order to add methods that are specific to a particular instantiation of a module inside of an ASRModel.

Each method should first check that the module is present within the subclass, and support additional functionality if the corresponding module is present.

change_attention_model(self_attention_model: Optional[str] = None, att_context_size: Optional[List[int]] = None, update_config: bool = True)

Update the self_attention_model if function is available in encoder.

Parameters

self_attention_model (str) –
type of the attention layer and positional encoding

’rel_pos’:
relative positional embedding and Transformer-XL

’rel_pos_local_attn’:
relative positional embedding and Transformer-XL with local attention using overlapping windows. Attention context is determined by att_context_size parameter.

’abs_pos’:
absolute positional embedding and Transformer

If None is provided, the self_attention_model isn’t changed. Defauts to None.
att_context_size (List[int]) – List of 2 ints corresponding to left and right attention context sizes, or None to keep as it is. Defauts to None.
update_config (bool) – Whether to update the config or not with the new attention model. Defaults to True.

change_conv_asr_se_context_window(context_window: int, update_config: bool = True)

Update the context window of the SqueezeExcitation module if the provided model contains an encoder which is an instance of ConvASREncoder.

Parameters

change_subsampling_conv_chunking_factor(subsampling_conv_chunking_factor: int, update_config: bool = True)

Update the conv_chunking_factor (int) if function is available in encoder. Default is 1 (auto) Set it to -1 (disabled) or to a specific value (power of 2) if you OOM in the conv subsampling layers

Parameters

conformer_stream_step(processed_signal: torch.Tensor, processed_signal_length: Optional[torch.Tensor] = None, cache_last_channel: Optional[torch.Tensor] = None, cache_last_time: Optional[torch.Tensor] = None, cache_last_channel_len: Optional[torch.Tensor] = None, keep_all_outputs: bool = True, previous_hypotheses: Optional[List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]] = None, previous_pred_out: Optional[torch.Tensor] = None, drop_extra_pre_encoded: Optional[int] = None, return_transcription: bool = True, return_log_probs: bool = False)

It simulates a forward step with caching for streaming purposes. It supports the ASR models where their encoder supports streaming like Conformer. :param processed_signal: the input audio signals :param processed_signal_length: the length of the audios :param cache_last_channel: the cache tensor for last channel layers like MHA :param cache_last_channel_len: engths for cache_last_channel :param cache_last_time: the cache tensor for last time layers like convolutions :param keep_all_outputs: if set to True, would not drop the extra outputs specified by encoder.streaming_cfg.valid_out_len :param previous_hypotheses: the hypotheses from the previous step for RNNT models :param previous_pred_out: the predicted outputs from the previous step for CTC models :param drop_extra_pre_encoded: number of steps to drop from the beginning of the outputs after the downsampling module. This can be used if extra paddings are added on the left side of the input. :param return_transcription: whether to decode and return the transcriptions. It can not get disabled for Transducer models. :param return_log_probs: whether to return the log probs, only valid for ctc model

Returns
Return type

transcribe_simulate_cache_aware_streaming(paths2audio_files: List[str], batch_size: int = 4, logprobs: bool = False, return_hypotheses: bool = False, online_normalization: bool = False)

Parameters
Returns

class nemo.collections.asr.parts.mixins.transcription.TranscriptionMixin

Bases: abc.ABC

An abstract class for transcribe-able models.

Creates a template function transcribe() that provides an interface to perform transcription of audio tensors or filepaths.

The following abstract classes must be implemented by the subclass:

_transcribe_input_manifest_processing():
Process the provided input arguments (filepaths only) and return a config dict for the dataloader. The data loader is should generally operate on NeMo manifests.
_setup_transcribe_dataloader():
Setup the dataloader for transcription. Receives the output from _transcribe_input_manifest_processing().
_transcribe_forward():
Implements the model’s custom forward pass to return outputs that are processed by _transcribe_output_processing().
_transcribe_output_processing():
Implements the post processing of the model’s outputs to return the results to the user. The result can be a list of objects, list of list of objects, tuple of objects, tuple of list of objects, or a dict of list of objects.

transcribe(audio: Union[str, List[str], numpy.ndarray], batch_size: int = 4, return_hypotheses: bool = False, num_workers: int = 0, channel_selector: Optional[Union[int, Iterable[int], str]] = None, augmentor: Optional[omegaconf.DictConfig] = None, verbose: bool = True, override_config: Optional[nemo.collections.asr.parts.mixins.transcription.TranscribeConfig] = None, **config_kwargs) → Union[List[Any], List[List[Any]], Tuple[Any], Tuple[List[Any]], Dict[str, List[Any]]]

Template function that defines the execution strategy for transcribing audio.

Parameters
Returns

transcribe_generator(audio, override_config: Optional[nemo.collections.asr.parts.mixins.transcription.TranscribeConfig])

class nemo.collections.asr.parts.mixins.transcription.TranscribeConfig(batch_size: int = 4, return_hypotheses: bool = False, num_workers: Optional[int] = None, channel_selector: Union[int, Iterable[int], str] = None, augmentor: Optional[omegaconf.DictConfig] = None, verbose: bool = True, partial_hypothesis: Optional[List[Any]] = None, _internal: Optional[nemo.collections.asr.parts.mixins.transcription.InternalTranscribeConfig] = None)

class nemo.collections.asr.parts.mixins.interctc_mixin.InterCTCMixin

Bases: object

Adds utilities for computing interCTC loss from https://arxiv.org/abs/2102.03216.

To use, make sure encoder accesses interctc['capture_layers'] property in the AccessMixin and registers interctc/layer_output_X and interctc/layer_length_X for all layers that we want to get loss from. Additionally, specify the following config parameters to set up loss:

Copy
Copied!

            
            interctc:
    # can use different values
    loss_weights: [0.3]
    apply_at_layers: [8]

Then call

self.setup_interctc(ctc_decoder_name, ctc_loss_name, ctc_wer_name) in the init method
self.add_interctc_losses after computing regular loss.
self.finalize_interctc_metrics(metrics, outputs, prefix="val_") in the multi_validation_epoch_end method.
self.finalize_interctc_metrics(metrics, outputs, prefix="test_") in the multi_test_epoch_end method.

add_interctc_losses(loss_value: torch.Tensor, transcript: torch.Tensor, transcript_len: torch.Tensor, compute_wer: bool, compute_loss: bool = True, log_wer_num_denom: bool = False, log_prefix: str = '') → Tuple[Optional[torch.Tensor], Dict]

Adding interCTC losses if required.

Will also register loss/wer metrics in the returned dictionary.

Parameters
Returns
Return type

finalize_interctc_metrics(metrics: Dict, outputs: List[Dict], prefix: str)

get_captured_interctc_tensors() → List[Tuple[torch.Tensor, torch.Tensor]]

get_interctc_param(param_name)

is_interctc_enabled() → bool

set_interctc_enabled(enabled: bool)

set_interctc_param(param_name, param_value)

setup_interctc(decoder_name, loss_name, wer_name)

Datasets

Character Encoding Datasets

class nemo.collections.asr.data.audio_to_text.AudioToCharDataset(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.data.audio_to_text._AudioTextDataset

Dataset that loads tensors via a json file containing paths to audio files, transcripts, and durations (in seconds). Each new line is a different sample. Example below: {“audio_filepath”: “/path/to/audio.wav”, “text_filepath”: “/path/to/audio.txt”, “duration”: 23.147} … {“audio_filepath”: “/path/to/audio.wav”, “text”: “the transcription”, “offset”: 301.75, “duration”: 0.82, “utt”: “utterance_id”, “ctm_utt”: “en_4156”, “side”: “A”}

Parameters

property output_types: Optional[Dict[str, nemo.core.neural_types.neural_type.NeuralType]]

class nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.data.audio_to_text._TarredAudioToTextDataset

A similar Dataset to the AudioToCharDataset, but which loads tarred audio files.

Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToCharDataset), as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should contain the information for one audio file, including at least the transcript and name of the audio file within the tarball.

Valid formats for the audio_tar_filepaths argument include: (1) a single string that can be brace-expanded, e.g. ‘path/to/audio.tar’ or ‘path/to/audio_{1..100}.tar.gz’, or (2) a list of file paths that will not be brace-expanded, e.g. [‘audio_1.tar’, ‘audio_2.tar’, …].

See the WebDataset documentation for more information about accepted data and input formats.

If using multiple workers the number of shards should be divisible by world_size to ensure an even split among workers. If it is not divisible, logging will give a warning but training will proceed. In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering is applied. We currently do not check for this, but your program may hang if the shards are uneven!

Notice that a few arguments are different from the AudioToCharDataset; for example, shuffle (bool) has been replaced by shuffle_n (int).

Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest after filtering. An incorrect manifest length may lead to some DataLoader issues down the line.

Parameters

Text-to-Text Datasets for Hybrid ASR-TTS models

class nemo.collections.asr.data.text_to_text.TextToTextDataset(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.data.text_to_text.TextToTextDatasetBase, nemo.core.classes.dataset.Dataset

Text-to-Text Map-style Dataset for hybrid ASR-TTS models

collate_fn(batch: List[Union[nemo.collections.asr.data.text_to_text.TextToTextItem, tuple]]) → Union[nemo.collections.asr.data.text_to_text.TextToTextBatch, nemo.collections.asr.data.text_to_text.TextOrAudioToTextBatch, tuple]

class nemo.collections.asr.data.text_to_text.TextToTextIterableDataset(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.data.text_to_text.TextToTextDatasetBase, nemo.core.classes.dataset.IterableDataset

Text-to-Text Iterable Dataset for hybrid ASR-TTS models Only part necessary for current process should be loaded and stored

collate_fn(batch: List[Union[nemo.collections.asr.data.text_to_text.TextToTextItem, tuple]]) → Union[nemo.collections.asr.data.text_to_text.TextToTextBatch, nemo.collections.asr.data.text_to_text.TextOrAudioToTextBatch, tuple]

Subword Encoding Datasets

class nemo.collections.asr.data.audio_to_text.AudioToBPEDataset(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.data.audio_to_text._AudioTextDataset

In practice, the dataset and manifest used for character encoding and byte pair encoding are exactly the same. The only difference lies in how the dataset tokenizes the text in the manifest.

Parameters

property output_types: Optional[Dict[str, nemo.core.neural_types.neural_type.NeuralType]]

class nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.data.audio_to_text._TarredAudioToTextDataset

A similar Dataset to the AudioToBPEDataset, but which loads tarred audio files.

Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToBPEDataset), as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should contain the information for one audio file, including at least the transcript and name of the audio file within the tarball.

See the WebDataset documentation for more information about accepted data and input formats.

Notice that a few arguments are different from the AudioToBPEDataset; for example, shuffle (bool) has been replaced by shuffle_n (int).

Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest after filtering. An incorrect manifest length may lead to some DataLoader issues down the line.

Parameters

Audio Preprocessors

class nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.modules.audio_preprocessing.AudioPreprocessor, nemo.core.classes.exportable.Exportable

Featurizer module that converts wavs to mel spectrograms.

Parameters

input_example(max_batch: int = 8, max_dim: int = 32000, min_length: int = 200)

property input_types

property output_types

Returns definitions of module output ports.

processed_signal:
processed_length:

classmethod restore_from(restore_path: str)

Restores model instance (weights and configuration) from a .nemo file

Parameters

save_to(save_path: str)

Standardized method to save a tarfile containing the checkpoint, config, and any additional artifacts. Implemented via nemo.core.connectors.save_restore_connector.SaveRestoreConnector.save_to().

Parameters

class nemo.collections.asr.modules.AudioToMFCCPreprocessor(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.modules.audio_preprocessing.AudioPreprocessor

Preprocessor that converts wavs to MFCCs. Uses torchaudio.transforms.MFCC.

Parameters

property input_types

property output_types

classmethod restore_from(restore_path: str)

Restores model instance (weights and configuration) from a .nemo file

Parameters

save_to(save_path: str)

Standardized method to save a tarfile containing the checkpoint, config, and any additional artifacts. Implemented via nemo.core.connectors.save_restore_connector.SaveRestoreConnector.save_to().

Parameters

Audio Augmentors

class nemo.collections.asr.modules.SpectrogramAugmentation(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule

Performs time and freq cuts in one of two ways. SpecAugment zeroes out vertical and horizontal sections as described in SpecAugment (https://arxiv.org/abs/1904.08779). Arguments for use with SpecAugment are freq_masks, time_masks, freq_width, and time_width. SpecCutout zeroes out rectangulars as described in Cutout (https://arxiv.org/abs/1708.04552). Arguments for use with Cutout are rect_masks, rect_freq, and rect_time.

Parameters

property input_types

property output_types

class nemo.collections.asr.modules.CropOrPadSpectrogramAugmentation(*args: Any, **kwargs: Any)

Bases: nemo.core.classes.module.NeuralModule

Pad or Crop the incoming Spectrogram to a certain shape.

Parameters

property input_types

property output_types

classmethod restore_from(restore_path: str)

Restores model instance (weights and configuration) from a .nemo file

Parameters

save_to(save_path: str)

Standardized method to save a tarfile containing the checkpoint, config, and any additional artifacts. Implemented via nemo.core.connectors.save_restore_connector.SaveRestoreConnector.save_to().

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.SpeedPerturbation(sr, resample_type, min_speed_rate=0.9, max_speed_rate=1.1, num_rates=5, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Performs Speed Augmentation by re-sampling the data to a different sampling rate, which does not preserve pitch.

Note: This is a very slow operation for online augmentation. If space allows, it is preferable to pre-compute and save the files to augment the dataset.

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.TimeStretchPerturbation(min_speed_rate=0.9, max_speed_rate=1.1, num_rates=5, n_fft=512, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Time-stretch an audio series by a fixed rate while preserving pitch, based on [1, 2].

Note: This is a simplified implementation, intended primarily for reference and pedagogical purposes. It makes no attempt to handle transients, and is likely to produce audible artifacts.

Reference [1] [Ellis, D. P. W. “A phase vocoder in Matlab.” Columbia University, 2002.] (http://www.ee.columbia.edu/~dpwe/resources/matlab/pvoc/) [2] [librosa.effects.time_stretch] (https://librosa.github.io/librosa/generated/librosa.effects.time_stretch.html)

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.GainPerturbation(min_gain_dbfs=- 10, max_gain_dbfs=10, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Applies random gain to the audio.

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.ImpulsePerturbation(manifest_path=None, audio_tar_filepaths=None, shuffle_n=128, normalize_impulse=False, shift_impulse=False, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Convolves audio with a Room Impulse Response.

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.ShiftPerturbation(min_shift_ms=- 5.0, max_shift_ms=5.0, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Perturbs audio by shifting the audio in time by a random amount between min_shift_ms and max_shift_ms. The final length of the audio is kept unaltered by padding the audio with zeros.

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.NoisePerturbation(manifest_path=None, min_snr_db=10, max_snr_db=50, max_gain_db=300.0, rng=None, audio_tar_filepaths=None, shuffle_n=100, orig_sr=16000)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Perturbation that adds noise to input audio.

Parameters

perturb(data, ref_mic=0)

Parameters

perturb_with_foreground_noise(data, noise, data_rms=None, max_noise_dur=2, max_additions=1, ref_mic=0)

Parameters

perturb_with_input_noise(data, noise, data_rms=None, ref_mic=0)

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.WhiteNoisePerturbation(min_level=- 90, max_level=- 46, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Perturbation that adds white noise to an audio file in the training dataset.

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.RirAndNoisePerturbation(rir_manifest_path=None, rir_prob=0.5, noise_manifest_paths=None, noise_prob=1.0, min_snr_db=0, max_snr_db=50, rir_tar_filepaths=None, rir_shuffle_n=100, noise_tar_filepaths=None, apply_noise_rir=False, orig_sample_rate=None, max_additions=5, max_duration=2.0, bg_noise_manifest_paths=None, bg_noise_prob=1.0, bg_min_snr_db=10, bg_max_snr_db=50, bg_noise_tar_filepaths=None, bg_orig_sample_rate=None, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

RIR augmentation with additive foreground and background noise. In this implementation audio data is augmented by first convolving the audio with a Room Impulse Response and then adding foreground noise and background noise at various SNRs. RIR, foreground and background noises should either be supplied with a manifest file or as tarred audio files (faster).

Different sets of noise audio files based on the original sampling rate of the noise. This is useful while training a mixed sample rate model. For example, when training a mixed model with 8 kHz and 16 kHz audio with a target sampling rate of 16 kHz, one would want to augment 8 kHz data with 8 kHz noise rather than 16 kHz noise.

Parameters

class nemo.collections.asr.parts.preprocessing.perturb.TranscodePerturbation(codecs=None, rng=None)

Bases: nemo.collections.asr.parts.preprocessing.perturb.Perturbation

Audio codec augmentation. This implementation uses sox to transcode audio with low rate audio codecs, so users need to make sure that the installed sox version supports the codecs used here (G711 and amr-nb).

Parameters

Miscellaneous Classes

CTC Decoding

class nemo.collections.asr.parts.submodules.ctc_decoding.CTCDecoding(decoding_cfg, vocabulary)

Bases: nemo.collections.asr.parts.submodules.ctc_decoding.AbstractCTCDecoding

Used for performing CTC auto-regressive / non-auto-regressive decoding of the logprobs for character based models.

Parameters

decoding_cfg –
A dict-like object which contains the following key-value pairs.
strategy:
str value which represents the type of decoding that can occur. Possible values are :
- greedy (for greedy decoding).
- beam (for DeepSpeed KenLM based decoding).
compute_timestamps:
A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrite intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary.

ctc_timestamp_type:
A str value, which represents the types of timestamps that should be calculated. Can take the following values - “char” for character/subword time stamps, “word” for word level time stamps and “all” (default), for both character level and word level time stamps.

word_seperator:
Str token representing the seperator between words.

preserve_alignments:
Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for logprobs in it. Here, logprobs is a torch.Tensors.

confidence_cfg:
A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize ctc_decoder_predictions_tensor function with the preserve_frame_confidence flag set to True.
preserve_frame_confidence:
Bool flag which preserves the history of per-frame confidence scores generated during decoding. When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, frame_confidence is a List of floats.

preserve_token_confidence:
Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for token_confidence in it. Here, token_confidence is a List of floats.

The length of the list corresponds to the number of recognized tokens.

preserve_word_confidence:
Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for word_confidence in it. Here, word_confidence is a List of floats.

The length of the list corresponds to the number of recognized words.

exclude_blank:
Bool flag indicating that blank token confidence scores are to be excluded from the token_confidence.

aggregation:
Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are mean, min, max, prod.

tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and
attached to the regular frame confidence, making TDT frame confidence element a pair: (prediction_confidence, duration_confidence).

method_cfg:
A dict-like object which contains the method name and settings to compute per-frame confidence scores.

name:
The method name (str). Supported values:

’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.

entropy_type:
Which type of entropy to use (str). Used if confidence_method_cfg.name is set to entropy. Supported values:

’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy

alpha:
Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm:
A mapping of the entropy value to the interval [0,1]. Supported values:

’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.
batch_dim_index:
Index of the batch dimension of targets and predictions parameters of ctc_decoder_predictions_tensor methods. Can be either 0 or 1.
The config may further contain the following sub-dictionaries:

”greedy”:
preserve_alignments: Same as above, overrides above value. compute_timestamps: Same as above, overrides above value. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg.

”beam”:

beam_size:
int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above.

return_best_hypothesis:
optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default.

beam_alpha:
float, the strength of the Language model on the final score of a token. final_score = acoustic_score + beam_alpha * lm_score + beam_beta * seq_length.

beam_beta:
float, the strength of the sequence length penalty on the final score of a token. final_score = acoustic_score + beam_alpha * lm_score + beam_beta * seq_length.

kenlm_path:
str, path to a KenLM ARPA or .binary file (depending on the strategy chosen). If the path is invalid (file is not found at path), will raise a deferred error at the moment of calculation of beam search, so that users may update / change the decoding strategy to point to the correct file.
blank_id – The id of the RNNT blank token.

decode_ids_to_tokens(tokens: List[int]) → List[str]

Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id.

Parameters
Returns

decode_tokens_to_str(tokens: List[int]) → str

Implemented by subclass in order to decoder a token list into a string.

Parameters
Returns

class nemo.collections.asr.parts.submodules.ctc_decoding.CTCBPEDecoding(decoding_cfg, tokenizer: nemo.collections.common.tokenizers.tokenizer_spec.TokenizerSpec)

Bases: nemo.collections.asr.parts.submodules.ctc_decoding.AbstractCTCDecoding

Used for performing CTC auto-regressive / non-auto-regressive decoding of the logprobs for subword based models.

Parameters

decoding_cfg –
A dict-like object which contains the following key-value pairs.
strategy:
str value which represents the type of decoding that can occur. Possible values are :
- greedy (for greedy decoding).
- beam (for DeepSpeed KenLM based decoding).
compute_timestamps:
A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrite intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary.

ctc_timestamp_type:
A str value, which represents the types of timestamps that should be calculated. Can take the following values - “char” for character/subword time stamps, “word” for word level time stamps and “all” (default), for both character level and word level time stamps.

word_seperator:
Str token representing the seperator between words.

preserve_alignments:
Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for logprobs in it. Here, logprobs is a torch.Tensors.

confidence_cfg:
A dict-like object which contains the following key-value pairs related to confidence scores. In order to obtain hypotheses with confidence scores, please utilize ctc_decoder_predictions_tensor function with the preserve_frame_confidence flag set to True.
preserve_frame_confidence:
Bool flag which preserves the history of per-frame confidence scores generated during decoding. When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, frame_confidence is a List of floats.

preserve_token_confidence:
Bool flag which preserves the history of per-token confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for token_confidence in it. Here, token_confidence is a List of floats.

The length of the list corresponds to the number of recognized tokens.

preserve_word_confidence:
Bool flag which preserves the history of per-word confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for word_confidence in it. Here, word_confidence is a List of floats.

The length of the list corresponds to the number of recognized words.

exclude_blank:
Bool flag indicating that blank token confidence scores are to be excluded from the token_confidence.

aggregation:
Which aggregation type to use for collapsing per-token confidence into per-word confidence. Valid options are mean, min, max, prod.

tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and
attached to the regular frame confidence, making TDT frame confidence element a pair: (prediction_confidence, duration_confidence).

method_cfg:
A dict-like object which contains the method name and settings to compute per-frame confidence scores.

name:
The method name (str). Supported values:

’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.

entropy_type:
Which type of entropy to use (str). Used if confidence_method_cfg.name is set to entropy. Supported values:

’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy

alpha:
Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm:
A mapping of the entropy value to the interval [0,1]. Supported values:

’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.
batch_dim_index:
Index of the batch dimension of targets and predictions parameters of ctc_decoder_predictions_tensor methods. Can be either 0 or 1.
The config may further contain the following sub-dictionaries:

”greedy”:
preserve_alignments: Same as above, overrides above value. compute_timestamps: Same as above, overrides above value. preserve_frame_confidence: Same as above, overrides above value. confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg.

”beam”:

beam_size:
int, defining the beam size for beam search. Must be >= 1. If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above.

return_best_hypothesis:
optional bool, whether to return just the best hypothesis or all of the hypotheses after beam search has concluded. This flag is set by default.

beam_alpha:
float, the strength of the Language model on the final score of a token. final_score = acoustic_score + beam_alpha * lm_score + beam_beta * seq_length.

beam_beta:
float, the strength of the sequence length penalty on the final score of a token. final_score = acoustic_score + beam_alpha * lm_score + beam_beta * seq_length.

kenlm_path:
str, path to a KenLM ARPA or .binary file (depending on the strategy chosen). If the path is invalid (file is not found at path), will raise a deferred error at the moment of calculation of beam search, so that users may update / change the decoding strategy to point to the correct file.
tokenizer – NeMo tokenizer object, which inherits from TokenizerSpec.

decode_ids_to_tokens(tokens: List[int]) → List[str]

Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id.

Parameters
Returns

decode_tokens_to_str(tokens: List[int]) → str

Implemented by subclass in order to decoder a token list into a string.

Parameters
Returns

class nemo.collections.asr.parts.submodules.ctc_greedy_decoding.GreedyCTCInfer(blank_id: int, preserve_alignments: bool = False, compute_timestamps: bool = False, preserve_frame_confidence: bool = False, confidence_method_cfg: Optional[omegaconf.DictConfig] = None)

Bases: nemo.core.classes.common.Typing, nemo.collections.asr.parts.utils.asr_confidence_utils.ConfidenceMethodMixin

A greedy CTC decoder.

Provides a common abstraction for sample level and batch level greedy decoding.

Parameters

blank_index – int index of the blank token. Can be 0 or len(vocabulary).
preserve_alignments – Bool flag which preserves the history of logprobs generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for logprobs in it. Here, logprobs is a torch.Tensors.
compute_timestamps – A bool flag, which determines whether to compute the character/subword, or word based timestamp mapping the output log-probabilities to discrite intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary.
preserve_frame_confidence – Bool flag which preserves the history of per-frame confidence scores generated during decoding. When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, frame_confidence is a List of floats.
confidence_method_cfg –
A dict-like object which contains the method name and settings to compute per-frame confidence scores.
name: The method name (str).
Supported values:
’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.
entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to entropy.
Supported values:
’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy
alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0.
When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm: A mapping of the entropy value to the interval [0,1].
Supported values:
’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.

forward(decoder_output: torch.Tensor, decoder_lengths: torch.Tensor)

Returns a list of hypotheses given an input batch of the encoder hidden embedding. Output token is generated auto-repressively.

Parameters
Returns

property input_types

property output_types

class nemo.collections.asr.parts.submodules.ctc_beam_decoding.BeamCTCInfer(blank_id: int, beam_size: int, search_type: str = 'default', return_best_hypothesis: bool = True, preserve_alignments: bool = False, compute_timestamps: bool = False, beam_alpha: float = 1.0, beam_beta: float = 0.0, kenlm_path: Optional[str] = None, flashlight_cfg: Optional[nemo.collections.asr.parts.submodules.ctc_beam_decoding.FlashlightConfig] = None, pyctcdecode_cfg: Optional[nemo.collections.asr.parts.submodules.ctc_beam_decoding.PyCTCDecodeConfig] = None)

Bases: nemo.collections.asr.parts.submodules.ctc_beam_decoding.AbstractBeamCTCInfer

A greedy CTC decoder.

Provides a common abstraction for sample level and batch level greedy decoding.

Parameters

default_beam_search(x: torch.Tensor, out_len: torch.Tensor) → List[Union[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis, nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses]]

Open Seq2Seq Beam Search Algorithm (DeepSpeed)

Parameters
Returns

flashlight_beam_search(x: torch.Tensor, out_len: torch.Tensor) → List[Union[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis, nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses]]

Flashlight Beam Search Algorithm. Should support Char and Subword models.

Parameters
Returns

forward(decoder_output: torch.Tensor, decoder_lengths: torch.Tensor) → Tuple[List[Union[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis, nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses]]]

Returns a list of hypotheses given an input batch of the encoder hidden embedding. Output token is generated auto-repressively.

Parameters
Returns

set_decoding_type(decoding_type: str)

Sets the decoding type of the framework. Can support either char or subword models.

Parameters

RNNT Decoding

class nemo.collections.asr.parts.submodules.rnnt_decoding.RNNTDecoding(decoding_cfg, decoder, joint, vocabulary)

Bases: nemo.collections.asr.parts.submodules.rnnt_decoding.AbstractRNNTDecoding

Used for performing RNN-T auto-regressive decoding of the Decoder+Joint network given the encoder state.

Parameters

decoding_cfg –
A dict-like object which contains the following key-value pairs.
strategy:
str value which represents the type of decoding that can occur. Possible values are :
- greedy, greedy_batch (for greedy decoding).
- beam, tsd, alsd (for beam search decoding).
compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded
tokens as well as the decoded string. Default is False in order to avoid double decoding unless required.

preserve_alignments: Bool flag which preserves the history of logprobs generated during
decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for logprobs in it. Here, alignments is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)).

In order to obtain this hypothesis, please utilize rnnt_decoder_predictions_tensor function with the return_hypotheses flag set to True.

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti.

confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence
scores. In order to obtain hypotheses with confidence scores, please utilize rnnt_decoder_predictions_tensor function with the preserve_frame_confidence flag set to True.
preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores
generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, alignments is a List of List of floats.

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti.

preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores
generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for token_confidence in it. Here, token_confidence is a List of floats.

The length of the list corresponds to the number of recognized tokens.

preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores
generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for word_confidence in it. Here, word_confidence is a List of floats.

The length of the list corresponds to the number of recognized words.

exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded
from the token_confidence.

aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence.
Valid options are mean, min, max, prod.

tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and
attached to the regular frame confidence, making TDT frame confidence element a pair: (prediction_confidence, duration_confidence).

method_cfg: A dict-like object which contains the method name and settings to compute per-frame
confidence scores.

name:
The method name (str). Supported values:

’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.

entropy_type:
Which type of entropy to use (str). Used if confidence_method_cfg.name is set to entropy. Supported values:

’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy

alpha:
Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0. When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm:
A mapping of the entropy value to the interval [0,1]. Supported values:

’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.
The config may further contain the following sub-dictionaries:

”greedy”:

max_symbols: int, describing the maximum number of target tokens to decode per
timestep during greedy decoding. Setting to larger values allows longer sentences to be decoded, at the cost of increased execution time.

preserve_frame_confidence: Same as above, overrides above value.

confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg.

”beam”:

beam_size: int, defining the beam size for beam search. Must be >= 1.
If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above.

score_norm: optional bool, whether to normalize the returned beam score in the hypotheses.
Set to True by default.

return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the
hypotheses after beam search has concluded. This flag is set by default.

tsd_max_sym_exp: optional int, determines number of symmetric expansions of the target symbols
per timestep of the acoustic model. Larger values will allow longer sentences to be decoded, at increased cost to execution time.

alsd_max_target_len: optional int or float, determines the potential maximum target sequence length.

If an integer is provided, it can decode sequences of that particular maximum length. If a float is provided, it can decode sequences of int(alsd_max_target_len * seq_len), where seq_len is the length of the acoustic model output (T).

NOTE:
If a float is provided, it can be greater than 1! By default, a float of 2.0 is used so that a target sequence can be at most twice as long as the acoustic model output length T.

maes_num_steps: Number of adaptive steps to take. From the paper, 2 steps is generally sufficient,
and can be reduced to 1 to improve decoding speed while sacrificing some accuracy. int > 0.

maes_prefix_alpha: Maximum prefix length in prefix search. Must be an integer, and is advised to keep this as 1
in order to reduce expensive beam search cost later. int >= 0.

maes_expansion_beta: Maximum number of prefix expansions allowed, in addition to the beam size.
Effectively, the number of hypothesis = beam_size + maes_expansion_beta. Must be an int >= 0, and affects the speed of inference since large values will perform large beam search in the next step.

maes_expansion_gamma: Float pruning threshold used in the prune-by-value step when computing the expansions.
The default (2.3) is selected from the paper. It performs a comparison (max_log_prob - gamma <= log_prob[v]) where v is all vocabulary indices in the Vocab set and max_log_prob is the “most” likely token to be predicted. Gamma therefore provides a margin of additional tokens which can be potential candidates for expansion apart from the “most likely” candidate. Lower values will reduce the number of expansions (by increasing pruning-by-value, thereby improving speed but hurting accuracy). Higher values will increase the number of expansions (by reducing pruning-by-value, thereby reducing speed but potentially improving accuracy). This is a hyper parameter to be experimentally tuned on a validation set.

softmax_temperature: Scales the logits of the joint prior to computing log_softmax.
decoder – The Decoder/Prediction network module.
joint – The Joint network module.
vocabulary – The vocabulary (excluding the RNNT blank token) which will be used for decoding.

decode_ids_to_langs(tokens: List[int]) → List[str]

Decode a token id list into language ID (LID) list.

Parameters
Returns

decode_ids_to_tokens(tokens: List[int]) → List[str]

Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id.

Parameters
Returns

decode_tokens_to_lang(tokens: List[int]) → str

Compute the most likely language ID (LID) string given the tokens.

Parameters
Returns

decode_tokens_to_str(tokens: List[int]) → str

Implemented by subclass in order to decoder a token list into a string.

Parameters
Returns

class nemo.collections.asr.parts.submodules.rnnt_decoding.RNNTBPEDecoding(decoding_cfg, decoder, joint, tokenizer: nemo.collections.common.tokenizers.tokenizer_spec.TokenizerSpec)

Bases: nemo.collections.asr.parts.submodules.rnnt_decoding.AbstractRNNTDecoding

Used for performing RNN-T auto-regressive decoding of the Decoder+Joint network given the encoder state.

Parameters

decoding_cfg –
A dict-like object which contains the following key-value pairs.
strategy:
str value which represents the type of decoding that can occur. Possible values are :
- greedy, greedy_batch (for greedy decoding).
- beam, tsd, alsd (for beam search decoding).
compute_hypothesis_token_set: A bool flag, which determines whether to compute a list of decoded
tokens as well as the decoded string. Default is False in order to avoid double decoding unless required.

preserve_alignments: Bool flag which preserves the history of logprobs generated during
decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for alignments in it. Here, alignments is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)).

In order to obtain this hypothesis, please utilize rnnt_decoder_predictions_tensor function with the return_hypotheses flag set to True.

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti.

compute_timestamps: A bool flag, which determines whether to compute the character/subword, or
word based timestamp mapping the output log-probabilities to discrete intervals of timestamps. The timestamps will be available in the returned Hypothesis.timestep as a dictionary.

compute_langs: a bool flag, which allows to compute language id (LID) information per token,
word, and the entire sample (most likely language id). The LIDS will be available in the returned Hypothesis object as a dictionary

rnnt_timestamp_type: A str value, which represents the types of timestamps that should be calculated.
Can take the following values - “char” for character/subword time stamps, “word” for word level time stamps and “all” (default), for both character level and word level time stamps.
word_seperator: Str token representing the seperator between words.
preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores
generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, alignments is a List of List of ints.

confidence_cfg: A dict-like object which contains the following key-value pairs related to confidence
scores. In order to obtain hypotheses with confidence scores, please utilize rnnt_decoder_predictions_tensor function with the preserve_frame_confidence flag set to True.
preserve_frame_confidence: Bool flag which preserves the history of per-frame confidence scores
generated during decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, alignments is a List of List of floats.

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti.

preserve_token_confidence: Bool flag which preserves the history of per-token confidence scores
generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for token_confidence in it. Here, token_confidence is a List of floats.

The length of the list corresponds to the number of recognized tokens.

preserve_word_confidence: Bool flag which preserves the history of per-word confidence scores
generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for word_confidence in it. Here, word_confidence is a List of floats.

The length of the list corresponds to the number of recognized words.

exclude_blank: Bool flag indicating that blank token confidence scores are to be excluded
from the token_confidence.

aggregation: Which aggregation type to use for collapsing per-token confidence into per-word confidence.
Valid options are mean, min, max, prod.

tdt_include_duration: Bool flag indicating that the duration confidence scores are to be calculated and
attached to the regular frame confidence, making TDT frame confidence element a pair: (prediction_confidence, duration_confidence).

method_cfg: A dict-like object which contains the method name and settings to compute per-frame
confidence scores.

name:
The method name (str). Supported values:

’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.

entropy_type: Which type of entropy to use (str).
Used if confidence_method_cfg.name is set to entropy. Supported values:

’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy

alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0.
When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm: A mapping of the entropy value to the interval [0,1].
Supported values:

’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.
The config may further contain the following sub-dictionaries:

”greedy”:

max_symbols: int, describing the maximum number of target tokens to decode per
timestep during greedy decoding. Setting to larger values allows longer sentences to be decoded, at the cost of increased execution time.

preserve_frame_confidence: Same as above, overrides above value.

confidence_method_cfg: Same as above, overrides confidence_cfg.method_cfg.

”beam”:

beam_size: int, defining the beam size for beam search. Must be >= 1.
If beam_size == 1, will perform cached greedy search. This might be slightly different results compared to the greedy search above.

score_norm: optional bool, whether to normalize the returned beam score in the hypotheses.
Set to True by default.

return_best_hypothesis: optional bool, whether to return just the best hypothesis or all of the
hypotheses after beam search has concluded.

tsd_max_sym_exp: optional int, determines number of symmetric expansions of the target symbols
per timestep of the acoustic model. Larger values will allow longer sentences to be decoded, at increased cost to execution time.

alsd_max_target_len: optional int or float, determines the potential maximum target sequence length.

If an integer is provided, it can decode sequences of that particular maximum length. If a float is provided, it can decode sequences of int(alsd_max_target_len * seq_len), where seq_len is the length of the acoustic model output (T).

NOTE:
If a float is provided, it can be greater than 1! By default, a float of 2.0 is used so that a target sequence can be at most twice as long as the acoustic model output length T.

maes_num_steps: Number of adaptive steps to take. From the paper, 2 steps is generally sufficient,
and can be reduced to 1 to improve decoding speed while sacrificing some accuracy. int > 0.

maes_prefix_alpha: Maximum prefix length in prefix search. Must be an integer, and is advised to keep this as 1
in order to reduce expensive beam search cost later. int >= 0.

maes_expansion_beta: Maximum number of prefix expansions allowed, in addition to the beam size.
Effectively, the number of hypothesis = beam_size + maes_expansion_beta. Must be an int >= 0, and affects the speed of inference since large values will perform large beam search in the next step.

maes_expansion_gamma: Float pruning threshold used in the prune-by-value step when computing the expansions.
The default (2.3) is selected from the paper. It performs a comparison (max_log_prob - gamma <= log_prob[v]) where v is all vocabulary indices in the Vocab set and max_log_prob is the “most” likely token to be predicted. Gamma therefore provides a margin of additional tokens which can be potential candidates for expansion apart from the “most likely” candidate. Lower values will reduce the number of expansions (by increasing pruning-by-value, thereby improving speed but hurting accuracy). Higher values will increase the number of expansions (by reducing pruning-by-value, thereby reducing speed but potentially improving accuracy). This is a hyper parameter to be experimentally tuned on a validation set.

softmax_temperature: Scales the logits of the joint prior to computing log_softmax.
decoder – The Decoder/Prediction network module.
joint – The Joint network module.
tokenizer – The tokenizer which will be used for decoding.

decode_hypothesis(hypotheses_list: List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]) → List[Union[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis, nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses]]

Decode a list of hypotheses into a list of strings. Overrides the super() method optionally adding lang information

Parameters
Returns

decode_ids_to_langs(tokens: List[int]) → List[str]

Decode a token id list into language ID (LID) list.

Parameters
Returns

decode_ids_to_tokens(tokens: List[int]) → List[str]

Implemented by subclass in order to decode a token id list into a token list. A token list is the string representation of each token id.

Parameters
Returns

decode_tokens_to_lang(tokens: List[int]) → str

Compute the most likely language ID (LID) string given the tokens.

Parameters
Returns

decode_tokens_to_str(tokens: List[int]) → str

Implemented by subclass in order to decoder a token list into a string.

Parameters
Returns

class nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyRNNTInfer(decoder_model: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTDecoder, joint_model: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTJoint, blank_index: int, max_symbols_per_step: Optional[int] = None, preserve_alignments: bool = False, preserve_frame_confidence: bool = False, confidence_method_cfg: Optional[omegaconf.DictConfig] = None)

Bases: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding._GreedyRNNTInfer

A greedy transducer decoder.

Sequence level greedy decoding, performed auto-regressively.

Parameters

decoder_model – rnnt_utils.AbstractRNNTDecoder implementation.
joint_model – rnnt_utils.AbstractRNNTJoint implementation.
blank_index – int index of the blank token. Can be 0 or len(vocabulary).
max_symbols_per_step – Optional int. The maximum number of symbols that can be added to a sequence in a single time step; if set to None then there is no limit.
preserve_alignments –
Bool flag which preserves the history of alignments generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for alignments in it. Here, alignments is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)).

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti.
preserve_frame_confidence –
Bool flag which preserves the history of per-frame confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, frame_confidence is a List of List of floats.

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti.
confidence_method_cfg –
A dict-like object which contains the method name and settings to compute per-frame confidence scores.
name: The method name (str).
Supported values:
’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.
entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to entropy.
Supported values:
’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy
alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0.
When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm: A mapping of the entropy value to the interval [0,1].
Supported values:
’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.

forward(encoder_output: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]] = None)

Returns a list of hypotheses given an input batch of the encoder hidden embedding. Output token is generated auto-regressively.

Parameters
Returns

class nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedRNNTInfer(decoder_model: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTDecoder, joint_model: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTJoint, blank_index: int, max_symbols_per_step: Optional[int] = None, preserve_alignments: bool = False, preserve_frame_confidence: bool = False, confidence_method_cfg: Optional[omegaconf.DictConfig] = None, loop_labels: bool = True, use_cuda_graph_decoder: bool = False)

Bases: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding._GreedyRNNTInfer

A batch level greedy transducer decoder.

Batch level greedy decoding, performed auto-regressively.

Parameters

decoder_model – rnnt_utils.AbstractRNNTDecoder implementation.
joint_model – rnnt_utils.AbstractRNNTJoint implementation.
blank_index – int index of the blank token. Can be 0 or len(vocabulary).
max_symbols_per_step – Optional int. The maximum number of symbols that can be added to a sequence in a single time step; if set to None then there is no limit.
preserve_alignments –
Bool flag which preserves the history of alignments generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for alignments in it. Here, alignments is a List of List of Tuple(Tensor (of length V + 1), Tensor(scalar, label after argmax)).

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti.
preserve_frame_confidence –
Bool flag which preserves the history of per-frame confidence scores generated during greedy decoding (sample / batched). When set to true, the Hypothesis will contain the non-null value for frame_confidence in it. Here, frame_confidence is a List of List of floats.

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more confidence scores. U is the number of target tokens for the current timestep Ti.
confidence_method_cfg –
A dict-like object which contains the method name and settings to compute per-frame confidence scores.
name: The method name (str).
Supported values:
’max_prob’ for using the maximum token probability as a confidence.

’entropy’ for using a normalized entropy of a log-likelihood vector.
entropy_type: Which type of entropy to use (str). Used if confidence_method_cfg.name is set to entropy.
Supported values:
’gibbs’ for the (standard) Gibbs entropy. If the alpha (α) is provided,
the formula is the following: H_α = -sum_i((p^α_i)*log(p^α_i)). Note that for this entropy, the alpha should comply the following inequality: (log(V)+2-sqrt(log^2(V)+4))/(2*log(V)) <= α <= (1+log(V-1))/log(V-1) where V is the model vocabulary size.

’tsallis’ for the Tsallis entropy with the Boltzmann constant one.
Tsallis entropy formula is the following: H_α = 1/(α-1)*(1-sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/Tsallis_entropy

’renyi’ for the Rényi entropy.
Rényi entropy formula is the following: H_α = 1/(1-α)*log_2(sum_i(p^α_i)), where α is a parameter. When α == 1, it works like the Gibbs entropy. More: https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy
alpha: Power scale for logsoftmax (α for entropies). Here we restrict it to be > 0.
When the alpha equals one, scaling is not applied to ‘max_prob’, and any entropy type behaves like the Shannon entropy: H = -sum_i(p_i*log(p_i))

entropy_norm: A mapping of the entropy value to the interval [0,1].
Supported values:
’lin’ for using the linear mapping.

’exp’ for using exponential mapping with linear shift.
loop_labels – Switching between decoding algorithms. Both algorithms produce equivalent results. loop_labels=True (default) algorithm is faster (especially for large batches) but can use a bit more memory (negligible overhead compared to the amount of memory used by the encoder). loop_labels=False is an implementation of a traditional decoding algorithm, which iterates over frames (encoder output vectors), and in the inner loop, decodes labels for the current frame one by one, stopping when <blank> is found. loop_labels=True iterates over labels, on each step finding the next non-blank label (evaluating Joint multiple times in inner loop); It uses a minimal possible amount of calls to prediction network (with maximum possible batch size), which makes it especially useful for scaling the prediction network.

forward(encoder_output: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]] = None)

Returns a list of hypotheses given an input batch of the encoder hidden embedding. Output token is generated auto-regressively.

Parameters
Returns

class nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamRNNTInfer(decoder_model: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTDecoder, joint_model: nemo.collections.asr.modules.rnnt_abstract.AbstractRNNTJoint, beam_size: int, search_type: str = 'default', score_norm: bool = True, return_best_hypothesis: bool = True, tsd_max_sym_exp_per_step: Optional[int] = 50, alsd_max_target_len: Union[int, float] = 1.0, nsc_max_timesteps_expansion: int = 1, nsc_prefix_alpha: int = 1, maes_num_steps: int = 2, maes_prefix_alpha: int = 1, maes_expansion_gamma: float = 2.3, maes_expansion_beta: int = 2, language_model: Optional[Dict[str, Any]] = None, softmax_temperature: float = 1.0, preserve_alignments: bool = False, ngram_lm_model: Optional[str] = None, ngram_lm_alpha: float = 0.0, hat_subtract_ilm: bool = False, hat_ilm_weight: float = 0.0)

Bases: nemo.core.classes.common.Typing

Beam Search implementation ported from ESPNet implementation - https://github.com/espnet/espnet/blob/master/espnet/nets/beam_search_transducer.py

Sequence level beam decoding or batched-beam decoding, performed auto-repressively depending on the search type chosen.

Parameters

decoder_model – rnnt_utils.AbstractRNNTDecoder implementation.
joint_model – rnnt_utils.AbstractRNNTJoint implementation.
beam_size –
number of beams for beam search. Must be a positive integer >= 1. If beam size is 1, defaults to stateful greedy search. This greedy search might result in slightly different results than the greedy results obtained by GreedyRNNTInfer due to implementation differences.

For accurate greedy results, please use GreedyRNNTInfer or GreedyBatchedRNNTInfer.
search_type (# The following arguments are specific to the chosen) –
str representing the type of beam search to perform. Must be one of [‘beam’, ‘tsd’, ‘alsd’]. ‘nsc’ is currently not supported.

Algoritm used:

beam - basic beam search strategy. Larger beams generally result in better decoding,
however the time required for the search also grows steadily.

tsd - time synchronous decoding. Please refer to the paper:
[Alignment-Length Synchronous Decoding for RNN Transducer](https://ieeexplore.ieee.org/document/9053040) for details on the algorithm implemented.

Time synchronous decoding (TSD) execution time grows by the factor T * max_symmetric_expansions. For longer sequences, T is greater, and can therefore take a long time for beams to obtain good results. This also requires greater memory to execute.

alsd - alignment-length synchronous decoding. Please refer to the paper:
[Alignment-Length Synchronous Decoding for RNN Transducer](https://ieeexplore.ieee.org/document/9053040) for details on the algorithm implemented.

Alignment-length synchronous decoding (ALSD) execution time is faster than TSD, with growth factor of T + U_max, where U_max is the maximum target length expected during execution.

Generally, T + U_max < T * max_symmetric_expansions. However, ALSD beams are non-unique, therefore it is required to use larger beam sizes to achieve the same (or close to the same) decoding accuracy as TSD.

For a given decoding accuracy, it is possible to attain faster decoding via ALSD than TSD.

maes = modified adaptive expansion searcn. Please refer to the paper:
[Accelerating RNN Transducer Inference via Adaptive Expansion Search](https://ieeexplore.ieee.org/document/9250505)

Modified Adaptive Synchronous Decoding (mAES) execution time is adaptive w.r.t the number of expansions (for tokens) required per timestep. The number of expansions can usually be constrained to 1 or 2, and in most cases 2 is sufficient.

This beam search technique can possibly obtain superior WER while sacrificing some evaluation time.
score_norm – bool, whether to normalize the scores of the log probabilities.
return_best_hypothesis – bool, decides whether to return a single hypothesis (the best out of N), or return all N hypothesis (sorted with best score first). The container class changes based this flag - When set to True (default), returns a single Hypothesis. When set to False, returns a NBestHypotheses container, which contains a list of Hypothesis.
search_type –
tsd_max_sym_exp_per_step – Used for search_type=tsd. The maximum symmetric expansions allowed per timestep during beam search. Larger values should be used to attempt decoding of longer sequences, but this in turn increases execution time and memory usage.
alsd_max_target_len – Used for search_type=alsd. The maximum expected target sequence length during beam search. Larger values allow decoding of longer sequences at the expense of execution time and memory.
stabilized. (# The following two flags are placeholders and unused until nsc implementation is) –
nsc_max_timesteps_expansion – Unused int.
nsc_prefix_alpha – Unused int.
flags (# mAES) –
maes_num_steps – Number of adaptive steps to take. From the paper, 2 steps is generally sufficient. int > 1.
maes_prefix_alpha – Maximum prefix length in prefix search. Must be an integer, and is advised to keep this as 1 in order to reduce expensive beam search cost later. int >= 0.
maes_expansion_beta – Maximum number of prefix expansions allowed, in addition to the beam size. Effectively, the number of hypothesis = beam_size + maes_expansion_beta. Must be an int >= 0, and affects the speed of inference since large values will perform large beam search in the next step.
maes_expansion_gamma – Float pruning threshold used in the prune-by-value step when computing the expansions. The default (2.3) is selected from the paper. It performs a comparison (max_log_prob - gamma <= log_prob[v]) where v is all vocabulary indices in the Vocab set and max_log_prob is the “most” likely token to be predicted. Gamma therefore provides a margin of additional tokens which can be potential candidates for expansion apart from the “most likely” candidate. Lower values will reduce the number of expansions (by increasing pruning-by-value, thereby improving speed but hurting accuracy). Higher values will increase the number of expansions (by reducing pruning-by-value, thereby reducing speed but potentially improving accuracy). This is a hyper parameter to be experimentally tuned on a validation set.
softmax_temperature – Scales the logits of the joint prior to computing log_softmax.
preserve_alignments –
Bool flag which preserves the history of alignments generated during beam decoding (sample). When set to true, the Hypothesis will contain the non-null value for alignments in it. Here, alignments is a List of List of Tensor (of length V + 1).

The length of the list corresponds to the Acoustic Length (T). Each value in the list (Ti) is a torch.Tensor (U), representing 1 or more targets from a vocabulary. U is the number of target tokens for the current timestep Ti.

NOTE: preserve_alignments is an invalid argument for any search_type other than basic beam search.
ngram_lm_model – str The path to the N-gram LM
ngram_lm_alpha – float Alpha weight of N-gram LM
tokens_type – str Tokenization type [‘subword’, ‘char’]

align_length_sync_decoding(h: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis] = None) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Alignment-length synchronous beam search implementation. Based on https://ieeexplore.ieee.org/document/9053040

Parameters
Returns
Return type

compute_ngram_score(current_lm_state: kenlm.State, label: int) → Tuple[float, kenlm.State]

default_beam_search(h: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis] = None) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Beam search implementation.

Parameters
Returns
Return type

greedy_search(h: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis] = None) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Greedy search implementation for transducer. Generic case when beam size = 1. Results might differ slightly due to implementation details as compared to GreedyRNNTInfer and GreedyBatchRNNTInfer.

Parameters
Returns
Return type

property input_types

modified_adaptive_expansion_search(h: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis] = None) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Based on/modified from https://ieeexplore.ieee.org/document/9250505

Parameters
Returns
Return type

property output_types

prefix_search(hypotheses: List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis], enc_out: torch.Tensor, prefix_alpha: int) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

recombine_hypotheses(hypotheses: List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Recombine hypotheses with equivalent output sequence.

Parameters
Returns
Return type

resolve_joint_output(enc_out: torch.Tensor, dec_out: torch.Tensor) → Tuple[torch.Tensor, torch.Tensor]

sort_nbest(hyps: List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Sort hypotheses by score or score given sequence length.

Parameters
Returns
Return type

time_sync_decoding(h: torch.Tensor, encoded_lengths: torch.Tensor, partial_hypotheses: Optional[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis] = None) → List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]

Time synchronous beam search implementation. Based on https://ieeexplore.ieee.org/document/9053040

Parameters
Returns
Return type

Hypotheses

class nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis(score: float, y_sequence: typing.Union[typing.List[int], torch.Tensor], text: typing.Optional[str] = None, dec_out: typing.Optional[typing.List[torch.Tensor]] = None, dec_state: typing.Optional[typing.Union[typing.List[typing.List[torch.Tensor]], typing.List[torch.Tensor]]] = None, timestep: typing.Union[typing.List[int], torch.Tensor] = <factory>, alignments: typing.Optional[typing.Union[typing.List[int], typing.List[typing.List[int]]]] = None, frame_confidence: typing.Optional[typing.Union[typing.List[float], typing.List[typing.List[float]]]] = None, token_confidence: typing.Optional[typing.List[float]] = None, word_confidence: typing.Optional[typing.List[float]] = None, length: typing.Union[int, torch.Tensor] = 0, y: typing.Optional[typing.List[torch.tensor]] = None, lm_state: typing.Optional[typing.Union[typing.Dict[str, typing.Any], typing.List[typing.Any]]] = None, lm_scores: typing.Optional[torch.Tensor] = None, ngram_lm_state: typing.Optional[typing.Union[typing.Dict[str, typing.Any], typing.List[typing.Any]]] = None, tokens: typing.Optional[typing.Union[typing.List[int], torch.Tensor]] = None, last_token: typing.Optional[torch.Tensor] = None)

Bases: object

Hypothesis class for beam search algorithms.

score: A float score obtained from an AbstractRNNTDecoder module’s score_hypothesis method.

y_sequence: Either a sequence of integer ids pointing to some vocabulary, or a packed torch.Tensor

dec_state: A list (or list of list) of LSTM-RNN decoder states. Can be None.

text: (Optional) A decoded string after processing via CTC / RNN-T decoding (removing the CTC/RNNT
timestep: (Optional) A list of integer indices representing at which index in the decoding
alignments: (Optional) Represents the CTC / RNNT token alignments as integer tokens along an axis of
frame_confidence: (Optional) Represents the CTC / RNNT per-frame confidence scores as token probabilities
token_confidence: (Optional) Represents the CTC / RNNT per-token confidence scores as token probabilities
word_confidence: (Optional) Represents the CTC / RNNT per-word confidence scores as token probabilities
length: Represents the length of the sequence (the original length without padding), otherwise

y: (Unused) A list of torch.Tensors representing the list of hypotheses.

lm_state: (Unused) A dictionary state cache used by an external Language Model.

lm_scores: (Unused) Score of the external Language Model.

ngram_lm_state: (Optional) State of the external n-gram Language Model.

tokens: (Optional) A list of decoded tokens (can be characters or word-pieces.

last_token (Optional): A token or batch of tokens which was predicted in the last step.

class nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses(n_best_hypotheses: Optional[List[nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis]])

Adapter Networks

class nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.MultiHeadAttentionAdapter(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.parts.submodules.multi_head_attention.MultiHeadAttention, nemo.collections.common.parts.adapter_modules.AdapterModuleUtil

Multi-Head Attention layer of Transformer.

Parameters

forward(query, key, value, mask, pos_emb=None, cache=None)

Compute ‘Scaled Dot Product Attention’. :param query: (batch, time1, size) :type query: torch.Tensor :param key: (batch, time2, size) :type key: torch.Tensor :param value: (batch, time2, size) :type value: torch.Tensor :param mask: (batch, time1, time2) :type mask: torch.Tensor :param cache: (batch, time_cache, size) :type cache: torch.Tensor

Returns
Return type

get_default_strategy_config() → dataclasses.dataclass

class nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.RelPositionMultiHeadAttentionAdapter(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.parts.submodules.multi_head_attention.RelPositionMultiHeadAttention, nemo.collections.common.parts.adapter_modules.AdapterModuleUtil

Multi-Head Attention layer of Transformer-XL with support of relative positional encoding. Paper: https://arxiv.org/abs/1901.02860

Parameters

forward(query, key, value, mask, pos_emb, cache=None)

Compute ‘Scaled Dot Product Attention’ with rel. positional encoding. :param query: (batch, time1, size) :type query: torch.Tensor :param key: (batch, time2, size) :type key: torch.Tensor :param value: (batch, time2, size) :type value: torch.Tensor :param mask: (batch, time1, time2) :type mask: torch.Tensor :param pos_emb: (batch, time1, size) :type pos_emb: torch.Tensor :param cache: (batch, time_cache, size) :type cache: torch.Tensor

Returns
Return type

get_default_strategy_config() → dataclasses.dataclass

class nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.PositionalEncodingAdapter(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.parts.submodules.multi_head_attention.PositionalEncoding, nemo.collections.common.parts.adapter_modules.AdapterModuleUtil

Absolute positional embedding adapter.

Note

Absolute positional embedding value is added to the input tensor without residual connection ! Therefore, the input is changed, if you only require the positional embedding, drop the returned x !

Parameters

get_default_strategy_config() → dataclasses.dataclass

class nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.RelPositionalEncodingAdapter(*args: Any, **kwargs: Any)

Bases: nemo.collections.asr.parts.submodules.multi_head_attention.RelPositionalEncoding, nemo.collections.common.parts.adapter_modules.AdapterModuleUtil

Relative positional encoding for TransformerXL’s layers See : Appendix B in https://arxiv.org/abs/1901.02860

Note

Relative positional embedding value is not added to the input tensor ! Therefore, the input should be updated changed, if you only require the positional embedding, drop the returned x !

Parameters

get_default_strategy_config() → dataclasses.dataclass

Adapter Strategies

class nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.MHAResidualAddAdapterStrategy(stochastic_depth: float = 0.0, l2_lambda: float = 0.0)

Bases: nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategy

An implementation of residual addition of an adapter module with its input for the MHA Adapters.

forward(input: torch.Tensor, adapter: torch.nn.Module, *, module: AdapterModuleMixin)

A basic strategy, comprising of a residual connection over the input, after forward pass by the underlying adapter. Additional work is done to pack and unpack the dictionary of inputs and outputs.

Note: The value tensor is added to the output of the attention adapter as the residual connection.

Parameters

input –
A dictionary of multiple input arguments for the adapter module.

query, key, value: Original output tensor of the module, or the output of the
previous adapter (if more than one adapters are enabled).

mask: Attention mask.

pos_emb: Optional positional embedding for relative encoding.
adapter – The adapter module that is currently required to perform the forward pass.
module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns

The result tensor, after one of the active adapters has finished its forward passes.

compute_output(input: torch.Tensor, adapter: torch.nn.Module, *, module: AdapterModuleMixin) → torch.Tensor

Compute the output of a single adapter to some input.

Parameters
Returns

Previous NeMo ASR Configuration Files

Next Example With MCV