Automatic Speech Recognition (ASR)

ASR, or Automatic Speech Recognition, refers to the problem of getting a program to automatically transcribe spoken language (speech-to-text). Our goal is usually to have a model that minimizes the Word Error Rate (WER) metric when transcribing speech input. In other words, given some audio file (e.g. a WAV file) containing speech, how do we transform this into the corresponding text with as few errors as possible?

Traditional speech recognition takes a generative approach, modeling the full pipeline of how speech sounds are produced in order to evaluate a speech sample. We would start from a language model that encapsulates the most likely orderings of words that are generated (e.g. an n-gram model), to a pronunciation model for each word in that ordering (e.g. a pronunciation table), to an acoustic model that translates those pronunciations to audio waveforms (e.g. a Gaussian Mixture Model).

Then, if we receive some spoken input, our goal would be to find the most likely sequence of text that would result in the given audio according to our generative pipeline of models. Overall, with traditional speech recognition, we try to model Pr(audio|transcript)*Pr(transcript), and take the argmax of this over possible transcripts.

Over time, neural nets advanced to the point where each component of the traditional speech recognition model could be replaced by a neural model that had better performance and that had a greater potential for generalization. For example, we could replace an n-gram model with a neural language model, and replace a pronunciation table with a neural pronunciation model, and so on. However, each of these neural models need to be trained individually on different tasks, and errors in any model in the pipeline could throw off the whole prediction.

Thus, we can see the appeal of end-to-end ASR architectures: discriminative models that simply take an audio input and give a textual output, and in which all components of the architecture are trained together towards the same goal. The model’s encoder would be akin to an acoustic model for extracting speech features, which can then be directly piped to a decoder which outputs text. If desired, we could integrate a language model that would improve our predictions, as well.

And the entire end-to-end ASR model can be trained at once–a much easier pipeline to handle!

The full documentation tree is as follows:

• Models
  • Jasper
  • QuartzNet
  • Citrinet
  • ContextNet
  • Conformer-CTC
  • Conformer-Transducer
  • References
• Datasets
  • LibriSpeech
  • Fisher English Training Speech
  • 2000 HUB5 English Evaluation Speech
  • AN4 Dataset
  • Aishell-1
  • Aishell-2
  • Preparing Custom ASR Data
  • Tarred Datasets
    • Conversion to Tarred Datasets
• ASR Language Modeling
  • N-gram Language Modeling
    • Train N-gram LM
    • Evaluate by Beam Search Decoding and N-gram LM
      • Hyperparameter Grid Search
  • Neural Rescoring
    • Train Neural Rescorer
    • Evaluation
      • Hyperparameter Linear Search
• Checkpoints
  • Loading Local Checkpoints
  • NGC Pretrained Checkpoints
    • Transcribing/Inference
    • Fine-tuning on Different Datasets
  • Automatic Speech Recognition Models
    • Language Models for ASR
  • Speech Recognition (Languages)
    • English
    • Mandarin
    • German
    • Polish
    • Italian
    • Russian
    • Spanish
    • Catalan
• NeMo ASR Configuration Files
  • Dataset Configuration
  • Preprocessor Configuration
  • Augmentation Configurations
  • Tokenizer Configurations
  • Model Architecture Configurations
    • Jasper and QuartzNet
    • Citrinet
    • Conformer-CTC
    • ContextNet
    • Conformer-Transducer
  • Transducer Configurations
    • Model Defaults
    • Acoustic Encoder Model
    • Decoder / Prediction Model
    • Joint Model
      • Effect of Batch Splitting / Fused Batch step
      • Effect of Fused Batch Step
    • Transducer Decoding
    • Transducer Loss
      • FastEmit Regularization
  • Fine-tuning Configurations
    • Fine-tuning via a NeMo model
    • Fine-tuning via a NeMo pretrained model name
    • Fine-tuning via a Pytorch Lightning checkpoint
• NeMo ASR collection API
  • Model Classes
  • Modules
  • Parts
  • Mixins
  • Datasets
    • Character Encoding Datasets
    • Subword Encoding Datasets
  • Audio Preprocessors
  • Audio Augmentors
  • Miscellaneous Classes
    • RNNT Decoding
    • Hypotheses

Resources and Documentation

Hands-on speech recognition tutorial notebooks can be found under the ASR tutorials folder. If you are a beginner to NeMo, consider trying out the ASR with NeMo tutorial. This and most other tutorials can be run on Google Colab by specifying the link to the notebooks’ GitHub pages on Colab.

If you are looking for information about a particular ASR model, or would like to find out more about the model architectures available in the nemo_asr collection, refer to the Models section.

NeMo includes preprocessing scripts for several common ASR datasets. The Datasets section contains instructions on running those scripts. It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.

Information about how to load model checkpoints (either local files or pretrained ones from NGC), as well as a list of the checkpoints available on NGC are located on the Checkpoints section.

Documentation regarding the configuration files specific to the nemo_asr models can be found on the Configuration Files section.