Speaker Diarization

Speaker Diarization Overview

Speaker diarization is the process of segmenting audio recordings by speaker labels and aims to answer the question “who spoke when?”. Speaker diarization makes a clear distinction when it is compared with speech recognition. As shown in the figure below, before we perform speaker diarization, we know “what is spoken” yet we do not know “who spoke it”. Therefore, speaker diarization is an essential feature for a speech recognition system to enrich the transcription with speaker labels.

To figure out “who spoke when”, speaker diarization systems need to capture the characteristics of unseen speakers and tell apart which regions in the audio recording belong to which speaker. To achieve this, speaker diarization systems extract voice characteristics, count the number of speakers, then assign the audio segments to the corresponding speaker index.

Diarize speech with 3 lines of code

After installing NeMo, you can diarize an audio file as follows:

from nemo.collections.asr.models import SortformerEncLabelModel
diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2.1").eval()
predicted_segments = diar_model.diarize(audio=["/path/to/your/audio.wav"], batch_size=1)

For help choosing the right model for your use case, see Choosing a Model.

Types of Speaker Diarization Systems

1. End-to-End Speaker Diarization System:

End-to-end speaker diarization systems pursue a much more simplified version of a system where a single neural network model accepts raw audio signals and outputs speaker activity for each audio frame. Therefore, end-to-end diarization models have an advantage in ease of optimization and deployments.

Currently, NeMo Speech AI provides the following end-to-end speaker diarization models:

• Sortformer Diarizer : A transformer-based model that estimates speaker labels from the given audio input giving the speaker indexes in arrival-time order.

Model	Type	HuggingFace Link
diar_sortformer_4spk-v1	Offline	nvidia/diar_sortformer_4spk-v1
diar_streaming_sortformer_4spk-v2	Streaming	nvidia/diar_streaming_sortformer_4spk-v2
diar_streaming_sortformer_4spk-v2.1	Streaming	nvidia/diar_streaming_sortformer_4spk-v2.1

2. Cascaded Speaker Diarization System:

Traditional cascaded (also referred to as modular or pipelined) speaker diarization systems consist of multiple modules such as voice activity detection (VAD), speaker embedding extraction, and clustering. Cascaded speaker diarization systems are more challenging to optimize and deploy together but still have the advantage of fewer restrictions on the number of speakers and session length.

Cascaded NeMo Speech AI speaker diarization system consists of the following modules:

• Voice Activity Detector (VAD): A trainable model which detects the presence or absence of speech to generate timestamps for speech activity from the given audio recording.

• Speaker Embedding Extractor: A trainable model that extracts speaker embedding vectors containing voice characteristics from raw audio signal.

• Clustering Module: A non-trainable module that groups speaker embedding vectors into a number of clusters.

Module	Model	Link
Voice Activity Detector	Frame VAD Multilingual MarbleNet v2.0	HuggingFace · NGC
Speaker Embedding Extractor	TitaNet-Large	HuggingFace · NGC
Clustering Module	Spectral Clustering	Non-trainable (no checkpoint)

The full documentation tree is as follows:

• Models
  • Sortformer Diarizer
  • Streaming Sortformer Diarizer
  • References
• Datasets
  • Data Preparation for Speaker Diarization Training (End-to-End Diarization)
    • Overview
    • Working with Long-Form Audio
    • Handling num_speakers and Partial Speaker Presence
    • Speaker-Count Coverage in Training Data
  • Data Preparation for Diarization Inference: for Both End-to-end and Cascaded Systems
• Checkpoints
• End-to-end Speaker Diarization Models
  • Sortformer Diarizer Training
  • Sortformer Diarizer Inference with Post-processing
  • Streaming Sortformer Diarizer Training
  • Streaming Sortformer Diarizer Inference with Post-processing
  • HuggingFace Pretrained Checkpoints
  • End-to-end Speaker Diarization Models
• Models for Cascaded Speaker Diarization Pipeline
  • Loading Local Checkpoints
  • Inference
  • NGC Pretrained Checkpoints
  • Models for Speaker Diarization Pipeline
• Speaker Diarization Configuration Files
  • Hydra Configurations for Sortformer Diarizer Training
  • Hydra Configurations for Streaming Sortformer Diarizer Training
  • Hydra Configurations for (Streaming) Sortformer Diarization Post-processing
• Hydra Configurations for Diarization Inference
  • Diarizer Configurations
    • Configurations for Voice Activity Detector
    • Configurations for Speaker Embedding in Diarization
    • Configurations for Clustering in Diarization
    • Configurations for Diarization with ASR
• NeMo Speaker Diarization API
  • Model Classes
    • ClusteringDiarizer
      • diarize()
      • list_available_models()
      • restore_from()
      • save_to()
      • verbose
    • SortformerEncLabelModel
      • add_rttms_mask_mats()
      • diarize()
      • forward()
      • forward_for_export()
      • forward_infer()
      • forward_streaming()
      • forward_streaming_step()
        • spkcache
        • spkcache_lengths
        • spkcache_preds
        • fifo
        • fifo_lengths
        • fifo_preds
        • spk_perm
      • frontend_encoder()
      • input_names
      • input_types
      • list_available_models()
      • multi_validation_epoch_end()
      • on_validation_epoch_end()
      • oom_safe_feature_extraction()
      • output_names
      • output_types
      • process_signal()
      • setup_test_data()
      • setup_training_data()
      • setup_validation_data()
      • streaming_export()
      • streaming_input_examples()
      • test_batch()
      • test_dataloader()
      • test_step()
      • training_step()
      • validation_step()
  • Mixins
    • DiarizationMixin
      • diarize()
    • SpkDiarizationMixin
      • diarize()
      • diarize_generator()
• Resource and Documentation Guide