Curate AudioProcess DataQuality Assessment

Quality Assessment for Audio Data

View as Markdown

Filter audio quality using transcription accuracy metrics, duration analysis, and custom quality measures to ensure high-quality speech datasets for ASR training.

How it Works

Audio quality assessment in NeMo Curator focuses on speech-specific metrics that correlate with training data quality:

  1. Transcription Accuracy: Word Error Rate (WER) and Character Error Rate (CER) between ground truth and ASR predictions
  2. Duration Analysis: Audio length validation and speech rate calculations
  3. Value-based Filtering: Configurable filtering using comparison operators

Quality Metrics

Word Error Rate (WER)

The primary metric for assessing transcription quality:

from nemo_curator.stages.audio.metrics.wer import GetPairwiseWerStage
# Calculate WER for each audio sample
wer_stage = GetPairwiseWerStage(
    text_key="text",           # Ground truth transcription
    pred_text_key="pred_text", # ASR prediction
    wer_key="wer_pct",         # Output WER percentage field
)

WER measures the percentage of words that differ between ground truth and predicted transcriptions:

  • WER = 0%: Perfect transcription match
  • WER = 25%: Good quality (1 in 4 words incorrect)
  • WER = 50%: Moderate quality
  • WER >75%: Poor quality (consider filtering)

Character Error Rate (CER)

For segment-oriented manifests, ComputeWERStage calculates CER together with WER and writes the results under the metrics object:

from nemo_curator.stages.audio.metrics.wer import ComputeWERStage
metrics_stage = ComputeWERStage(
    hypothesis_text_key="pred_text",
    reference_text_key="text",
)

metrics.cer.cer and metrics.wer.wer are nonnegative ratios, where 0 means an exact match. Insertions can make either ratio exceed 1.0. GetPairwiseWerStage instead writes the ratio multiplied by 100 in the top-level wer_pct field for simple threshold filtering.

Speech Rate Metrics

The same ComputeWERStage records speaking speed from the hypothesis text and segment duration:

metrics_stage = ComputeWERStage(
    hypothesis_text_key="pred_text",
    reference_text_key="text",
)
# Adds metrics.word_rate and metrics.char_rate to each processed entry or segment.

For a complete example of using speech rate metrics in a pipeline, refer to the Duration Filtering guide.

Filtering Strategies

WER-based Filtering

Filter audio samples based on transcription accuracy:

from nemo_curator.stages.audio.common import PreserveByValueStage
# Keep samples with WER <= 30% (high quality)
high_quality_filter = PreserveByValueStage(
    input_value_key="wer_pct",
    target_value=30.0,
    operator="le"  # less than or equal
)
# Remove samples with WER >= 80% (very poor quality)
poor_quality_filter = PreserveByValueStage(
    input_value_key="wer_pct",
    target_value=80.0,
    operator="lt"  # less than
)
# Preserves only entries with WER < 80%

Duration-based Filtering

Filter by audio length to remove short or long samples:

from nemo_curator.stages.audio.common import PreserveByValueStage
# Keep samples between 1-30 seconds
duration_min_filter = PreserveByValueStage(
    input_value_key="duration",
    target_value=1.0,
    operator="ge"  # greater than or equal
)
duration_max_filter = PreserveByValueStage(
    input_value_key="duration",
    target_value=30.0,
    operator="le"  # less than or equal
)

Combined Quality Filtering

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.audio.metrics.wer import GetPairwiseWerStage
from nemo_curator.stages.audio.common import GetAudioDurationStage, PreserveByValueStage
# Create multi-stage quality pipeline
quality_pipeline = Pipeline(name="audio_quality_assessment")
# Calculate all metrics
quality_pipeline.add_stage(GetPairwiseWerStage())
quality_pipeline.add_stage(GetAudioDurationStage(
    audio_filepath_key="audio_filepath",
    duration_key="duration"
))
# Apply filters in sequence
filters = [
    PreserveByValueStage("wer_pct", 50.0, "le"),    # WER &lt;= 50%
    PreserveByValueStage("duration", 1.0, "ge"),     # Duration &gt;= 1s
    PreserveByValueStage("duration", 20.0, "le"),    # Duration &lt;= 20s
]
for filter_stage in filters:
    quality_pipeline.add_stage(filter_stage)

Operator Options

The PreserveByValueStage supports several comparison operators:

OperatorDescriptionExample Use Case
"eq"Equal toExact duration matching
"ne"Not equal toExclude specific values
"lt"Less thanMax thresholds
"le"Less than or equalQuality thresholds
"gt"Greater thanMin thresholds
"ge"Greater than or equalMin requirements

Complete Quality Assessment Pipeline

Here’s a complete working example that demonstrates quality assessment:

from nemo_curator.pipeline import Pipeline
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.stages.audio.datasets.fleurs.create_initial_manifest import CreateInitialManifestFleursStage
from nemo_curator.stages.audio.inference.asr.stage import ASRStage
from nemo_curator.stages.audio.metrics.wer import GetPairwiseWerStage
from nemo_curator.stages.audio.common import GetAudioDurationStage, PreserveByValueStage
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.resources import Resources
# Create complete quality assessment pipeline
pipeline = Pipeline(name="audio_quality_assessment")
# 1. Load data
pipeline.add_stage(CreateInitialManifestFleursStage(
    lang="hy_am",
    split="dev",
    raw_data_dir="./audio_data"
).with_(batch_size=4))
# 2. ASR inference
pipeline.add_stage(ASRStage(
    adapter_target="nemo_curator.models.asr.nemo_asr.NeMoASRAdapter",
    model_id="nvidia/stt_hy_fastconformer_hybrid_large_pc",
    max_audio_sec_per_actor=240.0,
    max_inference_duration_s=120.0,
    local_bucketing=True,
    audio_filepath_key="audio_filepath",
).with_(resources=Resources(gpus=1.0)))
# 3. Calculate quality metrics
pipeline.add_stage(GetPairwiseWerStage())
pipeline.add_stage(GetAudioDurationStage(
    audio_filepath_key="audio_filepath",
    duration_key="duration"
))
# 4. Apply quality filters
pipeline.add_stage(PreserveByValueStage(
    input_value_key="wer_pct",
    target_value=75.0,
    operator="le"  # Keep WER &lt;= 75%
))
pipeline.add_stage(PreserveByValueStage(
    input_value_key="duration",
    target_value=1.0,
    operator="ge"  # Keep duration &gt;= 1s
))
pipeline.add_stage(PreserveByValueStage(
    input_value_key="duration",
    target_value=30.0,
    operator="le"  # Keep duration &lt;= 30s
))
# 5. Export high-quality results
pipeline.add_stage(AudioToDocumentStage())
pipeline.add_stage(JsonlWriter(path="./high_quality_audio"))
# Execute pipeline
executor = XennaExecutor()
pipeline.run(executor)