Text Integration | NeMo Curator

This guide covers how audio processing integrates with text curation workflows in NeMo Curator, enabling seamless multi-modal data preparation and cross-modal quality assessment.

Integration Architecture

Audio-text integration in NeMo Curator operates on several levels:

Data Structure Integration

Format Conversion: AudioTask to DocumentBatch

Serializable top-level fields remain available without remapping (text and pred_text remain unchanged)
Raw audio fields, segments, and top-level tensor values are removed during conversion

Retained Metadata: Serializable top-level metadata remains available unless the converter explicitly excludes it

File paths remain available for traceability and debugging
Top-level quality metrics such as wer_pct and duration remain available for filtering operations
Segment-level metrics require an AudioTask-native export path because conversion removes segments

Pipeline Integration

Sequential Processing: Audio to Text to Multi-Modal

The AudioToDocumentStage provides the conversion bridge between audio and text processing workflows.

Parallel Processing: Simultaneous audio and text analysis

Audio-Informed Text Quality

Use audio characteristics to enhance text quality assessment:

Speech Rate Analysis: Detect unnaturally fast or slow speech using the metrics.word_rate and metrics.char_rate fields emitted by ComputeWERStage

Duration-Text Consistency: Ensure transcription length matches audio duration

Short audio with long text: Potential transcription errors
Long audio with short text: Potential missing content
Optimal ratio: ~3-5 characters per second of audio

Text-Informed Audio Quality

Use text characteristics to assess audio quality:

Transcription Completeness: Detect incomplete or truncated speech

Sentence fragments without proper endings
Unusual punctuation patterns
Incomplete words or phrases

Content Coherence: Assess semantic consistency

Logical flow and coherence in transcriptions
Domain-appropriate vocabulary usage
Language consistency throughout sample

Workflow Patterns

Audio-First Workflows

Start with audio processing, then apply text curation:

Use Cases:

Speech dataset curation for ASR training
Podcast transcription and processing
Lecture and educational content preparation

Text-First Workflows

Start with text processing, then verify with audio:

Use Cases:

Validating existing transcriptions with audio
Creating audio-text pairs from separate sources
Quality control for crowd-sourced transcriptions

Data Flow Concepts

Conversion Mechanisms

AudioTask to DocumentBatch:

NeMo Curator provides the AudioToDocumentStage for converting audio processing results to text processing format:

from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.tasks import AudioTask
# Create the conversion stage
converter = AudioToDocumentStage()
# Example input AudioTask data
audio_data = {
    "audio_filepath": "/audio.wav",
    "text": "ground truth",
    "pred_text": "asr prediction",
    "wer_pct": 15.2,
    "duration": 3.4
}
audio_task = AudioTask(data=audio_data)
# The stage returns a list containing one DocumentBatch
# with the remaining serializable fields in a pandas DataFrame
document_batches = converter.process_batch([audio_task])
document_batch = document_batches[0]  # Extract the single DocumentBatch
# These top-level fields are preserved in the DocumentBatch:
# - audio_filepath, text, pred_text, wer_pct, duration

AudioToDocumentStage removes waveform, audio, audio_data, audio_array, segments, and any top-level tensor-valued field. To retain JSON-serializable segment data and segment-level metrics, write the AudioTask directly with ManifestWriterStage before conversion. A built-in DocumentBatch to AudioTask conversion stage is not provided; create a custom stage if you need reverse conversion.

For practical usage examples and step-by-step implementation, refer to Text integration.

Metadata Flow

Additive Processing: Processing stages typically add metadata without removing existing fields

# Stage 1: Initial loading
stage1_output = {"audio_filepath": "/audio.wav", "text": "transcription"}
# Stage 2: ASR inference
stage2_output = {**stage1_output, "pred_text": "asr result"}
# Stage 3: Quality assessment
stage3_output = {**stage2_output, "wer_pct": 15.2, "duration": 3.4}
# Stage 4: Text processing (after conversion)
stage4_output = {**stage3_output, "word_count": 6, "language": "en"}

Quality Assessment Integration

Available Quality Metrics

NeMo Curator provides these audio quality assessment capabilities:

Word Error Rate (WER) Analysis:

WER correlates with transcription accuracy
Available through GetPairwiseWerStage
Measures percentage of incorrect words between ground truth and ASR predictions

Duration and Speech Rate Analysis:

Duration validation using GetAudioDurationStage
Speech rate calculation through ComputeWERStage (metrics.word_rate)
Character rate calculation through ComputeWERStage (metrics.char_rate)

Individual Quality Dimensions:

Technical Quality: File integrity, format compliance, duration validation
Content Quality: Transcription accuracy via WER/CER metrics
Speech Rate Quality: Words/characters per second analysis

Performance and Scaling

Memory Considerations

AudioTask Memory Usage:

Metadata storage scales linearly with batch size
Audio files loaded on-demand, not cached in memory
Large batches increase processing efficiency but consume more RAM

Conversion Overhead:

AudioTask → DocumentBatch conversion is lightweight
Metadata copying has minimal performance impact
Batch size affects conversion performance

Processing Efficiency

Separate vs. Single-Pipeline Integration:

Sequential Processing: Audio to Text (lower memory, slower)

from pathlib import Path
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.audio.common import ManifestReader
from nemo_curator.stages.audio.inference.asr.stage import ASRStage
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter  # Example filter
# Define a text quality filter
text_quality_filter = WordCountFilter(min_words=10)
manifest_path = Path("/data/audio/manifest.jsonl").expanduser().resolve()
executor = XennaExecutor()
# Process audio completely first
audio_pipeline = Pipeline(
    name="audio_processing",
    stages=[
        ManifestReader(manifest_path=str(manifest_path)),
        ASRStage(
            adapter_target="nemo_curator.models.asr.nemo_asr.NeMoASRAdapter",
            model_id="nvidia/stt_en_fastconformer_transducer_large",
            max_audio_sec_per_actor=240.0,
            max_inference_duration_s=120.0,
            local_bucketing=True,
            audio_filepath_key="audio_filepath",
        ),
        AudioToDocumentStage()
    ]
)
audio_results = audio_pipeline.run(executor)
# Then process text
text_pipeline = Pipeline(
    name="text_processing",
    stages=[
        ScoreFilter(filter_obj=text_quality_filter)
    ]
)
final_results = text_pipeline.run(executor, initial_tasks=audio_results)

Single-Pipeline Processing: Keep audio and text stages in one execution graph

from pathlib import Path
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.audio.common import ManifestReader
from nemo_curator.stages.audio.inference.asr.stage import ASRStage
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.stages.audio.metrics.wer import GetPairwiseWerStage
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter
manifest_path = Path("/data/audio/manifest.jsonl").expanduser().resolve()
executor = XennaExecutor()
text_quality_filter = WordCountFilter(min_words=10)
pipeline = Pipeline(name="audio_text", stages=[
    ManifestReader(manifest_path=str(manifest_path)),
    ASRStage(
        adapter_target="nemo_curator.models.asr.nemo_asr.NeMoASRAdapter",
        model_id="nvidia/stt_en_fastconformer_transducer_large",
        max_audio_sec_per_actor=240.0,
        max_inference_duration_s=120.0,
        local_bucketing=True,
        audio_filepath_key="audio_filepath",
    ),
    GetPairwiseWerStage(text_key="text", pred_text_key="pred_text", wer_key="wer_pct"),
    AudioToDocumentStage(),
    ScoreFilter(filter_obj=text_quality_filter)
])
results = pipeline.run(executor)

Scaling Strategies

Horizontal Scaling: Distribute across several workers

Partition audio files across workers
Independent processing with final aggregation
Load balancing based on audio duration

Vertical Scaling: Optimize single-machine performance

GPU acceleration for ASR inference
Batch size optimization for hardware
Memory management for large datasets

Design Principles

Modularity

Separation of Concerns: Audio and text processing remain independent

Audio stages focus on speech-specific operations
Text stages handle language processing
Integration stages manage cross-modal operations

Modular Architecture: Mix and match audio and text processing stages

Flexible pipeline construction
Reusable stage components
Configurable integration points

Extensibility

Custom Integration Patterns: Support for domain-specific workflows

Custom conversion logic
Specialized quality metrics
Domain-specific filtering rules

Plugin Architecture: Easy addition of new integration methods

Custom stage implementations
External tool integration
Specialized format support