> This page is for version 26.09 · v1.4.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.4.0 (26.09) (default): https://docs.nvidia.com/nemo/curator/latest/llms.txt
> - Main · preview: https://docs.nvidia.com/nemo/curator/main/llms.txt
> - 26.09 · v1.4.0: https://docs.nvidia.com/nemo/curator/v26.09/llms.txt
> - 26.07 · v1.3.0: https://docs.nvidia.com/nemo/curator/v26.07/llms.txt
> - 26.04 · v1.2.0: https://docs.nvidia.com/nemo/curator/v26.04/llms.txt
> - 26.02 · v1.1.0: https://docs.nvidia.com/nemo/curator/v26.02/llms.txt
> - 25.09 · v1.0.0: https://docs.nvidia.com/nemo/curator/v25.09/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

> Concepts for integrating audio processing with text curation workflows in multimodal applications

# Audio-Text Integration Concepts

This guide covers how audio processing integrates with text curation workflows in NeMo Curator, enabling seamless multi-modal data preparation and cross-modal quality assessment.

## Integration Architecture

Audio-text integration in NeMo Curator operates on several levels:

### Data Structure Integration

**Format Conversion**: `AudioTask` to `DocumentBatch`

* Serializable top-level fields remain available without remapping (`text` and `pred_text` remain unchanged)
* Raw audio fields, `segments`, and top-level tensor values are removed during conversion

**Retained Metadata**: Serializable top-level metadata remains available unless the converter explicitly excludes it

* File paths remain available for traceability and debugging
* Top-level quality metrics such as `wer_pct` and `duration` remain available for filtering operations
* Segment-level metrics require an `AudioTask`-native export path because conversion removes `segments`

### Pipeline Integration

**Sequential Processing**: Audio to Text to Multi-Modal

```mermaid
flowchart LR
    A[Audio Files] --> B[ASRStage]
    B --> C[AudioToDocumentStage]
    C --> D[ScoreFilter<br />Text Processing]
    D --> E[Integrated Output]

    style A fill:#e1f5fe
    style C fill:#ffcc02
    style E fill:#fff3e0
```

The `AudioToDocumentStage` provides the conversion bridge between audio and text processing workflows.

**Parallel Processing**: Simultaneous audio and text analysis

```mermaid
flowchart LR
    A[Audio Files] --> B[ASRStage]
    C[Text Data] --> D[ScoreFilter<br />Text Processing]
    B --> E[Cross-Modal<br />Quality Assessment]
    D --> E
    E --> F[Filtered Output]

    style A fill:#e1f5fe
    style C fill:#e8f5e8
    style F fill:#fff3e0
```

## Cross-Modal Quality Assessment

### Audio-Informed Text Quality

Use audio characteristics to enhance text quality assessment:

**Speech Rate Analysis**: Detect unnaturally fast or slow speech using the `metrics.word_rate` and `metrics.char_rate` fields emitted by `ComputeWERStage`

**Duration-Text Consistency**: Ensure transcription length matches audio duration

* Short audio with long text: Potential transcription errors
* Long audio with short text: Potential missing content
* Optimal ratio: \~3-5 characters per second of audio

### Text-Informed Audio Quality

Use text characteristics to assess audio quality:

**Transcription Completeness**: Detect incomplete or truncated speech

* Sentence fragments without proper endings
* Unusual punctuation patterns
* Incomplete words or phrases

**Content Coherence**: Assess semantic consistency

* Logical flow and coherence in transcriptions
* Domain-appropriate vocabulary usage
* Language consistency throughout sample

## Workflow Patterns

### Audio-First Workflows

Start with audio processing, then apply text curation:

```mermaid
flowchart TD
    A[Audio Files] --> B[ASRStage<br />ASR Transcription]
    B --> C[GetPairwiseWerStage<br />Calculate WER Metrics]
    C --> D[ScoreFilter<br />WER-based Filtering]
    D --> E[AudioToDocumentStage<br />Convert to DocumentBatch]
    E --> F[ScoreFilter<br />Text Quality Assessment]
    F --> G[Filter<br />Metadata-based Filtering]
    G --> H[Text Enhancement Stages]
    H --> I[Processed Dataset]

    style A fill:#e1f5fe
    style E fill:#fff3e0
    style I fill:#e8f5e8

    classDef audioStage fill:#bbdefb
    classDef conversionStage fill:#ffcc02
    classDef textStage fill:#c8e6c9

    class B,C,D audioStage
    class E conversionStage
    class F,G,H textStage
```

**Use Cases**:

* Speech dataset curation for ASR training
* Podcast transcription and processing
* Lecture and educational content preparation

### Text-First Workflows

Start with text processing, then verify with audio:

```mermaid
flowchart TD
    A[Text Corpus] --> B[ScoreFilter<br />Text Quality Assessment]
    B --> C[Filter<br />Initial Text Filtering]
    C --> D[Audio File Matching<br />Custom Stage]
    D --> E[ASRStage<br />ASR Validation]
    E --> F[GetPairwiseWerStage<br />Cross-Modal Metrics]
    F --> G[ScoreFilter<br />Consistency Filtering]
    G --> H[Validated Dataset]

    style A fill:#e8f5e8
    style D fill:#fff3e0
    style H fill:#e1f5fe

    classDef textStage fill:#c8e6c9
    classDef matchingStage fill:#ffcc02
    classDef audioStage fill:#bbdefb

    class B,C textStage
    class D matchingStage
    class E,F,G audioStage
```

**Use Cases**:

* Validating existing transcriptions with audio
* Creating audio-text pairs from separate sources
* Quality control for crowd-sourced transcriptions

## Data Flow Concepts

### Conversion Mechanisms

**AudioTask to DocumentBatch**:

NeMo Curator provides the `AudioToDocumentStage` for converting audio processing results to text processing format:

```python
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.tasks import AudioTask

# Create the conversion stage
converter = AudioToDocumentStage()

# Example input AudioTask data
audio_data = {
    "audio_filepath": "/audio.wav",
    "text": "ground truth",
    "pred_text": "asr prediction",
    "wer_pct": 15.2,
    "duration": 3.4
}
audio_task = AudioTask(data=audio_data)

# The stage returns a list containing one DocumentBatch
# with the remaining serializable fields in a pandas DataFrame
document_batches = converter.process_batch([audio_task])
document_batch = document_batches[0]  # Extract the single DocumentBatch

# These top-level fields are preserved in the DocumentBatch:
# - audio_filepath, text, pred_text, wer_pct, duration
```

`AudioToDocumentStage` removes `waveform`, `audio`, `audio_data`, `audio_array`, `segments`, and any top-level tensor-valued field. To retain JSON-serializable segment data and segment-level metrics, write the `AudioTask` directly with `ManifestWriterStage` before conversion. A built-in `DocumentBatch` to `AudioTask` conversion stage is not provided; create a custom stage if you need reverse conversion.

For practical usage examples and step-by-step implementation, refer to [Text integration](/curate-audio/process-data/text-integration).

### Metadata Flow

**Additive Processing**: Processing stages typically add metadata without removing existing fields

```python
# Stage 1: Initial loading
stage1_output = {"audio_filepath": "/audio.wav", "text": "transcription"}

# Stage 2: ASR inference
stage2_output = {**stage1_output, "pred_text": "asr result"}

# Stage 3: Quality assessment
stage3_output = {**stage2_output, "wer_pct": 15.2, "duration": 3.4}

# Stage 4: Text processing (after conversion)
stage4_output = {**stage3_output, "word_count": 6, "language": "en"}
```

## Quality Assessment Integration

### Available Quality Metrics

NeMo Curator provides these audio quality assessment capabilities:

**Word Error Rate (WER) Analysis**:

* WER correlates with transcription accuracy
* Available through `GetPairwiseWerStage`
* Measures percentage of incorrect words between ground truth and ASR predictions

**Duration and Speech Rate Analysis**:

* Duration validation using `GetAudioDurationStage`
* Speech rate calculation through `ComputeWERStage` (`metrics.word_rate`)
* Character rate calculation through `ComputeWERStage` (`metrics.char_rate`)

**Individual Quality Dimensions**:

* **Technical Quality**: File integrity, format compliance, duration validation
* **Content Quality**: Transcription accuracy via WER/CER metrics
* **Speech Rate Quality**: Words/characters per second analysis

## Performance and Scaling

### Memory Considerations

**AudioTask Memory Usage**:

* Metadata storage scales linearly with batch size
* Audio files loaded on-demand, not cached in memory
* Large batches increase processing efficiency but consume more RAM

**Conversion Overhead**:

* AudioTask → DocumentBatch conversion is lightweight
* Metadata copying has minimal performance impact
* Batch size affects conversion performance

### Processing Efficiency

**Separate vs. Single-Pipeline Integration**:

**Sequential Processing**: Audio to Text (lower memory, slower)

```python
from pathlib import Path

from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.audio.common import ManifestReader
from nemo_curator.stages.audio.inference.asr.stage import ASRStage
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter  # Example filter

# Define a text quality filter
text_quality_filter = WordCountFilter(min_words=10)
manifest_path = Path("/data/audio/manifest.jsonl").expanduser().resolve()
executor = XennaExecutor()

# Process audio completely first
audio_pipeline = Pipeline(
    name="audio_processing",
    stages=[
        ManifestReader(manifest_path=str(manifest_path)),
        ASRStage(
            adapter_target="nemo_curator.models.asr.nemo_asr.NeMoASRAdapter",
            model_id="nvidia/stt_en_fastconformer_transducer_large",
            max_audio_sec_per_actor=240.0,
            max_inference_duration_s=120.0,
            local_bucketing=True,
            audio_filepath_key="audio_filepath",
        ),
        AudioToDocumentStage()
    ]
)
audio_results = audio_pipeline.run(executor)

# Then process text
text_pipeline = Pipeline(
    name="text_processing",
    stages=[
        ScoreFilter(filter_obj=text_quality_filter)
    ]
)
final_results = text_pipeline.run(executor, initial_tasks=audio_results)
```

**Single-Pipeline Processing**: Keep audio and text stages in one execution graph

```python
from pathlib import Path

from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.audio.common import ManifestReader
from nemo_curator.stages.audio.inference.asr.stage import ASRStage
from nemo_curator.stages.audio.io.convert import AudioToDocumentStage
from nemo_curator.stages.audio.metrics.wer import GetPairwiseWerStage
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter

manifest_path = Path("/data/audio/manifest.jsonl").expanduser().resolve()
executor = XennaExecutor()
text_quality_filter = WordCountFilter(min_words=10)

pipeline = Pipeline(name="audio_text", stages=[
    ManifestReader(manifest_path=str(manifest_path)),
    ASRStage(
        adapter_target="nemo_curator.models.asr.nemo_asr.NeMoASRAdapter",
        model_id="nvidia/stt_en_fastconformer_transducer_large",
        max_audio_sec_per_actor=240.0,
        max_inference_duration_s=120.0,
        local_bucketing=True,
        audio_filepath_key="audio_filepath",
    ),
    GetPairwiseWerStage(text_key="text", pred_text_key="pred_text", wer_key="wer_pct"),
    AudioToDocumentStage(),
    ScoreFilter(filter_obj=text_quality_filter)
])
results = pipeline.run(executor)
```

### Scaling Strategies

**Horizontal Scaling**: Distribute across several workers

* Partition audio files across workers
* Independent processing with final aggregation
* Load balancing based on audio duration

**Vertical Scaling**: Optimize single-machine performance

* GPU acceleration for ASR inference
* Batch size optimization for hardware
* Memory management for large datasets

## Design Principles

### Modularity

**Separation of Concerns**: Audio and text processing remain independent

* Audio stages focus on speech-specific operations
* Text stages handle language processing
* Integration stages manage cross-modal operations

**Modular Architecture**: Mix and match audio and text processing stages

* Flexible pipeline construction
* Reusable stage components
* Configurable integration points

### Extensibility

**Custom Integration Patterns**: Support for domain-specific workflows

* Custom conversion logic
* Specialized quality metrics
* Domain-specific filtering rules

**Plugin Architecture**: Easy addition of new integration methods

* Custom stage implementations
* External tool integration
* Specialized format support