Long-Form Audio Cutting for ALM Pretraining

Use the long-form audio pretraining pipeline to turn diarized, transcribed recordings into bounded-duration snippets for audio language model (ALM) pretraining. The pipeline preserves speaker and transcript metadata, shifts timestamps into each snippet’s time range, converts audio to mono, resamples it, and packages the results as a sorted tar archive plus JSONL manifest.

This workflow is different from the ALM windowing tutorial. The windowing pipeline selects metadata windows from an existing manifest. This pipeline also reads source audio, extracts the selected ranges, and produces reusable audio assets for interleaved audio/text continuation, ASR, TTS, or diarization training.

Pipeline flow

All seven stages are CPU stages. ReadLongFormManifestStage fans one manifest into an AudioTask per source recording, and SnippetExtractionStage fans each recording into one task per surviving snippet. Workers write unique shards; the driver merges and validates them after pipeline execution.

Stage contracts

_snippet_plan and pipeline counters are internal coordination data. They are removed from emitted snippet rows or retained in task metadata only; do not build downstream schemas around them.

Prerequisites and capacity planning

Run this workflow in a supported Linux environment and install NeMo Curator with the CPU audio dependencies:

$
uv sync --extra audio_cpu

The pipeline uses SoundFile/libsndfile for decoding and encoding, TorchAudio for resampling, and a Hugging Face fast tokenizer for transcript repetition detection. Input can use a format supported by the installed libsndfile build; output is restricted to WAV, FLAC, or Ogg. A GPU is not required.

Choose one tokenizer source:

• A local directory loadable by AutoTokenizer.from_pretrained(..., use_fast=True)
• A Hugging Face Hub repository ID, such as openai/whisper-large-v3

For a gated Hub repository, authenticate with HF_TOKEN, huggingface-cli login, or --hf-token. Remote tokenizers are downloaded once per node before workers load them from the local cache. A slow tokenizer fails stage setup because the repetition filter requires character offset mappings.

Plan storage for the source audio, worker tar shards, and final tar. During finalization, the worker shards and merged tar coexist, so free space should exceed approximately twice the expected final snippet-audio volume, plus manifests and metrics. Tar payloads are streamed during merge, but the member index grows with the number of snippets.

Create the input manifest

Create one JSON object per source recording. The following schema preserves the data needed by common ALM consumers:

1
{
2
  "id": "meeting-001",
3
  "audio_filepath": "recordings/meeting-001.wav",
4
  "language": "en",
5
  "audio_sample_rate": 48000,
6
  "audio_num_channels": 2,
7
  "duration": 3660.0,
8
  "segments": [
9
    {
10
      "speaker": "speaker_0",
11
      "start": 12.4,
12
      "end": 18.8,
13
      "text": "Welcome to the weekly project meeting.",
14
      "text_ITN": "Welcome to the weekly project meeting.",
15
      "words": [
16
        {"word": "Welcome", "start": 12.4, "end": 12.9},
17
        {"word": "meeting.", "start": 18.1, "end": 18.8}
18
      ]
19
    }
20
  ]
21
}

Choose path resolution deliberately

Invalid JSON lines, rows without a usable ID, and rows without the audio path field are logged and skipped. A duplicate basename in basename mode raises ValueError and stops the reader.

Preview the cut with a dry run

Start with --dry-run. It executes manifest reading, overlap filtering, snippet planning, transcript repetition filtering, manifest writing, and metrics aggregation without reading source audio or creating a tar:

$
uv run python -m tutorials.audio.audio_pretrain.run \
>
  --input-manifest /data/long-form/manifest.jsonl \
>
  --audio-dir /data/long-form/audio \
>
  --output-dir /data/pretrain/preview \
>
  --output-manifest /data/pretrain/preview/snippets.jsonl \
>
  --output-audio-tar-path /data/pretrain/preview/snippets.tar \
>
  --metrics-path /data/pretrain/preview/metrics.json \
>
  --tokenizer-path openai/whisper-large-v3 \
>
  --max-duration-sec 30 \
>
  --dry-run

The preview manifest’s audio_filepath still contains the tar member name that a real run would create, but snippets.tar does not exist. Preview duration is the planned end - start; a real run records the resampled frame-count duration, which can differ by at most one target-rate frame.

Review metrics.json for yield, duration distribution, overlap loss, and transcript repetition before paying the audio I/O and storage cost of a full run.

Run the full extraction

Remove --dry-run and use a new output directory so preview files cannot be mistaken for extracted data:

$
uv run python -m tutorials.audio.audio_pretrain.run \
>
  --input-manifest /data/long-form/manifest.jsonl \
>
  --audio-dir /data/long-form/audio \
>
  --output-dir /data/pretrain/run-001 \
>
  --output-manifest /data/pretrain/run-001/snippets.jsonl \
>
  --output-audio-tar-path /data/pretrain/run-001/snippets.tar \
>
  --metrics-path /data/pretrain/run-001/metrics.json \
>
  --tokenizer-path openai/whisper-large-v3 \
>
  --tokenizer-cache-dir /data/hf-cache \
>
  --max-duration-sec 30 \
>
  --min-duration-sec 2 \
>
  --max-segment-gap-in-snippet 5 \
>
  --target-sample-rate 16000 \
>
  --output-format flac

The default backend is Xenna in streaming mode. Use --backend ray_data for Ray Data. --execution-mode is passed only to Xenna; Ray Data ignores that CLI setting.

Understand cutting and filtering

1. Remove empty and overlapping segments

OverlapFilterStage first removes a segment only when it has neither nonempty text nor any words. It then sorts the remaining segments and removes both members of an overlapping pair when either condition is true:

• Their intersection is greater than or equal to min_overlap_sec (default 0.5 seconds).
• One interval fully contains the other, even when the intersection is shorter than the threshold.

Smaller incidental overlaps without containment are retained. A chain of pairwise overlaps can therefore remove every segment in the chain. The metrics distinguish empty and overlap drops.

2. Greedily pack contiguous content

SnippetCutPlannerStage walks the surviving time-sorted segments. It appends the next segment while both conditions hold:

• next.end - snippet.first.start <= max_duration_sec
• next.start - snippet.last.end <= max_segment_gap_in_snippet

When either condition fails, the current snippet closes and a new one begins with the next segment. Segments are never split. Consequently, a single segment longer than the maximum is dropped as too_long rather than cut internally. Candidate spans shorter than min_duration_sec are dropped as too_short, and candidates whose joined text is blank are dropped as no_text.

The default 30-second maximum gap avoids connecting content separated by a long silence, such as an advertisement or recording boundary. Tighten it when semantic continuity matters more than packing density.

3. Remove transcript loops before audio I/O

SnippetRepetitionFilterStage joins segments[].text, tokenizes it without special tokens, and counts contiguous token-ID n-grams. With the defaults, a snippet is dropped when any 10-token n-gram occurs more than three times—that is, four or more occurrences. Text shorter than ngram_n tokens is retained.

This stage runs before extraction, so rejected Whisper-style decoding loops do not incur audio reads, resampling, or writes. The metrics summary retains up to 1,000 rejected text examples across the run.

4. Extract and normalize audio

For each surviving plan, SnippetExtractionStage:

1. Converts seconds to source frames using floor for the start and ceil for the end, clamped to the source file.
2. Reads only that slice through SoundFile.
3. Averages multiple channels into one mono channel.
4. Resamples with TorchAudio when the source rate differs from target_sample_rate.
5. Encodes wav as PCM-16, flac as PCM-16, or ogg as Vorbis.
6. Writes <snippet_id>.<format> to a worker tar shard.
7. Shifts segment and word timestamps by the planned start and clamps them to [0, planned duration].

Snippet IDs use millisecond precision: meeting-001-12_400-42_100. Periods and path separators in the source ID become underscores so the tar member stays at the archive root and remains compatible with WebDataset/Energon key parsing.

An unreadable source, empty frame range, slice-read error, or tar-write error is logged. If no snippet can be emitted for a source, the extractor sends an internal stub downstream so its zero-output metrics are still counted; stubs are never written to the final manifest.

dropped.missing_audio is reserved for rows removed during final manifest-to-tar reconciliation. A source file that is absent before extraction instead appears as a source with zero output snippets and an extraction error in the logs; it does not increment that reconciliation counter.

Configuration reference

Use the Python API

When building the pipeline directly, call the driver-side prepare and finalize functions yourself. Keep finalization in finally so recoverable shards are merged after interruption or failure:

1
from nemo_curator.backends.xenna import XennaExecutor
2
from nemo_curator.stages.audio.alm.pretrain import (
3
    build_audio_pretrain_pipeline,
4
    finalize_audio_pretrain_outputs,
5
    prepare_audio_pretrain_outputs,
6
)
7
8
manifest = "/data/pretrain/run-001/snippets.jsonl"
9
audio_tar = "/data/pretrain/run-001/snippets.tar"
10
metrics = "/data/pretrain/run-001/metrics.json"
11
12
pipeline = build_audio_pretrain_pipeline(
13
    input_manifest="/data/long-form/manifest.jsonl",
14
    audio_dir="/data/long-form/audio",
15
    output_dir="/data/pretrain/run-001",
16
    output_manifest_path=manifest,
17
    output_audio_tar_path=audio_tar,
18
    metrics_path=metrics,
19
    max_duration_sec=30,
20
    min_duration_sec=2,
21
    max_segment_gap_in_snippet=5,
22
    tokenizer_path="openai/whisper-large-v3",
23
    target_sample_rate=16000,
24
    output_format="flac",
25
)
26
27
prepare_audio_pretrain_outputs(manifest, metrics, audio_tar)
28
try:
29
    pipeline.run(XennaExecutor(config={"execution_mode": "streaming"}))
30
finally:
31
    finalize_audio_pretrain_outputs(
32
        manifest,
33
        metrics,
34
        audio_tar,
35
        audio_filepath_key="audio_filepath",
36
    )

prepare_audio_pretrain_outputs removes stale *.shard-* files next to all three outputs. finalize_audio_pretrain_outputs merges manifest and metrics shards, streams readable tar members into lexicographic order, removes merged shards, and reconciles the manifest against the tar.

Read the output manifest

Each output line represents one snippet. Values below are illustrative:

1
{
2
  "id": "meeting-001",
3
  "language": "en",
4
  "snippet_id": "meeting-001-12_400-42_100",
5
  "audio_filepath": "meeting-001-12_400-42_100.flac",
6
  "audio_sample_rate": 16000,
7
  "audio_num_channels": 1,
8
  "duration": 29.7,
9
  "segments": [
10
    {
11
      "speaker": "speaker_0",
12
      "start": 0.0,
13
      "end": 6.4,
14
      "text": "Welcome to the weekly project meeting.",
15
      "text_ITN": "Welcome to the weekly project meeting.",
16
      "words": [
17
        {"word": "Welcome", "start": 0.0, "end": 0.5}
18
      ]
19
    }
20
  ]
21
}

audio_filepath is the tar-internal member name, not a standalone filesystem path. The source id remains unchanged so all snippets can be joined back to their original recording.

Metadata transformations

Interpret metrics

The final metrics file includes aggregate and per-source values:

1
{
2
  "num_input_audios": 100,
3
  "num_output_snippets": 820,
4
  "input_total_segments": 7400,
5
  "input_total_duration_sec": 285000.0,
6
  "output_total_segments": 6950,
7
  "output_total_duration_sec": 23840.4,
8
  "dropped": {
9
    "empty": 20,
10
    "overlap": 180,
11
    "too_long": 4,
12
    "too_short": 12,
13
    "no_text": 8,
14
    "repetition": 6,
15
    "missing_audio": 1
16
  },
17
  "snippet_duration_histogram_30s": {"0-30": 810, "30-60": 10},
18
  "dropped_repetition_examples": ["repeated transcript example"],
19
  "per_original": [
20
    {
21
      "id": "meeting-001",
22
      "in_segments": 80,
23
      "in_duration_sec": 3600.0,
24
      "dropped": {"empty": 1, "overlap": 2, "too_long": 0, "too_short": 0, "no_text": 0, "repetition": 0},
25
      "out_snippets": 9,
26
      "out_segments": 77,
27
      "out_duration_sec": 260.3
28
    }
29
  ]
30
}

Input duration is the wall-clock span from the earliest segment start to the latest segment end, not decoded file duration or summed speech duration. Output duration is also a snippet span and therefore includes silence between packed segments. Histogram bins are fixed 30-second ranges, and a duration exactly on a boundary enters the higher bin. Consequently, a snippet of exactly 30 seconds appears in 30-60 even when max_duration_sec=30; that bin does not indicate that the configured cap was exceeded.

The missing_audio and corrupted_audio counters are sparse: each key is written only when its own reconciliation count is nonzero. The example above shows missing_audio but omits corrupted_audio because no members were unreadable. Read these keys defensively—for example dropped.get("corrupted_audio", 0)—rather than indexing them directly, which raises KeyError when the count was zero. After reconciliation removes rows, output totals, histogram values, and each per-source out_* field are rebuilt from the authoritative manifest.

Retry, cleanup, and partial-output behavior

• Before every run, prepare_audio_pretrain_outputs deletes stale manifest, metrics, and tar shards matching the requested output names. Do not treat leftover shards as checkpoints.
• The CLI always calls finalization in finally. If workers wrote usable shards before an exception or interruption, those shards are merged into partial final outputs for diagnosis or recovery.
• If a run fails before producing any new shards, merge steps skip their outputs rather than truncating a previous successful final manifest, metrics file, or tar.
• If new shards exist, their merge replaces the corresponding final output. Use a new run directory when previous results must remain immutable.
• Malformed final JSONL shard lines, unreadable or truncated tar tails, and failed members are skipped while earlier valid data is recovered.
• Final reconciliation drops a manifest row when its tar member is missing, unreadable, empty, or has an invalid sample rate. Orphan tar members are harmless and are not removed.
• The merged tar is lexicographically sorted by member name for WebDataset/Energon compatibility.

Validate a production result by checking that manifest line count equals num_output_snippets, inspecting the tar member count, and reviewing all dropped categories. Treat dry-run manifests as plans only because their referenced tar does not exist.

Related topics

• ALM pipeline tutorial
• ALM pipeline concepts
• ALM data builder
• ALM overlap filtering
• Speaker separation and diarization
• Execution backends