Skip to navigation
ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsDatasetsnemo_automodel.components.datasets.audio

nemo_automodel.components.datasets.audio

View as Markdown

Submodules

Package Contents

Functions

NameDescription
make_cv17_datasetLoad and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning.
make_hf_audio_asr_datasetLazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning.
make_multi_en_asr_datasetBuild the multi-source English ASR training dataset for Qwen-Omni.
qwen2_5_omni_asr_collate_fnCollate Qwen2.5-Omni ASR conversations.
qwen3_omni_asr_collate_fnCollate Qwen3-Omni ASR conversations into model inputs without qwen_omni_utils.

API

nemo_automodel.components.datasets.audio.datasets.make_cv17_dataset(
    path_or_dataset: str = 'ysdede/commonvoice_17_tr_f...,
    split: str = 'train',
    sampling_rate: int = 16000,
    audio_column: str = 'audio',
    text_column: str = 'transcription',
    load_kwargs: typing.Any = {}
) -> datasets.Dataset

Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning.

Torchcodec-free: the audio column is cast with Audio(decode=False) and decoded lazily via soundfile (_decode_audio_cell_to_mono_float32) inside a with_transform callback, so no audio is decoded at construction time. Each accessed item is {"conversation": [...], "audio": (waveform, sampling_rate)}, matching what phi4_mm_collate_fn consumes.

Parameters:

path_or_dataset
strDefaults to 'ysdede/commonvoice_17_tr_fixed'

HuggingFace dataset id or local path.

split
strDefaults to 'train'

Dataset split to load.

sampling_rate
intDefaults to 16000

Target sampling rate in Hz for the decoded waveform.

audio_column
strDefaults to 'audio'

Name of the audio column.

text_column
strDefaults to 'transcription'

Name of the transcript column.

**load_kwargs
AnyDefaults to {}

Forwarded to datasets.load_dataset.

Returns: Dataset

A HuggingFace Dataset with a lazy decode transform attached.

Raises:

  • ValueError: When audio_column or text_column is missing.
nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset(
    path_or_dataset: str,
    split: str = 'train',
    name: str | None = None,
    sampling_rate: int = 16000,
    system_prompt: str | None = None,
    user_prompt: str | None = None,
    audio_column: str = 'audio',
    text_column: str = 'text',
    drop_empty_text: bool = True,
    min_audio_duration_seconds: float | None = None,
    max_audio_duration_seconds: float | None = None,
    load_kwargs: typing.Any = {}
) -> datasets.Dataset

Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning.

Loads any HuggingFace ASR dataset that exposes an audio column (Audio feature with bytes and/or path populated after cast_column(decode=False)) and a transcript column, and yields the Qwen-Omni chat-template conversation expected by the ASR collate functions. No audio is decoded at construction time — both the soundfile decode (mono mix + float32 cast + optional scipy.signal.resample_poly) and the conversation assembly run inside a HuggingFace with_transform callback, so the only fixed startup cost is the Arrow-level metadata read of the parquet shards (and the on-demand download of those shards if they are not already in the HF cache). Empty-transcript filtering happens via dataset.filter against the text column only — also Arrow-level — so audio bytes are never materialized at startup.

Defaults are tuned for the common case (audio / text columns, 16 kHz, no system turn). Datasets that diverge can override per-field via YAML; see :file:docs/guides/audio/qwen3-omni-asr.md for an override table.

Parameters:

path_or_dataset
str

HuggingFace dataset id or local path.

split
strDefaults to 'train'

Dataset split to load (e.g. "train", "train[:5000]").

name
str | NoneDefaults to None

Optional dataset configuration / subset. Forwarded to datasets.load_dataset(path, name=name, ...). Required by some datasets (e.g. edinburghcstr/ami needs "ihm" or "sdm"; CommonVoice needs the language code).

sampling_rate
intDefaults to 16000

Target sampling rate in Hz. Audio is resampled inside the lazy transform if the source rate differs.

system_prompt
str | NoneDefaults to None

Instruction placed in a system turn. Default None skips the system turn entirely; pass a string to emit one.

user_prompt
str | NoneDefaults to None

Instruction prepended to the audio inside the user turn. Pass None to emit a user turn with only the audio item.

audio_column
strDefaults to 'audio'

Name of the audio column in the source dataset (default "audio" — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech / CommonVoice).

text_column
strDefaults to 'text'

Name of the transcript column (default "text" — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech; override to "sentence" for CommonVoice).

drop_empty_text
boolDefaults to True

If True, samples whose transcript is empty or whitespace are dropped via dataset.filter (Arrow-level, no audio decode). If False, an empty transcript triggers a ValueError inside the transform at access time.

min_audio_duration_seconds
float | NoneDefaults to None

Optional minimum audio duration. Samples shorter than this threshold are dropped via dataset.filter using soundfile.info (header-only read, no full decode). The HF Qwen-Omni Whisper feature extractor has a known off-by-one between input_features and feature_attention_mask for sub-second clips (~0.27 s manifests as a 27-vs-26 frame mismatch); set this to 1.0 for AMI / CommonVoice-style corpora that contain very short utterances.

max_audio_duration_seconds
float | NoneDefaults to None

Optional maximum audio duration. Samples longer than this threshold are dropped via the same header-only soundfile.info probe (no decode). Use this to cap activation memory: long clips inflate the Whisper feature extractor’s input_features (mel frames) and can OOM a rank at larger batch sizes. None disables the cap.

**load_kwargs
AnyDefaults to {}

Forwarded to datasets.load_dataset (e.g. trust_remote_code=True).

Returns: Dataset

A HuggingFace Dataset whose elements are

Raises:

  • ValueError: When audio_column or text_column is missing, when an audio cell has neither bytes nor path, or when drop_empty_text=False and a transcript is empty.
nemo_automodel.components.datasets.audio.multi_en.make_multi_en_asr_dataset(
    sampling_rate: int = TARGET_SAMPLING_RATE,
    system_prompt: str | None = None,
    user_prompt: str | None = DEFAULT_USER_PROMPT,
    min_audio_duration_seconds: float | None = 1.0,
    max_audio_duration_seconds: float | None = 30.0,
    shuffle_seed: int | None = 42,
    sources: list[nemo_automodel.components.datasets.audio.multi_en.Source | collections.abc.Mapping[str, typing.Any]] | None = None,
    audio_column: str = 'audio',
    text_column: str = 'text'
) -> datasets.Dataset

Build the multi-source English ASR training dataset for Qwen-Omni.

Mixes the six public corpora in SOURCES (or a caller-supplied subset), normalises every transcript with normalize_text, and attaches the shared lazy ASR transform so audio is decoded with soundfile (and resampled to sampling_rate) only on item access — never at construction time and never via torchcodec.

Parameters:

sampling_rate
intDefaults to TARGET_SAMPLING_RATE

Target sampling rate in Hz for the decoded waveform.

system_prompt
str | NoneDefaults to None

Optional system-turn instruction. None omits it.

user_prompt
str | NoneDefaults to DEFAULT_USER_PROMPT

User-turn instruction placed before the audio. Defaults to a generic English ASR instruction.

min_audio_duration_seconds
float | NoneDefaults to 1.0

Drop clips shorter than this (header-only probe). Defaults to 1.0 to dodge the Qwen-Omni Whisper feature-extractor sub-second off-by-one.

max_audio_duration_seconds
float | NoneDefaults to 30.0

Drop clips longer than this (header-only probe). Defaults to 30.0 to cap activation memory and avoid per-rank OOM from long clips in the mix. None disables the cap.

shuffle_seed
int | NoneDefaults to 42

Global shuffle seed; None disables shuffling.

sources
list[Source | Mapping[str, Any]] | NoneDefaults to None

Optional subset of SOURCES (e.g. to skip gated corpora for a local smoke run). Defaults to all six.

audio_column
strDefaults to 'audio'

Name of the audio column to standardise on.

text_column
strDefaults to 'text'

Name of the transcript column to standardise on.

Returns: Dataset

A HuggingFace Dataset whose elements are

nemo_automodel.components.datasets.audio.collate_fns.qwen2_5_omni_asr_collate_fn(
    examples: typing.Sequence[typing.Dict[str, typing.Any]],
    processor: typing.Any
) -> typing.Dict[str, torch.Tensor]

Collate Qwen2.5-Omni ASR conversations.

Thin alias over qwen3_omni_asr_collate_fn: the body is processor- agnostic (it only depends on the processor exposing apply_chat_template and the audio= kwarg, both of which Qwen2_5OmniProcessor provides), so the entire Qwen3-Omni-ASR path works unchanged here. We expose a separate symbol so YAML configs can pick the right collate via _target_ without users having to know about the Qwen3-Omni name.

nemo_automodel.components.datasets.audio.collate_fns.qwen3_omni_asr_collate_fn(
    examples: typing.Sequence[typing.Dict[str, typing.Any]],
    processor: typing.Any
) -> typing.Dict[str, torch.Tensor]

Collate Qwen3-Omni ASR conversations into model inputs without qwen_omni_utils.

Unlike qwen3_omni_collate_fn (in vlm.collate_fns), this collate is intended for environments that lack qwen_omni_utils and torchcodec. It assumes audio waveforms are already attached to the conversation as 1-D np.ndarray items of the form {"type": "audio", "audio": waveform} (see nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset) and passes them directly to the processor’s audio= kwarg, which routes to the bundled WhisperFeatureExtractor.

Label masking is delegated to build_labels_from_template, which uses the marker-based fast path that already supports Qwen3OmniMoeProcessor via _IMSTART_TEMPLATE_PROCESSORS. The collate produces pre-shifted labels (labels[:, 1:]) and slices same-shape tensors to [:, :-1] so the downstream loss (MaskedCrossEntropy/FusedLinearCrossEntropy) consumes them without a second internal shift.

Parameters:

examples
Sequence[Dict[str, Any]]

Iterable of dicts each containing a conversation key, where the last turn MUST be an assistant turn with non-empty text.

processor
Any

A Qwen3OmniMoeProcessor instance (or compatible mock).

Returns: Dict[str, torch.Tensor]

Dict with input_ids, attention_mask, input_features,

Raises:

  • ValueError: If any conversation lacks a non-empty assistant turn at the end (the marker-based labeler would otherwise produce all--100 labels and a NaN loss).