> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/datasets/audio/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # nemo_automodel.components.datasets.audio ## Submodules * **[`nemo_automodel.components.datasets.audio.collate_fns`](/nemo-automodel/nemo_automodel/components/datasets/audio/collate_fns)** * **[`nemo_automodel.components.datasets.audio.datasets`](/nemo-automodel/nemo_automodel/components/datasets/audio/datasets)** * **[`nemo_automodel.components.datasets.audio.multi_en`](/nemo-automodel/nemo_automodel/components/datasets/audio/multi_en)** ## Package Contents ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | | [`make_cv17_dataset`](#nemo_automodel-components-datasets-audio-datasets-make_cv17_dataset) | Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning. | | [`make_hf_audio_asr_dataset`](#nemo_automodel-components-datasets-audio-datasets-make_hf_audio_asr_dataset) | Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning. | | [`make_multi_en_asr_dataset`](#nemo_automodel-components-datasets-audio-multi_en-make_multi_en_asr_dataset) | Build the multi-source English ASR training dataset for Qwen-Omni. | | [`qwen2_5_omni_asr_collate_fn`](#nemo_automodel-components-datasets-audio-collate_fns-qwen2_5_omni_asr_collate_fn) | Collate Qwen2.5-Omni ASR conversations. | | [`qwen3_omni_asr_collate_fn`](#nemo_automodel-components-datasets-audio-collate_fns-qwen3_omni_asr_collate_fn) | Collate Qwen3-Omni ASR conversations into model inputs without `qwen_omni_utils`. | ### API ```python nemo_automodel.components.datasets.audio.datasets.make_cv17_dataset( path_or_dataset: str = 'ysdede/commonvoice_17_tr_f..., split: str = 'train', sampling_rate: int = 16000, audio_column: str = 'audio', text_column: str = 'transcription', load_kwargs: typing.Any = {} ) -> datasets.Dataset ``` Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning. Torchcodec-free: the audio column is cast with `Audio(decode=False)` and decoded lazily via `soundfile` (`_decode_audio_cell_to_mono_float32`) inside a `with_transform` callback, so no audio is decoded at construction time. Each accessed item is `{"conversation": [...], "audio": (waveform, sampling_rate)}`, matching what `phi4_mm_collate_fn` consumes. **Parameters:** **`path_or_dataset`** `str` — default: 'ysdede/commonvoice\_17\_tr\_fixed' HuggingFace dataset id or local path. --- **`split`** `str` — default: 'train' Dataset split to load. --- **`sampling_rate`** `int` — default: 16000 Target sampling rate in Hz for the decoded waveform. --- **`audio_column`** `str` — default: 'audio' Name of the audio column. --- **`text_column`** `str` — default: 'transcription' Name of the transcript column. --- **`**load_kwargs`** `Any` — default: \{} Forwarded to `datasets.load_dataset`. --- **Returns:** `Dataset` A HuggingFace `Dataset` with a lazy decode transform attached. **Raises:** * `ValueError`: When `audio_column` or `text_column` is missing. ```python nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset( path_or_dataset: str, split: str = 'train', name: str | None = None, sampling_rate: int = 16000, system_prompt: str | None = None, user_prompt: str | None = None, audio_column: str = 'audio', text_column: str = 'text', drop_empty_text: bool = True, min_audio_duration_seconds: float | None = None, max_audio_duration_seconds: float | None = None, load_kwargs: typing.Any = {} ) -> datasets.Dataset ``` Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning. Loads any HuggingFace ASR dataset that exposes an audio column (`Audio` feature with `bytes` and/or `path` populated after `cast_column(decode=False)`) and a transcript column, and yields the Qwen-Omni chat-template conversation expected by the ASR collate functions. **No audio is decoded at construction time** — both the soundfile decode (mono mix + `float32` cast + optional `scipy.signal.resample_poly`) and the conversation assembly run inside a HuggingFace `with_transform` callback, so the only fixed startup cost is the Arrow-level metadata read of the parquet shards (and the on-demand download of those shards if they are not already in the HF cache). Empty-transcript filtering happens via `dataset.filter` against the text column only — also Arrow-level — so audio bytes are never materialized at startup. Defaults are tuned for the common case (`audio` / `text` columns, 16 kHz, no system turn). Datasets that diverge can override per-field via YAML; see :file:`docs/guides/audio/qwen3-omni-asr.md` for an override table. **Parameters:** **`path_or_dataset`** `str` HuggingFace dataset id or local path. --- **`split`** `str` — default: 'train' Dataset split to load (e.g. `"train"`, `"train[:5000]"`). --- **`name`** `str | None` — default: None Optional dataset configuration / subset. Forwarded to `datasets.load_dataset(path, name=name, ...)`. Required by some datasets (e.g. `edinburghcstr/ami` needs `"ihm"` or `"sdm"`; CommonVoice needs the language code). --- **`sampling_rate`** `int` — default: 16000 Target sampling rate in Hz. Audio is resampled inside the lazy transform if the source rate differs. --- **`system_prompt`** `str | None` — default: None Instruction placed in a `system` turn. Default `None` skips the system turn entirely; pass a string to emit one. --- **`user_prompt`** `str | None` — default: None Instruction prepended to the audio inside the user turn. Pass `None` to emit a user turn with only the audio item. --- **`audio_column`** `str` — default: 'audio' Name of the audio column in the source dataset (default `"audio"` — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech / CommonVoice). --- **`text_column`** `str` — default: 'text' Name of the transcript column (default `"text"` — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech; override to `"sentence"` for CommonVoice). --- **`drop_empty_text`** `bool` — default: True If True, samples whose transcript is empty or whitespace are dropped via `dataset.filter` (Arrow-level, no audio decode). If False, an empty transcript triggers a `ValueError` inside the transform at access time. --- **`min_audio_duration_seconds`** `float | None` — default: None Optional minimum audio duration. Samples shorter than this threshold are dropped via `dataset.filter` using `soundfile.info` (header-only read, no full decode). The HF Qwen-Omni Whisper feature extractor has a known off-by-one between `input_features` and `feature_attention_mask` for sub-second clips (\~0.27 s manifests as a 27-vs-26 frame mismatch); set this to `1.0` for AMI / CommonVoice-style corpora that contain very short utterances. --- **`max_audio_duration_seconds`** `float | None` — default: None Optional maximum audio duration. Samples longer than this threshold are dropped via the same header-only `soundfile.info` probe (no decode). Use this to cap activation memory: long clips inflate the Whisper feature extractor's `input_features` (mel frames) and can OOM a rank at larger batch sizes. `None` disables the cap. --- **`**load_kwargs`** `Any` — default: \{} Forwarded to `datasets.load_dataset` (e.g. `trust_remote_code=True`). --- **Returns:** `Dataset` A HuggingFace `Dataset` whose elements are **Raises:** * `ValueError`: When `audio_column` or `text_column` is missing, when an audio cell has neither `bytes` nor `path`, or when `drop_empty_text=False` and a transcript is empty. ```python nemo_automodel.components.datasets.audio.multi_en.make_multi_en_asr_dataset( sampling_rate: int = TARGET_SAMPLING_RATE, system_prompt: str | None = None, user_prompt: str | None = DEFAULT_USER_PROMPT, min_audio_duration_seconds: float | None = 1.0, max_audio_duration_seconds: float | None = 30.0, shuffle_seed: int | None = 42, sources: list[nemo_automodel.components.datasets.audio.multi_en.Source | collections.abc.Mapping[str, typing.Any]] | None = None, audio_column: str = 'audio', text_column: str = 'text' ) -> datasets.Dataset ``` Build the multi-source English ASR training dataset for Qwen-Omni. Mixes the six public corpora in `SOURCES` (or a caller-supplied subset), normalises every transcript with `normalize_text`, and attaches the shared lazy ASR transform so audio is decoded with `soundfile` (and resampled to `sampling_rate`) only on item access — never at construction time and never via `torchcodec`. **Parameters:** **`sampling_rate`** `int` — default: TARGET\_SAMPLING\_RATE Target sampling rate in Hz for the decoded waveform. --- **`system_prompt`** `str | None` — default: None Optional system-turn instruction. `None` omits it. --- **`user_prompt`** `str | None` — default: DEFAULT\_USER\_PROMPT User-turn instruction placed before the audio. Defaults to a generic English ASR instruction. --- **`min_audio_duration_seconds`** `float | None` — default: 1.0 Drop clips shorter than this (header-only probe). Defaults to `1.0` to dodge the Qwen-Omni Whisper feature-extractor sub-second off-by-one. --- **`max_audio_duration_seconds`** `float | None` — default: 30.0 Drop clips longer than this (header-only probe). Defaults to `30.0` to cap activation memory and avoid per-rank OOM from long clips in the mix. `None` disables the cap. --- **`shuffle_seed`** `int | None` — default: 42 Global shuffle seed; `None` disables shuffling. --- **`sources`** `list[Source | Mapping[str, Any]] | None` — default: None Optional subset of `SOURCES` (e.g. to skip gated corpora for a local smoke run). Defaults to all six. --- **`audio_column`** `str` — default: 'audio' Name of the audio column to standardise on. --- **`text_column`** `str` — default: 'text' Name of the transcript column to standardise on. --- **Returns:** `Dataset` A HuggingFace `Dataset` whose elements are ```python nemo_automodel.components.datasets.audio.collate_fns.qwen2_5_omni_asr_collate_fn( examples: typing.Sequence[typing.Dict[str, typing.Any]], processor: typing.Any ) -> typing.Dict[str, torch.Tensor] ``` Collate Qwen2.5-Omni ASR conversations. Thin alias over `qwen3_omni_asr_collate_fn`: the body is processor- agnostic (it only depends on the processor exposing `apply_chat_template` and the `audio=` kwarg, both of which `Qwen2_5OmniProcessor` provides), so the entire Qwen3-Omni-ASR path works unchanged here. We expose a separate symbol so YAML configs can pick the right collate via `_target_` without users having to know about the Qwen3-Omni name. ```python nemo_automodel.components.datasets.audio.collate_fns.qwen3_omni_asr_collate_fn( examples: typing.Sequence[typing.Dict[str, typing.Any]], processor: typing.Any ) -> typing.Dict[str, torch.Tensor] ``` Collate Qwen3-Omni ASR conversations into model inputs without `qwen_omni_utils`. Unlike `qwen3_omni_collate_fn` (in `vlm.collate_fns`), this collate is intended for environments that lack `qwen_omni_utils` and `torchcodec`. It assumes audio waveforms are already attached to the conversation as 1-D `np.ndarray` items of the form `{"type": "audio", "audio": waveform}` (see `nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset`) and passes them directly to the processor's `audio=` kwarg, which routes to the bundled `WhisperFeatureExtractor`. Label masking is delegated to `build_labels_from_template`, which uses the marker-based fast path that already supports `Qwen3OmniMoeProcessor` via `_IMSTART_TEMPLATE_PROCESSORS`. The collate produces pre-shifted labels (`labels[:, 1:]`) and slices same-shape tensors to `[:, :-1]` so the downstream loss (`MaskedCrossEntropy`/`FusedLinearCrossEntropy`) consumes them without a second internal shift. **Parameters:** **`examples`** `Sequence[Dict[str, Any]]` Iterable of dicts each containing a `conversation` key, where the last turn MUST be an `assistant` turn with non-empty text. --- **`processor`** `Any` A `Qwen3OmniMoeProcessor` instance (or compatible mock). --- **Returns:** `Dict[str, torch.Tensor]` Dict with `input_ids`, `attention_mask`, `input_features`, **Raises:** * `ValueError`: If any conversation lacks a non-empty assistant turn at the end (the marker-based labeler would otherwise produce all-`-100` labels and a NaN loss). ## Docs - [nemo_automodel.components.datasets.audio.collate_fns](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/datasets/audio/collate_fns.md) - [nemo_automodel.components.datasets.audio.datasets](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/datasets/audio/datasets.md) - [nemo_automodel.components.datasets.audio.multi_en](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/datasets/audio/multi_en.md)