> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.datasets.audio.datasets

Audio / ASR dataset builders for NeMo AutoModel.

All audio decoding here is intentionally `torchcodec`-free: HuggingFace audio
columns are cast with `Audio(decode=False)` and decoded on demand with
`soundfile` (resampling via `scipy.signal.resample_poly`). This matches the
inference-time pattern in `result/decode_vllm.py` and keeps the builders
usable inside environments that do not ship `torchcodec`.

## Module Contents

### Classes

| Name                                                                                        | Description                                                     |
| ------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| [`Cv17DatasetConfig`](#nemo_automodel-components-datasets-audio-datasets-Cv17DatasetConfig) | Construction-time configuration for the CommonVoice 17 dataset. |

### Functions

| Name                                                                                                                          | Description                                                                    |
| ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [`_attach_asr_transform`](#nemo_automodel-components-datasets-audio-datasets-_attach_asr_transform)                           | Attach the lazy decode-and-build-conversation transform to an ASR dataset.     |
| [`_build_asr_conversation`](#nemo_automodel-components-datasets-audio-datasets-_build_asr_conversation)                       | Assemble the Qwen-Omni ASR chat-template conversation for one sample.          |
| [`_decode_audio_cell_to_mono_float32`](#nemo_automodel-components-datasets-audio-datasets-_decode_audio_cell_to_mono_float32) | Decode a HuggingFace `Audio(decode=False)` cell to a 1-D float32 waveform.     |
| [`_filter_audio_duration`](#nemo_automodel-components-datasets-audio-datasets-_filter_audio_duration)                         | Drop rows outside `[min_seconds, max_seconds]` via header-only `sf.info`.      |
| [`_filter_nonempty_text`](#nemo_automodel-components-datasets-audio-datasets-_filter_nonempty_text)                           | Drop rows whose `text_column` is empty or whitespace (Arrow-level, no decode). |
| [`make_cv17_dataset`](#nemo_automodel-components-datasets-audio-datasets-make_cv17_dataset)                                   | Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning.  |
| [`make_hf_audio_asr_dataset`](#nemo_automodel-components-datasets-audio-datasets-make_hf_audio_asr_dataset)                   | Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning.     |

### Data

[`logger`](#nemo_automodel-components-datasets-audio-datasets-logger)

### API

```python
class nemo_automodel.components.datasets.audio.datasets.Cv17DatasetConfig(
    path_or_dataset: str = 'ysdede/commonvoice_17_tr_f...,
    split: str = 'train',
    sampling_rate: int = 16000,
    audio_column: str = 'audio',
    text_column: str = 'transcription'
)
```

Dataclass

Construction-time configuration for the CommonVoice 17 dataset.

Name of the audio column.

HuggingFace dataset id or local path.

Target audio sampling rate in Hz.

Dataset split to load.

Name of the transcript column.

```python
nemo_automodel.components.datasets.audio.datasets.Cv17DatasetConfig.build() -> datasets.Dataset
```

Build the lazily decoded CommonVoice 17 dataset.

```python
nemo_automodel.components.datasets.audio.datasets._attach_asr_transform(
    dataset: datasets.Dataset,
    audio_column: str = 'audio',
    text_column: str = 'text',
    sampling_rate: int = 16000,
    system_prompt: str | None = None,
    user_prompt: str | None = None
) -> datasets.Dataset
```

Attach the lazy decode-and-build-conversation transform to an ASR dataset.

The dataset must already expose an `audio_column` cast with
`Audio(decode=False)` and a (normalized, non-empty) `text_column`. The
returned dataset yields `{"conversation": <chat-template list>}` per item;
audio is decoded with `soundfile` (mono mix + `float32` + optional
`scipy.signal.resample_poly`) only on access, so the transform runs inside
dataloader workers rather than at construction time.

The conversation shape follows the prompt-presence matrix:

* both `system_prompt` and `user_prompt` set →
  `system → user(text+audio) → assistant`
* only `system_prompt` set → `system → user(audio) → assistant`
* only `user_prompt` set → `user(text+audio) → assistant`  (no system turn)
* neither set → `user(audio) → assistant`

Whitespace-only prompts are treated as absent.

**Parameters:**

A HuggingFace `Dataset` with `audio_column` (decode=False)
and `text_column`.

Name of the audio column.

Name of the transcript column.

Target sampling rate in Hz.

Optional system-turn instruction.

Optional user-turn instruction placed before the audio.

**Returns:** `Dataset`

The dataset with a `with_transform` callback attached.

```python
nemo_automodel.components.datasets.audio.datasets._build_asr_conversation(
    waveform: numpy.ndarray,
    transcript: str,
    system_prompt: str | None,
    user_prompt: str | None,
    has_system: bool,
    has_user_text: bool
) -> list[dict[str, typing.Any]]
```

Assemble the Qwen-Omni ASR chat-template conversation for one sample.

```python
nemo_automodel.components.datasets.audio.datasets._decode_audio_cell_to_mono_float32(
    audio_cell: dict[str, typing.Any],
    target_sampling_rate: int
) -> tuple[numpy.ndarray, int]
```

Decode a HuggingFace `Audio(decode=False)` cell to a 1-D float32 waveform.

Avoids `torchcodec` by using `soundfile` for both byte and path branches,
matching the pattern in `result/decode_vllm.py`.

**Parameters:**

Dict with `bytes` and/or `path` keys, as returned by
HuggingFace `datasets` when the column has `Audio(decode=False)`.

Desired output sampling rate (Hz). If the source
differs, the waveform is resampled via `scipy.signal.resample_poly`.

**Returns:** `tuple[np.ndarray, int]`

Tuple of `(waveform_float32_mono, target_sampling_rate)`.

**Raises:**

* `ValueError`: If both `bytes` and `path` are missing.

```python
nemo_automodel.components.datasets.audio.datasets._filter_audio_duration(
    dataset: datasets.Dataset,
    audio_column: str,
    min_seconds: float | None = None,
    max_seconds: float | None = None
) -> datasets.Dataset
```

Drop rows outside `[min_seconds, max_seconds]` via header-only `sf.info`.

Uses `soundfile.info` (header read, no PCM decode) on the `Audio(decode=False)`
cell. The bytes branch wraps the payload in `BytesIO`; the path branch passes the
path directly. Cells that fail to probe are dropped. Bounds are inclusive; pass
`None` to disable that side. The upper bound caps activation memory (long clips
inflate the Whisper feature extractor's `input_features` and can OOM a rank).

**Parameters:**

Dataset whose `audio_column` is cast with `Audio(decode=False)`.

Name of the audio column.

Inclusive lower bound on duration, or `None`.

Inclusive upper bound on duration, or `None`.

**Returns:** `Dataset`

The filtered dataset (unchanged if both bounds are `None`).

```python
nemo_automodel.components.datasets.audio.datasets._filter_nonempty_text(
    dataset: datasets.Dataset,
    text_column: str
) -> datasets.Dataset
```

Drop rows whose `text_column` is empty or whitespace (Arrow-level, no decode).

```python
nemo_automodel.components.datasets.audio.datasets.make_cv17_dataset(
    path_or_dataset: str = 'ysdede/commonvoice_17_tr_f...,
    split: str = 'train',
    sampling_rate: int = 16000,
    audio_column: str = 'audio',
    text_column: str = 'transcription',
    load_kwargs: typing.Any = {}
) -> datasets.Dataset
```

Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning.

Torchcodec-free: the audio column is cast with `Audio(decode=False)` and
decoded lazily via `soundfile` (`_decode_audio_cell_to_mono_float32`) inside
a `with_transform` callback, so no audio is decoded at construction time. Each
accessed item is `{"conversation": [...], "audio": (waveform, sampling_rate)}`,
matching what :func:`phi4_mm_collate_fn` consumes.

**Parameters:**

HuggingFace dataset id or local path.

Dataset split to load.

Target sampling rate in Hz for the decoded waveform.

Name of the audio column.

Name of the transcript column.

Forwarded to `datasets.load_dataset`.

**Returns:** `Dataset`

A HuggingFace `Dataset` with a lazy decode transform attached.

**Raises:**

* `ValueError`: When `audio_column` or `text_column` is missing.

```python
nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset(
    path_or_dataset: str,
    split: str = 'train',
    name: str | None = None,
    sampling_rate: int = 16000,
    system_prompt: str | None = None,
    user_prompt: str | None = None,
    audio_column: str = 'audio',
    text_column: str = 'text',
    drop_empty_text: bool = True,
    min_audio_duration_seconds: float | None = None,
    max_audio_duration_seconds: float | None = None,
    load_kwargs: typing.Any = {}
) -> datasets.Dataset
```

Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning.

Loads any HuggingFace ASR dataset that exposes an audio column (`Audio`
feature with `bytes` and/or `path` populated after
`cast_column(decode=False)`) and a transcript column, and yields the
Qwen-Omni chat-template conversation expected by the ASR collate functions.
**No audio is decoded at construction time** — both the soundfile decode
(mono mix + `float32` cast + optional `scipy.signal.resample_poly`) and
the conversation assembly run inside a HuggingFace `with_transform`
callback, so the only fixed startup cost is the Arrow-level metadata read of
the parquet shards (and the on-demand download of those shards if they are
not already in the HF cache). Empty-transcript filtering happens via
`dataset.filter` against the text column only — also Arrow-level — so audio
bytes are never materialized at startup.

Defaults are tuned for the common case (`audio` / `text` columns,
16 kHz, no system turn). Datasets that diverge can override per-field via
YAML; see :file:`docs/guides/audio/qwen3-omni-asr.md` for an override table.

**Parameters:**

HuggingFace dataset id or local path.

Dataset split to load (e.g. `"train"`, `"train[:5000]"`).

Optional dataset configuration / subset. Forwarded to
`datasets.load_dataset(path, name=name, ...)`. Required by some
datasets (e.g. `edinburghcstr/ami` needs `"ihm"` or `"sdm"`;
CommonVoice needs the language code).

Target sampling rate in Hz. Audio is resampled inside
the lazy transform if the source rate differs.

Instruction placed in a `system` turn. Default
`None` skips the system turn entirely; pass a string to emit one.

Instruction prepended to the audio inside the user turn.
Pass `None` to emit a user turn with only the audio item.

Name of the audio column in the source dataset (default
`"audio"` — works for AMI / LibriSpeech / GigaSpeech /
WenetSpeech / CommonVoice).

Name of the transcript column (default `"text"` —
works for AMI / LibriSpeech / GigaSpeech / WenetSpeech; override
to `"sentence"` for CommonVoice).

If True, samples whose transcript is empty or
whitespace are dropped via `dataset.filter` (Arrow-level, no
audio decode). If False, an empty transcript triggers a
`ValueError` inside the transform at access time.

Optional minimum audio duration. Samples
shorter than this threshold are dropped via `dataset.filter`
using `soundfile.info` (header-only read, no full decode). The
HF Qwen-Omni Whisper feature extractor has a known off-by-one
between `input_features` and `feature_attention_mask` for
sub-second clips (\~0.27 s manifests as a 27-vs-26 frame
mismatch); set this to `1.0` for AMI / CommonVoice-style
corpora that contain very short utterances.

Optional maximum audio duration. Samples
longer than this threshold are dropped via the same header-only
`soundfile.info` probe (no decode). Use this to cap activation
memory: long clips inflate the Whisper feature extractor's
`input_features` (mel frames) and can OOM a rank at larger batch
sizes. `None` disables the cap.

Forwarded to `datasets.load_dataset` (e.g.
`trust_remote_code=True`).

**Returns:** `Dataset`

A HuggingFace `Dataset` whose elements are

**Raises:**

* `ValueError`: When `audio_column` or `text_column` is missing, when
  an audio cell has neither `bytes` nor `path`, or when
  `drop_empty_text=False` and a transcript is empty.

```python
nemo_automodel.components.datasets.audio.datasets.logger = logging.getLogger(__name__)
```