> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # nemo_automodel.components.datasets.audio.datasets Audio / ASR dataset builders for NeMo AutoModel. All audio decoding here is intentionally `torchcodec`-free: HuggingFace audio columns are cast with `Audio(decode=False)` and decoded on demand with `soundfile` (resampling via `scipy.signal.resample_poly`). This matches the inference-time pattern in `result/decode_vllm.py` and keeps the builders usable inside environments that do not ship `torchcodec`. ## Module Contents ### Classes | Name | Description | | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | [`Cv17DatasetConfig`](#nemo_automodel-components-datasets-audio-datasets-Cv17DatasetConfig) | Construction-time configuration for the CommonVoice 17 dataset. | ### Functions | Name | Description | | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | [`_attach_asr_transform`](#nemo_automodel-components-datasets-audio-datasets-_attach_asr_transform) | Attach the lazy decode-and-build-conversation transform to an ASR dataset. | | [`_build_asr_conversation`](#nemo_automodel-components-datasets-audio-datasets-_build_asr_conversation) | Assemble the Qwen-Omni ASR chat-template conversation for one sample. | | [`_decode_audio_cell_to_mono_float32`](#nemo_automodel-components-datasets-audio-datasets-_decode_audio_cell_to_mono_float32) | Decode a HuggingFace `Audio(decode=False)` cell to a 1-D float32 waveform. | | [`_filter_audio_duration`](#nemo_automodel-components-datasets-audio-datasets-_filter_audio_duration) | Drop rows outside `[min_seconds, max_seconds]` via header-only `sf.info`. | | [`_filter_nonempty_text`](#nemo_automodel-components-datasets-audio-datasets-_filter_nonempty_text) | Drop rows whose `text_column` is empty or whitespace (Arrow-level, no decode). | | [`make_cv17_dataset`](#nemo_automodel-components-datasets-audio-datasets-make_cv17_dataset) | Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning. | | [`make_hf_audio_asr_dataset`](#nemo_automodel-components-datasets-audio-datasets-make_hf_audio_asr_dataset) | Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning. | ### Data [`logger`](#nemo_automodel-components-datasets-audio-datasets-logger) ### API ```python class nemo_automodel.components.datasets.audio.datasets.Cv17DatasetConfig( path_or_dataset: str = 'ysdede/commonvoice_17_tr_f..., split: str = 'train', sampling_rate: int = 16000, audio_column: str = 'audio', text_column: str = 'transcription' ) ``` Dataclass Construction-time configuration for the CommonVoice 17 dataset. **`audio_column`** `str = 'audio'` Name of the audio column. --- **`path_or_dataset`** `str = 'ysdede/commonvoice_17_tr_fixed'` HuggingFace dataset id or local path. --- **`sampling_rate`** `int = 16000` Target audio sampling rate in Hz. --- **`split`** `str = 'train'` Dataset split to load. --- **`text_column`** `str = 'transcription'` Name of the transcript column. --- ```python nemo_automodel.components.datasets.audio.datasets.Cv17DatasetConfig.build() -> datasets.Dataset ``` Build the lazily decoded CommonVoice 17 dataset. ```python nemo_automodel.components.datasets.audio.datasets._attach_asr_transform( dataset: datasets.Dataset, audio_column: str = 'audio', text_column: str = 'text', sampling_rate: int = 16000, system_prompt: str | None = None, user_prompt: str | None = None ) -> datasets.Dataset ``` Attach the lazy decode-and-build-conversation transform to an ASR dataset. The dataset must already expose an `audio_column` cast with `Audio(decode=False)` and a (normalized, non-empty) `text_column`. The returned dataset yields `{"conversation": <chat-template list>}` per item; audio is decoded with `soundfile` (mono mix + `float32` + optional `scipy.signal.resample_poly`) only on access, so the transform runs inside dataloader workers rather than at construction time. The conversation shape follows the prompt-presence matrix: * both `system_prompt` and `user_prompt` set → `system → user(text+audio) → assistant` * only `system_prompt` set → `system → user(audio) → assistant` * only `user_prompt` set → `user(text+audio) → assistant` (no system turn) * neither set → `user(audio) → assistant` Whitespace-only prompts are treated as absent. **Parameters:** **`dataset`** `Dataset` A HuggingFace `Dataset` with `audio_column` (decode=False) and `text_column`. --- **`audio_column`** `str` — default: 'audio' Name of the audio column. --- **`text_column`** `str` — default: 'text' Name of the transcript column. --- **`sampling_rate`** `int` — default: 16000 Target sampling rate in Hz. --- **`system_prompt`** `str | None` — default: None Optional system-turn instruction. --- **`user_prompt`** `str | None` — default: None Optional user-turn instruction placed before the audio. --- **Returns:** `Dataset` The dataset with a `with_transform` callback attached. ```python nemo_automodel.components.datasets.audio.datasets._build_asr_conversation( waveform: numpy.ndarray, transcript: str, system_prompt: str | None, user_prompt: str | None, has_system: bool, has_user_text: bool ) -> list[dict[str, typing.Any]] ``` Assemble the Qwen-Omni ASR chat-template conversation for one sample. ```python nemo_automodel.components.datasets.audio.datasets._decode_audio_cell_to_mono_float32( audio_cell: dict[str, typing.Any], target_sampling_rate: int ) -> tuple[numpy.ndarray, int] ``` Decode a HuggingFace `Audio(decode=False)` cell to a 1-D float32 waveform. Avoids `torchcodec` by using `soundfile` for both byte and path branches, matching the pattern in `result/decode_vllm.py`. **Parameters:** **`audio_cell`** `dict[str, Any]` Dict with `bytes` and/or `path` keys, as returned by HuggingFace `datasets` when the column has `Audio(decode=False)`. --- **`target_sampling_rate`** `int` Desired output sampling rate (Hz). If the source differs, the waveform is resampled via `scipy.signal.resample_poly`. --- **Returns:** `tuple[np.ndarray, int]` Tuple of `(waveform_float32_mono, target_sampling_rate)`. **Raises:** * `ValueError`: If both `bytes` and `path` are missing. ```python nemo_automodel.components.datasets.audio.datasets._filter_audio_duration( dataset: datasets.Dataset, audio_column: str, min_seconds: float | None = None, max_seconds: float | None = None ) -> datasets.Dataset ``` Drop rows outside `[min_seconds, max_seconds]` via header-only `sf.info`. Uses `soundfile.info` (header read, no PCM decode) on the `Audio(decode=False)` cell. The bytes branch wraps the payload in `BytesIO`; the path branch passes the path directly. Cells that fail to probe are dropped. Bounds are inclusive; pass `None` to disable that side. The upper bound caps activation memory (long clips inflate the Whisper feature extractor's `input_features` and can OOM a rank). **Parameters:** **`dataset`** `Dataset` Dataset whose `audio_column` is cast with `Audio(decode=False)`. --- **`audio_column`** `str` Name of the audio column. --- **`min_seconds`** `float | None` — default: None Inclusive lower bound on duration, or `None`. --- **`max_seconds`** `float | None` — default: None Inclusive upper bound on duration, or `None`. --- **Returns:** `Dataset` The filtered dataset (unchanged if both bounds are `None`). ```python nemo_automodel.components.datasets.audio.datasets._filter_nonempty_text( dataset: datasets.Dataset, text_column: str ) -> datasets.Dataset ``` Drop rows whose `text_column` is empty or whitespace (Arrow-level, no decode). ```python nemo_automodel.components.datasets.audio.datasets.make_cv17_dataset( path_or_dataset: str = 'ysdede/commonvoice_17_tr_f..., split: str = 'train', sampling_rate: int = 16000, audio_column: str = 'audio', text_column: str = 'transcription', load_kwargs: typing.Any = {} ) -> datasets.Dataset ``` Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning. Torchcodec-free: the audio column is cast with `Audio(decode=False)` and decoded lazily via `soundfile` (`_decode_audio_cell_to_mono_float32`) inside a `with_transform` callback, so no audio is decoded at construction time. Each accessed item is `{"conversation": [...], "audio": (waveform, sampling_rate)}`, matching what :func:`phi4_mm_collate_fn` consumes. **Parameters:** **`path_or_dataset`** `str` — default: 'ysdede/commonvoice\_17\_tr\_fixed' HuggingFace dataset id or local path. --- **`split`** `str` — default: 'train' Dataset split to load. --- **`sampling_rate`** `int` — default: 16000 Target sampling rate in Hz for the decoded waveform. --- **`audio_column`** `str` — default: 'audio' Name of the audio column. --- **`text_column`** `str` — default: 'transcription' Name of the transcript column. --- **`**load_kwargs`** `Any` — default: \{} Forwarded to `datasets.load_dataset`. --- **Returns:** `Dataset` A HuggingFace `Dataset` with a lazy decode transform attached. **Raises:** * `ValueError`: When `audio_column` or `text_column` is missing. ```python nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset( path_or_dataset: str, split: str = 'train', name: str | None = None, sampling_rate: int = 16000, system_prompt: str | None = None, user_prompt: str | None = None, audio_column: str = 'audio', text_column: str = 'text', drop_empty_text: bool = True, min_audio_duration_seconds: float | None = None, max_audio_duration_seconds: float | None = None, load_kwargs: typing.Any = {} ) -> datasets.Dataset ``` Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning. Loads any HuggingFace ASR dataset that exposes an audio column (`Audio` feature with `bytes` and/or `path` populated after `cast_column(decode=False)`) and a transcript column, and yields the Qwen-Omni chat-template conversation expected by the ASR collate functions. **No audio is decoded at construction time** — both the soundfile decode (mono mix + `float32` cast + optional `scipy.signal.resample_poly`) and the conversation assembly run inside a HuggingFace `with_transform` callback, so the only fixed startup cost is the Arrow-level metadata read of the parquet shards (and the on-demand download of those shards if they are not already in the HF cache). Empty-transcript filtering happens via `dataset.filter` against the text column only — also Arrow-level — so audio bytes are never materialized at startup. Defaults are tuned for the common case (`audio` / `text` columns, 16 kHz, no system turn). Datasets that diverge can override per-field via YAML; see :file:`docs/guides/audio/qwen3-omni-asr.md` for an override table. **Parameters:** **`path_or_dataset`** `str` HuggingFace dataset id or local path. --- **`split`** `str` — default: 'train' Dataset split to load (e.g. `"train"`, `"train[:5000]"`). --- **`name`** `str | None` — default: None Optional dataset configuration / subset. Forwarded to `datasets.load_dataset(path, name=name, ...)`. Required by some datasets (e.g. `edinburghcstr/ami` needs `"ihm"` or `"sdm"`; CommonVoice needs the language code). --- **`sampling_rate`** `int` — default: 16000 Target sampling rate in Hz. Audio is resampled inside the lazy transform if the source rate differs. --- **`system_prompt`** `str | None` — default: None Instruction placed in a `system` turn. Default `None` skips the system turn entirely; pass a string to emit one. --- **`user_prompt`** `str | None` — default: None Instruction prepended to the audio inside the user turn. Pass `None` to emit a user turn with only the audio item. --- **`audio_column`** `str` — default: 'audio' Name of the audio column in the source dataset (default `"audio"` — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech / CommonVoice). --- **`text_column`** `str` — default: 'text' Name of the transcript column (default `"text"` — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech; override to `"sentence"` for CommonVoice). --- **`drop_empty_text`** `bool` — default: True If True, samples whose transcript is empty or whitespace are dropped via `dataset.filter` (Arrow-level, no audio decode). If False, an empty transcript triggers a `ValueError` inside the transform at access time. --- **`min_audio_duration_seconds`** `float | None` — default: None Optional minimum audio duration. Samples shorter than this threshold are dropped via `dataset.filter` using `soundfile.info` (header-only read, no full decode). The HF Qwen-Omni Whisper feature extractor has a known off-by-one between `input_features` and `feature_attention_mask` for sub-second clips (\~0.27 s manifests as a 27-vs-26 frame mismatch); set this to `1.0` for AMI / CommonVoice-style corpora that contain very short utterances. --- **`max_audio_duration_seconds`** `float | None` — default: None Optional maximum audio duration. Samples longer than this threshold are dropped via the same header-only `soundfile.info` probe (no decode). Use this to cap activation memory: long clips inflate the Whisper feature extractor's `input_features` (mel frames) and can OOM a rank at larger batch sizes. `None` disables the cap. --- **`**load_kwargs`** `Any` — default: \{} Forwarded to `datasets.load_dataset` (e.g. `trust_remote_code=True`). --- **Returns:** `Dataset` A HuggingFace `Dataset` whose elements are **Raises:** * `ValueError`: When `audio_column` or `text_column` is missing, when an audio cell has neither `bytes` nor `path`, or when `drop_empty_text=False` and a transcript is empty. ```python nemo_automodel.components.datasets.audio.datasets.logger = logging.getLogger(__name__) ```