ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsDatasetsAudio

nemo_automodel.components.datasets.audio.datasets

View as Markdown

Audio / ASR dataset builders for NeMo AutoModel.

All audio decoding here is intentionally torchcodec-free: HuggingFace audio columns are cast with Audio(decode=False) and decoded on demand with soundfile (resampling via scipy.signal.resample_poly). This matches the inference-time pattern in result/decode_vllm.py and keeps the builders usable inside environments that do not ship torchcodec.

Module Contents

Functions

NameDescription
_attach_asr_transformAttach the lazy decode-and-build-conversation transform to an ASR dataset.
_build_asr_conversationAssemble the Qwen-Omni ASR chat-template conversation for one sample.
_decode_audio_cell_to_mono_float32Decode a HuggingFace Audio(decode=False) cell to a 1-D float32 waveform.
_filter_audio_durationDrop rows outside [min_seconds, max_seconds] via header-only sf.info.
_filter_nonempty_textDrop rows whose text_column is empty or whitespace (Arrow-level, no decode).
make_cv17_datasetLoad and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning.
make_hf_audio_asr_datasetLazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning.

Data

logger

API

nemo_automodel.components.datasets.audio.datasets._attach_asr_transform(
    dataset: datasets.Dataset,
    audio_column: str = 'audio',
    text_column: str = 'text',
    sampling_rate: int = 16000,
    system_prompt: str | None = None,
    user_prompt: str | None = None
) -> datasets.Dataset

Attach the lazy decode-and-build-conversation transform to an ASR dataset.

The dataset must already expose an audio_column cast with Audio(decode=False) and a (normalized, non-empty) text_column. The returned dataset yields {"conversation": <chat-template list>} per item; audio is decoded with soundfile (mono mix + float32 + optional scipy.signal.resample_poly) only on access, so the transform runs inside dataloader workers rather than at construction time.

The conversation shape follows the prompt-presence matrix:

  • both system_prompt and user_prompt set → system → user(text+audio) → assistant
  • only system_prompt set → system → user(audio) → assistant
  • only user_prompt set → user(text+audio) → assistant (no system turn)
  • neither set → user(audio) → assistant

Whitespace-only prompts are treated as absent.

Parameters:

dataset
Dataset

A HuggingFace Dataset with audio_column (decode=False) and text_column.

audio_column
strDefaults to 'audio'

Name of the audio column.

text_column
strDefaults to 'text'

Name of the transcript column.

sampling_rate
intDefaults to 16000

Target sampling rate in Hz.

system_prompt
str | NoneDefaults to None

Optional system-turn instruction.

user_prompt
str | NoneDefaults to None

Optional user-turn instruction placed before the audio.

Returns: Dataset

The dataset with a with_transform callback attached.

nemo_automodel.components.datasets.audio.datasets._build_asr_conversation(
    waveform: numpy.ndarray,
    transcript: str,
    system_prompt: str | None,
    user_prompt: str | None,
    has_system: bool,
    has_user_text: bool
) -> list[dict[str, typing.Any]]

Assemble the Qwen-Omni ASR chat-template conversation for one sample.

nemo_automodel.components.datasets.audio.datasets._decode_audio_cell_to_mono_float32(
    audio_cell: dict[str, typing.Any],
    target_sampling_rate: int
) -> tuple[numpy.ndarray, int]

Decode a HuggingFace Audio(decode=False) cell to a 1-D float32 waveform.

Avoids torchcodec by using soundfile for both byte and path branches, matching the pattern in result/decode_vllm.py.

Parameters:

audio_cell
dict[str, Any]

Dict with bytes and/or path keys, as returned by HuggingFace datasets when the column has Audio(decode=False).

target_sampling_rate
int

Desired output sampling rate (Hz). If the source differs, the waveform is resampled via scipy.signal.resample_poly.

Returns: tuple[np.ndarray, int]

Tuple of (waveform_float32_mono, target_sampling_rate).

Raises:

  • ValueError: If both bytes and path are missing.
nemo_automodel.components.datasets.audio.datasets._filter_audio_duration(
    dataset: datasets.Dataset,
    audio_column: str,
    min_seconds: float | None = None,
    max_seconds: float | None = None
) -> datasets.Dataset

Drop rows outside [min_seconds, max_seconds] via header-only sf.info.

Uses soundfile.info (header read, no PCM decode) on the Audio(decode=False) cell. The bytes branch wraps the payload in BytesIO; the path branch passes the path directly. Cells that fail to probe are dropped. Bounds are inclusive; pass None to disable that side. The upper bound caps activation memory (long clips inflate the Whisper feature extractor’s input_features and can OOM a rank).

Parameters:

dataset
Dataset

Dataset whose audio_column is cast with Audio(decode=False).

audio_column
str

Name of the audio column.

min_seconds
float | NoneDefaults to None

Inclusive lower bound on duration, or None.

max_seconds
float | NoneDefaults to None

Inclusive upper bound on duration, or None.

Returns: Dataset

The filtered dataset (unchanged if both bounds are None).

nemo_automodel.components.datasets.audio.datasets._filter_nonempty_text(
    dataset: datasets.Dataset,
    text_column: str
) -> datasets.Dataset

Drop rows whose text_column is empty or whitespace (Arrow-level, no decode).

nemo_automodel.components.datasets.audio.datasets.make_cv17_dataset(
    path_or_dataset: str = 'ysdede/commonvoice_17_tr_f...,
    split: str = 'train',
    sampling_rate: int = 16000,
    audio_column: str = 'audio',
    text_column: str = 'transcription',
    load_kwargs: typing.Any = {}
) -> datasets.Dataset

Load and preprocess the CommonVoice 17 dataset for audio-to-text fine-tuning.

Torchcodec-free: the audio column is cast with Audio(decode=False) and decoded lazily via soundfile (_decode_audio_cell_to_mono_float32) inside a with_transform callback, so no audio is decoded at construction time. Each accessed item is {"conversation": [...], "audio": (waveform, sampling_rate)}, matching what :func:phi4_mm_collate_fn consumes.

Parameters:

path_or_dataset
strDefaults to 'ysdede/commonvoice_17_tr_fixed'

HuggingFace dataset id or local path.

split
strDefaults to 'train'

Dataset split to load.

sampling_rate
intDefaults to 16000

Target sampling rate in Hz for the decoded waveform.

audio_column
strDefaults to 'audio'

Name of the audio column.

text_column
strDefaults to 'transcription'

Name of the transcript column.

**load_kwargs
AnyDefaults to {}

Forwarded to datasets.load_dataset.

Returns: Dataset

A HuggingFace Dataset with a lazy decode transform attached.

Raises:

  • ValueError: When audio_column or text_column is missing.
nemo_automodel.components.datasets.audio.datasets.make_hf_audio_asr_dataset(
    path_or_dataset: str,
    split: str = 'train',
    name: str | None = None,
    sampling_rate: int = 16000,
    system_prompt: str | None = None,
    user_prompt: str | None = None,
    audio_column: str = 'audio',
    text_column: str = 'text',
    drop_empty_text: bool = True,
    min_audio_duration_seconds: float | None = None,
    max_audio_duration_seconds: float | None = None,
    load_kwargs: typing.Any = {}
) -> datasets.Dataset

Lazy HuggingFace audio→text dataset builder for Qwen-Omni ASR fine-tuning.

Loads any HuggingFace ASR dataset that exposes an audio column (Audio feature with bytes and/or path populated after cast_column(decode=False)) and a transcript column, and yields the Qwen-Omni chat-template conversation expected by the ASR collate functions. No audio is decoded at construction time — both the soundfile decode (mono mix + float32 cast + optional scipy.signal.resample_poly) and the conversation assembly run inside a HuggingFace with_transform callback, so the only fixed startup cost is the Arrow-level metadata read of the parquet shards (and the on-demand download of those shards if they are not already in the HF cache). Empty-transcript filtering happens via dataset.filter against the text column only — also Arrow-level — so audio bytes are never materialized at startup.

Defaults are tuned for the common case (audio / text columns, 16 kHz, no system turn). Datasets that diverge can override per-field via YAML; see :file:docs/guides/audio/qwen3-omni-asr.md for an override table.

Parameters:

path_or_dataset
str

HuggingFace dataset id or local path.

split
strDefaults to 'train'

Dataset split to load (e.g. "train", "train[:5000]").

name
str | NoneDefaults to None

Optional dataset configuration / subset. Forwarded to datasets.load_dataset(path, name=name, ...). Required by some datasets (e.g. edinburghcstr/ami needs "ihm" or "sdm"; CommonVoice needs the language code).

sampling_rate
intDefaults to 16000

Target sampling rate in Hz. Audio is resampled inside the lazy transform if the source rate differs.

system_prompt
str | NoneDefaults to None

Instruction placed in a system turn. Default None skips the system turn entirely; pass a string to emit one.

user_prompt
str | NoneDefaults to None

Instruction prepended to the audio inside the user turn. Pass None to emit a user turn with only the audio item.

audio_column
strDefaults to 'audio'

Name of the audio column in the source dataset (default "audio" — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech / CommonVoice).

text_column
strDefaults to 'text'

Name of the transcript column (default "text" — works for AMI / LibriSpeech / GigaSpeech / WenetSpeech; override to "sentence" for CommonVoice).

drop_empty_text
boolDefaults to True

If True, samples whose transcript is empty or whitespace are dropped via dataset.filter (Arrow-level, no audio decode). If False, an empty transcript triggers a ValueError inside the transform at access time.

min_audio_duration_seconds
float | NoneDefaults to None

Optional minimum audio duration. Samples shorter than this threshold are dropped via dataset.filter using soundfile.info (header-only read, no full decode). The HF Qwen-Omni Whisper feature extractor has a known off-by-one between input_features and feature_attention_mask for sub-second clips (~0.27 s manifests as a 27-vs-26 frame mismatch); set this to 1.0 for AMI / CommonVoice-style corpora that contain very short utterances.

max_audio_duration_seconds
float | NoneDefaults to None

Optional maximum audio duration. Samples longer than this threshold are dropped via the same header-only soundfile.info probe (no decode). Use this to cap activation memory: long clips inflate the Whisper feature extractor’s input_features (mel frames) and can OOM a rank at larger batch sizes. None disables the cap.

**load_kwargs
AnyDefaults to {}

Forwarded to datasets.load_dataset (e.g. trust_remote_code=True).

Returns: Dataset

A HuggingFace Dataset whose elements are

Raises:

  • ValueError: When audio_column or text_column is missing, when an audio cell has neither bytes nor path, or when drop_empty_text=False and a transcript is empty.
nemo_automodel.components.datasets.audio.datasets.logger = logging.getLogger(__name__)