`core.models.audio.audio_processor`#

Waveform audio processor for the NeMo Transformer audio frontend.

NemoAudioProcessor is the concrete, data-side feature extractor that the multimodal data pipeline uses to (a) estimate how many encoder/projector embeddings an audio clip expands to (for placeholder expansion and packing) and (b) materialize log-mel features for the audio encoder. It composes the model-frontend descriptors (NemoAudioFeatureConfig and NemoTransformerAudioTokenEstimator) with the vendored standalone log-mel preprocessor.

The data pipeline depends on this only through a small duck-typed interface (compute_num_embeddings / compute_num_frames / materialize plus the cumulative-prefix num_*_from_num_samples primitives), so an audio_ref is treated structurally — this module has no dependency on the data library’s AudioRef type.

Module Contents#

Classes#

NemoAudioProcessor

Waveform audio processor with a NeMo log-mel frontend.

Functions#

`_load_waveform_from_spec`
`_resolve_lazy_media`
`_audio_clip_to_float32`
`_decoder_sample_rate`
`_decode_avdecoder`
`_resolve_sample_rate`
`_audio_num_sample_tolerance`
`_normalize_mono_waveform`
`_infer_num_samples`

Data#

_AUDIO_DURATION_MISMATCH_TOLERANCE_SECONDS

API#

core.models.audio.audio_processor._AUDIO_DURATION_MISMATCH_TOLERANCE_SECONDS#: 0.5

core.models.audio.audio_processor._load_waveform_from_spec( audio_spec: dict[str, Any], ) → tuple[torch.Tensor, int | None]#

core.models.audio.audio_processor._resolve_lazy_media(media: Any) → Any#

core.models.audio.audio_processor._audio_clip_to_float32(clip: torch.Tensor) → torch.Tensor#

core.models.audio.audio_processor._decoder_sample_rate( decoder: Any, sample_rate: int | None, ) → int | None#

core.models.audio.audio_processor._decode_avdecoder( decoder: Any, source_name: str, *, sample_rate: int | None = None, ) → tuple[torch.Tensor, int | None]#

core.models.audio.audio_processor._resolve_sample_rate( audio_ref: Any, decoded_sample_rate: int | None, ) → int | None#

core.models.audio.audio_processor._audio_num_sample_tolerance( audio_ref: Any, decoded_sample_rate: int | None, ) → int#

core.models.audio.audio_processor._normalize_mono_waveform( audio_ref: Any, ) → tuple[torch.Tensor, int | None]#

core.models.audio.audio_processor._infer_num_samples(audio_ref: Any) → int#

class core.models.audio.audio_processor.NemoAudioProcessor( *, token_estimator: megatron.core.models.audio.audio_feature_config.NemoTransformerAudioTokenEstimator, feature_config: megatron.core.models.audio.audio_feature_config.NemoAudioFeatureConfig | None = None, )#

Waveform audio processor with a NeMo log-mel frontend.

Initialization

property input_feature_dim: int#: Number of mel feature bins produced per frame (the encoder input dim).

property sample_rate: int#: Expected input waveform sample rate, in Hz.

_validate_sample_rate( audio_ref: Any, decoded_sample_rate: int | None = None, ) → None#

_compute_num_frames_from_num_samples(num_samples: int) → int#

compute_num_frames(audio_ref: Any) → int#: Number of feature frames the clip described by audio_ref expands to.

num_frames_from_num_samples(num_samples: int) → int#

Pure frame-count math for an audio prefix of num_samples samples.

Slice math: frames_in([s, e)) = num_frames_from_num_samples(e) - num_frames_from_num_samples(s).

num_embeddings_from_num_samples(num_samples: int) → int#

Pure embedding-count math for an audio prefix of num_samples samples.

Slice math: embeds_in([s, e)) = num_embeddings_from_num_samples(e) - num_embeddings_from_num_samples(s).

compute_num_embeddings(audio_ref: Any) → int#: Number of encoder/projector embeddings the clip in audio_ref expands to.

materialize(audio_ref: Any) → tuple[torch.Tensor, int]#: Decode audio_ref and return its (T, n_mels) log-mel features and frame count.

core.models.audio.audio_processor#