core.models.audio.audio_feature_config

Audio frontend descriptors for the NeMo Transformer audio encoder.

These are the model-side configuration and token-count primitives that describe the NeMo audio frontend:

• NemoAudioFeatureConfig mirrors the NeMo AudioToMelSpectrogramPreprocessor keyword arguments consumed by megatron.core.models.audio.nemo_audio_preprocessing.

• NemoTransformerAudioTokenEstimator is the pure frame-count -> expanded-token math implied by the encoder’s pre-encode/subsampling configuration.

They are deliberately dependency-free (stdlib + math only) so the audio model package carries no data-loader dependency. The data pipeline’s waveform processor (NemoAudioProcessor) composes these and is injected into the dataloader.

Module Contents

Classes

NemoAudioFeatureConfig	Mirrors NeMo AudioToMelSpectrogramPreprocessor.__init__ keyword args.
NemoTransformerAudioTokenEstimator	Pure math for NeMo TransformerEncoder expanded-token counts.

Functions

ceil_div

Returns the ceiling of value / divisor; raises if divisor <= 0.

API

core.models.audio.audio_feature_config.ceil_div(value: int, divisor: int) → int

Returns the ceiling of value / divisor; raises if divisor <= 0.

class core.models.audio.audio_feature_config.NemoAudioFeatureConfig

Mirrors NeMo AudioToMelSpectrogramPreprocessor.__init__ keyword args.

Defaults match the published NeMo transformer_stacking YAML (Slaney mel, per_feature normalize, 0.97 pre-emphasis, log(x + 2**-24)). NemoAudioProcessor forwards these verbatim (via to_nemo_kwargs) to the vendored standalone AudioToMelSpectrogramPreprocessor in megatron.core.models.audio.nemo_audio_preprocessing.

sample_rate: int

16000

window_size: float

0.025

window_stride: float

0.01

n_window_size: int | None

None

n_window_stride: int | None

None

window: str

‘hann’

normalize: str | None

‘per_feature’

n_fft: int | None

512

preemph: float | None

0.97

features: int

128

lowfreq: float

0.0

highfreq: float | None

None

log: bool

True

log_zero_guard_type: str

‘add’

log_zero_guard_value: Any

None

dither: float

1e-05

pad_to: int

0

frame_splicing: int

1

exact_pad: bool

False

pad_value: float

0.0

mag_power: float

2.0

nb_augmentation_prob: float

0.0

nb_max_freq: int

4000

mel_norm: str

‘slaney’

classmethod from_dict(

data: Dict[str, Any],

) → core.models.audio.audio_feature_config.NemoAudioFeatureConfig

Builds a config from data, ignoring keys that are not dataclass fields.

to_nemo_kwargs() → Dict[str, Any]

Returns the kwarg dict consumable by AudioToMelSpectrogramPreprocessor.

class core.models.audio.audio_feature_config.NemoTransformerAudioTokenEstimator

Pure math for NeMo TransformerEncoder expanded-token counts.

Conv-style pre-encoders update lengths with floor division after each strided convolution. The stacking pre-encoder pads the batch tensor to a multiple of encoder_time_stride, then keeps each sample’s partial final stack as one output token, so stacking counts are per-sample ceil divisions.

encoder_time_stride is required and has no default: it must be derived from the loaded encoder config (NemoTransformerAudioConfig.encoder_time_stride, which depends on pre_encode and subsampling_factor). Hard-coding a default here would silently lie for any checkpoint whose encoder downsamples by something other than that constant – e.g. a transformer_stacking encoder with subsampling_factor=8. Construct via the provider (examples/multimodal/v3/energon_multimodal_provider.py) or pass the value explicitly.

encoder_time_stride: int

None

stack_factor: int

1

pre_encode: str

‘conv’

_estimate_encoder_steps(

num_frames: int,

padded_num_frames: int | None = None,

) → int

estimate(num_frames: int, padded_num_frames: int | None = None) → int

Returns the expanded token count for num_frames after encoder and stacking.

estimate_from_num_frames(

num_frames: int,

padded_num_frames: int | None = None,

) → int

Alias for :meth:estimate taking a frame count.

__call__(num_frames: int, padded_num_frames: int | None = None) → int