bridge.recipes.nemotronh_multimodal.h100.nemotron_omni

Nemotron Omni SFT/PEFT recipes (CORD v2 VL, Valor32k-AVQA audio-visual, temporal video).

All recipes use nemotron_omni_step (pass --step_func nemotron_omni_step).

Module Contents

Functions

_make_nemotron_omni_energon_dataset	Create the declarative temporal-video Energon config used by Omni recipes.
nemotron_omni_cord_v2_sft_4gpu_h100_bf16_config	Return a VL SFT config for Nemotron Omni on CORD v2.
nemotron_omni_cord_v2_sft_8gpu_h100_bf16_config	Return the one-node H100 CORD v2 SFT config with natural-routing HybridEP.
nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config	Return an 8K CORD v2 SFT config with in-batch packing and CP2.
nemotron_omni_cord_v2_peft_4gpu_h100_bf16_config	Return a LoRA PEFT config for Nemotron Omni on CORD v2.
nemotron_omni_cord_v2_peft_8gpu_h100_bf16_config	Return the one-node H100 CORD v2 PEFT config with natural-routing HybridEP.
_nemotron_omni_base	Shared model/training config for all Nemotron Omni recipes.
nemotron_omni_valor32k_sft_4gpu_h100_bf16_config	Return an Energon SFT config with temporal video embedder enabled.
nemotron_omni_valor32k_peft_4gpu_h100_bf16_config	LoRA PEFT recipe on temporal-video Energon path (temporal_patch_dim=2).

Data

_DEFAULT_HF_PATH
_DEFAULT_HF_REVISION
_CORD_V2_REVISION
__all__

API

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni._DEFAULT_HF_PATH

‘nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16’

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni._DEFAULT_HF_REVISION

‘24e67ea000b7c2837fc8f9488aa2008524fac8ba’

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni._CORD_V2_REVISION

‘7f0115a4b758a71d6473b8d085751692da2fef98’

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni._make_nemotron_omni_energon_dataset(

micro_batch_size: int,

*,

hf_processor_path: str = _DEFAULT_HF_PATH,

hf_processor_revision: str | None = _DEFAULT_HF_REVISION,

) → megatron.bridge.data.builders.EnergonDatasetConfig

Create the declarative temporal-video Energon config used by Omni recipes.

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_cord_v2_sft_4gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a VL SFT config for Nemotron Omni on CORD v2.

Vision-language finetuning on the CORD v2 receipt parsing dataset. Sound modules are omitted because this dataset contains only image-text samples. Default configuration: 4 GPUs (TP=4). Uses nemotron_omni_step (pass –step_func nemotron_omni_step).

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_cord_v2_sft_8gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return the one-node H100 CORD v2 SFT config with natural-routing HybridEP.

This variant preserves the 4-GPU recipe’s real image-text samples, objective, global batch size, optimizer hyperparameters, and BF16 model arithmetic. TP2 and EP8 fold over the same eight ranks so dense projections and experts are both sharded. A larger microbatch, fused kernels, and selective activation recompute reduce launch overhead while preserving the training objective. Precision-aware Adam stores BF16 gradients/moments and scaled FP16 main parameters.

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return an 8K CORD v2 SFT config with in-batch packing and CP2.

In-batch packing requires a micro batch greater than one. The TP4/CP2 topology needs at least eight GPUs and aligns every packed row to the combined CP/SP multiple. Precision-aware Adam uses scaled FP16 main parameters, BF16 gradients, and BF16 moments so first-step optimizer-state initialization fits within 80 GB H100 memory. The standard all-to-all dispatcher is used because HybridEP did not make forward progress for the packed CP2 workload in the controlled backend screen.

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_cord_v2_peft_4gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a LoRA PEFT config for Nemotron Omni on CORD v2.

LoRA adapters are applied to attention, Mamba, and FC1/FC2 projections. Vision base modules remain frozen and sound modules are omitted. Default configuration: 4 GPUs (TP=4). Uses nemotron_omni_step (pass –step_func nemotron_omni_step).

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_cord_v2_peft_8gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return the one-node H100 CORD v2 PEFT config with natural-routing HybridEP.

The adapter set, freezing policy, real image-text samples, batch sizes, optimizer hyperparameters, and BF16 model arithmetic match the portable 4-GPU recipe. Precision-aware Adam stores BF16 gradients/moments and scaled FP16 main parameters.

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni._nemotron_omni_base() → megatron.bridge.training.config.ConfigContainer

Shared model/training config for all Nemotron Omni recipes.

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_valor32k_sft_4gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return an Energon SFT config with temporal video embedder enabled.

Uses RADIO’s separate_video_embedder to fuse temporal frame pairs (2 consecutive frames → 1 vision embedding) instead of discarding every other frame. The shard path must be set via CLI override: dataset.path=<path>.

Uses nemotron_omni_step (pass --step_func nemotron_omni_step).

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.nemotron_omni_valor32k_peft_4gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

LoRA PEFT recipe on temporal-video Energon path (temporal_patch_dim=2).

Adapters target attention, Mamba, and FC1/FC2 projections. Vision and sound base modules remain frozen while matching adapters are trainable.

bridge.recipes.nemotronh_multimodal.h100.nemotron_omni.__all__

[‘nemotron_omni_cord_v2_long_context_sft_8gpu_h100_bf16_config’, ‘nemotron_omni_cord_v2_peft_4gpu_h1…