Nemotron 3.5 Super VL#

Nemotron 3.5 Super VL combines a Nemotron-H hybrid language decoder with a RADIO vision encoder, a vision projector, and a separate temporal video embedder. It reuses the Nemotron Omni image/video stack without an audio encoder and retains shared Multi-Token Prediction (MTP) weights.

Megatron Bridge provides checkpoint conversion and pretraining, SFT, and PEFT recipes. The verification records below distinguish verified configurations from pending checks; adding a recipe does not imply that every workflow is verified.

Python recipes live in megatron.bridge.recipes.nemotronh_multimodal, the shared recipe package for Nemotron Omni and Super VL. Existing imports through megatron.bridge.recipes.nemotron_omni remain compatible aliases; CLI recipe names such as nemotron_35_super_vl_sft_config are unchanged.

The freeze_vision_model, freeze_vision_projection, and freeze_language_model model config options independently control which model components are trained. The Super-VL SFT recipes freeze the vision encoder while keeping the projection and language model trainable.

The records below contain refreshed conversion, inference, and training verification with the corrected training callbacks and shared-MTP objective. Verification is scoped to each recorded workflow and hardware configuration; the GB200 pretrain/resume comparison and stronger visual diagnostics are not all passing. The public checkpoint revision remains unbound, and functional throughput measurements are not optimized-performance claims.

For language-only conversion and training without media components, see the text-only companion guide.

Native Hugging Face processor compatibility#

With every supported Transformers release (5.21.1 through 5.23.0), AutoBridge.from_hf_pretrained(..., trust_remote_code=False) uses a Bridge-local processor loading adapter for Super VL. It removes a legacy read-only video configuration field in memory and does not require an audio feature extractor for this audio-free model. The source checkpoint and installed Transformers code are not modified; image/video processing and token expansion use the native HF implementation. Checkpoints that ship remote code still load through it with trust_remote_code=True.

The adapter is version-guarded to the Transformers releases from 5.21.1 through 5.23.0. Processor artifacts can be saved and reloaded through Bridge, but ordinary AutoProcessor.from_pretrained outside Bridge still encounters the native 5.21.1-5.23.0 audio-component requirement. This workaround does not imply standalone HF processor compatibility or full model forward parity. The wrapper’s hf_pretrained.model accessor selects AutoModelForImageTextToText for native Super VL and retains AutoModelForCausalLM for legacy remote-code loading. Ordinary text checkpoints continue to use AutoModelForCausalLM. Filtered text-only conversion views are not directly loadable for HF inference; load their standalone text exports instead. Full native forward parity remains a separate verification requirement.

Known native HF runtime limitations for this checkpoint, as observed with Transformers 5.19.0:

  • The installed mamba-ssm execution path produced non-finite text logits in verification. With that package unavailable to the HF process, the native PyTorch fallback produced bitwise-identical source/export logits and identical 16-token greedy continuations on three prompts. This is a diagnostic runtime condition, not an automatic Bridge workaround; do not remove dependencies from a shared Megatron training environment.

  • Image/video inputs can be prepared with the Bridge processor adapter and passed to the native HF model. This is not standalone HF processor support. Native video forward currently fails because it concatenates an already-tensor pooler_output as though it were a sequence. Video inference is not verified.

  • Native HF does not consume the source checkpoint’s vision_projector.vision_final_layernorm keys. The tested Base checkpoint has identity affine values, but this does not validate loading non-identity values from a fine-tuned checkpoint. Bridge preserves these source weights during conversion; native HF visual reload remains a separate compatibility gap.

These limitations and checks do not update or supersede the historical verification-card results below.

Verified configurations#

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration#

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import · CPU

â—‹ Unverified
Hardware
not specified
Precision
BF16
Last verified
—
Exact command

No runnable command is recorded for this status.

Expected result

CPU-only import is not claimed in this card; the distributed GPU import below is the verified conversion path for this checkpoint.

Import · GPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-10-26
Exact command
Command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 2 --gpus-per-node 4 --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 --hf-revision 1f8953f7f0935358e3605150d22bc15df062061c --megatron-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron --torch-dtype bfloat16 --tp 1 --pp 2 --ep 4 --etp 1 --trust-remote-code
Expected result

Two-node, eight-GPU distributed BF16 import and paired export passed on H100 at this commit with all 43,079 source tensors exact in keys, shapes, dtypes and values (GB200 passed at 3a5b4f61, which has identical conversion code). Compared 248,906,833,960 source bytes, including language, MTP, image encoder, projector, temporal embedder and the vision summary-index buffer. This establishes conversion, not training or numerical forward parity.

Export · CPU

â—‹ Unverified
Hardware
not specified
Precision
BF16
Last verified
—
Exact command

No runnable command is recorded for this status.

Expected result

CPU-only export is not claimed in this card; the distributed GPU export below is the verified conversion path for this checkpoint.

Export · GPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-10-26
Exact command
Command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 2 --gpus-per-node 4 --hf-model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 --megatron-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron --hf-path work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 2 --ep 4 --etp 1 --trust-remote-code --distributed-save
Expected result

Saved round-trip comparison passed on H100 at this commit for all 43,079 tensors with no missing or extra keys and zero value differences (GB200 passed at 3a5b4f61, which has identical conversion code). The H100 exported package reloaded through native Transformers and performed bounded image generation. Exact conversion is separate from forward-logit correlation. The export command reads the source package from the Hub default branch; the card revision does not pin that command.

Pretrain · H100

✓ Verified
Hardware
H100
Precision
BF16
Last verified
2026-10-21
Recorded metrics
Initial loss
3.403948
Final loss
1.916613
Step time · last 10 avg
53,961.150 ms
Model throughput · last 10 avg
127.110 TFLOP/s/GPU
Token throughput · last 10 avg
1,518.129 tokens/s/GPU
Exact command
Command
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8 --recipe nemotron_35_super_vl_pretrain_64gpu_h100_bf16_config --mode pretrain --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000 --max_steps 200 --save_interval 100 --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-pretrain dataset.path=work/data/datacomp-525k-energon dataset.do_validation=false dataset.num_workers=2 model.tensor_model_parallel_size=2 model.sequence_parallel=true model.cross_entropy_fusion_impl=native validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200 checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-pretrain-config/ConfigContainer.yaml ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true
Expected result

Completed 200 finite optimizer steps on 64 H100 GPUs with zero skipped or NaN iterations and complete checkpoints at steps 100 and 200. DataComp image-caption training warm-started from the card's GPU import of the pinned revision at sequence length 4096, GBS/MBS 1280/1, TP2/PP2/EP32/ETP1, natural HybridEP routing, selective recompute and FP32 optimizer state. Shared MTP uses two prediction depths and loss scale 0.3. Losses are the logged LM loss. Rates are padded token slots per GPU, not supervised-token counts. This is bounded support verification, not full convergence.

Pretrain · GB200

â—‹ Unverified
Hardware
GB200
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

Refreshed DataComp pretraining used 64 GB200 GPUs, pretrained weights, natural HybridEP routing, FP32 optimizer state, TP2/PP1/EP64/ETP1, sequence length 8192, GBS/MBS 512/1 and selective moe_act recompute. The original 500-step schedule reached step 475 before its four-hour limit. A fresh-root continuation from step 250 reached step 500 and saved a complete final checkpoint, whose native model reload passed. This is not an uninterrupted 500-step reference. One of 1125 matched loss observations exceeded the unchanged resume tolerance; independent replay remains pending, so no verified pretrain/resume pair is claimed.

SFT · H100

✓ Verified
Hardware
H100
Precision
BF16
Last verified
2026-10-21
Recorded metrics
Initial loss
3.40958
Final loss
1.957656
Step time · last 10 avg
43,528.770 ms
Model throughput · last 10 avg
158.250 TFLOP/s/GPU
Token throughput · last 10 avg
1,881.974 tokens/s/GPU
Exact command
Command
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8 --recipe nemotron_35_super_vl_sft_64gpu_h100_bf16_config --mode sft --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000 --max_steps 200 --save_interval 100 --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft dataset.path=work/data/datacomp-525k-energon dataset.do_validation=false dataset.num_workers=2 model.tensor_model_parallel_size=2 model.sequence_parallel=true model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 model.recompute_modules=[] validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200 checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-sft-config/ConfigContainer.yaml ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true
Expected result

Completed 200 finite optimizer steps on 64 H100 GPUs with zero skipped or NaN iterations and complete checkpoints at steps 100 and 200, starting from the card's GPU import of the pinned revision. DataComp image-caption training uses sequence length 4096, GBS/MBS 1280/1, TP2/PP2/EP32/ETP1, natural (not forced) HybridEP routing with the recipe's expert capacity factor 1.1 and padded expert inputs (token dropping), full recompute, FP32 optimizer state and two shared MTP prediction depths at loss scale 0.3. The vision encoder is frozen; the language model and projection are trainable. Losses are the logged LM loss. Token rates are padded slots, not supervised-token counts. Post-SFT export is checked separately. This verifies image-caption training, not video training or full convergence.

SFT · GB200

✓ Verified
Hardware
GB200
Precision
BF16
Last verified
2026-10-21
Recorded metrics
Initial loss
3.465676
Final loss
1.938715
Step time · last 10 avg
43,286.690 ms
Model throughput · last 10 avg
158.440 TFLOP/s/GPU
Token throughput · last 10 avg
1,892.499 tokens/s/GPU
Exact command
Command
./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4 --recipe nemotron_35_super_vl_sft_64gpu_gb200_bf16_config --mode sft --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000 --max_steps 200 --save_interval 100 --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft dataset.path=work/data/datacomp-525k-energon dataset.do_validation=false dataset.num_workers=2 validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=200 checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-sft-config/ConfigContainer.yaml ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true
Expected result

Completed 200 finite optimizer steps on 64 GB200 GPUs with zero skipped or NaN iterations and complete checkpoints at steps 100 and 200, starting from the card's GPU import of the pinned revision. DataComp image-caption training uses sequence length 4096, GBS/MBS 1280/1, TP2/PP1/EP64/ETP1, natural (not forced) HybridEP routing with the recipe's expert capacity factor 1.1 and padded expert inputs (token dropping), FP32 optimizer state and two shared MTP prediction depths at loss scale 0.3. The vision encoder is frozen; the language model and projection are trainable. Losses are the logged LM loss. Token rates are padded slots, not supervised-token counts. Post-SFT export is checked separately. This verifies image-caption training, not video training or full convergence.

Long Context · H100

â—‹ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

No H100 long-context SFT result is claimed in this card.

Long Context · GB200

â—‹ Unverified
Hardware
GB200
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

No GB200 long-context SFT result is claimed in this card.

LoRA · H100

✓ Verified
Hardware
H100
Precision
BF16
Last verified
2026-10-21
Recorded metrics
Initial loss
2.486903
Final loss
2.589456
Step time · last 10 avg
5,964.470 ms
Model throughput · last 10 avg
57.490 TFLOP/s/GPU
Token throughput · last 10 avg
686.733 tokens/s/GPU
Exact command
Command
./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 8 --recipe nemotron_35_super_vl_peft_16gpu_h100_bf16_config --mode lora --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000 --max_steps 500 --save_interval 250 --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-peft dataset.path=work/data/datacomp-525k-energon dataset.do_validation=false dataset.num_workers=2 validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=500 checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/h100-peft-config/ConfigContainer.yaml ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true
Expected result

Completed 500 finite LoRA optimizer steps on 16 H100 GPUs with zero skipped or NaN iterations and complete checkpoints at steps 250 and 500, starting from the card's GPU import of the pinned revision. DataComp image-caption training uses sequence length 4096, GBS/MBS 16/1, TP4/PP2/EP8/ETP1, natural all-to-all routing and FP32 optimizer state. LoRA uses rank 32, alpha 32 and zero dropout on language-model projections only; the vision encoder and projection are frozen. Losses are the logged LM loss. Token rates are padded slots, not supervised-token counts. This is bounded support verification, not full convergence.

LoRA · GB200

✓ Verified
Hardware
GB200
Precision
BF16
Last verified
2026-10-21
Recorded metrics
Initial loss
3.499214
Final loss
2.041462
Step time · last 10 avg
2,941.140 ms
Model throughput · last 10 avg
116.600 TFLOP/s/GPU
Token throughput · last 10 avg
1,392.657 tokens/s/GPU
Exact command
Command
./scripts/training/train.sh --wait --nodes 4 --gpus-per-node 4 --recipe nemotron_35_super_vl_peft_16gpu_gb200_bf16_config --mode lora --pretrained_checkpoint work/model-verification/nemotron-3.5-super-vl-120b-a12b/gpu-megatron/iter_0000000 --max_steps 500 --save_interval 250 --save_dir work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-peft dataset.path=work/data/datacomp-525k-energon dataset.do_validation=false dataset.num_workers=2 validation.eval_iters=0 validation.eval_interval=0 scheduler.lr_decay_iters=500 checkpoint.load=null checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.async_save=false checkpoint.exit_on_missing_checkpoint=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3.5-super-vl-120b-a12b/gb200-peft-config/ConfigContainer.yaml ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true
Expected result

Completed 500 finite LoRA optimizer steps on 16 GB200 GPUs with zero skipped or NaN iterations and complete checkpoints at steps 250 and 500, starting from the card's GPU import of the pinned revision. DataComp image-caption training uses sequence length 4096, GBS/MBS 16/1, TP2/PP1/EP16/ETP1, natural all-to-all routing and FP32 optimizer state. LoRA uses rank 32, alpha 32 and zero dropout on language-model projections only; the vision encoder and projection are frozen. Losses are the logged LM loss. Token rates are padded slots, not supervised-token counts. This is bounded support verification, not full convergence.