Nemotron 3.5 Super VL

NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained pairs a Nemotron Super NemotronH (hybrid Mamba-2 and Attention) Mixture-of-Experts (MoE) language backbone with a RADIO v2.5-H vision encoder, sharing its NemotronH_Omni_Reasoning_V3 architecture and vision stack with Nemotron-3-Nano-Omni. This midtrain checkpoint is vision-pretrained only (sound_config is absent). There is no audio encoder, so audio inputs are not supported on this checkpoint.


Task	Multimodal (Text·Image)
Architecture	`NemotronH_Omni_Reasoning_V3`
Parameters	121B total (MoE, 512 routed experts, 22 active per token; 124B with the MTP head)
HF Org	nvidia

Available Models

NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained: 121B total (MoE), vision-pretrained, no audio encoder

Architecture

NemotronH_Omni_Reasoning_V3 reuses the same NemotronOmniForConditionalGeneration wrapper, vision tower, and state-dict adapter as NemotronH_Nano_Omni_Reasoning_V3; only the LLM sub-config scale differs (Super compared to Nano). The sound_encoder and sound_projection submodules are constructed only when the checkpoint’s sound_config is present, so this checkpoint loads with audio disabled.

Example Recipes

Recipe	Dataset	Description
nemotron_3_5_super_vl_tulu3_text_100steps.yaml	Tulu-3	Full SFT, text-only (vision/audio towers frozen and unused)
nemotron_3_5_super_vl_tulu3_packed_cp8_100steps.yaml	Tulu-3 (packed, THD)	Context-parallel parity test, `cp_size=8`; text-only packs (`inject_fake_images: false`)
nemotron_3_5_super_vl_cord_v2_peft.yaml	CORD-v2	LoRA (rank 64) on the LLM linear projections, base model frozen; receipt image to structured field tokens; single node

LoRA on CORD-v2

nemotron_3_5_super_vl_cord_v2_peft.yaml fine-tunes the model on CORD-v2 receipt field extraction (the same task as the NemotronOmni CORD-v2 guide) with LoRA adapters instead of full SFT. Because the base weights are frozen, there are no fp32 master weights or optimizer states for them, so the 121B model trains on a single node of 8 H100 (cp_size 1, ep_size 8) instead of the 4 nodes that full SFT needs.


LoRA targets	272 linear projections: Mamba `in_proj`/`out_proj`, attention `q/k/v/o_proj`, MoE `fc1/fc2_latent_proj` and shared experts (routed experts, vision tower, projector, `lm_head` frozen)
Trainable parameters	177M (0.15%), rank 64, alpha 128; TE `FusedAdam` with fp32 master weights, lr 2e-4
Peak memory	38.8 GiB/GPU
Adapter checkpoint	355 MB (`adapter_model.safetensors` + `adapter_config.json`)
Training	400 steps (global batch 8, 4 epochs of the 800 training receipts) in 13 min on one node; train loss 0.51 → 0.01
Validation loss	0.041 / 0.039 / 0.043 / 0.040 at steps 99 / 199 / 299 / 399
Field extraction (first 20 validation receipts, greedy, adapter merged into the base model)	step 199 (`LOWEST_VAL`): 8/20 exact match, mean similarity 0.954 (median 0.997), 20/20 structured outputs

Try with NeMo AutoModel

1. Clone and install from source (full instructions):

git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel
uv sync --locked --all-groups --all-extras --extra vlm-media

2. Run a recipe from inside the repo — LoRA on CORD-v2 on a single node (adapter-only checkpoints, ~39 GiB/GPU):

uv run automodel examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml --nproc-per-node 8

The Tulu-3 text recipe runs on 4 nodes:

uv run automodel examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_tulu3_text_100steps.yaml --nnodes 4 --nproc-per-node 8