Nemotron 3.5 Super VL

A step-by-step guide for fine-tuning Nemotron 3.5 Super VL (121B hybrid Mamba and Attention Mixture-of-Experts (MoE) VLM) to extract structured receipt data from scanned images using NeMo AutoModel. Full SFT on 4 nodes or LoRA on a single node, then inference and evaluation on the CORD-v2 validation split.

What is Nemotron 3.5 Super VL?

Nemotron 3.5 Super VL (nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained, architecture NemotronH_Omni_Reasoning_V3) is the 121B member of the Nemotron 3.5 family.

Key architectural details:

• LLM backbone: NemotronV3 hybrid, 88 layers (40 Mamba2, 8 attention, 40 MoE), hidden dim 4096
• MoE: 512 routed experts per MoE layer, top-22 routing plus 1 shared expert
• Vision encoder: RADIO v2.5-H (ViT-Huge), 512x512 tiles, 256 vision tokens per tile
• Audio: This vision-pretrained checkpoint ships no sound_config, so no audio tower is built
• MTP: The checkpoint ships one multi-token-prediction head; the recipe below keeps it off (num_nextn_predict_layers: 0)
• Parameters: 120.7B trainable with the vision tower frozen (99.31% of the model); ~227 GB in bf16

Fine-Tune for Receipt Field Extraction

This guide fine-tunes the model on CORD-v2 (Consolidated Receipt Dataset) to extract structured fields from scanned receipts:

The base model produces free-form descriptions. After fine-tuning, it outputs structured XML-like token sequences matching the receipt fields.

Guide Overview

Hardware Requirements

• 4 nodes x 8 H100 80 GB (512 experts sharded with ep_size=32; cp_size=4 so the 8-sample global batch is one sample per data-parallel rank)
• SFT memory: ~65 GiB per GPU (fp32 master weights, bf16 Adam moments, activation checkpointing)
• Training time: ~21 min on 4x8 H100 for 400 steps (4 epochs of the 800 training samples), including four validation passes with checkpoint saves; about 26 min wall clock with model loading and the final consolidated export
• LoRA PEFT: one node x 8 H100 (cp_size 1 / ep_size 8, ~39 GiB per GPU). The frozen base needs no fp32 master copy or optimizer state.

Step 0 — Set Up the Environment

# Inside the NeMo AutoModel container (26.04+):
cd /opt/Automodel
uv pip install --python /opt/venv/bin/python ".[vlm-media]"
# Or from a source checkout:
git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel
uv sync --locked --all-groups --all-extras

Step 1 — Explore the CORD-v2 Dataset

CORD-v2 contains scanned receipts with structured ground-truth JSON labels.

import json
from datasets import load_dataset
dataset = load_dataset("naver-clova-ix/cord-v2")
print(f"Train      : {len(dataset['train'])} samples")
print(f"Validation : {len(dataset['validation'])} samples")
print(f"Test       : {len(dataset['test'])} samples")
ex = dataset["train"][0]
gt = json.loads(ex["ground_truth"])["gt_parse"]
print(f"\nGround-truth keys: {list(gt.keys())}")

Expected output:

Train      : 800 samples
Validation : 100 samples
Test       : 100 samples
Ground-truth keys: ['menu', 'sub_total', 'total']

Target Format for JSON-to-Token Conversion

NeMo AutoModel converts structured JSON into an XML-like token sequence using the json2token() function. This is the format the model is trained to produce:

<s_total><s_total_price>45,500</s_total_price><s_changeprice>4,500</s_changeprice>
<s_cashprice>50,000</s_cashprice></s_total><s_menu><s_price>16,500</s_price>
<s_nm>REAL GANACHE</s_nm><s_cnt>1</s_cnt><sep/><s_price>13,000</s_price>
<s_nm>EGG TART</s_nm><s_cnt>1</s_cnt></s_menu>

Step 2 — Training Configuration

Config file: examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2.yaml

recipe: FinetuneRecipeForVLM
step_scheduler:
  global_batch_size: 8
  local_batch_size: 1
  ckpt_every_steps: 400
  val_every_steps: 100
  max_steps: 400
model:
  _target_: nemo_automodel.NeMoAutoModelForImageTextToText.from_pretrained
  pretrained_model_name_or_path: nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained
  trust_remote_code: true
  num_nextn_predict_layers: 0
  torch_dtype: torch.bfloat16
  backend:
    _target_: nemo_automodel.components.models.common.BackendConfig
    attn: te
    linear: te
    rms_norm: torch_fp32
    rope_fusion: false
    fake_balanced_gate: false
    enable_hf_state_dict_adapter: true
checkpoint:
  enabled: true
  checkpoint_dir: vlm_checkpoints/nemotron_3_5_super_vl_cord_v2
  model_save_format: safetensors
  save_consolidated: final
  cpu_offload: true     # stage model/optimizer state dicts on CPU before writing
distributed:
  strategy: fsdp2
  cp_size: 4            # dp 8 on 32 GPUs -> one sample per rank per step
  ep_size: 32           # 512 MoE experts across 32 GPUs
  activation_checkpointing: true
  reshard_after_forward: true
  moe:
    reshard_after_forward: true
  multimodal:
    frozen_sharding: replicate   # frozen RADIO tower kept whole on every rank
freeze_config:
  freeze_vision_tower: true
  freeze_audio_tower: true
  freeze_language_model: false
dataset:
  _target_: nemo_automodel.components.datasets.vlm.datasets.make_cord_v2_dataset
  path_or_dataset: naver-clova-ix/cord-v2
  split: train
dataloader:
  collate_fn:
    _target_: nemo_automodel.components.datasets.vlm.collate_fns.nemotron_omni_collate_fn
    max_length: 4096
optimizer:
  _target_: transformer_engine.pytorch.optimizers.fused_adam.FusedAdam
  lr: 5.0e-5
  weight_decay: 0.01
  betas: [0.9, 0.95]
  master_weights: true
  store_param_remainders: true
  exp_avg_dtype: torch.bfloat16
  exp_avg_sq_dtype: torch.bfloat16

LoRA PEFT Config

Config file: examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml

Same data pipeline and schedule; the base model stays frozen and rank-64 LoRA adapters are trained on the LLM’s linear projections (Mamba in_proj/out_proj, attention q/k/v/o_proj, MoE latent projections and shared-expert MLPs). The vision tower, projector and lm_head are excluded:

peft:
  _target_: nemo_automodel.components._peft.lora.PeftConfig
  match_all_linear: false
  exclude_modules:
    - "*vision_tower*"
    - "*vision_model*"
    - "*vision_projector*"
    - "*audio*"
    - "*sound*"
    - "*lm_head*"
    - "*mlp1*"
  dim: 64
  alpha: 128
  use_triton: true
checkpoint:
  enabled: true
  checkpoint_dir: vlm_checkpoints/nemotron_3_5_super_vl_cord_v2_peft
  model_save_format: safetensors
  save_consolidated: false    # LoRA checkpoints hold the adapter weights only
distributed:
  strategy: fsdp2
  cp_size: 1            # dp 8 on 8 GPUs -> one sample per rank per step
  ep_size: 8
optimizer:
  _target_: transformer_engine.pytorch.optimizers.fused_adam.FusedAdam
  lr: 2e-4
  weight_decay: 0.01
  betas: [0.9, 0.95]
  master_weights: true              # fp32 master copy of the bf16 LoRA weights
  store_param_remainders: true
  exp_avg_dtype: torch.bfloat16
  exp_avg_sq_dtype: torch.bfloat16

Without fp32 master weights and optimizer state for the frozen 121B parameters, LoRA fits on a single node of 8 H100 (~39 GiB per GPU). 177M trainable parameters (0.15%).

Collate Function

Nemotron 3.5 Super VL uses InternVL-style image handling: each <image> token is expanded to 256 vision embeddings per tile during the forward pass. The collate function extracts the images from the conversation, applies the chat template (which adds the <think></think> prefix for the assistant turn), runs the processor, and builds the training labels.

Step 3 — Launch Fine-Tuning

On a Slurm cluster (one task per GPU, 4 nodes):

srun --nodes=4 --ntasks-per-node=8 --gpus-per-node=8 \
    python examples/vlm_finetune/finetune.py \
    -c examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2.yaml

You can also launch the same recipe with the automodel launcher: automodel examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2.yaml --nproc-per-node 8 --nnodes 4. W&B logging is opt-in: add --wandb.enable=true --wandb.entity=<entity> --wandb.project=<project> to either command.

LoRA PEFT

srun --nodes=1 --ntasks-per-node=8 --gpus-per-node=8 \
    python examples/vlm_finetune/finetune.py \
    -c examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml

You can also launch it on any 8-GPU machine with the automodel launcher: automodel examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml --nproc-per-node 8.

Training Log for LoRA PEFT

Trainable parameters: 177,012,736
Trainable parameters percentage: 0.15%
step    0 | loss 0.5054 | grad_norm  1.35 | lr 2.00e-04 | mem 38.60 GiB
step   10 | loss 0.1082 | grad_norm  0.79 | lr 2.00e-04 | mem 38.76 GiB
step   50 | loss 0.0657 | grad_norm  0.49 | lr 2.00e-04 | mem 38.76 GiB
step  100 | loss 0.0268 | grad_norm  0.23 | lr 2.00e-04 | mem 38.76 GiB
step  200 | loss 0.0087 | grad_norm  0.12 | lr 2.00e-04 | mem 38.76 GiB
step  300 | loss 0.0112 | grad_norm  0.18 | lr 2.00e-04 | mem 38.78 GiB
step  399 | loss 0.0067 | grad_norm  0.13 | lr 2.00e-04 | mem 38.76 GiB
Validation:
  step  99 | val_loss 0.0406
  step 199 | val_loss 0.0386  <-- LOWEST_VAL
  step 299 | val_loss 0.0434
  step 399 | val_loss 0.0398

400 steps take ~13 min on 8 H100 (including the four validation passes). Each checkpoint holds the adapter only (~355 MB), so vlm_checkpoints/nemotron_3_5_super_vl_cord_v2_peft/ stays small even with a checkpoint per validation step.

Training Log for Full SFT

Trainable parameters: 120,668,687,360
Trainable parameters percentage: 99.31%
step    0 | loss 0.5057 | grad_norm  6.11 | lr 5.00e-05 | mem 65.02 GiB
step   10 | loss 0.1518 | grad_norm  3.28 | lr 5.00e-05 | mem 65.02 GiB
step   50 | loss 0.0806 | grad_norm  1.01 | lr 5.00e-05 | mem 65.02 GiB
step  100 | loss 0.0839 | grad_norm  1.96 | lr 5.00e-05 | mem 65.02 GiB
step  200 | loss 0.0219 | grad_norm  0.58 | lr 5.00e-05 | mem 65.02 GiB
step  300 | loss 0.0298 | grad_norm  0.88 | lr 5.00e-05 | mem 65.02 GiB
step  399 | loss 0.0114 | grad_norm  0.39 | lr 5.00e-05 | mem 65.02 GiB
Validation:
  step  99 | val_loss 0.0753
  step 199 | val_loss 0.0710
  step 299 | val_loss 0.0684  <-- LOWEST_VAL
  step 399 | val_loss 0.0796

Training loss keeps falling through epoch 4 while validation loss bottoms out at epoch 3 and rises in the last epoch — the 800-receipt training set is small enough for a 121B model to start memorizing it.

Checkpoints Saved

vlm_checkpoints/nemotron_3_5_super_vl_cord_v2/
  epoch_0_step_99/             # sharded weights + optimizer state (~1.4 TB each) + model/consolidate.sh
  epoch_1_step_199/
  epoch_2_step_299/
  epoch_3_step_399/
    model/
      consolidate.sh
      consolidated/            # inline HF export (save_consolidated: final), ~227 GB bf16
        config.json
        model.safetensors.index.json
        model-00001-of-00063.safetensors
        ...
    optim/
    rng/
    dataloader/
  LATEST -> epoch_3_step_399
  LOWEST_VAL -> epoch_2_step_299
  training.jsonl
  validation.jsonl

A checkpoint is written at the end of every epoch (every 100 steps for the 800-receipt training set) and at the final step, so LOWEST_VAL can point at any of them; with fp32 master weights and Adam moments each sharded checkpoint of this model is ~1.4 TB, so budget ~6 TB for the run. Intermediate checkpoints stay sharded; run their model/consolidate.sh to get an HF export of LOWEST_VAL.

Step 4 — Run Inference and Evaluation

Full SFT Inference

Load the consolidated HF export across all visible GPUs (device_map="auto"; the bf16 weights need ~227 GB, so use one node with 8x H100), greedy-decode CORD-v2 validation receipts with the training prompt (<image>\nDescribe this image., enable_thinking=False) and score each prediction against json2token(gt_parse, sort_json_key=True):

• exact match — prediction identical to the ground-truth token sequence
• similarity — normalized sequence similarity (difflib ratio; 1.0 = identical)

# LOWEST_VAL is an intermediate checkpoint, so export it first (the final step is exported inline)
bash vlm_checkpoints/nemotron_3_5_super_vl_cord_v2/LOWEST_VAL/model/consolidate.sh

import difflib, json
import torch
from datasets import load_dataset
from transformers import AutoModel, AutoProcessor
from nemo_automodel.components.datasets.vlm.utils import json2token
CKPT = "vlm_checkpoints/nemotron_3_5_super_vl_cord_v2/LOWEST_VAL/model/consolidated"  # pragma: allowlist secret
NUM_SAMPLES = 20
processor = AutoProcessor.from_pretrained(CKPT, trust_remote_code=True)
model = AutoModel.from_pretrained(CKPT, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto")
# RADIO's `summary_idxs` is a non-persistent buffer and can come back as a meta tensor after loading.
model.vision_model.radio_model.summary_idxs = None
model.eval()
# Same single-turn prompt the collate function trained on; enable_thinking=False emits the empty
# <think></think> assistant prefix.
prompt = processor.tokenizer.apply_chat_template(
    [{"role": "user", "content": "<image>\nDescribe this image."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
# v3 processors return placeholder-expansion metadata that is NOT a generate() kwarg.
PROCESSOR_METADATA_KEYS = ("num_patches", "num_tokens", "imgs_sizes")
@torch.no_grad()
def predict(image, max_new_tokens=1024):
    inputs = processor(text=prompt, images=[image], return_tensors="pt")
    for key in PROCESSOR_METADATA_KEYS:
        inputs.pop(key, None)
    device = next(model.parameters()).device
    inputs = {k: (v.to(device) if isinstance(v, torch.Tensor) else v) for k, v in inputs.items()}
    output_ids = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return processor.tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
dataset = load_dataset("naver-clova-ix/cord-v2", split="validation")
exact, similarity = 0, 0.0
for i in range(NUM_SAMPLES):
    sample = dataset[i]
    target = json2token(json.loads(sample["ground_truth"])["gt_parse"], sort_json_key=True)
    pred = predict(sample["image"].convert("RGB"))
    ratio = difflib.SequenceMatcher(None, pred, target).ratio()
    exact += pred == target
    similarity += ratio
    print(f"[{i}] exact={pred == target} similarity={ratio:.3f}\n  target: {target}\n  pred:   {pred}")
print(f"exact match {exact}/{NUM_SAMPLES} | mean similarity {similarity / NUM_SAMPLES:.3f}")

Pointing CKPT at the base checkpoint id (nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained) scores the un-tuned model the same way.

LoRA PEFT Inference

LoRA checkpoints contain only adapter_model.safetensors + adapter_config.json, saved under the training wrapper’s module names (language_model.model.layers.N.mixer.in_proj, …), which match the HF module tree. Load the base checkpoint as above, fold the adapters into the base weights (W += (B @ A) * alpha / r), then run the same predict loop:

import re
from pathlib import Path
from safetensors import safe_open
BASE = "nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained"
ADAPTER = "vlm_checkpoints/nemotron_3_5_super_vl_cord_v2_peft/LOWEST_VAL/model"  # pragma: allowlist secret
processor = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModel.from_pretrained(BASE, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto")
model.vision_model.radio_model.summary_idxs = None
model.eval()
cfg = json.loads((Path(ADAPTER) / "adapter_config.json").read_text())
scale = cfg["lora_alpha"] / cfg["r"]
pairs = {}
with safe_open(str(Path(ADAPTER) / "adapter_model.safetensors"), framework="pt") as f:
    for key in f.keys():
        m = re.match(r"^(?:base_model\.model\.)?(.+)\.lora_(A|B)\.weight$", key)
        if m:
            pairs.setdefault(m.group(1), {})[m.group(2)] = f.get_tensor(key)
modules = dict(model.named_modules())
with torch.no_grad():
    for fqn, ab in pairs.items():
        weight = modules[fqn].weight
        delta = (ab["B"].to(weight.device, torch.float32) @ ab["A"].to(weight.device, torch.float32)) * scale
        weight.add_(delta.to(weight.dtype))
print(f"merged {len(pairs)} LoRA modules (r={cfg['r']}, alpha={cfg['lora_alpha']})")
# ... then build `prompt` and run the `predict` loop from the SFT example above.

Resources: One node with 8x H100 (bf16 weights spread over the GPUs). Runtime: About 2–4 min to load the 227 GB export, then about 26 s per receipt (greedy, up to 1024 new tokens); 20 samples take about 13 min. The un-tuned base model is slower (~55 s per sample) because its free-form descriptions run to the token limit.

Step 5 — Results

Evaluation on the First 20 CORD-v2 Validation Samples

The checkpoint with the lowest validation loss also decodes best, so evaluate LOWEST_VAL (run its model/consolidate.sh first; the final step is the only one exported inline).

The base model answers Describe this image. with prose (“This is a photograph of a receipt from a restaurant, likely a pizzeria …”). After fine-tuning every prediction is a well-formed <s_total>...</s_total><s_menu>...</s_menu> sequence; the remaining errors are field-level (a dropped <s_cnt> or <s_vatyn>, a <s_sub> block where the label has none) or OCR digits.

Sample Predictions (Fine-Tuned, Step 299, LOWEST_VAL)

Summary