> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # Nemotron 3.5 Super VL > Fine-tune Nemotron 3.5 Super VL on CORD-v2 for receipt field extraction, then run inference and evaluation. **A step-by-step guide for fine-tuning Nemotron 3.5 Super VL (121B hybrid Mamba and Attention Mixture-of-Experts (MoE) VLM) to extract structured receipt data from scanned images using [NeMo AutoModel](https://github.com/NVIDIA-NeMo/Automodel). Full SFT on 4 nodes or LoRA on a single node, then inference and evaluation on the CORD-v2 validation split.** --- ## What is Nemotron 3.5 Super VL? Nemotron 3.5 Super VL (`nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained`, architecture `NemotronH_Omni_Reasoning_V3`) is the 121B member of the Nemotron 3.5 family. Key architectural details: * **LLM backbone**: NemotronV3 hybrid, 88 layers (40 Mamba2, 8 attention, 40 MoE), hidden dim 4096 * **MoE**: 512 routed experts per MoE layer, top-22 routing plus 1 shared expert * **Vision encoder**: RADIO v2.5-H (ViT-Huge), 512x512 tiles, 256 vision tokens per tile * **Audio**: This vision-pretrained checkpoint ships no `sound_config`, so no audio tower is built * **MTP**: The checkpoint ships one multi-token-prediction head; the recipe below keeps it off (`num_nextn_predict_layers: 0`) * **Parameters**: 120.7B trainable with the vision tower frozen (99.31% of the model); \~227 GB in bf16 ## Fine-Tune for Receipt Field Extraction This guide fine-tunes the model on **CORD-v2** (Consolidated Receipt Dataset) to extract structured fields from scanned receipts: | Field | Example | | ----------- | ------------------------------ | | `menu` | Item names, quantities, prices | | `sub_total` | Subtotal, tax, discount | | `total` | Total price, cash paid, change | The **base model** produces free-form descriptions. After fine-tuning, it outputs **structured XML-like token sequences** matching the receipt fields. ## Guide Overview | Step | Description | | ---------- | ----------------------------------------------------- | | **Step 0** | Environment setup | | **Step 1** | Explore the CORD-v2 dataset | | **Step 2** | Training configuration (full SFT and LoRA) | | **Step 3** | Launch fine-tuning (4 nodes for SFT, 1 node for LoRA) | | **Step 4** | Run inference and evaluation on the fine-tuned model | | **Step 5** | Results | ## Hardware Requirements * **4 nodes x 8 H100 80 GB** (512 experts sharded with `ep_size=32`; `cp_size=4` so the 8-sample global batch is one sample per data-parallel rank) * **SFT memory**: \~65 GiB per GPU (fp32 master weights, bf16 Adam moments, activation checkpointing) * **Training time**: \~21 min on 4x8 H100 for 400 steps (4 epochs of the 800 training samples), including four validation passes with checkpoint saves; about 26 min wall clock with model loading and the final consolidated export * **LoRA PEFT**: one node x 8 H100 (`cp_size 1` / `ep_size 8`, \~39 GiB per GPU). The frozen base needs no fp32 master copy or optimizer state. --- ## Step 0 — Set Up the Environment ```bash # Inside the NeMo AutoModel container (26.04+): cd /opt/Automodel uv pip install --python /opt/venv/bin/python ".[vlm-media]" # Or from a source checkout: git clone https://github.com/NVIDIA-NeMo/Automodel.git cd Automodel uv sync --locked --all-groups --all-extras ``` Nemotron 3.5 Super VL requires `mamba_ssm` and `causal_conv1d` (the `cuda` extra, pre-built in the NeMo AutoModel container), Transformer Engine (`attn: te`, `linear: te`), and DeepEP for expert dispatch. The checkpoint is \~227 GB in bf16 (63 safetensors shards); download it into `$HF_HOME` before launching so that all 32 ranks read from the shared cache. --- ## Step 1 — Explore the CORD-v2 Dataset [CORD-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) contains scanned receipts with structured ground-truth JSON labels. ```python import json from datasets import load_dataset dataset = load_dataset("naver-clova-ix/cord-v2") print(f"Train : {len(dataset['train'])} samples") print(f"Validation : {len(dataset['validation'])} samples") print(f"Test : {len(dataset['test'])} samples") ex = dataset["train"][0] gt = json.loads(ex["ground_truth"])["gt_parse"] print(f"\nGround-truth keys: {list(gt.keys())}") ``` Expected output: ``` Train : 800 samples Validation : 100 samples Test : 100 samples Ground-truth keys: ['menu', 'sub_total', 'total'] ``` ### Target Format for JSON-to-Token Conversion NeMo AutoModel converts structured JSON into an XML-like **token sequence** using the `json2token()` function. This is the format the model is trained to produce: ``` 45,5004,500 50,00016,500 REAL GANACHE113,000 EGG TART1 ``` --- ## Step 2 — Training Configuration **Config file**: `examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2.yaml` ```yaml recipe: FinetuneRecipeForVLM step_scheduler: global_batch_size: 8 local_batch_size: 1 ckpt_every_steps: 400 val_every_steps: 100 max_steps: 400 model: _target_: nemo_automodel.NeMoAutoModelForImageTextToText.from_pretrained pretrained_model_name_or_path: nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained trust_remote_code: true num_nextn_predict_layers: 0 torch_dtype: torch.bfloat16 backend: _target_: nemo_automodel.components.models.common.BackendConfig attn: te linear: te rms_norm: torch_fp32 rope_fusion: false fake_balanced_gate: false enable_hf_state_dict_adapter: true checkpoint: enabled: true checkpoint_dir: vlm_checkpoints/nemotron_3_5_super_vl_cord_v2 model_save_format: safetensors save_consolidated: final cpu_offload: true # stage model/optimizer state dicts on CPU before writing distributed: strategy: fsdp2 cp_size: 4 # dp 8 on 32 GPUs -> one sample per rank per step ep_size: 32 # 512 MoE experts across 32 GPUs activation_checkpointing: true reshard_after_forward: true moe: reshard_after_forward: true multimodal: frozen_sharding: replicate # frozen RADIO tower kept whole on every rank freeze_config: freeze_vision_tower: true freeze_audio_tower: true freeze_language_model: false dataset: _target_: nemo_automodel.components.datasets.vlm.datasets.make_cord_v2_dataset path_or_dataset: naver-clova-ix/cord-v2 split: train dataloader: collate_fn: _target_: nemo_automodel.components.datasets.vlm.collate_fns.nemotron_omni_collate_fn max_length: 4096 optimizer: _target_: transformer_engine.pytorch.optimizers.fused_adam.FusedAdam lr: 5.0e-5 weight_decay: 0.01 betas: [0.9, 0.95] master_weights: true store_param_remainders: true exp_avg_dtype: torch.bfloat16 exp_avg_sq_dtype: torch.bfloat16 ``` ### LoRA PEFT Config **Config file**: `examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml` Same data pipeline and schedule; the base model stays frozen and rank-64 LoRA adapters are trained on the LLM's linear projections (Mamba `in_proj`/`out_proj`, attention `q/k/v/o_proj`, MoE latent projections and shared-expert MLPs). The vision tower, projector and `lm_head` are excluded: ```yaml peft: _target_: nemo_automodel.components._peft.lora.PeftConfig match_all_linear: false exclude_modules: - "*vision_tower*" - "*vision_model*" - "*vision_projector*" - "*audio*" - "*sound*" - "*lm_head*" - "*mlp1*" dim: 64 alpha: 128 use_triton: true checkpoint: enabled: true checkpoint_dir: vlm_checkpoints/nemotron_3_5_super_vl_cord_v2_peft model_save_format: safetensors save_consolidated: false # LoRA checkpoints hold the adapter weights only distributed: strategy: fsdp2 cp_size: 1 # dp 8 on 8 GPUs -> one sample per rank per step ep_size: 8 optimizer: _target_: transformer_engine.pytorch.optimizers.fused_adam.FusedAdam lr: 2e-4 weight_decay: 0.01 betas: [0.9, 0.95] master_weights: true # fp32 master copy of the bf16 LoRA weights store_param_remainders: true exp_avg_dtype: torch.bfloat16 exp_avg_sq_dtype: torch.bfloat16 ``` Without fp32 master weights and optimizer state for the frozen 121B parameters, LoRA fits on **a single node of 8 H100** (\~39 GiB per GPU). 177M trainable parameters (0.15%). ### Collate Function Nemotron 3.5 Super VL uses InternVL-style image handling: each `` token is expanded to 256 vision embeddings per tile during the forward pass. The collate function extracts the images from the conversation, applies the chat template (which adds the `` prefix for the assistant turn), runs the processor, and builds the training labels. --- ## Step 3 — Launch Fine-Tuning On a Slurm cluster (one task per GPU, 4 nodes): ```bash srun --nodes=4 --ntasks-per-node=8 --gpus-per-node=8 \ python examples/vlm_finetune/finetune.py \ -c examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2.yaml ``` You can also launch the same recipe with the `automodel` launcher: `automodel examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2.yaml --nproc-per-node 8 --nnodes 4`. W\&B logging is opt-in: add `--wandb.enable=true --wandb.entity= --wandb.project=` to either command. ### LoRA PEFT ```bash srun --nodes=1 --ntasks-per-node=8 --gpus-per-node=8 \ python examples/vlm_finetune/finetune.py \ -c examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml ``` You can also launch it on any 8-GPU machine with the `automodel` launcher: `automodel examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_cord_v2_peft.yaml --nproc-per-node 8`. ### Training Log for LoRA PEFT ``` Trainable parameters: 177,012,736 Trainable parameters percentage: 0.15% step 0 | loss 0.5054 | grad_norm 1.35 | lr 2.00e-04 | mem 38.60 GiB step 10 | loss 0.1082 | grad_norm 0.79 | lr 2.00e-04 | mem 38.76 GiB step 50 | loss 0.0657 | grad_norm 0.49 | lr 2.00e-04 | mem 38.76 GiB step 100 | loss 0.0268 | grad_norm 0.23 | lr 2.00e-04 | mem 38.76 GiB step 200 | loss 0.0087 | grad_norm 0.12 | lr 2.00e-04 | mem 38.76 GiB step 300 | loss 0.0112 | grad_norm 0.18 | lr 2.00e-04 | mem 38.78 GiB step 399 | loss 0.0067 | grad_norm 0.13 | lr 2.00e-04 | mem 38.76 GiB Validation: step 99 | val_loss 0.0406 step 199 | val_loss 0.0386 <-- LOWEST_VAL step 299 | val_loss 0.0434 step 399 | val_loss 0.0398 ``` 400 steps take \~13 min on 8 H100 (including the four validation passes). Each checkpoint holds the adapter only (\~355 MB), so `vlm_checkpoints/nemotron_3_5_super_vl_cord_v2_peft/` stays small even with a checkpoint per validation step. ### Training Log for Full SFT ``` Trainable parameters: 120,668,687,360 Trainable parameters percentage: 99.31% step 0 | loss 0.5057 | grad_norm 6.11 | lr 5.00e-05 | mem 65.02 GiB step 10 | loss 0.1518 | grad_norm 3.28 | lr 5.00e-05 | mem 65.02 GiB step 50 | loss 0.0806 | grad_norm 1.01 | lr 5.00e-05 | mem 65.02 GiB step 100 | loss 0.0839 | grad_norm 1.96 | lr 5.00e-05 | mem 65.02 GiB step 200 | loss 0.0219 | grad_norm 0.58 | lr 5.00e-05 | mem 65.02 GiB step 300 | loss 0.0298 | grad_norm 0.88 | lr 5.00e-05 | mem 65.02 GiB step 399 | loss 0.0114 | grad_norm 0.39 | lr 5.00e-05 | mem 65.02 GiB Validation: step 99 | val_loss 0.0753 step 199 | val_loss 0.0710 step 299 | val_loss 0.0684 <-- LOWEST_VAL step 399 | val_loss 0.0796 ``` Training loss keeps falling through epoch 4 while validation loss bottoms out at epoch 3 and rises in the last epoch — the 800-receipt training set is small enough for a 121B model to start memorizing it. ### Checkpoints Saved ``` vlm_checkpoints/nemotron_3_5_super_vl_cord_v2/ epoch_0_step_99/ # sharded weights + optimizer state (~1.4 TB each) + model/consolidate.sh epoch_1_step_199/ epoch_2_step_299/ epoch_3_step_399/ model/ consolidate.sh consolidated/ # inline HF export (save_consolidated: final), ~227 GB bf16 config.json model.safetensors.index.json model-00001-of-00063.safetensors ... optim/ rng/ dataloader/ LATEST -> epoch_3_step_399 LOWEST_VAL -> epoch_2_step_299 training.jsonl validation.jsonl ``` A checkpoint is written at the end of every epoch (every 100 steps for the 800-receipt training set) and at the final step, so `LOWEST_VAL` can point at any of them; with fp32 master weights and Adam moments each sharded checkpoint of this model is \~1.4 TB, so budget \~6 TB for the run. Intermediate checkpoints stay sharded; run their `model/consolidate.sh` to get an HF export of `LOWEST_VAL`. --- ## Step 4 — Run Inference and Evaluation ### Full SFT Inference Load the consolidated HF export across all visible GPUs (`device_map="auto"`; the bf16 weights need \~227 GB, so use one node with 8x H100), greedy-decode CORD-v2 validation receipts with the training prompt (`\nDescribe this image.`, `enable_thinking=False`) and score each prediction against `json2token(gt_parse, sort_json_key=True)`: * **exact match** — prediction identical to the ground-truth token sequence * **similarity** — normalized sequence similarity (difflib ratio; 1.0 = identical) ```bash # LOWEST_VAL is an intermediate checkpoint, so export it first (the final step is exported inline) bash vlm_checkpoints/nemotron_3_5_super_vl_cord_v2/LOWEST_VAL/model/consolidate.sh ``` ```python import difflib, json import torch from datasets import load_dataset from transformers import AutoModel, AutoProcessor from nemo_automodel.components.datasets.vlm.utils import json2token CKPT = "vlm_checkpoints/nemotron_3_5_super_vl_cord_v2/LOWEST_VAL/model/consolidated" # pragma: allowlist secret NUM_SAMPLES = 20 processor = AutoProcessor.from_pretrained(CKPT, trust_remote_code=True) model = AutoModel.from_pretrained(CKPT, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto") # RADIO's `summary_idxs` is a non-persistent buffer and can come back as a meta tensor after loading. model.vision_model.radio_model.summary_idxs = None model.eval() # Same single-turn prompt the collate function trained on; enable_thinking=False emits the empty # assistant prefix. prompt = processor.tokenizer.apply_chat_template( [{"role": "user", "content": "\nDescribe this image."}], tokenize=False, add_generation_prompt=True, enable_thinking=False, ) # v3 processors return placeholder-expansion metadata that is NOT a generate() kwarg. PROCESSOR_METADATA_KEYS = ("num_patches", "num_tokens", "imgs_sizes") @torch.no_grad() def predict(image, max_new_tokens=1024): inputs = processor(text=prompt, images=[image], return_tensors="pt") for key in PROCESSOR_METADATA_KEYS: inputs.pop(key, None) device = next(model.parameters()).device inputs = {k: (v.to(device) if isinstance(v, torch.Tensor) else v) for k, v in inputs.items()} output_ids = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False) return processor.tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip() dataset = load_dataset("naver-clova-ix/cord-v2", split="validation") exact, similarity = 0, 0.0 for i in range(NUM_SAMPLES): sample = dataset[i] target = json2token(json.loads(sample["ground_truth"])["gt_parse"], sort_json_key=True) pred = predict(sample["image"].convert("RGB")) ratio = difflib.SequenceMatcher(None, pred, target).ratio() exact += pred == target similarity += ratio print(f"[{i}] exact={pred == target} similarity={ratio:.3f}\n target: {target}\n pred: {pred}") print(f"exact match {exact}/{NUM_SAMPLES} | mean similarity {similarity / NUM_SAMPLES:.3f}") ``` Pointing `CKPT` at the base checkpoint id (`nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained`) scores the un-tuned model the same way. ### LoRA PEFT Inference LoRA checkpoints contain only `adapter_model.safetensors` + `adapter_config.json`, saved under the training wrapper's module names (`language_model.model.layers.N.mixer.in_proj`, ...), which match the HF module tree. Load the base checkpoint as above, fold the adapters into the base weights (`W += (B @ A) * alpha / r`), then run the same `predict` loop: ```python import re from pathlib import Path from safetensors import safe_open BASE = "nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained" ADAPTER = "vlm_checkpoints/nemotron_3_5_super_vl_cord_v2_peft/LOWEST_VAL/model" # pragma: allowlist secret processor = AutoProcessor.from_pretrained(BASE, trust_remote_code=True) model = AutoModel.from_pretrained(BASE, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="auto") model.vision_model.radio_model.summary_idxs = None model.eval() cfg = json.loads((Path(ADAPTER) / "adapter_config.json").read_text()) scale = cfg["lora_alpha"] / cfg["r"] pairs = {} with safe_open(str(Path(ADAPTER) / "adapter_model.safetensors"), framework="pt") as f: for key in f.keys(): m = re.match(r"^(?:base_model\.model\.)?(.+)\.lora_(A|B)\.weight$", key) if m: pairs.setdefault(m.group(1), {})[m.group(2)] = f.get_tensor(key) modules = dict(model.named_modules()) with torch.no_grad(): for fqn, ab in pairs.items(): weight = modules[fqn].weight delta = (ab["B"].to(weight.device, torch.float32) @ ab["A"].to(weight.device, torch.float32)) * scale weight.add_(delta.to(weight.dtype)) print(f"merged {len(pairs)} LoRA modules (r={cfg['r']}, alpha={cfg['lora_alpha']})") # ... then build `prompt` and run the `predict` loop from the SFT example above. ``` **Resources:** One node with 8x H100 (bf16 weights spread over the GPUs). **Runtime:** About 2–4 min to load the 227 GB export, then about 26 s per receipt (greedy, up to 1024 new tokens); 20 samples take about 13 min. The un-tuned base model is slower (\~55 s per sample) because its free-form descriptions run to the token limit. --- ## Step 5 — Results ### Evaluation on the First 20 CORD-v2 Validation Samples | Model | Exact match | Mean similarity | Structured output | | ----------------------------------------------------------------- | ----------- | ----------------------------------------- | ----------------------------------------- | | Base (`NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained`) | 0/20 | 0.048 | 0/20 (free-form description of the photo) | | Full SFT, step 299 (`LOWEST_VAL`, val loss 0.068) | **8/20** | **0.965** (median 0.997; 16/20 above 0.9) | 20/20 | | Full SFT, step 399 (`LATEST`, lr=5e-5, 400 steps) | 5/20 | 0.923 (median 0.989; 14/20 above 0.9) | 20/20 | | LoRA PEFT, step 199 (`LOWEST_VAL`, val loss 0.039) | 8/20 | 0.954 (median 0.997; 15/20 above 0.9) | 20/20 | The checkpoint with the lowest validation loss also decodes best, so evaluate `LOWEST_VAL` (run its `model/consolidate.sh` first; the final step is the only one exported inline). The base model answers `Describe this image.` with prose ("This is a photograph of a receipt from a restaurant, likely a pizzeria ..."). After fine-tuning every prediction is a well-formed `......` sequence; the remaining errors are field-level (a dropped `` or ``, a `` block where the label has none) or OCR digits. ### Sample Predictions (Fine-Tuned, Step 299, `LOWEST_VAL`) | Sample | Ground-truth menu items | Prediction vs. ground truth | Similarity | | ------ | -------------------------------------------------- | ------------------------------------------------------------------------ | ---------- | | 1 | `REAL GANACHE, EGG TART, PIZZA TOAST` | Exact match | 1.00 | | 2 | `Kopi Susu Kolonel` | Same fields, one value differs | 1.00 | | 3 | `S-Ovaltine 50%` | Missing `` | 0.93 | | 4 | `70%, Less Ice, M-Caramel Black Tea` | Exact match | 1.00 | | 5 | `Sedang, BBQ Chicken` | Same fields, one value differs | 1.00 | | 6 | `LE MINERAL` | Same fields, one value differs | 0.99 | | 7 | `POTATO SAUSAGE BREAD, OREO GREEN TEA SPREAD, ...` | Exact match | 1.00 | | 8 | `Choco Devil` | Missing ``; an `` field where the label has none | 0.88 | ### Summary | | Full SFT | LoRA PEFT | | ---------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------- | | Trainable params | 120.7B (99.31%) | 177M (0.15%) | | Learning rate | 5e-5 | 2e-4 (both TE FusedAdam, fp32 master weights) | | GPU memory | \~65 GiB / GPU (32 GPUs) | \~39 GiB / GPU (8 GPUs) | | Training time | \~21 min on 4x8 H100 (\~26 min wall clock) | \~13 min on 1x8 H100 | | Best val loss | 0.068 (step 299) | 0.039 (step 199) | | Final train loss | 0.0114 (step 399; mean of last 10 steps 0.029) | 0.0067 (step 399; mean of last 10 steps 0.014) | | Checkpoint size | \~227 GB (consolidated bf16) | \~355 MB (adapter only) | | Exact matches (20 val) | 8/20 at step 299 (`LOWEST_VAL`, mean similarity 0.96); 5/20 at step 399 (0.92) | 8/20 at step 199 (`LOWEST_VAL`, mean similarity 0.95) |