Nemotron 3.5 Super VL: Continued Pretraining on FineWeb

This guide shows how to continue training the language backbone of Nemotron 3.5 Super VL on raw FineWeb text. It follows the setup, training, checkpoint, and results structure of the CORD-v2 tutorial and uses a next-token pretraining objective and Megatron-indexed .bin/.idx data.

The model is loaded from nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained. Despite the historical 67B in its name, this checkpoint contains approximately 121B parameters. The complete VL model is retained; its RADIO vision tower and vision projector are frozen and unused during this text-only run. The language backbone is trained with full-parameter updates, with no LoRA adapters. The MTP head is disabled.

What This Run Measures

The recipe uses the first 100,000 documents from one shard in FineWeb’s sample/10BT configuration, producing 69,756,226 tokens, including one EOS/EOD token per document. This is a bounded continued-pretraining example, not a run over all of FineWeb or the full 10-billion-token sample.

The Megatron loader splits documents into 99,000 training and 1,000 held-out validation documents before building sample indices. It concatenates document token streams, uses EOS as the document separator, and samples 4,096-token windows. A window can span multiple documents. The default GPT dataset does not reset attention or Mamba state at EOS within a window.

There is no chat template, user/assistant conversation, or assistant-only loss mask. For tokens [t0, t1, ..., t4096], one training sample has:

input_ids: [t0, t1, ..., t4095]
labels:    [t1, t2, ..., t4096]
loss_mask: [ 1,  1, ...,     1]

The model learns to predict all next tokens, including document-ending EOS tokens. Validation loss is computed on held-out FineWeb text using the same tokenizer and objective. A lower loss on this sample does not establish improved downstream task accuracy or retained image understanding.

Guide Overview

Hardware Requirements

• 8 nodes × 8 H100 80 GB GPUs (64 GPUs) required for this recipe
• Storage: Approximately 232 GiB for the base BF16 checkpoint, and approximately 6 TiB of additional shared storage for checkpoint retention

Run Configuration

Step 0 — Set Up the Environment

Run preparation and training in the NeMo AutoModel container on allocated compute nodes. For a source overlay inside an existing container:

cd /path/to/Automodel
export PYTHONPATH="$PWD:${PYTHONPATH:-}"
# Install these only if they are absent from the environment.
uv pip install --python "$(command -v python3)" huggingface_hub pyarrow wandb

The training environment also needs Transformer Engine, DeepEP, mamba_ssm, causal_conv1d, and a C++ compiler for the Megatron dataset indexing helper. The installation guide describes the complete environment.

Set shared paths accessible from every training node:

export MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-midtrain-67B-vision-pretrained
export MODEL_REVISION=2a6837af6b67067b5a89ceb73cb6d01dfdc71ea8
export HF_HOME=/path/to/shared/hf_cache
export CPT_DATA_ROOT=/path/to/shared/super35_fineweb
export FINEWEB_DATA_DIR="$CPT_DATA_ROOT/megatron"
mkdir -p "$CPT_DATA_ROOT" "$FINEWEB_DATA_DIR"
export WANDB_ENTITY=your-wandb-entity
export WANDB_PROJECT=your-wandb-project
export WANDB_MODE=online
export WANDB_CONSOLE=off

Replace the W&B entity and project placeholders with your values. Authenticate Hugging Face if the model requires access. Download the base model once into the shared cache before starting a multi-node allocation:

MODEL_SNAPSHOT=$(hf download "$MODEL" \
  --revision "$MODEL_REVISION" \
  --cache-dir "$HF_HOME/hub" \
  --quiet)
export MODEL_SNAPSHOT

The base BF16 checkpoint is approximately 232 GiB (249 GB). A full sharded training checkpoint in this configuration occupies approximately 1.35 TiB, including about 227 GiB of model shards and 1.1 TiB of optimizer state. Allow approximately 6 TiB of additional storage for two recent checkpoints, a potentially separate best checkpoint, and a new save in progress.

Step 1 — Download a Reproducible FineWeb Sample

FineWeb is distributed as Parquet containing a text column. The raw download is not yet Megatron training data. This example pins the dataset revision and takes the first 100,000 rows from sample/10BT/000_00000.parquet. The source shard download is about 2.15 GB.

Save the following as prepare_fineweb.py in the repository root:

import hashlib
import json
import os
from pathlib import Path
import pyarrow as pa
import pyarrow.parquet as pq
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
root = Path(os.environ["CPT_DATA_ROOT"])
revision = "9bb295ddab0e05d785b879661af7260fed5140fc"  # pragma: allowlist secret (public FineWeb commit)
source_name = "sample/10BT/000_00000.parquet"
source = hf_hub_download(
    "HuggingFaceFW/fineweb",
    source_name,
    repo_type="dataset",
    revision=revision,
    local_dir=root / "download",
)
batches, remaining = [], 100_000
for batch in pq.ParquetFile(source).iter_batches(batch_size=10_000, columns=["text", "id"]):
    batch = batch.slice(0, min(remaining, batch.num_rows))
    batches.append(batch)
    remaining -= batch.num_rows
    if remaining == 0:
        break
assert remaining == 0
raw = root / "fineweb_100k.parquet"
table = pa.Table.from_batches(batches)
pq.write_table(table, raw)
# Save only tokenizer assets for the standalone preprocessing tool.
tokenizer = AutoTokenizer.from_pretrained(os.environ["MODEL_SNAPSHOT"], trust_remote_code=True)
tokenizer.save_pretrained(root / "tokenizer")
with raw.open("rb") as handle:
    checksum = hashlib.file_digest(handle, "sha256").hexdigest()
manifest = {
    "dataset": "HuggingFaceFW/fineweb",
    "revision": revision,
    "source_file": source_name,
    "selected_rows": [0, 100000],
    "documents": table.num_rows,
    "tokenizer": os.environ["MODEL"],
    "tokenizer_revision": os.environ["MODEL_REVISION"],
    "vocab_size": len(tokenizer),
    "eos_token_id": tokenizer.eos_token_id,
    "raw_sha256": checksum,
}
(root / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
print(json.dumps(manifest, indent=2))

Run the script from the same shell where you exported the shared paths:

python prepare_fineweb.py

Use this model’s tokenizer for preprocessing. Megatron files created for another tokenizer, including GPT-2 or a different Nemotron checkpoint, are not interchangeable.

Step 2 — Tokenize into Megatron .bin/.idx

Run the repository’s existing preprocessing tool. --append-eod appends the tokenizer’s EOS token to each document. Do not apply a chat template or split documents into sentences.

python tools/preprocess_megatron_dataset.py \
  --input "$CPT_DATA_ROOT/fineweb_100k.parquet" \
  --input-type parquet \
  --json-keys text \
  --output-prefix fineweb \
  --output-path "$FINEWEB_DATA_DIR" \
  --pretrained-model-name-or-path "$CPT_DATA_ROOT/tokenizer" \
  --workers 16 \
  --append-eod

Expected outputs:

super35_fineweb/
├── manifest.json
├── fineweb_100k.parquet
├── tokenizer/
└── megatron/
    ├── fineweb_0_text_document.bin
    └── fineweb_0_text_document.idx

The binary file stores token IDs; the index stores document offsets and lengths. Here the token IDs use 32 bits because the vocabulary has 131,072 entries. The output is about 267 MiB of token data plus 2 MiB of index metadata. index_mapping/ is created when the dataset loader builds deterministic document, sample, and shuffle indices.

Save the following as inspect_fineweb.py in the repository root to verify the indexed data:

import hashlib
import json
import os
from pathlib import Path
import numpy as np
import pyarrow.parquet as pq
from transformers import AutoTokenizer
from nemo_automodel.components.datasets.llm.megatron.indexed_dataset import IndexedDataset
root = Path(os.environ["CPT_DATA_ROOT"])
prefix = Path(os.environ["FINEWEB_DATA_DIR"]) / "fineweb_0_text_document"
dataset = IndexedDataset(str(prefix))
tokenizer = AutoTokenizer.from_pretrained(root / "tokenizer")
text = pq.read_table(root / "fineweb_100k.parquet", columns=["text"])["text"][0].as_py()
expected = tokenizer(text).input_ids + [tokenizer.eos_token_id]
assert np.array_equal(dataset[0], expected)
assert len(dataset) == 100_000
token_count = int(dataset.sequence_lengths.sum())
assert token_count == 69_756_226
# Record the derived files so all nodes can use the same verified dataset.
checksums = {}
for suffix in [".bin", ".idx"]:
    path = Path(str(prefix) + suffix)
    with path.open("rb") as handle:
        checksums[path.name] = hashlib.file_digest(handle, "sha256").hexdigest()
manifest_path = root / "manifest.json"
manifest = json.loads(manifest_path.read_text())
manifest["megatron"] = {
    "prefix": str(prefix),
    "documents": len(dataset),
    "tokens_including_eod": token_count,
    "sha256": checksums,
}
manifest_path.write_text(json.dumps(manifest, indent=2) + "\n")
print("documents:", len(dataset))
print("tokens including EOD:", token_count)
print("first document round-trip: passed")
print(json.dumps(manifest["megatron"], indent=2))

Run the script from the same shell where you exported the shared paths:

python inspect_fineweb.py

Expected counts are 100,000 documents and 69,756,226 tokens. The manifest now records the source dataset revision, tokenizer revision, row selection, and SHA-256 checksums of the indexed training files. Set FINEWEB_DATA_DIR to this same shared directory on every training node.

Step 3 — Review the Continued-Pretraining YAML

Use nemotron_3_5_super_vl_fineweb_cpt.yaml.

The TrainFinetuneRecipeForNextTokenPrediction recipe is also the existing LLM pretraining entry point. Its name does not imply an SFT objective: MegatronPretraining supplies raw, shifted next-token targets. from_pretrained loads the existing weights rather than randomly initializing a model. Architecture registration resolves the causal-LM entry point to the full NemotronOmniForConditionalGeneration implementation.

The important data fields are:

dataset:
  _target_: nemo_automodel.components.datasets.llm.megatron_dataset.MegatronPretraining
  paths: /path/to/shared/super35_fineweb/megatron/fineweb_0_text_document
  index_mapping_dir: /path/to/shared/super35_fineweb/megatron/index_mapping
  seq_length: 4096
  seed: 1234
  split: "99,1,0"
  splits_to_build: train

Both dataset sections in the complete YAML use the same prefix, tokenizer, seed, and split ratio. The validation section selects splits_to_build: validation and fixes num_val_samples: 128. The split is applied to documents before sampling windows, so train and validation do not share documents. This held-out split is local to the selected FineWeb sample; FineWeb itself does not supply the validation set used here.

Set paths to the common prefix without .bin or .idx. These are Megatron indexed files, not NanoGPT binary shards. The loader already constructs fixed-length pretraining sequences, so this YAML does not enable the separate VLM/chat packing pipeline.

Create an effective YAML with the downloaded model snapshot, dataset paths, and W&B settings. The script reads the exported environment variables and writes their values directly into the YAML. Both training tokenizers use the same snapshot as preprocessing:

export CPT_RECIPE="$PWD/fineweb_cpt_effective.yaml"

Save the following as prepare_cpt_config.py in the repository root:

import os
from pathlib import Path
import yaml
source = Path("examples/llm_pretrain/nemotron_3_5_super_vl/nemotron_3_5_super_vl_fineweb_cpt.yaml")
config = yaml.safe_load(source.read_text())
snapshot = os.environ["MODEL_SNAPSHOT"]
data_dir = Path(os.environ["FINEWEB_DATA_DIR"])
config["model"]["pretrained_model_name_or_path"] = snapshot
for section in ["dataset", "validation_dataset"]:
    config[section]["paths"] = str(data_dir / "fineweb_0_text_document")
    config[section]["index_mapping_dir"] = str(data_dir / "index_mapping")
    config[section]["tokenizer"]["pretrained_model_name_or_path"] = snapshot
config["wandb"]["enable"] = True
config["wandb"]["entity"] = os.environ["WANDB_ENTITY"]
config["wandb"]["project"] = os.environ["WANDB_PROJECT"]
Path(os.environ["CPT_RECIPE"]).write_text(yaml.safe_dump(config, sort_keys=False))
print(os.environ["CPT_RECIPE"])

Generate the effective YAML:

python prepare_cpt_config.py

The global batch calculation is:

world size = 8 nodes × 8 GPUs = 64
DP size    = world / (TP × CP × PP) = 64 / 4 = 16
accumulation = global batch / (DP size × local batch) = 32 / (16 × 1) = 2
tokens per optimizer step = 32 × 4096 = 131072

The measured results cover 200 completed optimizer steps, corresponding to 26,214,400 target-token positions. Keep max_steps=10000 when reproducing the recorded data stream and learning-rate schedule: the Megatron loader builds its sample indices for that horizon, and the cosine scheduler uses the same horizon after a 10-step warmup. The 200-step curve covers the beginning of this schedule.

Step 4 — Launch with Slurm and Online W&B

The recipe’s example default is wandb.enable: false, following the repository’s opt-in convention. The commands in this tutorial explicitly enable online W&B.

Example fineweb_cpt.sub:

#!/usr/bin/env bash
#SBATCH --account=YOUR_SLURM_ACCOUNT
#SBATCH --partition=YOUR_GPU_PARTITION
#SBATCH --nodes=8
#SBATCH --gpus-per-node=8
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --mem=0
#SBATCH --exclusive
#SBATCH --time=00:45:00
#SBATCH --job-name=super35-fineweb-cpt
#SBATCH --output=super35-cpt-%j.out
#SBATCH --error=super35-cpt-%j.err
set -euo pipefail
export AUTOMODEL_REPO=/path/to/shared/Automodel
export CPT_RECIPE=/path/to/shared/Automodel/fineweb_cpt_effective.yaml
export HF_HOME=/path/to/shared/hf_cache
export FINEWEB_DATA_DIR=/path/to/shared/super35_fineweb/megatron
export CPT_CHECKPOINT_DIR=/path/to/shared/checkpoints/super35_cpt_${SLURM_JOB_ID}
export WANDB_ENTITY=your-wandb-entity
export WANDB_PROJECT=your-wandb-project
export WANDB_MODE=online
export WANDB_CONSOLE=off
export WANDB_RUN_ID=super35cpt${SLURM_JOB_ID}
export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
export MASTER_PORT=29577
: "${WANDB_API_KEY:?Export WANDB_API_KEY before sbatch}"
srun --kill-on-bad-exit=1 \
  --container-image=/path/to/shared/automodel.sqsh \
  --container-mounts=/path/to/shared:/path/to/shared \
  --no-container-mount-home \
  bash -c '
    set -euo pipefail
    cd "$AUTOMODEL_REPO"
    export PYTHONPATH="$AUTOMODEL_REPO:${PYTHONPATH:-}"
    export RANK=$SLURM_PROCID LOCAL_RANK=$SLURM_LOCALID WORLD_SIZE=$SLURM_NTASKS
    export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1
    export PYTORCH_ALLOC_CONF=expandable_segments:True
    export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=8
    python3 examples/llm_pretrain/pretrain.py \
      -c "$CPT_RECIPE" \
      --checkpoint.checkpoint_dir="$CPT_CHECKPOINT_DIR" \
      --wandb.enable=true \
      --wandb.name="super35-fineweb-cpt-${SLURM_JOB_ID}"
  '

Submit:

read -rsp "W&B API key: " WANDB_API_KEY
export WANDB_API_KEY
sbatch fineweb_cpt.sub

The rank-zero log prints the online W&B run URL. View the loss panel against Step with smoothing set to 0 to inspect the raw training loss.

Step 5 — Inspect the Checkpoint and Resume

The checkpoint directory contains training.jsonl and validation.jsonl plus saved training state. The recipe configures a sharded checkpoint every 100 steps. The retention window keeps the latest two complete checkpoints; a checkpoint protected by a pointer such as LOWEST_VAL can also be retained. Consolidated export is disabled in this recipe.

Select a complete checkpoint referenced by LATEST. To create an HF export, run the selected checkpoint’s model/consolidate.sh on a suitably sized compute allocation.

Keep the exact launch command with each run. The checkpoint’s config.yaml records the source YAML and does not include CLI overrides in the version used here. For the measured run, those overrides are materialized in a per-run YAML before submission.

Use the original sharded training checkpoint with checkpoint.restore_from to resume optimizer, scheduler, and dataloader state. Add the following override to the per-rank training command after checking that LATEST resolves to a complete checkpoint:

--checkpoint.restore_from=/path/to/checkpoints/LATEST

Keep the tokenizer, dataset prefix, dataset revision, training and validation split, seed, and batch semantics fixed when resuming training. The learning-rate scheduler restores its checkpointed decay schedule by default: increasing step_scheduler.max_steps alone does not extend the saved learning-rate decay. For this recipe, keep the original 10,000-step schedule when continuing training.

Step 6 — View the Training Loss in W&B

Open your W&B run and select Charts. In the loss panel, use Step for the X axis and loss for the Y axis. Set Smoothing to 0 in the panel’s settings. Open the panel in full-screen view to inspect all 200 training measurements and capture the chart for the tutorial.

Measured Results for 200 Steps

The DFW experiment completed 200 optimizer steps and 8 held-out validation passes, processing 26,214,400 target-token positions.

All 200 training losses and eight validation losses were recorded in W&B and checked against the corresponding Slurm log entries within their printed precision. The screenshot below shows the W&B loss panel with smoothing set to 0.

The screenshot uses W&B’s logged steps 0–199, corresponding to 200 completed optimizer updates. Each point is the raw training loss for that step.