Nemotron 3.5 Lightning

Nemotron 3 Superis a large language model (LLM) trained by NVIDIA, designed to deliver strong agentic, reasoning, and conversational capabilities. It is employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. Distinct from the Nano model, the Super model incorporates Multi-Token Prediction (MTP) layers for faster text generation and improved quality, and it is trained using NVFP4 quantization to maximize compute efficiency. The model has 12B active parameters and 120B parameters in total.

NeMo Megatron Bridge supports pretraining, full parameters finetuning, and LoRA finetuning this model. The finetuned model can be converted back to the 🤗 Hugging Face format for downstream evaluation.

Important

Run all commands from /opt/Megatron-Bridge (e.g. docker run -w /opt/Megatron-Bridge ...)

Verified configurations

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/cpu-megatron --torch-dtype bfloat16

Expected result

The pinned checkpoint imports on CPU, creates a reloadable iter_0000000, and preserves every MTP tensor for the paired CPU export. After export, all 6,513 tensors and 32,913,266,240 parameters match the pinned source exactly in name, shape, dtype, and value.

Import · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --tp 1 --pp 1 --ep 8 --etp 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/imported-megatron --torch-dtype bfloat16

Expected result

The command exits successfully and creates a reloadable iter_0000000. After the paired GPU export, all 6,513 tensors and 32,913,266,240 parameters match the pinned source exactly in name, shape, dtype, and value, including the MTP layers.

Export · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/cpu-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3.5-lightning/cpu-hf-export --torch-dtype bfloat16

Expected result

Strict CPU export writes a native NemotronHForCausalLM checkpoint that reloads with Transformers. All 6,513 tensors and 32,913,266,240 parameters match the pinned source bitwise with maximum difference zero.

Export · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --tp 1 --pp 1 --ep 8 --etp 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3.5-lightning/hf-export --torch-dtype bfloat16 --distributed-save

Expected result

Strict export exits successfully; all 6,513 tensors match the pinned source bitwise with original dtypes preserved, maximum difference zero, and Transformers reloads the output natively as NemotronHForCausalLM.

Pretrain · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-07-28

Recorded metrics

Initial loss

12.18927

Final loss

7.239111

Step time · last 10 avg

36,869.270 ms

Model throughput · last 10 avg

200.180 TFLOP/s/GPU

Token throughput · last 10 avg

7,110.095 tokens/s/GPU

Peak allocated memory

68.547 GiB

Peak reserved memory

73.893 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_pretrain_config --mode pretrain --dataset megatron-indexed --seq_length 8192 --max_steps 20 --lr 3e-4 --min_lr 3e-5 --warmup_iters 8 'dataset.blend=[["work/data/rp2/perp_middle/middle_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3.5-lightning-h100/rp2 rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null --save_dir work/model-verification/nemotron-3.5-lightning-h100/pretrain-reference --save_interval 10 checkpoint.async_save=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120

Expected result

On two complete eight-H100 nodes, the real-data RP2 CP2/p2p run completes exactly 20 sequence-length-8,192, GBS/MBS 512/1 optimizer steps with natural routing and finite LM/MTP-1/MTP-2 losses. LM loss is 12.18927 -> 7.239111, steps 11-20 average 36,869.270 ms / 200.180 TFLOP/s/GPU, no iteration is skipped or NaN, and complete 20-file, 430-GiB checkpoints are written at steps 10 and 20.

Pretrain · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-07-28

Recorded metrics

Initial loss

12.17197

Final loss

5.798433

Step time · last 10 avg

21,483.630 ms

Model throughput · last 10 avg

686.920 TFLOP/s/GPU

Token throughput · last 10 avg

24,404.070 tokens/s/GPU

Peak allocated memory

158.220 GiB

Peak reserved memory

162.290 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_pretrain_8k_config --mode pretrain --dataset megatron-indexed --seq_length 8192 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 8 'dataset.blend=[["work/data/rp2/perp_middle/middle_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3.5-lightning-gb200/rp2 rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/pretrain-reference logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, the sequence-length-8,192, GBS/MBS 512/2 real-data RP2 convergence-verification run completes exactly 100 finite-loss optimizer steps with natural routing, Transformer Engine fused cross entropy, CUDA graphs disabled, finite LM/MTP-1/MTP-2 losses, and no skipped or NaN iterations. LM loss is 12.171970 -> 5.798433; real-data steps 91-100 average 21,483.630 ms / 686.920 TFLOP/s/GPU with 158.220 GiB peak allocated and 162.290 GiB peak reserved memory. Complete 15-file, 430-GiB checkpoints with distributed metadata are written at steps 50 and 100.

Pretrain · FSDP · GB200 · bf16

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-07-28

Recorded metrics

Initial loss

12.17197

Final loss

5.797082

Step time · last 10 avg

26,665.830 ms

Model throughput · last 10 avg

553.430 TFLOP/s/GPU

Token throughput · last 10 avg

19,661.417 tokens/s/GPU

Peak allocated memory

160.760 GiB

Peak reserved memory

166.230 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_pretrain_8k_fsdp_config --mode pretrain --dataset megatron-indexed --seq_length 8192 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 8 'dataset.blend=[["work/data/rp2/perp_middle/middle_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3.5-lightning-gb200/rp2 rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/pretrain-fsdp logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, the sequence-length-8,192, GBS/MBS 512/2 BF16 real-data RP2 FSDP convergence-verification run completes exactly 100 finite-loss optimizer steps with natural routing, Transformer Engine fused cross entropy, active MTP-1/MTP-2 losses, and no skipped or NaN iterations. LM loss is 12.171970 -> 5.797082; real-data steps 91-100 average 26,665.830 ms / 553.430 TFLOP/s/GPU with 160.760 GiB peak allocated and 166.230 GiB peak reserved memory. Complete 14-file, 369-GiB FSDP checkpoints with distributed metadata are written at steps 50 and 100. Do not use this run to claim convergence neutrality against ordinary BF16 pretraining: 188 of 300 shared LM/MTP loss values exceed the 1% loss gate, with a maximum relative difference of 12.95%, despite the similar final loss.

Pretrain · FSDP · GB200 · fp8_mx

✓ Verified

Hardware

GB200

Precision

FP8 MX

Last verified

2026-07-24

Recorded metrics

Initial loss

12.19034

Final loss

3.913218

Step time · last 10 avg

13,917.000 ms

Model throughput · last 10 avg

795.390 TFLOP/s/GPU

Token throughput · last 10 avg

28,254.365 tokens/s/GPU

Peak allocated memory

169.540 GiB

Peak reserved memory

173.860 GiB

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_pretrain_8gpu_gb200_fp8mx_fsdp_config --mode pretrain --max_steps 20

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, the sequence-length-8,192 MXFP8 FSDP recipe completes exactly 20 finite-loss optimizer steps with GBS/MBS 384/3, active MTP-1 and MTP-2 losses, and no skipped or NaN iterations. LM loss is 12.19034 -> 3.913218, and steps 11-20 average 13,917.000 ms / 795.390 TFLOP/s/GPU with 169.54 GiB peak allocated and 173.86 GiB peak reserved memory. This run must not be compared with the BF16 real-data pretrain or BF16 FSDP runs: it uses MXFP8, mock data, forced routing, GBS/MBS 384/3, and no CUDA graphs.

SFT · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-08-03

Recorded metrics

Initial loss

0.4406566

Final loss

0.2757806

Step time · last 10 avg

9,887.800 ms

Model throughput · last 10 avg

79.240 TFLOP/s/GPU

Token throughput · last 10 avg

3,313.983 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_sft_openmathinstruct2_packed_config --mode sft --max_steps 100 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning-h100/imported-megatron/iter_0000000 --save_dir work/model-verification/nemotron-3.5-lightning-h100/sft-checkpoints

Expected result

Packed OpenMathInstruct-2 full SFT completes 100 finite-loss steps with active MTP gradients, LM loss 0.4406566 -> 0.2757806, finite MTP-1 loss 0.7371141 -> 0.4133309, and no skipped or NaN iterations. Steps 91-100 average 9,887.800 ms / 79.240 TFLOP/s/GPU, and the complete reloadable full-model checkpoint is written at step 100.

SFT · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-08-04

Recorded metrics

Initial loss

0.4407797

Final loss

0.2762039

Step time · last 10 avg

6,795.830 ms

Model throughput · last 10 avg

256.990 TFLOP/s/GPU

Token throughput · last 10 avg

9,643.561 tokens/s/GPU

Peak allocated memory

101.510 GiB

Peak reserved memory

103.960 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_sft_openmathinstruct2_packed_tp1_config --mode sft --max_steps 100 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning-gb200/imported-megatron/iter_0000000 --save_dir work/model-verification/nemotron-3.5-lightning-gb200/sft-checkpoints

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, packed OpenMathInstruct-2 full SFT completes exactly 100 finite-loss optimizer steps with active MTP gradients. LM loss is 0.4407797 -> 0.2762039, MTP-1 loss is 0.7373838 -> 0.4146959, and MTP-2 loss is 0.8092746 -> 0.4744196. Steps 91-100 average 6,795.830 ms / 256.990 TFLOP/s/GPU with 101.510 GiB peak allocated and 103.960 GiB peak reserved memory. No iteration is skipped or NaN, and the complete 12-file reloadable full-model checkpoint is written at step 100.

Long Context · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-08-03

Recorded metrics

Initial loss

0.4268321

Final loss

0.2650192

Step time · last 10 avg

40,863.010 ms

Model throughput · last 10 avg

188.950 TFLOP/s/GPU

Token throughput · last 10 avg

6,415.191 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_sft_config --mode sft --dataset openmathinstruct2 --max_steps 100 --seq_length 32768 --lr 5e-6 --min_lr 0 --warmup_iters 10 --tensor_model_parallel_size 2 --context_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning-h100/imported-megatron/iter_0000000 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.sequence_parallel=true model.cp_comm_type=p2p model.cross_entropy_loss_fusion=true model.calculate_per_token_loss=true model.recompute_granularity=full model.recompute_modules=null model.recompute_method=uniform model.recompute_num_layers=1 model.moe_router_force_load_balancing=false ddp.average_in_collective=false ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-h100/long-sft-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed OpenMathInstruct-2 CP2 SFT completes exactly 100 finite-loss 32K steps with LM loss 0.4268321 -> 0.2650192 and MTP-1 loss 0.7322379 -> 0.4439273. All 100 steps have finite LM/MTP losses and no skipped or NaN iterations. Steps 91-100 average 40,863.010 ms / 188.950 TFLOP/s/GPU. Complete step-50 and step-100 checkpoints each contain 20 files, distributed metadata, saved run configuration and training state, with latest marker 100.

Long Context · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-08-04

Recorded metrics

Initial loss

0.4268995

Final loss

0.2649925

Step time · last 10 avg

129,692.530 ms

Model throughput · last 10 avg

119.030 TFLOP/s/GPU

Token throughput · last 10 avg

4,042.546 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe nemotron_3_5_lightning_sft_config --mode sft --dataset openmathinstruct2 --max_steps 100 --seq_length 32768 --lr 5e-6 --min_lr 0 --warmup_iters 10 --context_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning-gb200/imported-megatron/iter_0000000 model.moe_flex_dispatcher_backend=hybridep 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false model.calculate_per_token_loss=true model.recompute_granularity=selective 'model.recompute_modules=[moe,layernorm,core_attn,mlp]' model.recompute_method=null model.recompute_num_layers=null model.moe_router_force_load_balancing=false ddp.average_in_collective=false ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/long-sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed OpenMathInstruct-2 CP2 SFT completes exactly 100 finite-loss 32K steps with LM loss 0.4268995 -> 0.2649925 and MTP-1 loss 0.7322624 -> 0.4439029. All 100 steps have finite LM/MTP losses and no skipped or NaN iterations. Steps 91-100 average 129,692.530 ms / 119.030 TFLOP/s/GPU. The complete step-100 checkpoint has distributed metadata, saved run configuration and training state, 12 files, and a latest marker of 100.

LoRA · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-08-03

Recorded metrics

Initial loss

0.4406696

Final loss

0.2854756

Step time · last 10 avg

16,900.360 ms

Model throughput · last 10 avg

92.890 TFLOP/s/GPU

Token throughput · last 10 avg

3,877.787 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe nemotron_3_5_lightning_peft_config --mode lora --dataset openmathinstruct2 --max_steps 100 --seq_length 4096 --lr 1e-4 --min_lr 0 --warmup_iters 10 --tensor_model_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning-h100/imported-megatron/iter_0000000 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:4096,pad_seq_to_mult:2}' model.sequence_parallel=true ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-h100/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed rank-32, alpha-32, zero-dropout LoRA completes 100 finite-loss steps with adapters on the MTP layers, LM loss 0.4406696 -> 0.2854756, finite MTP-1 loss 0.7370828 -> 0.4362708, and no skipped or NaN iterations. The complete adapter checkpoint is written at step 100 and reloads over the pinned base checkpoint at step 100.

LoRA · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-08-04

Recorded metrics

Initial loss

0.4406088

Final loss

0.2855029

Step time · last 10 avg

19,230.400 ms

Model throughput · last 10 avg

81.780 TFLOP/s/GPU

Token throughput · last 10 avg

3,407.937 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe nemotron_3_5_lightning_peft_config --mode lora --dataset openmathinstruct2 --max_steps 100 --seq_length 4096 --lr 1e-4 --min_lr 0 --warmup_iters 10 --tensor_model_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning-gb200/imported-megatron/iter_0000000 model.moe_flex_dispatcher_backend=hybridep 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:4096,pad_seq_to_mult:2}' model.sequence_parallel=true ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed rank-32, alpha-32, zero-dropout LoRA completes 100 finite-loss steps with 48 rank-local adapter attachment records covering MTP attention, expert, and shared-expert layers, LM loss 0.4406088 -> 0.2855029, finite MTP-1 loss 0.7371238 -> 0.4362065, and no skipped or NaN iterations. The complete adapter checkpoint is written at step 100 and reloads over the pinned base checkpoint at step 100.

Benchmark · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-08-03

Recorded metrics

Initial loss

12.16197

Final loss

0.01217863

Step time · last 10 avg

18,467.280 ms

Model throughput · last 10 avg

399.580 TFLOP/s/GPU

Token throughput · last 10 avg

14,195.052 tokens/s/GPU

Peak allocated memory

72.066 GiB

Peak reserved memory

75.340 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_pretrain_16gpu_h100_bf16_config --max_steps 50

Expected result

On 16x H100, the exact mock-data, force-balanced BF16 performance recipe completes exactly 50 steps with finite LM/MTP-1/MTP-2 losses and no skipped or NaN iterations. Transformer Engine CUDA graphs cover attention and Mamba scopes, while expert-FC1 activations are offloaded. Steps 41-50 average 18,467.280 ms and 399.580 model TFLOP/s/GPU; peak rank-0 allocated/reserved memory is 72.066/75.340 GiB. This benchmark-only run stores Adam moments in BF16, so its losses are not convergence-comparable with FP32 optimizer-state runs.

Benchmark · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-07-28

Recorded metrics

Initial loss

12.1825

Final loss

0.03325584

Step time · last 10 avg

18,236.060 ms

Model throughput · last 10 avg

809.220 TFLOP/s/GPU

Token throughput · last 10 avg

28,750.070 tokens/s/GPU

Peak allocated memory

156.850 GiB

Peak reserved memory

161.260 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe nemotron_3_5_lightning_pretrain_8gpu_gb200_bf16_config --max_steps 50

Expected result

On 8x GB200, the exact mock-data BF16 performance recipe completes exactly 50 steps with finite LM/MTP-1/MTP-2 losses and no skipped or NaN iterations. Transformer Engine CUDA graphs cover attention, Mamba, MoE-router, and MoE-preprocess scopes. Steps 41-50 average 18,236.060 ms and 809.220 model TFLOP/s/GPU; peak rank-0 allocated/reserved memory is 156.850/161.260 GiB.