Nemotron 3.5 Lightning

NVIDIA Nemotron 3.5 Lightning 30B-A3B BF16 is a large language model trained by NVIDIA for reasoning, agentic, and conversational workloads. It uses a hybrid Mixture-of-Experts architecture with interleaved Mamba-2, MoE, and Attention layers, together with Multi-Token Prediction (MTP). This BF16 reference checkpoint has 30B total parameters and 3B active parameters.

NeMo Megatron Bridge supports pretraining, full-parameter fine-tuning, and LoRA fine-tuning of this model. The fine-tuned model can be converted back to the 🤗 Hugging Face format for downstream evaluation.

Important

Run all commands from /opt/Megatron-Bridge (e.g. docker run -w /opt/Megatron-Bridge ...)

Verified configurations

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/cpu-megatron --torch-dtype bfloat16

Expected result

The pinned checkpoint imports on CPU, creates a reloadable iter_0000000, and preserves every MTP tensor for the paired CPU export. After export, all 6,513 tensors and 32,913,266,240 parameters match the pinned source exactly in name, shape, dtype, and value.

Import · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --tp 1 --pp 1 --ep 8 --etp 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/imported-megatron --torch-dtype bfloat16

Expected result

The command exits successfully and creates a reloadable iter_0000000. After the paired GPU export, all 6,513 tensors and 32,913,266,240 parameters match the pinned source exactly in name, shape, dtype, and value, including the MTP layers.

Export · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/cpu-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3.5-lightning/cpu-hf-export --torch-dtype bfloat16

Expected result

Strict CPU export writes a native NemotronHForCausalLM checkpoint that reloads with Transformers. All 6,513 tensors and 32,913,266,240 parameters match the pinned source bitwise with maximum difference zero.

Export · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-23

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --tp 1 --pp 1 --ep 8 --etp 1 --hf-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --hf-revision b3caaabed0263651a17dc1f2d4ce97e794f76c44 --megatron-path work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3.5-lightning/hf-export --torch-dtype bfloat16 --distributed-save

Expected result

Strict export exits successfully; all 6,513 tensors match the pinned source bitwise with original dtypes preserved, maximum difference zero, and Transformers reloads the output natively as NemotronHForCausalLM.

Pretrain · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

12.18931

Final loss

7.238761

Step time · last 10 avg

36,909.780 ms

Model throughput · last 10 avg

199.960 TFLOP/s/GPU

Token throughput · last 10 avg

7,102.291 tokens/s/GPU

Peak allocated memory

67.753 GiB

Peak reserved memory

70.758 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_pretrain_config --mode pretrain --dataset megatron-indexed --seq_length 8192 --max_steps 20 --lr 3e-4 --min_lr 3e-5 --warmup_iters 8 'dataset.blend=[["work/data/rp2/perp_middle/middle_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3.5-lightning-h100/rp2 rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null --save_dir work/model-verification/nemotron-3.5-lightning-h100/pretrain-reference --save_interval 10 checkpoint.async_save=false logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120

Expected result

On two complete eight-H100 nodes, the real-data RP2 CP2/p2p run completes exactly 20 sequence-length-8,192, GBS/MBS 512/1 optimizer steps with natural routing and finite LM/MTP-1/MTP-2 losses. LM loss is 12.18931 -> 7.238761, steps 11-20 average 36,909.780 ms / 199.960 TFLOP/s/GPU, no iteration is skipped or NaN, and complete 23-file, 430-GiB checkpoints are written at steps 10 and 20.

Pretrain · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

12.17197

Final loss

5.785674

Step time · last 10 avg

21,296.790 ms

Model throughput · last 10 avg

692.930 TFLOP/s/GPU

Token throughput · last 10 avg

24,618.170 tokens/s/GPU

Peak allocated memory

162.430 GiB

Peak reserved memory

166.630 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_pretrain_8k_config --mode pretrain --dataset megatron-indexed --seq_length 8192 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 8 'dataset.blend=[["work/data/rp2/perp_middle/middle_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3.5-lightning-gb200/rp2 rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/pretrain-reference logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, the sequence-length-8,192, GBS/MBS 512/2 real-data RP2 convergence-verification run completes exactly 100 finite-loss optimizer steps with natural routing, Transformer Engine fused cross entropy, CUDA graphs disabled, finite LM/MTP-1/MTP-2 losses, and no skipped or NaN iterations. LM loss is 12.171970 -> 5.785674; real-data steps 91-100 average 21,296.790 ms / 692.930 TFLOP/s/GPU with 162.430 GiB peak allocated and 166.630 GiB peak reserved memory. Complete 15-file, 430-GiB checkpoints with distributed metadata are written at steps 50 and 100.

Pretrain · FSDP · GB200 · bf16

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

12.17197

Final loss

5.812089

Step time · last 10 avg

26,049.240 ms

Model throughput · last 10 avg

566.530 TFLOP/s/GPU

Token throughput · last 10 avg

20,126.806 tokens/s/GPU

Peak allocated memory

164.230 GiB

Peak reserved memory

169.840 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_pretrain_8k_fsdp_config --mode pretrain --dataset megatron-indexed --seq_length 8192 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 8 'dataset.blend=[["work/data/rp2/perp_middle/middle_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3.5-lightning-gb200/rp2 rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/pretrain-fsdp logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null dist.distributed_timeout_minutes=120

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, the sequence-length-8,192, GBS/MBS 512/2 BF16 real-data RP2 FSDP convergence-verification run completes exactly 100 finite-loss optimizer steps with natural routing, Transformer Engine fused cross entropy, active MTP-1/MTP-2 losses, and no skipped or NaN iterations. LM loss is 12.171970 -> 5.812089; real-data steps 91-100 average 26,049.240 ms / 566.530 TFLOP/s/GPU with 164.230 GiB peak allocated and 169.840 GiB peak reserved memory. Complete 14-file, 369-GiB FSDP checkpoints with distributed metadata are written at steps 50 and 100. Do not use this run to claim convergence neutrality against ordinary BF16 pretraining: 155 of 300 shared LM/MTP loss values exceed the 1% loss gate, with a maximum relative difference of 9.80%, despite the similar final loss.

Pretrain · FSDP · GB200 · fp8_mx

✓ Verified

Hardware

GB200

Precision

FP8 MX

Last verified

2026-10-09

Recorded metrics

Initial loss

12.19034

Final loss

1.745152

Step time · last 10 avg

14,203.880 ms

Model throughput · last 10 avg

779.280 TFLOP/s/GPU

Token throughput · last 10 avg

27,683.703 tokens/s/GPU

Peak allocated memory

175.150 GiB

Peak reserved memory

180.030 GiB

Exact command

Command Copy

./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_pretrain_8gpu_gb200_fp8mx_fsdp_config --mode pretrain --max_steps 20

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, the sequence-length-8,192 MXFP8 FSDP recipe completes exactly 20 finite-loss optimizer steps with GBS/MBS 384/3, active MTP-1 and MTP-2 losses, and no skipped or NaN iterations. LM loss is 12.19034 -> 1.745152, and steps 11-20 average 14,203.880 ms / 779.280 TFLOP/s/GPU with 175.150 GiB peak allocated and 180.030 GiB peak reserved memory. This run must not be compared with the BF16 real-data pretrain or BF16 FSDP runs: it uses MXFP8, mock data, forced routing, GBS/MBS 384/3, and no CUDA graphs.

SFT · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

0.4406465

Final loss

0.2761714

Step time · last 10 avg

9,414.880 ms

Model throughput · last 10 avg

92.740 TFLOP/s/GPU

Token throughput · last 10 avg

3,480.448 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_sft_openmathinstruct2_packed_config --mode sft --max_steps 100 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 --save_dir work/model-verification/nemotron-3.5-lightning-h100/sft-checkpoints

Expected result

Packed OpenMathInstruct-2 full SFT completes 100 finite-loss steps with active MTP gradients, LM loss 0.4406465 -> 0.2761714, finite MTP-1 loss 0.7372081 -> 0.4146557, and no skipped or NaN iterations. Steps 91-100 average 9,414.880 ms / 92.740 TFLOP/s/GPU, and the complete reloadable full-model checkpoint is written at step 100.

SFT · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

0.4407797

Final loss

0.2761015

Step time · last 10 avg

9,533.730 ms

Model throughput · last 10 avg

183.250 TFLOP/s/GPU

Token throughput · last 10 avg

6,874.120 tokens/s/GPU

Peak allocated memory

101.510 GiB

Peak reserved memory

103.910 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --additional-slurm-params segment=2 --recipe nemotron_3_5_lightning_sft_openmathinstruct2_packed_tp1_config --mode sft --max_steps 100 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 --save_dir work/model-verification/nemotron-3.5-lightning-gb200/sft-checkpoints

Expected result

On two complete four-GPU GB200 nodes in one segment-2 NVLink domain, packed OpenMathInstruct-2 full SFT completes exactly 100 finite-loss optimizer steps with active MTP gradients. LM loss is 0.4407797 -> 0.2761015, MTP-1 loss is 0.7373838 -> 0.4145924, and MTP-2 loss is 0.8092746 -> 0.4743617. Steps 91-100 average 9,533.730 ms / 183.250 TFLOP/s/GPU with 101.510 GiB peak allocated and 103.910 GiB peak reserved memory. No iteration is skipped or NaN, and the complete 12-file reloadable full-model checkpoint is written at step 100.

Long Context · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-10-12

Recorded metrics

Initial loss

0.4268879

Final loss

0.2653436

Step time · last 10 avg

20,822.740 ms

Model throughput · last 10 avg

207.600 TFLOP/s/GPU

Token throughput · last 10 avg

6,294.657 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 4 --gpus-per-node 8 --recipe nemotron_3_5_lightning_sft_config --mode sft --dataset openmathinstruct2 --max_steps 100 --seq_length 32768 --lr 5e-6 --min_lr 0 --warmup_iters 10 --tensor_model_parallel_size 2 --context_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.sequence_parallel=true model.cp_comm_type=p2p model.cross_entropy_loss_fusion=true model.calculate_per_token_loss=true model.recompute_granularity=full model.recompute_modules=null model.recompute_method=uniform model.recompute_num_layers=1 model.moe_router_force_load_balancing=false ddp.average_in_collective=false ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-h100/long-sft-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

On four complete eight-H100 80GB nodes, packed OpenMathInstruct-2 CP2 SFT completes exactly 100 finite-loss 32K steps with LM loss 0.4268879 -> 0.2653436 and MTP-1 loss 0.7321755 -> 0.4469606. All 100 steps have finite LM/MTP losses and no skipped or NaN iterations. Steps 91-100 average 20,822.740 ms / 207.600 TFLOP/s/GPU. Complete step-50 and step-100 checkpoints each contain 36 files, distributed metadata, saved run configuration and training state, with latest marker 100. The two-node layout runs out of memory in iteration 2 on H100 80GB.

Long Context · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-12

Recorded metrics

Initial loss

0.4268849

Final loss

0.2653473

Step time · last 10 avg

41,101.830 ms

Model throughput · last 10 avg

420.700 TFLOP/s/GPU

Token throughput · last 10 avg

12,755.831 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe nemotron_3_5_lightning_sft_config --mode sft --dataset openmathinstruct2 --max_steps 100 --seq_length 32768 --lr 5e-6 --min_lr 0 --warmup_iters 10 --context_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 model.moe_flex_dispatcher_backend=hybridep 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false model.calculate_per_token_loss=true model.recompute_granularity=selective 'model.recompute_modules=[moe,layernorm,core_attn,mlp]' model.recompute_method=null model.recompute_num_layers=null model.moe_router_force_load_balancing=false ddp.average_in_collective=false ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save_optim=false checkpoint.save_rng=false checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/long-sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed OpenMathInstruct-2 CP2 SFT completes exactly 100 finite-loss 32K steps with LM loss 0.4268849 -> 0.2653473 and MTP-1 loss 0.7321853 -> 0.4469699. All 100 steps have finite LM/MTP losses and no skipped or NaN iterations. Steps 91-100 average 41,101.830 ms / 420.700 TFLOP/s/GPU. The complete step-100 checkpoint has distributed metadata, saved run configuration and training state, 12 files, and a latest marker of 100.

LoRA · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

0.4407612

Final loss

0.2858826

Step time · last 10 avg

19,558.310 ms

Model throughput · last 10 avg

89.580 TFLOP/s/GPU

Token throughput · last 10 avg

3,350.801 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe nemotron_3_5_lightning_peft_config --mode lora --dataset openmathinstruct2 --max_steps 100 --seq_length 4096 --lr 1e-4 --min_lr 0 --warmup_iters 10 --tensor_model_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:4096,pad_seq_to_mult:2}' model.sequence_parallel=true ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-h100/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed rank-32, alpha-32, zero-dropout LoRA completes 100 finite-loss steps with 48 rank-local adapter attachment records covering MTP attention, expert, and shared-expert layers, LM loss 0.4407612 -> 0.2858826, finite MTP-1 loss 0.7371775 -> 0.4370083, and no skipped or NaN iterations. Steps 91-100 average 19,558.310 ms / 89.580 TFLOP/s/GPU, and the complete 12-file adapter checkpoint is written at step 100 with a latest marker of 100.

LoRA · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

0.4406565

Final loss

0.2857773

Step time · last 10 avg

21,602.620 ms

Model throughput · last 10 avg

80.940 TFLOP/s/GPU

Token throughput · last 10 avg

3,033.706 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe nemotron_3_5_lightning_peft_config --mode lora --dataset openmathinstruct2 --max_steps 100 --seq_length 4096 --lr 1e-4 --min_lr 0 --warmup_iters 10 --tensor_model_parallel_size 2 --expert_tensor_parallel_size 1 --expert_model_parallel_size 8 --pretrained_checkpoint work/model-verification/nemotron-3.5-lightning/imported-megatron/iter_0000000 model.moe_flex_dispatcher_backend=hybridep 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:4096,pad_seq_to_mult:2}' model.sequence_parallel=true ddp.overlap_param_gather=false optimizer.overlap_param_gather=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.async_save=false --save_dir work/model-verification/nemotron-3.5-lightning-gb200/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

Packed rank-32, alpha-32, zero-dropout LoRA completes 100 finite-loss steps with 48 rank-local adapter attachment records covering MTP attention, expert, and shared-expert layers, LM loss 0.4406565 -> 0.2857773, finite MTP-1 loss 0.7370399 -> 0.4370756, and no skipped or NaN iterations. Steps 91-100 average 21,602.620 ms / 80.940 TFLOP/s/GPU, and the complete 12-file adapter checkpoint is written at step 100 with a latest marker of 100.

Benchmark · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

12.16197

Final loss

0.01383163

Step time · last 10 avg

18,073.460 ms

Model throughput · last 10 avg

408.260 TFLOP/s/GPU

Token throughput · last 10 avg

14,504.362 tokens/s/GPU

Peak allocated memory

74.170 GiB

Peak reserved memory

77.500 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe nemotron_3_5_lightning_pretrain_16gpu_h100_bf16_config --max_steps 50

Expected result

On 16x H100, the exact mock-data, force-balanced BF16 performance recipe completes exactly 50 steps with finite LM/MTP-1/MTP-2 losses and no skipped or NaN iterations. Transformer Engine CUDA graphs cover attention and Mamba scopes, while expert-FC1 activations are offloaded. Steps 41-50 average 18,073.460 ms and 408.260 model TFLOP/s/GPU; peak rank-0 allocated/reserved memory is 74.170/77.500 GiB. This benchmark-only run stores Adam moments in BF16, so its losses are not convergence-comparable with FP32 optimizer-state runs.

Benchmark · GB200

✓ Verified

Hardware

GB200

Precision

BF16

Last verified

2026-10-09

Recorded metrics

Initial loss

12.1825

Final loss

0.04329855

Step time · last 10 avg

18,902.370 ms

Model throughput · last 10 avg

780.730 TFLOP/s/GPU

Token throughput · last 10 avg

27,736.628 tokens/s/GPU

Peak allocated memory

161.060 GiB

Peak reserved memory

165.390 GiB

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe nemotron_3_5_lightning_pretrain_8gpu_gb200_bf16_config --max_steps 50

Expected result

On 8x GB200, the exact mock-data BF16 performance recipe completes exactly 50 steps with finite LM/MTP-1/MTP-2 losses and no skipped or NaN iterations. Transformer Engine CUDA graphs cover attention, Mamba, MoE-router, and MoE-preprocess scopes. Steps 41-50 average 18,902.370 ms and 780.730 model TFLOP/s/GPU; peak rank-0 allocated/reserved memory is 161.060/165.390 GiB.