Nemotron 3 Super#
Nemotron 3 Superis a large language model (LLM) trained by NVIDIA, designed to deliver strong agentic, reasoning, and conversational capabilities. It is employs a hybrid Latent Mixture-of-Experts (LatentMoE) architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. Distinct from the Nano model, the Super model incorporates Multi-Token Prediction (MTP) layers for faster text generation and improved quality, and it is trained using NVFP4 quantization to maximize compute efficiency. The model has 12B active parameters and 120B parameters in total.
NeMo Megatron Bridge supports pretraining, full parameters finetuning, and LoRA finetuning this model. The finetuned model can be converted back to the 🤗 Hugging Face format for downstream evaluation.
Important
Run all commands from /opt/Megatron-Bridge (e.g. docker run -w /opt/Megatron-Bridge ...)
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
â—‹ UnverifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 --hf-revision d51eab0d1f979ebc26b546e634a04f450d99158e --megatron-path work/model-verification/nemotron-3-super-120b-a12b/cpu-megatron --torch-dtype bfloat16Expected result
The pinned CPU import must exit successfully, create iter_0000000, and preserve the complete source checkpoint. A subsequent CPU export from that CPU-imported checkpoint must audit every supported tensor for exact keys, shapes, dtypes, and values; the currently verified CPU export uses the independently verified GPU-imported checkpoint instead.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 --hf-revision d51eab0d1f979ebc26b546e634a04f450d99158e --megatron-path work/model-verification/nemotron-3-super-120b-a12b/imported-megatron --torch-dtype bfloat16 --tp 1 --pp 1 --ep 8 --etp 1Expected result
The pinned eight-GPU BF16 import exits successfully and creates a reloadable TP1/PP1/EP8/ETP1 iter_0000000 checkpoint. Its paired export exactly reproduces all 42,683 source tensors and 123,611,033,088 elements, including the 41 FP32 router correction-bias tensors.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 --hf-revision d51eab0d1f979ebc26b546e634a04f450d99158e --megatron-path work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3-super-120b-a12b/cpu-hf-export --torch-dtype bfloat16Expected result
The single-process CPU export exits successfully and writes 247,222,108,160 bytes across 42,683 tensors and 123,611,033,088 values. Every exported key, shape, dtype, and value is bitwise identical to the pinned Hugging Face BF16 source, with no dtype conversions, and Transformers strictly reloads all 763 state tensors natively as NemotronHForCausalLM.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 --hf-revision d51eab0d1f979ebc26b546e634a04f450d99158e --megatron-path work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3-super-120b-a12b/hf-export --torch-dtype bfloat16 --distributed-save --tp 1 --pp 1 --ep 8 --etp 1Expected result
Strict distributed export exits successfully and writes 247,222,108,160 bytes across all 42,683 tensors. Keys, shapes, dtypes, values, and critical configuration match the pinned source with zero mismatches, including 41 FP32 router correction-bias tensors, and Transformers reloads the result natively as NemotronHForCausalLM with 88 layers.
Pretrain · H100
✓ VerifiedRecorded metrics
- Initial loss
- 12.19922
- Final loss
- 5.706429
- Step time · last 10 avg
- 29,368.690 ms
- Model throughput · last 10 avg
- 236.330 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,789.365 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8 --recipe nemotron_3_super_pretrain_config --mode pretrain --dataset megatron-indexed --max_steps 100 'dataset.blend=[["work/data/rp2/head_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3-super-120b-a12b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2-tokenizer/tokenizer.model rng.seed=1234 dataset.random_seed=1234 validation.eval_iters=0 validation.eval_interval=0 checkpoint.ckpt_format=torch_dist checkpoint.async_save=false checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/h100-pretrain-reference --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/nemotron-3-super-120b-a12b/h100-pretrain-config/ConfigContainer.yamlExpected result
On exactly 64 H100s, the BF16 support run completes 100 bounded RP2 optimizer steps at TP1/PP2/CP1/EP32/ETP1, GBS/MBS 1280/1, natural routing, HybridEP flex dispatch with 1.10x fixed expert capacity, eager execution, selective recompute of layernorm, moe_act, moe, and core_attn, no optimizer-state CPU offload, and ordered gradient reduction. Loss remains finite from 12.19922 to 5.706429 with zero skipped or NaN iterations. The final ten steps average 29368.690 ms and 236.330 TFLOP/s/GPU. Peak device memory is 58.079 GiB allocated and 70.336 GiB reserved. The resolved configuration persists, and complete iter_0000050 and iter_0000100 checkpoints each contain 64 distributed rank shards plus checkpoint metadata and training state.
Pretrain · GB200
â—‹ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 2 --recipe nemotron_3_super_pretrain_16gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --max_steps 100 --seq_length 4096 model.cuda_graph_impl=none --lr 4.5e-4 --min_lr 4.5e-6 --warmup_iters 10 'dataset.blend=[["work/data/rp2/head_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3-super-120b-a12b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2-tokenizer/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 optimizer.optimizer_cpu_offload=true optimizer.optimizer_offload_fraction=0.4 optimizer.overlap_cpu_optimizer_d2h_h2d=false model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 checkpoint.ckpt_format=torch_dist checkpoint.async_save=false checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/gb200-pretrain-reference --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 16 GB200s, the BF16 support run must complete exactly 100 bounded RP2 optimizer steps at TP8/PP1/CP1/EP16/ETP1 and the recipe's cohort GBS/MBS 1024/1. Full activation recompute, 40% optimizer CPU offload, and eager execution keep the model and optimizer checkpoint within memory. Loss must stay finite with no skipped or NaN iteration, and complete iter_0000050 and iter_0000100 checkpoints must save. The recipe moved from GBS 16 to the cohort GBS 1024, so the earlier metrics no longer describe this command and it needs a rerun.
SFT · H100
â—‹ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 8 --recipe nemotron_3_super_sft_16gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 8192 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="2d59de1cbd51c0adf384eb906b766d1aee0e0517"' dataset.hf_output_root=work/data/tulu3/nemotron-3-super-120b-a12b-pad8 dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:8192,pad_seq_to_mult:8}' model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 optimizer.optimizer_cpu_offload=true optimizer.optimizer_offload_fraction=0.75 optimizer.overlap_cpu_optimizer_d2h_h2d=true checkpoint.ckpt_format=torch_dist checkpoint.async_save=false checkpoint.save_optim=false checkpoint.save_rng=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/h100-sft --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
The immutable-revision Tulu3 run must complete exactly 100 full-SFT steps on 16 H100s at sequence length 8192, TP8/EP16, the recipe's cohort GBS/MBS 32/1, and pad-8 offline packing. Loss must stay finite with no skipped or NaN iteration, and the step-100 checkpoint must save completely for export. The recipe moved from GBS 16 to the cohort GBS 32, so the earlier metrics no longer describe this command and it needs a rerun.
SFT · GB200
â—‹ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 4 --gpus-per-node 4 --recipe nemotron_3_super_sft_16gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 8192 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="2d59de1cbd51c0adf384eb906b766d1aee0e0517"' dataset.hf_output_root=work/data/tulu3/nemotron-3-super-120b-a12b-pad8 dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:8192,pad_seq_to_mult:8}' model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 optimizer.optimizer_cpu_offload=true optimizer.optimizer_offload_fraction=0.75 optimizer.overlap_cpu_optimizer_d2h_h2d=true checkpoint.ckpt_format=torch_dist checkpoint.async_save=false checkpoint.save_optim=false checkpoint.save_rng=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/gb200-sft --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
The immutable-revision Tulu3 run must complete exactly 100 full-SFT steps on 16 GB200s at sequence length 8192, TP8/EP16, the recipe's cohort GBS/MBS 32/1, and pad-8 offline packing. Loss must stay finite with no skipped or NaN iteration, and the step-100 checkpoint must save completely. The recipe moved from GBS 16 to the cohort GBS 32, so the earlier metrics no longer describe this command and it needs a rerun.
Long Context · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.168852
- Final loss
- 1.011637
- Step time · last 10 avg
- 123,114.400 ms
- Model throughput · last 10 avg
- 6.890 TFLOP/s/GPU
- Token throughput · last 10 avg
- 33.270 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 8 --recipe nemotron_3_super_sft_16gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --max_steps 10 --seq_length 32768 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="2d59de1cbd51c0adf384eb906b766d1aee0e0517"' dataset.hf_output_root=work/data/tulu3/nemotron-3-super-120b-a12b-long dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.attention_backend=auto model.cross_entropy_loss_fusion=false model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 optimizer.optimizer_cpu_offload=true optimizer.optimizer_offload_fraction=1.0 optimizer.overlap_cpu_optimizer_d2h_h2d=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.ckpt_format=torch_dist checkpoint.async_save=false checkpoint.save_optim=false checkpoint.save_rng=false checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/h100-long-sft --save_interval 10 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 16 H100s, the dedicated TP1/PP8/CP2/EP2 recipe completes exactly ten packed 32K SFT steps with pad-4 alignment, full recompute, and full optimizer CPU offload. Loss is finite from 1.168852 to 1.011637 with no skipped or NaN iterations. The ten-step metric window includes the first-step cold-start cost by policy, the resolved configuration is persisted, and the final model-only checkpoint saves successfully.
Long Context · GB200
â—‹ UnverifiedRecorded metrics
- Initial loss
- 7.022207
- Final loss
- 6.406034
- Step time · last 10 avg
- 21,544.740 ms
- Model throughput · last 10 avg
- 15.930 TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 4 --gpus-per-node 4 --recipe nemotron_3_super_sft_16gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --max_steps 10 --seq_length 32768 --lr 5e-6 --min_lr 0 --warmup_iters 10 dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/gb200-long-sft --save_interval 10 logger.log_interval=1 logger.log_throughput=trueExpected result
A dedicated GB200 run must complete at least ten packed 32K SFT steps with CP2, finite loss, no skipped or NaN iterations, all five metrics, a persisted resolved configuration, and a reloadable final checkpoint.
LoRA · H100
â—‹ UnverifiedRecorded metrics
- Initial loss
- None
- Final loss
- None
- Step time · last 10 avg
- None ms
- Model throughput · last 10 avg
- None TFLOP/s/GPU
- Token throughput · last 10 avg
- None tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 8 --recipe nemotron_3_super_peft_16gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 8192 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="2d59de1cbd51c0adf384eb906b766d1aee0e0517"' dataset.hf_output_root=work/data/tulu3/nemotron-3-super-120b-a12b-peft-pad8 dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:8192,pad_seq_to_mult:8}' model.recompute_granularity=selective 'model.recompute_modules=[core_attn]' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.ckpt_format=torch_dist checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/h100-peft --save_interval 100 logger.log_interval=1 logger.log_throughput=trueExpected result
H100 training has completed 100 finite LoRA steps and saved an adapter, but this leaf remains unverified until the exported adapter reloads natively over the pinned Hugging Face base and produces a nonzero, finite logits effect.
LoRA · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 1.23861
- Final loss
- 0.8878498
- Step time · last 10 avg
- 12,699.460 ms
- Model throughput · last 10 avg
- 54.120 TFLOP/s/GPU
- Token throughput · last 10 avg
- 645.067 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 4 --gpus-per-node 4 --recipe nemotron_3_super_peft_16gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-super-120b-a12b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 8192 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="2d59de1cbd51c0adf384eb906b766d1aee0e0517"' dataset.hf_output_root=work/data/tulu3/nemotron-3-super-120b-a12b-peft-pad8 dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:8192,pad_seq_to_mult:8}' model.recompute_granularity=selective 'model.recompute_modules=[core_attn]' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.ckpt_format=torch_dist checkpoint.async_save=false checkpoint.load=null --save_dir work/model-verification/nemotron-3-super-120b-a12b/gb200-peft --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 16 GB200s, the immutable-revision Tulu3 run completes exactly 100 pad-8 packed LoRA steps with finite loss from 1.23861 to 0.8878498, no skipped or NaN iterations, all five metrics, and a final adapter checkpoint. The adapter contains only 82,304 LoRA tensors targeting linear_qkv, linear_proj, linear_fc1, linear_fc2, in_proj, and out_proj. Native PEFT reload over the pinned base succeeds; all logits are finite and the adapter changes them by maximum absolute 3.1875 and mean absolute 0.472428.
Benchmark · H100
✓ VerifiedRecorded metrics
- Initial loss
- 12.19134
- Final loss
- 0.006422824
- Step time · last 10 avg
- 27,588.580 ms
- Model throughput · last 10 avg
- 248.610 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,969.345 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 8 --gpus-per-node 8 --recipe nemotron_3_super_pretrain_64gpu_h100_bf16_config --max_steps 50 logger.log_throughput=true logger.log_device_memory_used=true logger.save_config_filepath=work/model-verification/nemotron-3-super-120b-a12b/h100-performance/ConfigContainer.yamlExpected result
On exactly 64 H100s, the canonical BF16 performance recipe must complete 50 mock-data steps at TP1/PP2/CP1/EP32/ETP1, GBS/MBS 1280/1, forced routing, HybridEP flex dispatch with 1.10x fixed expert capacity, eager execution, selective recompute of layernorm, moe_act, moe, and core_attn, no optimizer-state CPU offload, and ordered gradient reduction. All keyed rows have finite loss and throughput with zero skipped or NaN iterations. The final ten steps average 27588.580 ms and 248.610 TFLOP/s/GPU, passing gates of at most 28000 ms and at least 240 TFLOP/s/GPU. Peak device memory is 59.000 GiB allocated and 69.344 GiB reserved, and the resolved configuration persists. Mock data and forced routing make the loss a finiteness check rather than convergence evidence.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 12.17778
- Final loss
- 0.01120061
- Step time · last 10 avg
- 9,926.350 ms
- Model throughput · last 10 avg
- 559.390 TFLOP/s/GPU
- Token throughput · last 10 avg
- 3,301.113 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4 --recipe nemotron_3_super_pretrain_64gpu_gb200_bf16_config --max_steps 50 logger.log_throughput=trueExpected result
On exactly 64 GB200s, the canonical BF16 performance recipe completes 50 mock-data steps at TP2/PP1/CP1/EP64, GBS/MBS 512/1, HybridEP flex dispatch, and Transformer Engine CUDA graph scopes attn, mamba, moe_router, and moe_preprocess. All 50 keyed rows have finite loss and throughput with zero skipped or NaN iterations. The final ten steps average 9926.350 ms and 559.390 TFLOP/s/GPU, passing gates of at most 11000 ms and at least 500 TFLOP/s/GPU, and the resolved configuration is persisted. Mock data and forced routing make the loss a finiteness check rather than convergence evidence. These numbers are GB200-only evidence.
Benchmark · GB300
✓ VerifiedRecorded metrics
- Initial loss
- 12.17656
- Final loss
- 0.01398012
- Step time · last 10 avg
- 6,357.940 ms
- Model throughput · last 10 avg
- 873.330 TFLOP/s/GPU
- Token throughput · last 10 avg
- 10,307.741 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4 --recipe nemotron_3_super_pretrain_64gpu_gb300_nvfp4_config --mode pretrain --max_steps 50 --seq_length 8192 logger.save_config_filepath=work/model-verification/nemotron-3-super-120b-a12b/gb300-performance/ConfigContainer.yamlExpected result
On exactly 64 GB300s, the canonical NVFP4 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP64/ETP1, GBS/MBS 512/1, and sequence length 8192. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.17656 to 0.01398012; the final ten steps average 6357.940 ms, 873.330 TFLOP/s/GPU, and 10307.741 tokens/s/GPU. The resolved configuration persists. Mock data and forced routing make the loss a finiteness check rather than convergence evidence.