Nemotron 3 Ultra#

Nemotron 3 Ultra is a 550B total / A55B active hybrid Mamba-Transformer MoE model.

Megatron Bridge provides Nemotron 3 Ultra recipes and examples for Hugging Face to Megatron conversion, inference, DCLM pretraining, packed OpenMathInstruct-2 full SFT, and packed OpenMathInstruct-2 LoRA PEFT.

Use the main example README for setup and scripts: examples/models/nemotron/nemotron_3_ultra/README.md.

Verified configurations#

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration#

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import · CPU

â—‹ Unverified
Hardware
not specified
Precision
BF16
Last verified
—
Exact command
Command 
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 --hf-revision 624ba927cfbef0427354998700de3d51173c8c04 --megatron-path work/model-verification/nemotron-3-ultra-550b-a55b/cpu-megatron --torch-dtype bfloat16
Expected result

A machine with enough host memory must import the complete pinned BF16 checkpoint, persist iter_0000000, and reload it without missing, unexpected, shape-mismatched, dtype-mismatched, or value-mismatched tensors.

Import · GPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-07-25
Exact command
Command 
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 3 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 --hf-revision 624ba927cfbef0427354998700de3d51173c8c04 --megatron-path work/model-verification/nemotron-3-ultra-550b-a55b/imported-megatron --torch-dtype bfloat16 --tp 1 --pp 3 --ep 8 --etp 1
Expected result

The command exited successfully and persisted a reloadable iter_0000000 at TP1/PP3/EP8/ETP1. Exact distributed export back to the immutable source revision accounted for all 51,023 Nemotron-H, LatentMoE, Mamba, attention, and MTP tensors and 560,524,603,904 parameters with matching names, shapes, dtypes, and values.

Export · CPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-08-26
Exact command
Command 
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 --hf-revision 624ba927cfbef0427354998700de3d51173c8c04 --megatron-path work/model-verification/nemotron-3-ultra-550b-a55b/imported-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3-ultra-550b-a55b/cpu-hf-export --torch-dtype bfloat16
Expected result

The single-process CPU export exits successfully and writes all 51,023 tensors, 560,524,603,904 values, and 1,121,049,257,984 tensor-payload bytes. Keys, shapes, dtypes, and values are bitwise identical to the pinned source with no dtype conversions, preserving its BF16 model weights and FP32 expert-score correction biases. Transformers strictly reloads the output natively as NemotronHForCausalLM with all 927 state tensors and no loading discrepancies.

Export · GPU

✓ Verified
Hardware
not specified
Precision
BF16
Last verified
2026-07-25
Exact command
Command 
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 3 --gpus-per-node 8 --hf-model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 --hf-revision 624ba927cfbef0427354998700de3d51173c8c04 --megatron-path work/model-verification/nemotron-3-ultra-550b-a55b/imported-megatron/iter_0000000 --hf-path work/model-verification/nemotron-3-ultra-550b-a55b/hf-export --torch-dtype bfloat16 --distributed-save --tp 1 --pp 3 --ep 8 --etp 1
Expected result

The command exited successfully and wrote 225 source-native Hugging Face shards. All 51,023 tensors and 560,524,603,904 parameters matched the pinned source exactly in name, shape, dtype, and value, preserving BF16 model weights and FP32 expert-score correction biases. Semantic configuration, generation configuration, the 131,072-token vocabulary, special tokens, chat template, and representative encodings also matched, and the export reloaded natively with NemotronHConfig and architecture NemotronHForCausalLM.

Pretrain · H100

â—‹ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Peak allocated memory
None GiB
Peak reserved memory
None GiB
Exact command
Command 
./scripts/training/train.sh --nodes 32 --gpus-per-node 8 --recipe nemotron_3_ultra_pretrain_256gpu_h100_bf16_fsdp_config --mode pretrain --dataset megatron-indexed --max_steps 10 --wait 'dataset.blend=[["work/data/rp2/head_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3-ultra-550b-a55b/rp2-h100 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model dataset.random_seed=1234 dataset.num_workers=8 rng.seed=1234 scheduler.lr_decay_iters=100 checkpoint.load=null checkpoint.save=null checkpoint.async_save=false validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.save_config_filepath=work/model-verification/nemotron-3-ultra-550b-a55b/pretrain-h100-resolved.yaml logger.tensorboard_dir=null
Expected result

In the card's current base container, the public BF16 library recipe must complete all 10 bounded RedPajama2 head_01 optimizer steps at TP4/PP1/CP1/EP64/ETP1, GBS/MBS 512/1, and sequence length 8192. Megatron-FSDP, natural expert routing, standard all-to-all dispatch, FP32 gradient communication, numerical checks, full-layer block recompute, and the library optimizer and constant-LR schedule must remain active. Every keyed step must have finite loss, throughput, and gradient norm with zero skipped and zero NaN iterations.

Pretrain · GB200

✓ Verified
Hardware
GB200
Precision
BF16
Last verified
2026-09-03
Recorded metrics
Initial loss
12.25656
Final loss
8.224941
Step time · last 10 avg
10,125.530 ms
Model throughput · last 10 avg
291.790 TFLOP/s/GPU
Token throughput · last 10 avg
809.044 tokens/s/GPU
Peak allocated memory
178.170 GiB
Peak reserved memory
181.380 GiB
Exact command
Command 
./scripts/training/train.sh --nodes 64 --gpus-per-node 4 --recipe nemotron_3_ultra_pretrain_256gpu_gb200_bf16_ep16_config --mode pretrain --dataset megatron-indexed --max_steps 20 --wait 'dataset.blend=[["work/data/rp2/head_01_text_document"],null]' dataset.path_to_cache=work/cache/nemotron-3-ultra-550b-a55b/rp2-gb200-ep16 dataset.random_seed=1234 dataset.num_workers=8 rng.seed=1234 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model tokenizer.use_tokenizer_vocab_size=false model.moe_router_force_load_balancing=false checkpoint.load=null checkpoint.pretrained_checkpoint=work/model-verification/nemotron-3-ultra-550b-a55b/imported-megatron checkpoint.finetune=true checkpoint.load_optim=false checkpoint.load_rng=false checkpoint.dist_ckpt_strictness=log_all checkpoint.save=null checkpoint.async_save=false validation.eval_global_batch_size=256 validation.eval_micro_batch_size=1 validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.save_config_filepath=work/model-verification/nemotron-3-ultra-550b-a55b/pretrain-gb200-ep16-resolved.yaml logger.tensorboard_dir=null logger.wandb_project=nemotron-ultra-verification logger.wandb_exp_name=gb200-bf16-ep16-20step logger.wandb_save_dir=work/model-verification/nemotron-3-ultra-550b-a55b/wandb
Expected result

In the card's current base container, the public checkpoint-initialized BF16 library recipe completed all 20 bounded RedPajama2 head_01 optimizer steps at TP4/PP4/CP1/EP16/ETP1, GBS/MBS 256/1, and sequence length 8192. Natural expert routing, non-FSDP distributed optimizer, HybridEP, targeted expert-activation offload, selective attention/layernorm recompute, numerical checks, and the library optimizer and constant-LR schedule remained active. Every keyed step had finite loss, throughput, and gradient norm with zero skipped and zero NaN iterations, and durable Weights & Biases telemetry was produced.

SFT · H100

â—‹ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command
Command 
./scripts/training/train.sh --nodes 48 --gpus-per-node 8 --recipe nemotron_3_ultra_sft_384gpu_h100_bf16_openmathinstruct2_packed_config --mode sft --wait --pretrained_checkpoint work/model-verification/nemotron-3-ultra-550b-a55b/imported-megatron/iter_0000000 --max_steps 100 --warmup_iters 10 scheduler.lr_decay_iters=100 'dataset.hf_dataset.split="train_1M[:10%]"' 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.hf_output_root=work/data/openmathinstruct2-ultra-4096-v1 dataset.seed=1234 dataset.offline_packing_specs.pad_seq_to_mult=2 validation.eval_iters=0 validation.eval_interval=0 --save_dir work/model-verification/nemotron-3-ultra-550b-a55b/sft-checkpoints --save_interval 100 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/nemotron-3-ultra-550b-a55b/sft-resolved.yaml logger.tensorboard_dir=null
Expected result

The public 384-H100 packed OpenMathInstruct-2 full-SFT recipe must use the pinned dataset revision and bounded train_1M first-ten-percent selection, complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1, and preserve pad_seq_to_mult 2. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, a persisted post-setup configuration, audited supervised-token counts, and a complete reloadable final checkpoint.

SFT · GB200

â—‹ Unverified
Hardware
GB200
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

No GB200-specific Ultra full-SFT configuration is verified by this PR. The H100 library recipe must be adapted and validated independently before GB200 SFT support is claimed.

Long Context · H100

× Unsupported
Hardware
H100
Precision
NOT SPECIFIED
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

Megatron Bridge does not publish a dedicated Ultra packed long-context recipe. The only published Ultra full-SFT recipe fixes sequence length 4096 and context parallelism one, so it cannot provide the required joint sequence-packing and CP-greater-than-one evidence and is not relabeled as long-context support.

Long Context · GB200

× Unsupported
Hardware
GB200
Precision
NOT SPECIFIED
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

Megatron Bridge does not publish a dedicated Ultra packed long-context recipe for GB200.

LoRA · H100

â—‹ Unverified
Hardware
H100
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command
Command 
./scripts/training/train.sh --nodes 4 --gpus-per-node 8 --recipe nemotron_3_ultra_peft_32gpu_h100_bf16_openmathinstruct2_packed_config --mode lora --wait --pretrained_checkpoint work/model-verification/nemotron-3-ultra-550b-a55b/imported-megatron/iter_0000000 --max_steps 100 --warmup_iters 10 scheduler.lr_decay_iters=100 'dataset.hf_dataset.split="train_1M[:10%]"' 'dataset.hf_dataset.load_kwargs={revision:"469216e3f46f4dacf476b382e192485ea51a143e"}' dataset.hf_output_root=work/data/openmathinstruct2-ultra-4096-v1 dataset.seed=1234 dataset.offline_packing_specs.pad_seq_to_mult=2 validation.eval_iters=0 validation.eval_interval=0 --save_dir work/model-verification/nemotron-3-ultra-550b-a55b/peft-checkpoints --save_interval 100 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/nemotron-3-ultra-550b-a55b/peft-resolved.yaml logger.tensorboard_dir=null
Expected result

The public 32-H100 packed OpenMathInstruct-2 LoRA recipe must use the same pinned bounded dataset and pad_seq_to_mult 2 as full SFT, match all declared attention, MLP, and Mamba adapter targets, and complete 100 optimizer steps at sequence length 4096 and GBS/MBS 128/1. This published Ultra contract is support verification rather than the 8K/GBS8 qwen3_30b_a3b_convergence_v2 cohort. Loss must remain finite with no skipped or NaN iterations, all five metrics, audited supervised-token counts, a persisted post-setup configuration, and a reloadable adapter checkpoint.

LoRA · GB200

â—‹ Unverified
Hardware
GB200
Precision
BF16
Last verified
—
Recorded metrics
Initial loss
None
Final loss
None
Step time · last 10 avg
None ms
Model throughput · last 10 avg
None TFLOP/s/GPU
Token throughput · last 10 avg
None tokens/s/GPU
Exact command

No runnable command is recorded for this status.

Expected result

No GB200-specific Ultra PEFT configuration is verified by this PR.

Benchmark · GB200

✓ Verified
Hardware
GB200
Precision
FP8 MX
Last verified
2026-08-19
Recorded metrics
Initial loss
12.1668
Final loss
6.738161
Step time · last 10 avg
3,213.970 ms
Model throughput · last 10 avg
917.750 TFLOP/s/GPU
Token throughput · last 10 avg
2,548.873 tokens/s/GPU
Exact command
Command 
./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe nemotron_3_ultra_pretrain_256gpu_gb200_fp8mx_config --mode pretrain --max_steps 50 --seq_length 8192 logger.save_config_filepath=work/model-verification/nemotron-3-ultra-550b-a55b/gb200-performance/ConfigContainer.yaml
Expected result

On exactly 256 GB200s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP2/PP1/CP1/EP64/ETP1, GBS/MBS 256/1, and sequence length 8192 using Megatron-FSDP HSDP. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.1668 to 6.738161; the final ten steps average 3213.970 ms, 917.750 TFLOP/s/GPU, and 2548.873 tokens/s/GPU. The resolved configuration persists.

Benchmark · GB300

✓ Verified
Hardware
GB300
Precision
FP8 MX
Last verified
2026-07-25
Recorded metrics
Initial loss
12.20118
Final loss
7.019335
Step time · last 10 avg
2,282.400 ms
Model throughput · last 10 avg
1,306.850 TFLOP/s/GPU
Token throughput · last 10 avg
3,589.204 tokens/s/GPU
Exact command
Command 
./scripts/training/train.sh --wait --nodes 64 --gpus-per-node 4 --recipe nemotron_3_ultra_pretrain_256gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 8192 logger.save_config_filepath=work/model-verification/nemotron-3-ultra-550b-a55b/gb300-performance/ConfigContainer.yaml
Expected result

On exactly 256 GB300s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP64/ETP1, GBS/MBS 256/1, and sequence length 8192. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.20118 to 7.019335; the final ten steps average 2282.400 ms, 1306.850 TFLOP/s/GPU, and 3589.204 tokens/s/GPU. The resolved configuration persists.