Nemotron 3 Nano 4B#
Nemotron 3 Nano is a large language model (LLM) trained from scratch by NVIDIA, and designed as a unified model for both reasoning and non-reasoning tasks. The model employs a hybrid Mixture-of-Experts (MoE) architecture, consisting of 23 Mamba-2 and MoE layers, along with 6 Attention layers. Each MoE layer includes 128 experts plus 1 shared expert, with 6 experts activated per token. The model has 3.5B active parameters and 30B parameters in total.
NeMo Megatron Bridge supports pretraining, full parameters finetuning, and LoRA finetuning this model. The finetuned model can be converted back to the 🤗 Hugging Face format for downstream evaluation.
Important
Run all commands from /opt/Megatron-Bridge (e.g. docker run -w /opt/Megatron-Bridge ...)
Tip
We use the following environment variables throughout this page
HF_MODEL_ID=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16MEGATRON_MODEL_PATH=/models/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16(feel free to set your own path)
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/cpu-megatron --torch-dtype bfloat16Expected result
The command exits successfully and creates iter_0000000. After the paired CPU export, all 263 tensors and 3,973,556,832 parameters match the pinned source exactly in name, shape, dtype, and value, with maximum difference zero.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/imported-megatron --torch-dtype bfloat16Expected result
The command exits successfully and creates a reloadable iter_0000000. After the paired GPU export, all 263 tensors and 3,973,556,832 parameters match the pinned source exactly in name, shape, dtype, and value.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/cpu-megatron --hf-path work/model-verification/nemotron-3-nano-4b/cpu-hf-export --torch-dtype bfloat16Expected result
Strict export exits successfully; all 263 BF16 tensors match the pinned source bitwise, the 171-byte generation configuration is preserved byte-for-byte, and Transformers reloads the output natively as NemotronHForCausalLM.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 1 --tp 1 --hf-model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --hf-revision dfaf35de3e30f1867dd8dbc38a7fc9fb52d3914f --megatron-path work/model-verification/nemotron-3-nano-4b/imported-megatron --hf-path work/model-verification/nemotron-3-nano-4b/hf-export --torch-dtype bfloat16Expected result
Strict export exits successfully; all 263 BF16 tensors match the pinned source bitwise, the generation configuration is exact, and Transformers reloads the output natively as NemotronHForCausalLM. A_log, D, dt_bias, and out_proj are exact for all 21 Mamba layers.
Pretrain · H100
✓ VerifiedRecorded metrics
- Initial loss
- 12.4325
- Final loss
- 5.511951
- Step time · last 10 avg
- 26,775.110 ms
- Model throughput · last 10 avg
- 402.510 TFLOP/s/GPU
- Token throughput · last 10 avg
- 19,581.171 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_pretrain_8gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[["work/data/rp2/head_01"],null]' dataset.path_to_cache=work/cache/nemotron-3-nano-4b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 dist.distributed_timeout_minutes=30 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/pretrain-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
The uninterrupted bounded random-initialization RP2 reference finishes all 100 steps at recipe-owned GBS/MBS 1024/1 with the five recorded metrics, finite loss, and no skipped or NaN iterations. It reaches peak learning rate 3e-4 at step 40 and completes cosine decay to 3e-5 at step 100. Both iter_0000050 and iter_0000100 contain all eight distributed shards plus metadata, optimizer/RNG train state, run config, and tokenizer; iter_0000100 reloads at step 100 without an extra optimizer step.
SFT · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.839282
- Final loss
- 1.234906
- Step time · last 10 avg
- 627.000 ms
- Model throughput · last 10 avg
- 262.890 TFLOP/s/GPU
- Token throughput · last 10 avg
- 13,065.391 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_sft_8gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-sft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:1}' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=trueExpected result
Pad-1 offline packing is 99.27% efficient. The TP1/DP8 run uses four microbatches per optimizer step and completes all 100 steps with the four recorded metrics, finite loss, and no skipped or NaN iterations. Across 6,553,600 token slots, assistant-only loss masks contain 4,187,630 supervised tokens. The complete eight-shard iter_0000100 full-model checkpoint is saved and reloads for export.
Long Context · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.65866
- Final loss
- 1.100197
- Step time · last 10 avg
- 5,575.590 ms
- Model throughput · last 10 avg
- 142.060 TFLOP/s/GPU
- Token throughput · last 10 avg
- 5,877.046 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_sft_8gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 32768 --tensor_model_parallel_size 2 --context_parallel_size 2 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-long-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:32768,pad_seq_to_mult:4}' model.sequence_parallel=true model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false model.calculate_per_token_loss=true ddp.average_in_collective=false scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/long-sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=trueExpected result
Pad-4 offline packing is 99.35% efficient. The TP2/CP2/SP-on/DP2 run uses four microbatches per optimizer step and completes all 100 32K steps with the five recorded metrics, finite loss, and no skipped or NaN iterations. Across 26,214,400 token slots, assistant-only loss masks contain 16,734,347 supervised tokens. The complete eight-shard iter_0000100 checkpoint is saved and reloads at the same topology.
LoRA · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.769892
- Final loss
- 1.256916
- Step time · last 10 avg
- 381.690 ms
- Model throughput · last 10 avg
- 431.900 TFLOP/s/GPU
- Token throughput · last 10 avg
- 21,462.443 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 1 --gpus-per-node 8 --recipe nemotron_3_nano_4b_peft_8gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/nemotron-3-nano-4b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' dataset.hf_output_root=work/data/tulu3/nemotron-3-nano-4b-peft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true 'dataset.offline_packing_specs={packed_sequence_size:2048,pad_seq_to_mult:4}' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/nemotron-3-nano-4b/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=trueExpected result
Pad-4 offline packing is 99.36% efficient. The frozen-base LoRA run uses rank 8, alpha 16, zero dropout, and linear_qkv/linear_proj targets. Its TP1/DP8 layout uses four microbatches per optimizer step and completes all 100 steps with finite loss and no skipped or NaN iterations. Across 6,553,600 token slots, assistant-only loss masks contain 4,178,503 supervised tokens. The complete eight-shard iter_0000100 adapter checkpoint reloads over the pinned base model at step 100.