Qwen3-30B-A3B#
Qwen is a family of large language models developed by Alibaba Cloud, including dense models (Qwen2, Qwen2.5, Qwen3) and Mixture-of-Experts models (Qwen3 MoE, Qwen3-Next). The models feature innovations like QK layernorm, Gated-Delta Networks, and Zero-Centered RMSNorm for improved training stability and performance.
Qwen family models are supported via the Bridge system with auto-detected configuration and weight mapping.
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model Qwen/Qwen3-30B-A3B --megatron-path work/model-verification/qwen3-30b-a3b/cpu-megatron --torch-dtype bfloat16Expected result
The command exits successfully, creates iter_0000000, and the checkpoint round-trips through CPU export with all 18,867 BF16 tensors matching the recorded HF revision bitwise.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model Qwen/Qwen3-30B-A3B --megatron-path work/model-verification/qwen3-30b-a3b/imported-megatron --torch-dtype bfloat16 --tp 4 --pp 2 --ep 4Expected result
The command exits successfully, creates iter_0000000, and all 18,867 BF16 tensors reload with keys, shapes, dtypes, and values exactly matching the recorded HF revision.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model Qwen/Qwen3-30B-A3B --megatron-path work/model-verification/qwen3-30b-a3b/cpu-megatron/iter_0000000 --hf-path work/model-verification/qwen3-30b-a3b/cpu-hf-export --torch-dtype bfloat16Expected result
The command exits successfully; all 18,867 exported BF16 tensors match the recorded HF revision bitwise, norm_topk_prob remains true, and the export reloads as Qwen3MoeForCausalLM without missing or unexpected keys.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model Qwen/Qwen3-30B-A3B --megatron-path work/model-verification/qwen3-30b-a3b/imported-megatron/iter_0000000 --hf-path work/model-verification/qwen3-30b-a3b/hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16 --distributed-save --tp 4 --pp 2 --ep 4Expected result
Strict export exits successfully, all 18,867 written BF16 tensors match the recorded HF revision bitwise, norm_topk_prob remains true, and Transformers reloads the output as Qwen3MoeForCausalLM without missing or unexpected keys.
Pretrain · H100
✓ VerifiedRecorded metrics
- Initial loss
- 12.41145
- Final loss
- 6.139116
- Step time · last 10 avg
- 30,289.550 ms
- Model throughput · last 10 avg
- 199.120 TFLOP/s/GPU
- Token throughput · last 10 avg
- 8,654.602 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen3_30b_a3b_pretrain_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[["work/data/rp2/head_01"],null]' dataset.path_to_cache=work/cache/qwen3-30b-a3b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model scheduler.lr_decay_iters=100 model.moe_router_force_load_balancing=false ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null validation.eval_iters=0 validation.eval_interval=0 dataset.random_seed=1234 dataset.num_workers=8 rng.seed=1234 dist.distributed_timeout_minutes=30 --save_dir work/model-verification/qwen3-30b-a3b/pretrain-convergence-v1-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 16x H100, the public alias resolves to the 16-GPU recipe and completes exactly 100 bounded RP2 optimizer steps with TP1/PP1/CP1/EP16/ETP1, DP16, SP off, GBS/MBS 1024/1, and 64-way gradient accumulation. Natural routing, HybridEP, and Transformer Engine CUDA graphs for moe_router and moe_preprocess remain active. Loss is finite from 12.41145 to 6.139116 with no skipped or NaN iterations, all five metrics are recorded, and complete iter_0000050 and iter_0000100 checkpoints are saved.
Pretrain · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 12.41293
- Final loss
- 6.183484
- Step time · last 10 avg
- 12,362.340 ms
- Model throughput · last 10 avg
- 487.880 TFLOP/s/GPU
- Token throughput · last 10 avg
- 21,205.047 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb200_fp8mx_functional_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[["work/data/rp2/head_01"],null]' dataset.path_to_cache=work/cache/qwen3-30b-a3b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model scheduler.lr_decay_iters=100 model.moe_router_force_load_balancing=false ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null checkpoint.save_optim=true checkpoint.save_rng=true checkpoint.load_optim=true checkpoint.load_rng=true checkpoint.finetune=false validation.eval_iters=0 validation.eval_interval=0 dataset.random_seed=1234 dataset.num_workers=8 rng.seed=1234 dist.distributed_timeout_minutes=30 --save_dir work/model-verification/qwen3-30b-a3b/pretrain-mxfp8-gb200-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 8x GB200, this support-verification workload completes exactly 100 bounded RP2 optimizer steps with TP1/PP1/CP1/EP8/ETP1, DP8, SP off, GBS/MBS 512/4, and 16-way gradient accumulation. MXFP8 compute, natural routing, HybridEP, Transformer Engine CUDA graphs for moe_router and moe_preprocess, communication overlap, functional safety checks, and MXFP8 parameter all-gather remain active. Loss is finite from 12.41293 to 6.183484 with no skipped or NaN iterations, all five metrics are recorded, and complete eight-shard iter_0000050 and iter_0000100 checkpoints are saved. Timing and throughput are support sanity checks, not cross-model convergence or tuned performance claims.
SFT · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.70438
- Final loss
- 0.9030643
- Step time · last 10 avg
- 1,206.320 ms
- Model throughput · last 10 avg
- 64.020 TFLOP/s/GPU
- Token throughput · last 10 avg
- 3,395.451 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen3_30b_a3b_sft_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/qwen3-30b-a3b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="ad44e777bcd18fa416d9da3bd8f70d33ebb85d39"' dataset.hf_output_root=work/data/tulu3/qwen3-30b-a3b-sft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=1 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true --save_dir work/model-verification/qwen3-30b-a3b/sft-convergence-v1-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
The immutable-revision 16-GPU run completes exactly 100 full-SFT steps at TP1/PP1/CP1/EP16/ETP1, DP16, SP off, GBS/MBS 32/1, and two-way gradient accumulation with natural routing. Pad-1 offline packing is 99.30% efficient, and the sampled 6,553,600 token slots contain 4,350,004 supervised tokens after label masking. LM loss is finite from 1.704380 to 0.9030643 with no skipped or NaN iterations, all five metrics are recorded, and the complete sixteen-shard iter_0000100 full-model checkpoint reloads successfully.
Long Context · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.645009
- Final loss
- 1.468103
- Step time · last 10 avg
- 142,663.710 ms
- Model throughput · last 10 avg
- 26.120 TFLOP/s/GPU
- Token throughput · last 10 avg
- 459.374 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen3_30b_a3b_sft_8gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/qwen3-30b-a3b/imported-megatron/iter_0000000 --max_steps 20 --seq_length 32768 --context_parallel_size 2 -tp 8 -pp 1 -ep 8 --lr 1e-6 --min_lr 0 --warmup_iters 2 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="ad44e777bcd18fa416d9da3bd8f70d33ebb85d39"' dataset.hf_output_root=work/data/tulu3/qwen3-30b-a3b-long-context-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=16 model.cp_comm_type=p2p model.cross_entropy_loss_fusion=false model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save=null logger.log_interval=1 logger.log_throughput=trueExpected result
The immutable-revision 16-GPU run completes exactly 20 Tulu3 SFT steps at sequence length 32768 with TP8/PP1/CP2/EP8/SP-on, DeepEP, and explicit pad-16 offline packing. LM loss is 1.645009 to 1.468103; skipped/NaN totals are 0/0. The persisted post-setup runtime config matches the command, packing is 99.28%, and the sampled training window contains 13,573,663 actual supervised tokens. PP=1 keeps tokens, labels, loss masks, and packed-sequence boundaries on one pipeline stage.
LoRA · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.575987
- Final loss
- 1.113119
- Step time · last 10 avg
- 22,347.640 ms
- Model throughput · last 10 avg
- 13.840 TFLOP/s/GPU
- Token throughput · last 10 avg
- 733.142 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 4 --recipe qwen3_30b_a3b_peft_4gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/qwen3-30b-a3b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="ad44e777bcd18fa416d9da3bd8f70d33ebb85d39"' dataset.hf_output_root=work/data/tulu3/qwen3-30b-a3b-peft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=4 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false checkpoint.load=null ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true --save_dir work/model-verification/qwen3-30b-a3b/peft-tp4-ep4-pad4-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=trueExpected result
The immutable-revision 4-GPU run completes exactly 100 PEFT steps at TP4/PP1/CP1/EP4/ETP1, DP1, SP on, GBS/MBS 32/1, and 32-way gradient accumulation with natural routing and DeepEP. Pad-4 offline packing is 99.39% efficient, and the sampled 6,553,600 token slots contain 4,332,480 supervised tokens after label masking. Only rank-8, alpha-16, zero-dropout LoRA on linear_qkv and linear_proj is trainable. LM loss is finite from 1.575987 to 1.113119 with no skipped or NaN iterations, all five metrics are recorded, and the complete four-shard iter_0000100 adapter checkpoint covers all 192 expected adapter entries.
Benchmark · H100
✓ VerifiedRecorded metrics
- Initial loss
- 12.34643
- Final loss
- 8.145514
- Step time · last 10 avg
- 20,147.290 ms
- Model throughput · last 10 avg
- 299.352 TFLOP/s/GPU
- Token throughput · last 10 avg
- 13,011.378 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen3_30b_a3b_pretrain_16gpu_h100_bf16_config --mode pretrain --max_steps 50Expected result
On two nodes with 16x H100, the exact mock-data performance recipe completes exactly 50 steps with finite losses, no skipped or NaN iterations, and all five metrics recorded. HybridEP permute fusion uses 32 SMs and 64-token combine chunks, expert-parallel communication overlap is active with delayed weight-gradient compute disabled, and Transformer Engine CUDA graph capture completes for all 48 graphable layers with moe_router and moe_preprocess scopes. Over steps 41-50, the run averages at most 20.50 seconds per step and at least 295 model TFLOP/s/GPU, while peak allocated memory remains below 65 GiB/GPU.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 12.34754
- Final loss
- 8.112733
- Step time · last 10 avg
- 6,501.120 ms
- Model throughput · last 10 avg
- 927.680 TFLOP/s/GPU
- Token throughput · last 10 avg
- 40,322.898 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb200_fp8mx_config --max_steps 50Expected result
On two nodes with 8x GB200, the exact mock-data performance recipe completes exactly 50 steps with finite losses, no skipped or NaN iterations, and all five metrics recorded. The benchmark uses forced load balancing, HybridEP, MXFP8, and full-iteration CUDA graphs. Over steps 41-50, the run averages at most 7 seconds per step and at least 900 model TFLOP/s/GPU.
Benchmark · GB300
✓ VerifiedRecorded metrics
- Initial loss
- 12.34753
- Final loss
- 8.125566
- Step time · last 10 avg
- 5,854.910 ms
- Model throughput · last 10 avg
- 1,030.070 TFLOP/s/GPU
- Token throughput · last 10 avg
- 44,773.361 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 2 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/qwen3-30b-a3b/gb300-performance/ConfigContainer.yamlExpected result
On exactly 8 GB300s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP8/ETP1, GBS/MBS 512/8, and sequence length 4096. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.34753 to 8.125566; the final ten steps average 5854.910 ms, 1030.070 TFLOP/s/GPU, and 44773.361 tokens/s/GPU. The resolved configuration persists.
Pretrain · Weak scaling · GB300 · 8-gpu
✓ VerifiedRecorded metrics
- Initial loss
- 12.34753
- Final loss
- 8.131992
- Step time · last 10 avg
- 5,889.420 ms
- Model throughput · last 10 avg
- 1,024.020 TFLOP/s/GPU
- Token throughput · last 10 avg
- 44,511.004 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 --global_batch_size 512Expected result
On GB300, the exact mock-data MXFP8 recipe completes 50 steps at each of 8, 32, 128, and 256 GPUs with TP1/PP1/CP1/EP8/ETP1, MBS8, sequence length 4096, and GBS proportional to GPU count. Every point has finite losses and performance values, zero skipped or NaN iterations, a persisted post-setup runtime config, and all five metrics recorded from complete keyed optimizer-step rows.
Pretrain · Weak scaling · GB300 · 32-gpu
✓ VerifiedRecorded metrics
- Initial loss
- 12.3471
- Final loss
- 8.131451
- Step time · last 10 avg
- 5,814.290 ms
- Model throughput · last 10 avg
- 1,037.270 TFLOP/s/GPU
- Token throughput · last 10 avg
- 45,086.158 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 8 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 --global_batch_size 2048Expected result
On GB300, the exact mock-data MXFP8 recipe completes 50 steps at each of 8, 32, 128, and 256 GPUs with TP1/PP1/CP1/EP8/ETP1, MBS8, sequence length 4096, and GBS proportional to GPU count. Every point has finite losses and performance values, zero skipped or NaN iterations, a persisted post-setup runtime config, and all five metrics recorded from complete keyed optimizer-step rows.
Pretrain · Weak scaling · GB300 · 128-gpu
✓ VerifiedRecorded metrics
- Initial loss
- 12.34679
- Final loss
- 8.126566
- Step time · last 10 avg
- 5,831.720 ms
- Model throughput · last 10 avg
- 1,034.180 TFLOP/s/GPU
- Token throughput · last 10 avg
- 44,951.404 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 32 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 --global_batch_size 8192Expected result
On GB300, the exact mock-data MXFP8 recipe completes 50 steps at each of 8, 32, 128, and 256 GPUs with TP1/PP1/CP1/EP8/ETP1, MBS8, sequence length 4096, and GBS proportional to GPU count. Every point has finite losses and performance values, zero skipped or NaN iterations, a persisted post-setup runtime config, and all five metrics recorded from complete keyed optimizer-step rows.
Pretrain · Weak scaling · GB300 · 256-gpu
✓ VerifiedRecorded metrics
- Initial loss
- 12.34698
- Final loss
- 8.129371
- Step time · last 10 avg
- 5,818.620 ms
- Model throughput · last 10 avg
- 1,036.490 TFLOP/s/GPU
- Token throughput · last 10 avg
- 45,052.607 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 64 --gpus-per-node 4 --recipe qwen3_30b_a3b_pretrain_8gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 --global_batch_size 16384Expected result
On GB300, the exact mock-data MXFP8 recipe completes 50 steps at each of 8, 32, 128, and 256 GPUs with TP1/PP1/CP1/EP8/ETP1, MBS8, sequence length 4096, and GBS proportional to GPU count. Every point has finite losses and performance values, zero skipped or NaN iterations, a persisted post-setup runtime config, and all five metrics recorded from complete keyed optimizer-step rows.