Qwen3-8B

Qwen is a family of large language models developed by Alibaba Cloud, including dense models (Qwen2, Qwen2.5, Qwen3) and Mixture-of-Experts models (Qwen3 MoE, Qwen3-Next). The models feature innovations like QK layernorm, Gated-Delta Networks, and Zero-Centered RMSNorm for improved training stability and performance.

Qwen family models are supported via the Bridge system with auto-detected configuration and weight mapping.

Verified configurations

Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.

Run a configuration

Choose a workflow, precision, and exact recorded combination. The command and expected result update below.

Import · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-16

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model Qwen/Qwen3-8B --megatron-path work/model-verification/qwen3-8b/cpu-megatron --torch-dtype bfloat16

Expected result

The command exits successfully, creates iter_0000000, and the checkpoint round-trips through CPU export with all 399 HF tensors matching the recorded HF revision exactly in keys, shapes, dtypes, and values.

Import · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-16

Exact command

Command Copy

./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 4 --hf-model Qwen/Qwen3-8B --megatron-path work/model-verification/qwen3-8b/imported-megatron --torch-dtype bfloat16 --tp 4

Expected result

The command exits successfully, creates iter_0000000, and the checkpoint reloads at TP=4 with weights exactly matching the recorded HF revision.

Export · CPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-16

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model Qwen/Qwen3-8B --megatron-path work/model-verification/qwen3-8b/cpu-megatron/iter_0000000 --hf-path work/model-verification/qwen3-8b/cpu-hf-export

Expected result

The command exits successfully; all 399 exported tensors match the recorded HF revision exactly, and the export reloads on CPU as Qwen3ForCausalLM.

Export · GPU

✓ Verified

Hardware

not specified

Precision

BF16

Last verified

2026-07-16

Exact command

Command Copy

./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 4 --hf-model Qwen/Qwen3-8B --megatron-path work/model-verification/qwen3-8b/imported-megatron/iter_0000000 --hf-path work/model-verification/qwen3-8b/hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16 --distributed-save --tp 4

Expected result

Strict export exits successfully and the Hugging Face output reloads with AutoModelForCausalLM as Qwen3ForCausalLM.

Pretrain · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-07-21

Recorded metrics

Initial loss

12.73617

Final loss

6.190218

Step time · last 10 avg

25,070.320 ms

Model throughput · last 10 avg

512.700 TFLOP/s/GPU

Token throughput · last 10 avg

10,456.348 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen3_8b_pretrain_16gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[["work/data/rp2/head_01"],null]' dataset.path_to_cache=work/cache/qwen3-8b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model rng.seed=1234 dataset.random_seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 dist.distributed_timeout_minutes=30 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null --save_dir work/model-verification/qwen3-8b/pretrain-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=null

Expected result

The uninterrupted 16-GPU bounded RP2 run completes exactly 100 steps at TP1/PP1/CP1, DP16, GBS/MBS 1024/1, and 64-way gradient accumulation. It reaches peak learning rate at step 40 and completes cosine decay at step 100. LM loss is finite from 12.73617 to 6.190218 with no skipped or NaN iterations, all five metrics are recorded, the post-setup configuration persists, and complete 16-shard iter_0000050 and iter_0000100 checkpoints are saved.

SFT · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-07-19

Recorded metrics

Initial loss

1.657044

Final loss

0.9733383

Step time · last 10 avg

7,424.240 ms

Model throughput · last 10 avg

101.630 TFLOP/s/GPU

Token throughput · last 10 avg

2,206.825 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 1 --gpus-per-node 4 --recipe qwen3_8b_sft_4gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/qwen3-8b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="b968826d9c46dd6066d109eabc6255188de91218"' dataset.hf_output_root=work/data/tulu3/qwen3-8b-sft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=1 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/qwen3-8b/sft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true

Expected result

Immutable Tulu3 pad-1 offline packing is 99.30% efficient. Full SFT uses DP=1 with 32 gradient-accumulation steps and reaches step 100 with the four recorded metrics, finite loss, and no skipped or NaN iterations. Across 6,553,600 token slots, the sampled assistant-only loss masks contain 4,350,004 supervised tokens. The complete four-shard iter_0000100 full-model checkpoint is saved.

Long Context · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-07-20

Recorded metrics

Initial loss

1.611622

Final loss

1.490618

Step time · last 10 avg

71,668.780 ms

Model throughput · last 10 avg

34.080 TFLOP/s/GPU

Token throughput · last 10 avg

457.214 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe qwen3_8b_sft_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/qwen3-8b/imported-megatron/iter_0000000 --max_steps 20 --seq_length 32768 --context_parallel_size 2 --lr 1e-6 --min_lr 0 --warmup_iters 2 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="b968826d9c46dd6066d109eabc6255188de91218"' dataset.hf_output_root=work/data/tulu3/qwen3-8b-long-context-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=8 model.sequence_parallel=true model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save=null logger.log_interval=1 logger.log_throughput=true

Expected result

The immutable-revision 8-GPU run completes exactly 20 Tulu3 SFT steps at sequence length 32768 with recipe-owned TP4/PP1/CP2/SP-on, GBS/MBS 8/1, and explicit pad-8 offline packing. LM loss is 1.611622 to 1.490618; skipped/NaN totals are 0/0. The persisted post-setup runtime config matches the command, packing is 99.97%, and the sampled training window contains 3,444,917 actual supervised tokens.

LoRA · H100

✓ Verified

Hardware

H100

Precision

BF16

Last verified

2026-07-19

Recorded metrics

Initial loss

1.527745

Final loss

1.170038

Step time · last 10 avg

7,774.790 ms

Model throughput · last 10 avg

387.430 TFLOP/s/GPU

Token throughput · last 10 avg

8,429.295 tokens/s/GPU

Exact command

Command Copy

./scripts/training/train.sh --nodes 1 --gpus-per-node 1 --recipe qwen3_8b_peft_1gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/qwen3-8b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="b968826d9c46dd6066d109eabc6255188de91218"' dataset.hf_output_root=work/data/tulu3/qwen3-8b-peft-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=4 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/qwen3-8b/peft-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true

Expected result

Immutable Tulu3 pad-4 offline packing is 99.39% efficient. The 100 LoRA steps use DP=1 with 32 gradient-accumulation steps and complete with the five recorded metrics, finite loss, and no skipped or NaN iterations. Across 6,553,600 token slots, the sampled assistant-only loss masks contain 4,332,480 supervised tokens. The complete single-shard iter_0000100 adapter checkpoint is saved.