Qwen3.6-35B-A3B#
Qwen3.5 is a family of vision-language models supporting multimodal understanding across text, images, and videos. Qwen3.5-VL includes both dense models and Mixture-of-Experts (MoE) variants for improved efficiency at scale.
Qwen3.6 shares the same architecture as Qwen3.5 VL MoE (Qwen3_5MoeForConditionalGeneration) and is supported through the same bridge implementation.
Qwen 3.5/3.6 models feature a hybrid architecture combining GDN (Gated DeltaNet) layers with standard attention layers, SwiGLU activations, and RMSNorm. MoE variants use top-k routing with shared experts for better quality.
Qwen 3.5/3.6 models are supported via Megatron Bridge with auto-detected configuration and weight mapping.
Important
Use transformers >= 5.2.0 for Qwen3.5 and >= 5.8.1 for Qwen3.6.
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --gpus-per-node 1 --mem 512G --hf-model Qwen/Qwen3.6-35B-A3B --hf-revision 995ad96eacd98c81ed38be0c5b274b04031597b0 --megatron-path work/model-verification/qwen3.6-35b-a3b/cpu-megatronExpected result
The pinned-revision import exited successfully in 14m39s and created iter_0000000. Model weights remained on CPU; exactly one shared runtime GPU was required because Transformer Engine constructs the Qwen vision modules through CUDA even for CPU-weight conversion. The resulting checkpoint was reloadable by the subsequent verified CPU export.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 2 --gpus-per-node 8 --hf-model Qwen/Qwen3.6-35B-A3B --hf-revision 995ad96eacd98c81ed38be0c5b274b04031597b0 --megatron-path work/model-verification/qwen3.6-35b-a3b/imported-megatron --tp 2 --ep 8Expected result
The 16-H100 pinned-revision import exited successfully at TP2/PP1/EP8/ETP1 and created a 16-shard iter_0000000 distributed checkpoint. Subsequent verified GPU export and inference workloads reloaded this checkpoint successfully.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --gpus-per-node 1 --mem 512G --hf-model Qwen/Qwen3.6-35B-A3B --hf-revision 995ad96eacd98c81ed38be0c5b274b04031597b0 --megatron-path work/model-verification/qwen3.6-35b-a3b/cpu-megatron/iter_0000000 --hf-path work/model-verification/qwen3.6-35b-a3b/cpu-hf-exportExpected result
Export exited successfully in 6m42s with weights on CPU and exactly one shared runtime GPU for Transformer Engine vision-module construction. Transformers strictly reloaded all 1,045 BF16 tensors in 26 indexed shards as Qwen3_5MoeForConditionalGeneration. The exported config preserves 40 language layers, 256 experts, top-8 routing, 27 vision layers, and one MTP layer with dedicated embeddings disabled.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 2 --gpus-per-node 8 --hf-model Qwen/Qwen3.6-35B-A3B --hf-revision 995ad96eacd98c81ed38be0c5b274b04031597b0 --megatron-path work/model-verification/qwen3.6-35b-a3b/imported-megatron/iter_0000000 --hf-path work/model-verification/qwen3.6-35b-a3b/hf-export --tp 2 --ep 8Expected result
The 16-H100 distributed export exited successfully and wrote 26 indexed shards containing 1,045 BF16 tensors and 71,903,645,408 serialized tensor bytes. Transformers independently reloaded the persisted export as Qwen3_5MoeForConditionalGeneration with empty missing, unexpected, mismatched, and error sets.
Pretrain · H100
✓ VerifiedRecorded metrics
- Initial loss
- 3.610127
- Final loss
- 2.261159
- Step time · last 10 avg
- 330,730.470 ms
- Model throughput · last 10 avg
- 18.090 TFLOP/s/GPU
- Token throughput · last 10 avg
- 792.621 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe qwen35_vl_35b_a3b_pretrain_mock_config --dataset energon --deterministic --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/imported-megatron/iter_0000000 --max_steps 20 --warmup_iters 2 --save_dir work/model-verification/qwen3.6-35b-a3b/datacomp-deterministic-reference/checkpoints --save_interval 10 dataset.path=work/data/datacomp/energon dataset.task_encoder.hf_processor_path=Qwen/Qwen3.6-35B-A3B dataset.task_encoder.hf_processor_revision=995ad96eacd98c81ed38be0c5b274b04031597b0 dataset.do_validation=false scheduler.lr_decay_iters=20 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 checkpoint.load=null validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/datacomp-deterministic-reference/resolved-config.yamlExpected result
At the recorded bridge commit, this bounded projection-pretraining support run used the official DataComp download pipeline at repository commit 4a8df1992566, the DataComp-1B metadata at revision 086ebeee20d4, and img2dataset 1.40.0. The Energon dataset contains 519,827 training and 5,173 validation image-caption samples; each training record maps an image plus "Describe this image." to its caption. The deterministic 20-step reference consumed 10,240 training samples without wrapping. It warm-started the imported Qwen3.6 checkpoint, froze the language and vision towers, and trained the vision projection at TP1/PP1/CP1/EP8/ETP1, DP8, MBS1/GBS512, and 64-way gradient accumulation with one-layer full-uniform recompute. Loss was finite from 3.610127 to 2.261159 with zero skipped or NaN iterations; steps 11-20 averaged 330,730.470 ms and 18.090 TFLOP/s/GPU. Complete 72,366,371,530-byte eight-shard model/optimizer/RNG checkpoints and eight nonempty Energon rank states were saved at both steps 10 and 20, and the process exited successfully in 1h54m51s. This verifies the DataComp workflow and bounded Qwen VLM pretraining support, not canonical DataComp/CLIP convergence.
SFT · H100
✓ VerifiedRecorded metrics
- Initial loss
- 2.211317
- Final loss
- 1.006886
- Step time · last 10 avg
- 3,268.140 ms
- Model throughput · last 10 avg
- 57.690 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,506.625 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen35_vl_35b_a3b_sft_16gpu_h100_bf16_config --dataset medpix --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/imported-megatron/iter_0000000 --max_steps 100 --warmup_iters 10 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=Qwen/Qwen3.6-35B-A3B 'dataset.hf_processor_kwargs={revision:995ad96eacd98c81ed38be0c5b274b04031597b0}' dataset.do_validation=false dataset.pad_to_max_length=true scheduler.lr_decay_iters=100 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true --save_dir work/model-verification/qwen3.6-35b-a3b/sft-support-v11/checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/sft-support-v11/resolved-config.yamlExpected result
The exact 16-H100 command completed all 100 MedPix SFT steps at TP1/PP2/CP1/EP8/ETP1, MBS1/GBS32, natural expert routing, HybridEP, one-layer full-uniform recompute, graph-free language and vision stacks, FP32 optimizer state, and Transformer Engine fused cross entropy. Loss was finite from 2.211317 to 1.006886 with zero skipped or NaN iterations. Steps 91-100 averaged 3,268.140 ms and 57.690 model TFLOP/s/GPU. Step-50 and step-100 saves both completed; the final checkpoint contains 20 nonempty files, including 16 distributed shards, totaling 509,636,716,209 bytes, and the tracker selects iteration 100.
SFT · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 2.212866
- Final loss
- 0.9978729
- Step time · last 10 avg
- 9,899.640 ms
- Model throughput · last 10 avg
- 38.180 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,655.010 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe qwen35_vl_35b_a3b_sft_8gpu_gb200_bf16_functional_config --dataset medpix --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/gb200-imported/iter_0000000 --max_steps 100 --warmup_iters 10 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=Qwen/Qwen3.6-35B-A3B 'dataset.hf_processor_kwargs={revision:995ad96eacd98c81ed38be0c5b274b04031597b0}' scheduler.lr_decay_iters=100 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/gb200-sft/resolved-config.yamlExpected result
On 8x GB200, the shared Qwen3.5/Qwen3.6-VL functional recipe completed exactly 100 MedPix SFT steps at TP1/PP1/CP1/EP8/ETP1, MBS1/GBS32, natural routing, HybridEP, no activation recompute, and BF16 gradient reduction. Loss was finite from 2.212866 to 0.9978729 with zero skipped or NaN iterations. Steps 91-100 averaged 9,899.640 ms and 38.180 model TFLOP/s/GPU. The recipe intentionally disables checkpoint output for this bounded functional run; reload/export coverage remains represented by the H100 SFT and sft_export_inference items.
Long Context · H100
✓ VerifiedRecorded metrics
- Initial loss
- 1.997653
- Final loss
- 0.997894
- Step time · last 10 avg
- 36,847.110 ms
- Model throughput · last 10 avg
- 84.830 TFLOP/s/GPU
- Token throughput · last 10 avg
- 3,557.185 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 4 --gpus-per-node 8 --recipe qwen35_vl_35b_a3b_sft_long_context_32gpu_h100_bf16_config --dataset medpix --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/imported-megatron/iter_0000000 --max_steps 20 --warmup_iters 2 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=Qwen/Qwen3.6-35B-A3B 'dataset.hf_processor_kwargs={revision:995ad96eacd98c81ed38be0c5b274b04031597b0}' dataset.do_validation=false scheduler.lr_decay_iters=20 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 checkpoint.load=null checkpoint.save=null validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/medpix-long-context/resolved-config.yamlExpected result
The pinned 32-H100 run completed exactly 20 full-SFT steps at TP1/PP4/CP2/EP8/ETP1, dense DP4 and expert DP1, MBS2/GBS512, and 64-way gradient accumulation. It used 8192-token processor-native deferred in-batch packing with pad-to-multiple-of-4, fixed 448-by-448 MedPix images, per-token loss, and one-layer full-uniform activation recompute. The persisted post-setup config confirms every setting. Loss was finite from 1.997653 to 0.997894 with zero skipped or NaN iterations. After the compile-heavy first step, steps 11-20 averaged 36,847.110 ms and 84.830 TFLOP/s/GPU; the process exited successfully.
LoRA · H100
✓ VerifiedRecorded metrics
- Initial loss
- 2.212696
- Final loss
- 1.270382
- Step time · last 10 avg
- 1,714.300 ms
- Model throughput · last 10 avg
- 109.940 TFLOP/s/GPU
- Token throughput · last 10 avg
- 4,778.627 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen35_vl_35b_a3b_peft_16gpu_h100_bf16_config --dataset medpix --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/imported-megatron/iter_0000000 --max_steps 100 --warmup_iters 10 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=Qwen/Qwen3.6-35B-A3B 'dataset.hf_processor_kwargs={revision:995ad96eacd98c81ed38be0c5b274b04031597b0}' dataset.do_validation=false dataset.pad_to_max_length=true scheduler.lr_decay_iters=100 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true --save_dir work/model-verification/qwen3.6-35b-a3b/peft-support-v5/checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/peft-support-v5/resolved-config.yamlExpected result
The exact 16-H100 command completed all 100 MedPix LoRA steps at TP1/PP2/CP1/EP8/ETP1, MBS1/GBS32, natural expert routing, HybridEP, scoped Transformer Engine CUDA graphs, and no activation recompute. The recipe disables dispatch-backward/expert-wgrad overlap because LoRA-wrapped expert linears do not expose Megatron Core's backward_dw hook. Loss was finite from 2.212696 to 1.270382 with zero skipped or NaN iterations. Steps 91-100 averaged 1,714.300 ms and 109.940 model TFLOP/s/GPU. The step-100 checkpoint contains 20 nonempty files, including 16 distributed shards, totaling 5,004,722,270 bytes.
LoRA · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 2.214827
- Final loss
- 1.254138
- Step time · last 10 avg
- 5,511.530 ms
- Model throughput · last 10 avg
- 68.410 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,972.677 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe qwen35_vl_35b_a3b_peft_8gpu_gb200_bf16_functional_config --dataset medpix --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/gb200-imported/iter_0000000 --max_steps 100 --warmup_iters 10 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=Qwen/Qwen3.6-35B-A3B 'dataset.hf_processor_kwargs={revision:995ad96eacd98c81ed38be0c5b274b04031597b0}' scheduler.lr_decay_iters=100 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/gb200-peft/resolved-config.yamlExpected result
On the same 8x GB200 topology and MedPix schedule as the controlled SFT item, LoRA completed exactly 100 finite steps with zero skipped or NaN iterations. The persisted configs differ only in the evidence-output path, the intended PEFT LR/min-LR, and the LoRA block; model, data, topology, batch, schedule, and safety settings match. Step-1 loss 2.214827 differs from SFT's 2.212866 by only 0.001961, or 0.088618%, ruling out an initial-forward configuration mismatch. The later loss difference reflects LoRA's trainable parameter set and learning rate. Steps 91-100 averaged 5,511.530 ms and 68.410 model TFLOP/s/GPU.
Benchmark · H100
✓ VerifiedRecorded metrics
- Initial loss
- 3.604611
- Final loss
- 3.458849
- Step time · last 10 avg
- 22,271.220 ms
- Model throughput · last 10 avg
- 135.610 TFLOP/s/GPU
- Token throughput · last 10 avg
- 5,885.264 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen35_vl_35b_a3b_pretrain_16gpu_h100_bf16_functional_config --dataset energon --pretrained_checkpoint work/model-verification/qwen3.6-35b-a3b/imported-megatron/iter_0000000 --max_steps 50 --warmup_iters 10 dataset.path=work/data/datacomp/energon dataset.task_encoder.hf_processor_path=Qwen/Qwen3.6-35B-A3B dataset.task_encoder.hf_processor_revision=995ad96eacd98c81ed38be0c5b274b04031597b0 dataset.task_encoder.max_pixels=200704 dataset.do_validation=false dataset.pad_to_max_length=true scheduler.lr_decay_iters=50 model.hf_model_id=Qwen/Qwen3.6-35B-A3B model.bos_token_id=248044 checkpoint.load=null checkpoint.save=null validation.eval_iters=0 validation.eval_interval=0 logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/h100-performance/resolved-config.yamlExpected result
The exact 16-H100 command completed all 50 DataComp Energon steps at TP1/PP2/CP1/EP8/ETP1, a 16/24 language-layer pipeline split, and MBS1/GBS512. The persisted config confirmed the 248,320-token model vocabulary, natural expert routing, HybridEP, selective core-attention, GDN output-norm, and MoE-activation recompute, scoped Transformer Engine CUDA graphs for language attention/router/preprocessing, a graph-free variable-shape vision encoder, and Transformer Engine fused cross entropy. Loss remained finite from 3.604611 to 3.458849 with zero skipped or NaN iterations. Steps 41-50 averaged 22,271.220 ms and 135.610 model TFLOP/s/GPU, and the process exited successfully without checkpoint output. The separate fixed-shape forced-balance tuning screen remains the source of the 185.150-TFLOP/s/GPU leader policy; it is not substituted for this real-data measurement.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 12.81499
- Final loss
- 3.166235
- Step time · last 10 avg
- 26,880.420 ms
- Model throughput · last 10 avg
- 210.360 TFLOP/s/GPU
- Token throughput · last 10 avg
- 9,142.714 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 4 --recipe qwen35_vl_35b_a3b_pretrain_8gpu_gb200_fp8mx_config --max_steps 50 logger.save_config_filepath=work/model-verification/qwen3.6-35b-a3b/gb200-performance/resolved-config.yamlExpected result
On 8x GB200, the exact architecture-shared Qwen3.5/Qwen3.6-VL fixed-shape mock-data recipe completed 50 steps at TP1/PP1/CP1/EP8/ETP1, dense DP8 and expert DP1, MBS3/GBS480, and 20 microbatches per optimizer step. The persisted post-setup config confirmed forced expert balance, HybridEP, MXFP8, cuDNN LayerNorm, and Transformer Engine CUDA graphs for the MoE router and preprocessing scopes. Loss was finite from 12.81499 to 3.166235 with zero skipped or NaN iterations. Steps 41-50 averaged 26,880.420 ms and 210.360 model TFLOP/s/GPU, and the process exited successfully. This synthetic result is a throughput measurement, not a convergence claim.