Qwen3.8-27B#
Qwen3.5 is a family of vision-language models supporting multimodal understanding across text, images, and videos. Qwen3.5-VL includes both dense models and Mixture-of-Experts (MoE) variants for improved efficiency at scale.
Qwen3.6 shares the same architecture as Qwen3.5 VL MoE (Qwen3_5MoeForConditionalGeneration) and is supported through the same bridge implementation.
Qwen 3.5/3.6 models feature a hybrid architecture combining GDN (Gated DeltaNet) layers with standard attention layers, SwiGLU activations, and RMSNorm. MoE variants use top-k routing with shared experts for better quality.
Qwen 3.5/3.6 models are supported via Megatron Bridge with auto-detected configuration and weight mapping.
Important
Use transformers >= 5.2.0 for Qwen3.5 and >= 5.8.1 for Qwen3.6.
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --gpus-per-node 1 --mem 512G --hf-model Qwen/Qwen3.8-27B --hf-revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --megatron-path work/model-verification/qwen3.8-27b/cpu-megatronExpected result
The pinned-revision CPU import completed in 3m11s and created a reloadable iter_0000000 containing 8 nonempty files totaling 55,586,366,001 bytes. The converter selected Qwen35VLBridge, completed all 956 mappings, kept model weights on CPU, and reported 27,781,427,952 parameters. One shared runtime GB200 was used for Transformer Engine vision-module construction.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 4 --hf-model Qwen/Qwen3.8-27B --hf-revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --megatron-path work/model-verification/qwen3.8-27b/gpu-megatron --tp 4Expected result
The pinned-revision four-GB200 import completed in 2m01s at TP4/PP1 and created an 11-file iter_0000000 totaling 55,592,033,255 bytes, including four distributed checkpoint shards. All 956 conversion mappings completed, and subsequent export and two independent inference runs reloaded it.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --gpus-per-node 1 --mem 512G --hf-model Qwen/Qwen3.8-27B --hf-revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --megatron-path work/model-verification/qwen3.8-27b/cpu-megatron/iter_0000000 --hf-path work/model-verification/qwen3.8-27b/cpu-hf-exportExpected result
CPU export completed in 2m16s, wrote every source tensor, and produced 26 files totaling 55,583,128,197 bytes. The index preserves the original 1,199-key set and exact 18-shard placement with 55,562,855,904 BF16 tensor bytes. Transformers strictly reloaded the export as Qwen3_5ForConditionalGeneration with empty missing, unexpected, mismatched, and error sets, then completed processor-native GB200 inference. Export normalizes vision_config.model_type from qwen3_5 to qwen3_5_vision and updates producer-version metadata without changing the architecture.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 4 --hf-model Qwen/Qwen3.8-27B --hf-revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --megatron-path work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --hf-path work/model-verification/qwen3.8-27b/gpu-hf-export --tp 4Expected result
The four-GB200 distributed export completed in 1m55s, wrote every source tensor, and produced 26 files totaling 55,583,128,222 bytes. Its index has the same 1,199 keys, 18-shard placement, and 55,562,855,904 BF16 tensor bytes as the pinned source and CPU export. Transformers strictly reloaded it with empty missing, unexpected, mismatched, and error sets and completed processor-native inference. Relative to CPU export, config.json only adds the metadata field vision_config.dtype=bfloat16.
Pretrain · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 6.852052
- Final loss
- 3.831639
- Step time · last 10 avg
- 3,551.780 ms
- Model throughput · last 10 avg
- 390.360 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,306.449 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 4 --gpus-per-node 4 --recipe qwen35_vl_27b_pretrain_16gpu_h100_bf16_mock_config --mode pretrain --pretrained_checkpoint work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --max_steps 20 --warmup_iters 2 --save_dir work/model-verification/qwen3.8-27b/pretrain/checkpoints --save_interval 10 dataset.hf_processor_path=work/model-verification/qwen3.8-27b/gpu-hf-export tokenizer.tokenizer_model=work/model-verification/qwen3.8-27b/gpu-hf-export model.hf_model_id=Qwen/Qwen3.8-27B model.bos_token_id=248044 checkpoint.load=null validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.8-27b/pretrain/resolved-config.yaml scheduler.lr_decay_iters=20Expected result
The 16-GB200 MockVLM pretrain warm-started from the imported checkpoint and completed exactly 20 steps at TP4/PP4, sequence length 4096, and MBS2/GBS32. The language model and multimodal vision merger trained while the vision encoder remained frozen. Deterministic mode is omitted: it replaces the fused Gated DeltaNet kernels with a torch-native reference path and is rejected alongside the recipe's pre-gated-delta-rule fusion. Loss remained finite from 6.852052 to 3.831639 with finite gradients and zero skipped or NaN iterations. Steps 11-20 averaged 3,551.780 ms, 390.360 model TFLOP/s/GPU, and 2,306.449 token slots/s/GPU; step 11, the first step after the iteration-10 save, took 4,593.5 ms. Checkpoints at iterations 10 and 20 each contain 20 files totaling 399,255,348,187 bytes, and the tracker records iteration 20. The built-in 32-iteration validation and test passes completed at loss 3.824842 and 3.824848. The process exited successfully in 16m03s.
SFT · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 2.116404
- Final loss
- 1.429567
- Step time · last 10 avg
- 24,160.490 ms
- Model throughput · last 10 avg
- 58.010 TFLOP/s/GPU
- Token throughput · last 10 avg
- 339.066 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 4 --gpus-per-node 4 --recipe qwen35_vl_27b_sft_16gpu_h100_bf16_config --mode sft --dataset medpix --deterministic --pretrained_checkpoint work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --max_steps 100 --warmup_iters 10 --save_interval 50 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=work/model-verification/qwen3.8-27b/gpu-hf-export dataset.do_validation=false dataset.pad_to_max_length=true tokenizer.tokenizer_model=work/model-verification/qwen3.8-27b/gpu-hf-export model.hf_model_id=Qwen/Qwen3.8-27B model.bos_token_id=248044 model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 model.recompute_modules=null scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true --save_dir work/model-verification/qwen3.8-27b/sft-recompute/checkpoints logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.8-27b/sft-recompute/resolved-config.yamlExpected result
The 16-GB200 run completed exactly 100 full-SFT MedPix steps at TP4/PP4, MBS4/GBS32, deterministic mode, and one-layer full-uniform activation recompute. Loss remained finite from 2.116404 to 1.429567 with zero skipped or NaN iterations. Steps 91-100 averaged 24,160.490 ms and 58.010 model TFLOP/s/GPU. Step-50 and step-100 saves each contain 20 files totaling 404,422,605,133 bytes, the tracker selects iteration 100, and the process exited successfully in 45m42s. The unchanged stock TP4/PP4 recipe without activation recompute reached the training loop but OOMed near the approximately 184.3-GiB per-GPU device limit; the successful command records the required config-only recompute overrides.
Long Context · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 2.343622
- Final loss
- 1.378064
- Step time · last 10 avg
- 8,164.790 ms
- Model throughput · last 10 avg
- 362.530 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,006.665 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 4 --gpus-per-node 4 --recipe qwen35_vl_27b_sft_16gpu_h100_bf16_config --mode sft --dataset medpix --pretrained_checkpoint work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --max_steps 20 --warmup_iters 2 --seq_length 8192 --pipeline_model_parallel_size 2 --context_parallel_size 2 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=work/model-verification/qwen3.8-27b/gpu-hf-export dataset.do_validation=false dataset.enable_in_batch_packing=true dataset.defer_in_batch_packing_to_step=true dataset.in_batch_packing_pad_to_multiple_of=4 tokenizer.tokenizer_model=work/model-verification/qwen3.8-27b/gpu-hf-export model.hf_model_id=Qwen/Qwen3.8-27B model.bos_token_id=248044 model.calculate_per_token_loss=true model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 ddp.average_in_collective=false scheduler.lr_decay_iters=20 checkpoint.load=null checkpoint.save=null validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.8-27b/sft-long/resolved-config.yamlExpected result
The 16-GB200 run completed exactly 20 full-SFT steps at TP4/PP2/CP2, MBS4/GBS32, 8192 tokens, deferred processor-native in-batch packing with pad-to-multiple-of-4, per-token loss, and one-layer full-uniform activation recompute. Loss remained finite from 2.343622 to 1.378064 with zero skipped or NaN iterations. Steps 11-20 averaged 8,164.790 ms and 362.530 model TFLOP/s/GPU. The process exited successfully without checkpoint output. An earlier otherwise identical attempt with --deterministic reached the training loop but triggered the explicit guard that packed sequence does not support deterministic mode; the successful command omits that incompatible flag, as does the Qwen 3.5 MoE reference card.
LoRA · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 2.115661
- Final loss
- 1.558339
- Step time · last 10 avg
- 74,048.550 ms
- Model throughput · last 10 avg
- 75.790 TFLOP/s/GPU
- Token throughput · last 10 avg
- 442.520 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 4 --recipe qwen35_vl_27b_peft_2gpu_h100_bf16_config --mode lora --tensor_model_parallel_size 4 --dataset medpix --deterministic --pretrained_checkpoint work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --max_steps 100 --warmup_iters 10 --save_interval 50 'dataset.source.load_kwargs={revision:69eb219d477ab8699296da1fd6b89beb768042d3}' 'dataset.source.adapter_kwargs={resized_height:448,resized_width:448}' dataset.hf_processor_path=work/model-verification/qwen3.8-27b/gpu-hf-export dataset.do_validation=false dataset.pad_to_max_length=true tokenizer.tokenizer_model=work/model-verification/qwen3.8-27b/gpu-hf-export model.hf_model_id=Qwen/Qwen3.8-27B model.bos_token_id=248044 model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 model.recompute_modules=null scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_large_grads=true --save_dir work/model-verification/qwen3.8-27b/peft-tp4-recompute/checkpoints logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.8-27b/peft-tp4-recompute/resolved-config.yamlExpected result
The four-GB200 run completed exactly 100 deterministic MedPix LoRA steps at TP4/PP1, MBS4/GBS32, and one-layer full-uniform activation recompute. Loss remained finite from 2.115661 to 1.558339 with zero skipped or NaN iterations. Steps 91-100 averaged 74,048.550 ms and 75.790 model TFLOP/s/GPU. Step-50 and step-100 adapter checkpoints each contain 8 files totaling 2,294,105,245 bytes, the tracker selects iteration 100, and the process exited successfully in 2h01m43s. The unchanged stock TP2 two-GPU recipe without activation recompute OOMed; a TP4 retry without recompute also OOMed near the approximately 184.3-GiB per-GPU device limit. The successful command records only the required topology and recompute config overrides, with no code change.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 6.852387
- Final loss
- 3.884668
- Step time · last 10 avg
- 1,679.460 ms
- Model throughput · last 10 avg
- 819.190 TFLOP/s/GPU
- Token throughput · last 10 avg
- 4,877.758 tokens/s/GPU
- Peak allocated memory
- 174.120 GiB
Exact command
./scripts/training/train.sh --nodes 4 --gpus-per-node 4 --recipe qwen35_vl_27b_pretrain_16gpu_gb200_bf16_mock_config --mode pretrain --pretrained_checkpoint work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --max_steps 20 --warmup_iters 2 dataset.hf_processor_path=work/model-verification/qwen3.8-27b/gpu-hf-export tokenizer.tokenizer_model=work/model-verification/qwen3.8-27b/gpu-hf-export model.hf_model_id=Qwen/Qwen3.8-27B model.bos_token_id=248044 logger.save_config_filepath=work/model-verification/qwen3.8-27b/pretrain-performance/resolved-config.yaml scheduler.lr_decay_iters=20Expected result
On 16 GB200 GPUs, the exact canonical BF16 language-and-projector MockVLM recipe completes 20 steps at TP2/PP1/CP1, DP8, sequence length 4096, and MBS2/GBS32. The language model and multimodal vision merger train while the vision encoder remains frozen. The recipe uses fused Gated DeltaNet execution without activation recompute and disables evaluation and checkpoint output for the bounded run. Loss remains finite from 6.852387 to 3.884668 with finite gradients and zero skipped or NaN iterations. Excluding first-iteration kernel compilation, steps 11-20 average 1,679.460 ms, 819.190 corrected model TFLOP/s/GPU, and 4,877.758 token slots/s/GPU; peak allocated memory is 174.120 GiB. The process exits successfully. This mock-data result is throughput evidence, not convergence evidence.
Benchmark · H100
✓ VerifiedRecorded metrics
- Initial loss
- 6.860321
- Final loss
- 4.485981
- Step time · last 10 avg
- 4,591.290 ms
- Model throughput · last 10 avg
- 299.620 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,784.248 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe qwen35_vl_27b_pretrain_16gpu_h100_bf16_mock_config --mode pretrain --pretrained_checkpoint work/model-verification/qwen3.8-27b/gpu-megatron/iter_0000000 --max_steps 20 --warmup_iters 2 model.sequence_parallel=true dataset.hf_processor_path=Qwen/Qwen3.8-27B tokenizer.tokenizer_model=Qwen/Qwen3.8-27B model.hf_model_id=Qwen/Qwen3.8-27B model.bos_token_id=248044 scheduler.lr_decay_iters=20 train.eval_iters=0 validation.eval_iters=0 validation.eval_interval=0 checkpoint.save=null checkpoint.load=null logger.log_interval=1 logger.log_throughput=true logger.save_config_filepath=work/model-verification/qwen3.8-27b/pretrain-performance-h100/resolved-config.yamlExpected result
On two nodes with 16x H100, the BF16 language-and-projector MockVLM recipe warm-starts from the card's imported GPU Megatron checkpoint and completes 20 steps at TP4/PP4/CP1, DP1, sequence length 4096, and MBS2/GBS32. Sequence parallelism is enabled by the explicit model.sequence_parallel=true override; the recipe default leaves it disabled, which exhausts H100 memory on the first pipeline stage in bounded runs. The language model, including its single MTP layer, and the multimodal vision merger train while the vision encoder remains frozen. Gated DeltaNet runs fused, without activation recompute or CUDA graphs, and evaluation and checkpoint output are disabled. Loss remains finite from 6.860321 to 4.485981 with zero skipped or NaN iterations. Excluding first-iteration Triton compilation and autotuning, steps 11-20 average 4,591.290 ms, 299.620 model TFLOP/s/GPU, and 1,784.248 token slots/s/GPU. The run used a NeMo 26.10 release-candidate base container with the pinned Megatron-Core submodule, and the process exits successfully. This mock-data result is throughput evidence, not convergence evidence.