Qwen 3.5 / 3.6#

Qwen3.5 is a family of vision-language models supporting multimodal understanding across text, images, and videos. Qwen3.5-VL includes both dense models and Mixture-of-Experts (MoE) variants for improved efficiency at scale.

Qwen3.6 shares the same architecture as Qwen3.5 VL MoE (Qwen3_5MoeForConditionalGeneration) and is supported through the same bridge implementation.

Qwen 3.5/3.6 models feature a hybrid architecture combining GDN (Gated DeltaNet) layers with standard attention layers, SwiGLU activations, and RMSNorm. MoE variants use top-k routing with shared experts for better quality.

Qwen 3.5/3.6 models are supported via Megatron Bridge with auto-detected configuration and weight mapping.

Important

Use transformers >= 5.2.0 for Qwen3.5 and >= 5.8.1 for Qwen3.6.

Available Models#

Dense Models#

  • Qwen3.5 0.8B (Qwen/Qwen3.5-0.8B): 0.8B parameter vision-language model

    • Recommended: 1 node, 8 GPUs

  • Qwen3.5 2B (Qwen/Qwen3.5-2B): 2B parameter vision-language model

    • Recommended: 1 node, 8 GPUs

  • Qwen3.5 4B (Qwen/Qwen3.5-4B): 4B parameter vision-language model

    • Recommended: 1 node, 8 GPUs

  • Qwen3.5 9B (Qwen/Qwen3.5-9B): 9B parameter vision-language model

    • Recommended: 1 node, 8 GPUs

  • Qwen3.5 27B (Qwen/Qwen3.5-27B): 27B parameter vision-language model

    • Recommended: 2 nodes, 16 GPUs

Mixture-of-Experts (MoE) Models#

  • Qwen3.5 35B-A3B (Qwen/Qwen3.5-35B-A3B): 35B total parameters, 3B activated per token

    • Recommended: 2 nodes, 16 GPUs

  • Qwen3.5 122B-A10B (Qwen/Qwen3.5-122B-A10B): 122B total parameters, 10B activated per token

    • Recommended: 4 nodes, 32 GPUs

  • Qwen3.5 397B-A17B (Qwen/Qwen3.5-397B-A17B): 397B total parameters, 17B activated per token

    • 512 experts with top-10 routing and shared experts

    • Recommended: 16 nodes, 128 GPUs

Qwen3.6 (same bridge)#

  • Qwen3.6 35B-A3B (Qwen/Qwen3.6-35B-A3B): 35B total parameters, 3B activated per token

    • 256 experts with top-8 routing and shared experts

    • 40 layers: 10 groups × (3 GDN + 1 Attention)

    • Uses Qwen3_5MoeForConditionalGeneration architecture — auto-detected by AutoBridge

    • Recommended: 1 node, 8 GPUs (EP=8)

Examples#

For checkpoint conversion, inference, finetuning recipes, and step-by-step training guides, see the Qwen 3.5 Examples.

Text-only pretraining#

The Qwen3.5 9B and 35B-A3B recipes can pretrain only the language-model component of the unified models. They derive the registered language-model provider from the nested Hugging Face text_config; no vision model, projection, processor, or multimodal dataset is created.

from megatron.bridge.recipes.qwen import qwen35_text_9b_pretrain_config

config = qwen35_text_9b_pretrain_config()

The canonical aliases select eight-GPU GB200 BF16 library recipes. The dense 9B recipe, qwen35_text_9b_pretrain_8gpu_gb200_bf16_config, uses the same data parallel topology as the Llama 3 8B GB200 performance recipe, with module-scoped CUDA graphs so library correctness checks remain enabled. The MoE recipe, qwen35_text_35b_a3b_pretrain_8gpu_gb200_bf16_config, uses the applicable Qwen3.5-VL GB200 HybridEP settings with learned routing. Both retain library-recipe training, evaluation, logging, checkpointing, and correctness defaults. Set config.dataset.blend (or config.dataset.data_path) to use a prepared Megatron indexed text dataset.

Hugging Face Model Cards#

  • Qwen3.5 0.8B: https://huggingface.co/Qwen/Qwen3.5-0.8B

  • Qwen3.5 2B: https://huggingface.co/Qwen/Qwen3.5-2B

  • Qwen3.5 4B: https://huggingface.co/Qwen/Qwen3.5-4B

  • Qwen3.5 9B: https://huggingface.co/Qwen/Qwen3.5-9B

  • Qwen3.5 27B: https://huggingface.co/Qwen/Qwen3.5-27B

  • Qwen3.5 35B-A3B (MoE): https://huggingface.co/Qwen/Qwen3.5-35B-A3B

  • Qwen3.5 122B-A10B (MoE): https://huggingface.co/Qwen/Qwen3.5-122B-A10B

  • Qwen3.5 397B-A17B (MoE): https://huggingface.co/Qwen/Qwen3.5-397B-A17B

  • Qwen3.6 35B-A3B (MoE): https://huggingface.co/Qwen/Qwen3.6-35B-A3B