Qwen1.5-MoE-A2.7B | NVIDIA NeMo AutoModel

Qwen1.5-MoE is a Mixture-of-Experts variant from Alibaba Cloud that activates only a fraction of parameters per token, enabling efficient training and inference at scale.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Fine-Tune Qwen1.5-MoE-A2.7B

From the repository root, run:

uv run automodel --nproc-per-node=8 examples/llm_finetune/qwen/qwen1_5_moe_a2_7b_qlora.yaml

Choose a Workflow

Goal	Start Here
Quantized low-rank adaptation (QLoRA) - Qwen1.5 MoE A2.7B	Use qwen1_5_moe_a2_7b_qlora.yaml.

Model Reference

Model Architecture

Property	Value
Task	Text Generation (MoE)
Architecture	`Qwen2MoeForCausalLM`
Parameters	14.3B total / 2.7B active
Hugging Face Organization	Qwen

Available Models

Model	HF ID
Qwen1.5 MoE A2.7B	`Qwen/Qwen1.5-MoE-A2.7B`
Qwen1.5 MoE A2.7B Chat	`Qwen/Qwen1.5-MoE-A2.7B-Chat`

LLM Fine-Tuning Guide
Large MoE Fine-Tuning Guide

Fine-Tune Qwen1.5-MoE-A2.7B

Choose a Workflow

Model Reference

Model Architecture

Available Models

Related Resources