Qwen2#

Qwen2 is Alibaba Cloud’s second-generation large language model series. It features grouped query attention, YARN-based long-context extension, and dual chunk attention for long sequences. QwQ-32B-Preview, a reasoning-focused model, also uses this architecture.

Task

Text Generation

Architecture

Qwen2ForCausalLM

Parameters

0.5B – 72B

HF Org

Qwen

Available Models#

  • Qwen2.5: 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B

  • Qwen2: 0.5B, 1.5B, 7B, 57B-A14B (MoE), 72B

  • QwQ-32B-Preview — reasoning model

Architecture#

  • Qwen2ForCausalLM

Example HF Models#

Model

HF ID

Qwen2.5 7B Instruct

Qwen/Qwen2.5-7B-Instruct

Qwen2.5 72B Instruct

Qwen/Qwen2.5-72B-Instruct

Qwen2 7B Instruct

Qwen/Qwen2-7B-Instruct

QwQ 32B Preview

Qwen/QwQ-32B-Preview

Example Recipes#

Recipe

Description

qwen2_5_7b_squad.yaml

SFT — Qwen2.5 7B on SQuAD

qwq_32b_squad_peft.yaml

LoRA — QwQ 32B on SQuAD

Try with NeMo AutoModel#

1. Install (full instructions):

pip install nemo-automodel

2. Clone the repo to get the example recipes:

git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel

3. Run the recipe from inside the repo:

automodel --nproc-per-node=8 examples/llm_finetune/qwen/qwen2_5_7b_squad.yaml
Run with Docker

1. Pull the container and mount a checkpoint directory:

docker run --gpus all -it --rm \
  --shm-size=8g \
  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
  nvcr.io/nvidia/nemo-automodel:26.02.00

2. Navigate to the AutoModel directory (where the recipes are):

cd /opt/Automodel

3. Run the recipe:

automodel --nproc-per-node=8 examples/llm_finetune/qwen/qwen2_5_7b_squad.yaml

See the Installation Guide and LLM Fine-Tuning Guide.

Fine-Tuning#

See the LLM Fine-Tuning Guide for full SFT and LoRA instructions.

Hugging Face Model Cards#