GLM-5 / GLM-5.1 / GLM-5.2 (MoE + DSA)

GLM-5, GLM-5.1, and GLM-5.2 are Zhipu AI’s open-source large Mixture-of-Experts models featuring a DeepSeek-style MLA (Multi-head Latent Attention) + DSA (Dynamic Sparse Attention) architecture. GLM-5.2 uses the same glm_moe_dsa AutoModel architecture with IndexShare DSA support, optional TileLang or cuDNN sparse kernels, packed-sequence training, and context-parallel long-context recipes.

Key Features

GLM-5 family models in NeMo AutoModel include the following capabilities:

• Mixture-of-Experts (MoE): DeepSeek-style MLA and sparse DSA attention
• IndexShare DSA for GLM-5.2: Shared DSA layers reuse the previous full layer’s top-k sparse attention selection, including pipeline-parallel carry support
• Optional TileLang kernels: GLM-5.2 DSA indexer and sparse MLA paths through backend.attn: tilelang
• Optional cuDNN DSA indexer and FlashMLA sparse attention: Available on SM90+ GPUs through backend.attn: cudnn
• Packed-sequence and context-parallel recipes: Support for 4K and 32K Tulu3-style SFT runs
• Large-scale parallelism: FSDP2, Expert Parallelism, Pipeline Parallelism, and optional Context Parallelism

Available Models

• GLM-5 (GlmMoeDsaForCausalLM)
• GLM-5.1 (GlmMoeDsaForCausalLM): Updated weights
• GLM-5.2 (GlmMoeDsaForCausalLM): IndexShare DSA, TileLang and cuDNN sparse kernels, and long-context CP coverage

Example HF Models

Example Recipes

Parallel Setup

The HellaSwag recipes scale training using Expert Parallelism and Pipeline Parallelism (EP=64, PP=4 across 32 nodes of 8× H100 GPUs). The long-context GLM-5.2 recipe also enables CP=8, packed THD batches, and the TileLang DSA path.

1
distributed:
2
  strategy: fsdp2
3
  tp_size: 1
4
  cp_size: 8
5
  pp_size: 4
6
  ep_size: 64
7
  sequence_parallel: false
8
  activation_checkpointing: true
9
  reshard_after_forward: true
10
  pipeline:
11
    pp_schedule: interleaved1f1b
12
    pp_microbatch_size: 1
13
    round_virtual_stages_to_pp_multiple: down
14
    scale_grads_in_schedule: false
15
    patch_inner_model: false
16
    patch_causal_lm_model: false
17
    layers_per_stage: 2
18
    pp_seq_len: 32768
19
  moe:
20
    reshard_after_forward: true
21
    wrap_outer_model: false
22
    ignore_router_for_ac: true

1
model:
2
  pretrained_model_name_or_path: zai-org/GLM-5.2
3
  backend:
4
    attn: tilelang  # Use cudnn for the cuDNN DSA + FlashMLA path.
5
    linear: torch
6
    rms_norm: torch_fp32
7
    rope_fusion: false
8
    dispatcher: hybridep
9
    gate_precision: float32

Try with NeMo AutoModel

1. Clone and install from source (full instructions):

$
git clone https://github.com/NVIDIA-NeMo/Automodel.git
$
cd Automodel
$
uv sync --locked --all-groups --all-extras

The cuDNN backend also requires FlashMLA’s flash_mla_sparse_fwd, which is not published as a complete source distribution on PyPI. Install the tested revision with its submodules before selecting backend.attn: cudnn:

$
git clone --recursive https://github.com/deepseek-ai/FlashMLA.git /tmp/FlashMLA
$
git -C /tmp/FlashMLA checkout b7643bd54521f563b839b98289b5cd048c062ba2
$
git -C /tmp/FlashMLA submodule update --init --recursive
$
uv pip install --no-build-isolation /tmp/FlashMLA

2. Run the recipe from inside the repo:

$
uv run automodel --nproc-per-node=8 examples/llm_finetune/glm/glm_5.2_tulu3_32k_tilelang_cp8.yaml

$
docker run --gpus all -it --rm \
>
  --shm-size=8g \
>
  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
>
  nvcr.io/nvidia/nemo-automodel:26.06.00

$
cd /opt/Automodel

$
automodel --nproc-per-node=8 examples/llm_finetune/glm/glm_5.2_tulu3_32k_tilelang_cp8.yaml

See the Installation Guide and LLM Fine-Tuning Guide.

Fine-Tuning

See the LLM Fine-Tuning Guide and the Large MoE Fine-Tuning Guide.

Hugging Face Model Cards

• zai-org/GLM-5
• zai-org/GLM-5.1
• zai-org/GLM-5.2