Model CoverageLarge Language Models

GLM-5 / GLM-5.1 / GLM-5.2 (MoE + DSA)

View as Markdown

GLM-5, GLM-5.1, and GLM-5.2 are Zhipu AI’s open-source large Mixture-of-Experts models featuring a DeepSeek-style MLA (Multi-head Latent Attention) + DSA (Dynamic Sparse Attention) architecture. GLM-5.2 uses the same glm_moe_dsa AutoModel architecture with IndexShare DSA support, optional TileLang sparse kernels, packed-sequence training, and context-parallel long-context recipes.

TaskText Generation (MoE)
ArchitectureGlmMoeDsaForCausalLM
Parameterslarge MoE, routed sparse experts
HF Orgzai-org

Key Features

  • Mixture of Experts (MoE) with DeepSeek-style MLA and sparse DSA attention
  • IndexShare DSA for GLM-5.2: shared DSA layers reuse the previous full layer’s top-k sparse attention selection, including pipeline-parallel carry support
  • Optional TileLang kernels for GLM-5.2 DSA indexer and sparse MLA paths through backend.attn: tilelang
  • Packed-sequence and context-parallel recipes for 4K and 32K Tulu3-style SFT runs
  • Large-scale parallelism with FSDP2, Expert Parallelism, Pipeline Parallelism, and optional Context Parallelism

Available Models

  • GLM-5 (GlmMoeDsaForCausalLM)
  • GLM-5.1 (GlmMoeDsaForCausalLM): updated weights
  • GLM-5.2 (GlmMoeDsaForCausalLM): IndexShare DSA, TileLang sparse-kernel recipes, and long-context CP coverage

Example HF Models

ModelHF ID
GLM-5zai-org/GLM-5
GLM-5.1zai-org/GLM-5.1
GLM-5.2zai-org/GLM-5.2

Example Recipes

RecipeDescription
glm_5.2_tulu3_32k_tilelang_cp8.yamlSFT — GLM-5.2 on Tulu3 with packed 32K sequences, CP=8, TileLang DSA, EP=64, PP=4
glm_5.2_tulu3_4k_tilelang_100k.yamlSFT — GLM-5.2 on Tulu3 with packed 4K sequences and TileLang DSA
glm_5.2_hellaswag_pp.yamlSFT — GLM-5.2 on HellaSwag with EP=64, PP=4 on 32 nodes
glm_5_hellaswag_pp.yamlSFT — GLM-5 with EP=64, PP=4 on 32 nodes
glm_5.1_hellaswag_pp.yamlSFT — GLM-5.1 with EP=64, PP=4 on 32 nodes

Parallel Setup

The HellaSwag recipes scale training using Expert Parallelism and Pipeline Parallelism (EP=64, PP=4 across 32 nodes of 8× H100 GPUs). The long-context GLM-5.2 recipe additionally enables CP=8, packed THD batches, and the TileLang DSA path.

1distributed:
2  strategy: fsdp2
3  tp_size: 1
4  cp_size: 8
5  pp_size: 4
6  ep_size: 64
7  sequence_parallel: false
8  activation_checkpointing: true
9  reshard_after_forward: true
10  pipeline:
11    pp_schedule: interleaved1f1b
12    pp_microbatch_size: 1
13    round_virtual_stages_to_pp_multiple: down
14    scale_grads_in_schedule: false
15    patch_inner_model: false
16    patch_causal_lm_model: false
17    layers_per_stage: 2
18    pp_seq_len: 32768
19  moe:
20    reshard_after_forward: true
21    wrap_outer_model: false
22    ignore_router_for_ac: true
1model:
2  pretrained_model_name_or_path: zai-org/GLM-5.2
3  backend:
4    attn: tilelang
5    linear: torch
6    rms_norm: torch_fp32
7    rope_fusion: false
8    enable_deepep: true
9    gate_precision: float32

Try with NeMo AutoModel

1. Clone and install from source (full instructions):

$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel
$uv sync --locked --all-groups --all-extras

This is a long-context multi-node recipe that uses CP=8, EP=64, and PP=4. See the Launcher Guide for multi-node setup.

2. Run the recipe from inside the repo:

$uv run automodel --nproc-per-node=8 examples/llm_finetune/glm/glm_5.2_tulu3_32k_tilelang_cp8.yaml

1. Pull the container and mount a checkpoint directory:

$docker run --gpus all -it --rm \
>  --shm-size=8g \
>  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
>  nvcr.io/nvidia/nemo-automodel:26.06.00

2. Navigate to the AutoModel directory (where the recipes are):

$cd /opt/Automodel

3. Run the recipe:

$automodel --nproc-per-node=8 examples/llm_finetune/glm/glm_5.2_tulu3_32k_tilelang_cp8.yaml

See the Installation Guide and LLM Fine-Tuning Guide.

Fine-Tuning

See the LLM Fine-Tuning Guide and the Large MoE Fine-Tuning Guide.

Hugging Face Model Cards