GLM-5 / GLM-5.1 / GLM-5.2 / GLM-5.3 (MoE + DSA)

GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3 are Z.ai’s open-weight Mixture-of-Experts language models with Multi-head Latent Attention (MLA) and Dynamic Sparse Attention (DSA). GLM-5.3 uses the same base model and GlmMoeDsaForCausalLM architecture as GLM-5.2, with its gains coming from post-training.

Key Features

GLM-5 family models in NeMo AutoModel support:

• Mixture-of-Experts (MoE) with 256 routed experts, top-8 routing, and one shared expert in the GLM-5.2/5.3 configuration. The first three layers use dense feed-forward networks.
• IndexShare DSA for GLM-5.2 and GLM-5.3. Shared DSA layers reuse the previous full layer’s top-k sparse-attention selection.
• Optional cuDNN DSA and FlashMLA sparse attention on SM90 or later through backend.attn: cudnn.
• Optional TileLang kernels for the DSA indexer and sparse MLA path through backend.attn: tilelang.
• Packed-sequence training and distributed execution with FSDP2, expert parallelism, HybridEP dispatch, and optional context parallelism.

Available Models

• GLM-5 (GlmMoeDsaForCausalLM)
• GLM-5.1 (GlmMoeDsaForCausalLM): Updated weights
• GLM-5.2 (GlmMoeDsaForCausalLM): IndexShare DSA with cuDNN and TileLang backends
• GLM-5.3 (GlmMoeDsaForCausalLM): Same base architecture as GLM-5.2; updated post-training

Example HF Models

Example Recipes

Validated GLM-5.3 Configuration

The relevant model and expert-distribution settings are:

distributed:
  strategy: fsdp2
  ep_size: 64
  activation_checkpointing: true
  moe:
    reshard_after_forward: true
    wrap_outer_model: false
    ignore_router_for_ac: true
model:
  pretrained_model_name_or_path: zai-org/GLM-5.3
  backend:
    attn: cudnn
    dispatcher: hybridep
    gate_precision: float32

Install and Run

Clone and install NeMo AutoModel from source:

git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel
uv sync --locked --all-groups --all-extras

The cuDNN backend also requires FlashMLA’s flash_mla_sparse_fwd, which is not published as a complete source distribution on PyPI. Install the tested revision with its submodules:

git clone --recursive https://github.com/deepseek-ai/FlashMLA.git /tmp/FlashMLA
git -C /tmp/FlashMLA checkout b7643bd54521f563b839b98289b5cd048c062ba2
git -C /tmp/FlashMLA submodule update --init --recursive
uv pip install --no-build-isolation /tmp/FlashMLA

The published GLM-5.3 recipe requires 32 nodes with 8 GPUs per node. Launch it through the cluster launcher from inside the repository:

uv run automodel examples/llm_finetune/glm/glm_5.3_tulu3_4k_cudnn_100step.yaml \\
  --nproc-per-node=8 \\
  --wandb.enable=true

Numerical Validation

Hugging Face Logit Parity

The first four GLM-5.3 layers were compared with the Hugging Face reference. Representative loaded weights matched their source tensors exactly.

100-Step Training

The full 32-node / 256-GPU configuration completed all 100 optimizer steps. Throughput statistics exclude the first 10 warmup steps.

The run completed without non-finite metrics or critical rank errors.

Current Scope

• The published GLM-5.3 path covers full-parameter SFT with cuDNN DSA, FlashMLA, HybridEP, and EP64 at 4K sequence length.
• GLM-5.3 context-parallel and TileLang configurations were not validated by this recipe. Use the GLM-5.2 recipes above for the established CP8 and TileLang paths.
• The cuDNN sparse-attention path requires SM90 or later, cuDNN Frontend, and FlashMLA.

See the LLM Fine-Tuning Guide and the Large MoE Fine-Tuning Guide for dataset and training configuration details.

References

• zai-org/GLM-5
• zai-org/GLM-5.1
• zai-org/GLM-5.2
• zai-org/GLM-5.3
• GLM-5 technical report
• FlashMLA