GLM-5.3-Flash
GLM-5.3-Flash
GLM-5.3-Flash is a mixture-of-experts vision-language model with a hybrid Kimi Delta Attention and Dynamic Sparse Attention language backbone.
Supported Training Path
NeMo AutoModel provides a native configuration, image processor, vision tower, language model, and Hugging Face state-dict adapter for GLM-5.3-Flash. The supported full-model initialization path loads the base checkpoint through distributed checkpointing; single-GPU full-checkpoint loading is not supported.
The current onboarding supports image training. Video inputs, tensor parallelism, and pipeline parallelism are not enabled for this model. The validated recipe uses FSDP2 with expert parallelism, contiguous packed context parallelism, and HybridEP dispatch.
Attention Backends
- KDA layers use Flash Linear Attention kernels.
- Sparse DSA layers support the SDPA reference path.
- On SM90 or later,
backend.attn: cudnnuses FlashMLA forward with cuDNN sparse-attention backward. This optional path requires compatible FlashMLA and cuDNN Frontend installations.
Example Recipe
The recipe is sized for 9 nodes with 8 GPUs per node. It uses local batch size 1 and four gradient-accumulation microsteps to form a global batch of 144 packed samples.