Kimi K3

Kimi K3 is Moonshot AI’s 2.8T-parameter, 104B-active Mixture-of-Experts model. NeMo AutoModel provides a native implementation of the text model and loads its public checkpoint without trust_remote_code.

Architecture

• Hybrid attention alternates groups of three Kimi Delta Attention (KDA) layers with a Gated Multi-head Latent Attention (MLA) layer.
• Attention Residuals operate over 12-layer blocks. Pipeline stages preserve these block boundaries and carry the residual state between stages.
• Stable LatentMoE routes each token to 16 of 896 routed experts and includes two shared experts. The expert MLPs use SiTU-GLU.
• The released checkpoint uses MXFP4 weights. The example recipe dequantizes the base checkpoint while loading it for BF16 training.

Parallelism

The combined EP32 and PP8 configuration is validated on 256 GB200 GPUs.

Example Recipe

Run the Recipe

Allocate 64 GB200 nodes with four GPUs per node through your cluster launcher, then run:

$
uv run automodel --nproc-per-node=4 examples/llm_finetune/kimi/k3_hellaswag.yaml

See the Slurm Launcher Guide for multi-node setup and the Large MoE Fine-Tuning Guide for distributed MoE configuration.

References

• Kimi K3 model card
• Kimi K3 technical report
• NeMo AutoModel Kimi K3 implementation