Model CoverageLarge Language Models

Kimi Linear

View as Markdown

Kimi Linear is a hybrid-attention Mixture-of-Experts language model from Moonshot AI. Most layers use Kimi Delta Attention (KDA), a gated linear-attention variant with a recurrent state, and the remaining layers use full Multi-Head Latent Attention (MLA). NeMo AutoModel ships a native KimiLinear48BForCausalLM implementation with expert parallelism, packed sequences, and context parallelism.

TaskText Generation (hybrid linear attention, MoE)
ArchitectureKimiLinear48BForCausalLM
Parameters48B total / 3B active
HF Orgmoonshotai

Available Models

  • Kimi-Linear-48B-A3B-Instruct: 48B total, 3B activated

Architecture

  • KimiLinear48BForCausalLM (model_type: kimi_linear_48b_a3b)
  • Hybrid attention stack: KDA linear-attention layers interleaved with MLA layers
  • MoE feed-forward blocks with sigmoid routing and grouped top-k selection

Moonshot publishes this model and the Kimi K3 text backbone under the same model_type: kimi_linear and the same architectures: ["KimiLinearForCausalLM"], so neither field identifies the model on its own. NeMo AutoModel gives this implementation a distinct identity, kimi_linear_48b_a3b / KimiLinear48BForCausalLM, and leaves kimi_linear to the K3 text config. The example recipes name KimiLinear48BConfig explicitly, which is what a published Moonshot checkpoint needs; checkpoints saved by NeMo AutoModel already carry the distinct identity and load without that override.

Supported Parallelism

FeatureNotes
FSDP2Default sharding strategy for the recipes below
Expert parallelism (EP)Shards the MoE experts across ranks
Context parallelism (CP)Contiguous per-rank sequence shards. KDA passes its recurrent state from rank to rank, and MLA gathers the compressed KV latent
Packed sequencesDocument boundaries from the THD collater are honored by both layer types, including under CP

Example Hugging Face Models

ModelHF ID
Kimi Linear 48B A3B Instructmoonshotai/Kimi-Linear-48B-A3B-Instruct

Example Recipes

RecipeDescription
kimi_linear_48b_a3b_hellaswag.yamlSFT: Kimi Linear 48B A3B on HellaSwag with EP=8
kimi_linear_48b_a3b_longcontext_cp2.yamlSFT: 32k packed sequences with CP=2 and EP=8

Try with NeMo AutoModel

  1. Clone and install from source. For full instructions, refer to Installation:
$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel
$uv sync --locked --all-groups --all-extras
  1. Run the recipe from inside the repository:
$uv run automodel --nproc-per-node=8 examples/llm_finetune/kimi/kimi_linear_48b_a3b_hellaswag.yaml
  1. Pull the container and mount a checkpoint directory:
$docker run --gpus all -it --rm \
>  --shm-size=8g \
>  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
>  nvcr.io/nvidia/nemo-automodel:26.06.00
  1. Navigate to the AutoModel directory where the recipes are located:
$cd /opt/Automodel
  1. Run the recipe:
$automodel --nproc-per-node=8 examples/llm_finetune/kimi/kimi_linear_48b_a3b_hellaswag.yaml

Refer to the Installation Guide and the LLM Fine-Tuning Guide.

Fine-Tuning

Refer to the LLM Fine-Tuning Guide.

Hugging Face Model Cards