> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Kimi Linear

[Kimi Linear](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct) is a hybrid-attention Mixture-of-Experts language model from Moonshot AI. Most layers use Kimi Delta Attention (KDA), a gated linear-attention variant with a recurrent state, and the remaining layers use full Multi-Head Latent Attention (MLA). NeMo AutoModel ships a native `KimiLinear48BForCausalLM` implementation with expert parallelism, packed sequences, and context parallelism.

|                  |                                                 |
| ---------------- | ----------------------------------------------- |
| **Task**         | Text Generation (hybrid linear attention, MoE)  |
| **Architecture** | `KimiLinear48BForCausalLM`                      |
| **Parameters**   | 48B total / 3B active                           |
| **HF Org**       | [moonshotai](https://huggingface.co/moonshotai) |

## Available Models

* **Kimi-Linear-48B-A3B-Instruct**: 48B total, 3B activated

## Architecture

* `KimiLinear48BForCausalLM` (`model_type: kimi_linear_48b_a3b`)
* Hybrid attention stack: KDA linear-attention layers interleaved with MLA layers
* MoE feed-forward blocks with sigmoid routing and grouped top-k selection

Moonshot publishes this model and the Kimi K3 text backbone under the same
`model_type: kimi_linear` and the same `architectures: ["KimiLinearForCausalLM"]`, so
neither field identifies the model on its own. NeMo AutoModel gives this implementation a
distinct identity, `kimi_linear_48b_a3b` / `KimiLinear48BForCausalLM`, and leaves
`kimi_linear` to the K3 text config. The example recipes name `KimiLinear48BConfig`
explicitly, which is what a published Moonshot checkpoint needs; checkpoints saved by
NeMo AutoModel already carry the distinct identity and load without that override.

## Supported Parallelism

| Feature                  | Notes                                                                                                                           |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| FSDP2                    | Default sharding strategy for the recipes below                                                                                 |
| Expert parallelism (EP)  | Shards the MoE experts across ranks                                                                                             |
| Context parallelism (CP) | Contiguous per-rank sequence shards. KDA passes its recurrent state from rank to rank, and MLA gathers the compressed KV latent |
| Packed sequences         | Document boundaries from the THD collater are honored by both layer types, including under CP                                   |

## Example Hugging Face Models

| Model                        | HF ID                                                                                                       |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Kimi Linear 48B A3B Instruct | [`moonshotai/Kimi-Linear-48B-A3B-Instruct`](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct) |

## Example Recipes

| Recipe                                                                                                                                                                  | Description                                     |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| [kimi\_linear\_48b\_a3b\_hellaswag.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/kimi/kimi_linear_48b_a3b_hellaswag.yaml)              | SFT: Kimi Linear 48B A3B on HellaSwag with EP=8 |
| [kimi\_linear\_48b\_a3b\_longcontext\_cp2.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/kimi/kimi_linear_48b_a3b_longcontext_cp2.yaml) | SFT: 32k packed sequences with CP=2 and EP=8    |

## Try with NeMo AutoModel

1. Clone and install from source. For full instructions, refer to [Installation](/get-started/installation):

```bash
git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel
uv sync --locked --all-groups --all-extras
```

2. Run the recipe from inside the repository:

```bash
uv run automodel --nproc-per-node=8 examples/llm_finetune/kimi/kimi_linear_48b_a3b_hellaswag.yaml
```

#### Run with Docker

1. Pull the container and mount a checkpoint directory:

```bash
docker run --gpus all -it --rm \
  --shm-size=8g \
  -v $(pwd)/checkpoints:/opt/Automodel/checkpoints \
  nvcr.io/nvidia/nemo-automodel:26.06.00
```

2. Navigate to the AutoModel directory where the recipes are located:

```bash
cd /opt/Automodel
```

3. Run the recipe:

```bash
automodel --nproc-per-node=8 examples/llm_finetune/kimi/kimi_linear_48b_a3b_hellaswag.yaml
```

Refer to the [Installation Guide](/get-started/installation) and the [LLM Fine-Tuning Guide](/recipes-e2e-examples/sft-peft).

## Fine-Tuning

Refer to the [LLM Fine-Tuning Guide](/recipes-e2e-examples/sft-peft).

## Hugging Face Model Cards

* [moonshotai/Kimi-Linear-48B-A3B-Instruct](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)