> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v1.6.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt
> - dev: https://docs.nvidia.com/dynamo/dev/llms.txt
> - v1.6.1: https://docs.nvidia.com/dynamo/v1.6.1/llms.txt
> - v1.6.0: https://docs.nvidia.com/dynamo/v1.6.0/llms.txt
> - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt
> - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt
> - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt
> - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt
> - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt
> - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt
> - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt
> - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt
> - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt
> - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt
> - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt
> - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt
> - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt
> - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt
> - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt
> - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt
> - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt
> - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt
> - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Kimi-K3.1-Pro

Serve `moonshotai/Kimi-K3.1-Pro` with NVIDIA Dynamo using vLLM or SGLang. Pick your framework, GPU, and serving topology; deployment commands update to match. SGLang targets are experimental, text-only, and functional-only; no performance claims are made.

Choose your deployment target

Framework

vLLM

<input type="radio" id="recipe-framework-sglang" name="recipe-framework" value="sglang" />

SGLang

GPU

<input type="radio" id="recipe-sku-b200" name="recipe-sku" value="b200" />

B200

<input type="radio" id="recipe-sku-gb300" name="recipe-sku" value="gb300" />

GB300

VR200

Topology

Aggregated

<input type="radio" id="recipe-variant-disagg" name="recipe-variant" value="disagg" />

Disaggregated

**GPUs** 8x GB300 (1 worker) **Parallelism** TP8 across 2 nodes per worker **Speculative decoding** Disabled

**GPUs** 24x GB300 (2P1D) **Parallelism** TP8 across 2 nodes per worker **Speculative decoding** Disabled

**GPUs** 8x GB300 (1 worker) **Parallelism** TP8 across 2 nodes per worker **Speculative decoding** Disabled

**GPUs** 16x GB300 (1P1D) **Parallelism** TP8 across 2 nodes per worker **Speculative decoding** Disabled

**GPUs** 16x B200 (1 worker) **Parallelism** TP8 × PP2 across 2 nodes **Speculative decoding** DSpark (7 draft tokens)

**GPUs** 16x VR200 (2 workers) **Parallelism** TP8 across 2 nodes per worker **Speculative decoding** DSpark (7 draft tokens)

**GPUs** 24x VR200 (2P1D) **Parallelism** TP8 across 2 nodes per worker **Speculative decoding** DSpark (7 draft tokens)

> **Note**
>
> No B200 disaggregated recipe is published. Select aggregated serving or another GPU.

> **Note**
>
> SGLang recipes on this page cover GB300 only. Select GB300 or switch to vLLM.

## Prerequisites

* A Kubernetes cluster with the Dynamo platform installed and 8 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

- A Kubernetes cluster with the Dynamo platform installed and 24 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

* A Kubernetes cluster with the Dynamo platform installed and 8 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

- A Kubernetes cluster with the Dynamo platform installed and 16 GB300 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

* A Kubernetes cluster with the Dynamo platform installed and 16 B200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

- A Kubernetes cluster with the Dynamo platform installed and 16 VR200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

* A Kubernetes cluster with the Dynamo platform installed and 24 VR200 GPUs available. See the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart).

- A Hugging Face token with access to `moonshotai/Kimi-K3.1-Pro`.
- Access to the runtime images referenced by the selected manifest.

* The NVIDIA DRA driver with ComputeDomain support for multi-node NVLink serving. The hardware overlays include a ComputeDomain and worker channel claims.

Create the namespace and token secret:

```bash
export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}
```

> **Warning**
>
> Review storage paths, node placement, registry credentials, and resource claims before applying the selected manifest.

## Deploy

Create the shared model cache and download the main checkpoint:

> **Warning**
>
> Edit `storageClassName` in `recipes/kimi-k3.1-pro/model-cache/model-cache.yaml` to a ReadWriteMany class on your cluster before you apply it.

```bash
kubectl apply -f recipes/kimi-k3.1-pro/model-cache/model-cache.yaml -n ${NAMESPACE}
kubectl apply -f recipes/kimi-k3.1-pro/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s
```

The weights are stored at `/shared-model-cache/moonshotai/Kimi-K3.1-Pro` on the PVC. The download Job downloads only the main model.

For B200, also stage the `Inferact/Kimi-K3-DSpark` draft checkpoint at the path in the worker's `SPECULATIVE_CONFIG` on the same PVC, or update that setting to your staged checkpoint path.

For VR200, also stage the `Inferact/Kimi-K3-DSpark` draft checkpoint at `/shared-model-cache/Inferact/Kimi-K3-DSpark` on the same PVC. Both topologies mount `shared-model-cache` for the frontend and all workers; no node-local weight staging is required.

Deploy the selected DynamoGraphDeployment (DGD):

```bash
export DGD=kimi-k3-1-pro-vllm-aggregate-gb300
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}
```

For image and video input, apply the multimodal overlay instead:

```bash
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300-multimodal.yaml -n ${NAMESPACE}
```

```bash
export DGD=kimi-k3-1-pro-vllm-disagg-gb300
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}
```

For image and video input, apply the multimodal overlay instead:

```bash
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300-multimodal.yaml -n ${NAMESPACE}
```

```bash
export DGD=kimi-k3-1-pro-sglang-aggregate-gb300
kubectl apply -f recipes/kimi-k3.1-pro/sglang/agg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}
```

```bash
export DGD=kimi-k3-1-pro-sglang-disagg-gb300
kubectl apply -f recipes/kimi-k3.1-pro/sglang/disagg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}
```

```bash
export DGD=kimi-k3-1-pro-vllm-aggregate-b200
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200.yaml -n ${NAMESPACE}
```

```bash
export DGD=kimi-k3-1-pro-vllm-aggregate-vr200
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}
```

```bash
export DGD=kimi-k3-1-pro-vllm-disagg-vr200
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}
```

Each target also has a `generic` Kustomize overlay for customization. It inherits the base model paths and hardware requirements. B200 and VR200 targets enable DSpark; separate `*-synthetic-al` overlays change draft acceptance for benchmarking.

## Smoke Test

The deploy command sets `DGD` to the selected resource name. Wait for readiness and forward the frontend port:

```bash
kubectl wait --for=condition=Ready dynamographdeployment/${DGD} \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}
```

Send a chat completion with the served model name `moonshotai/Kimi-K3.1-Pro`:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"moonshotai/Kimi-K3.1-Pro","messages":[{"role":"user","content":"Reply with exactly: READY"}],"temperature":0,"max_tokens":256}'
```

## Benchmark

Use [perf.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/perf/perf.yaml) to run AIPerf 0.13.0 against the deployment. It runs `aiperf profile` with the chat endpoint type and streaming on the AgentX agentic trace (`semianalysis_cc_traces_weka_062126`), with an agentic warmup and a 1200 s profiling phase. The tokenizer is read from the model path on the `shared-model-cache` PVC.

The performance results use text-only AgentX traffic. Apply the selected benchmark variant before running AIPerf; normal serving commands above use natural acceptance.

```bash
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200-synthetic-al.yaml -n ${NAMESPACE}
```

```bash
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200-synthetic-al.yaml -n ${NAMESPACE}
```

```bash
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200-synthetic-al.yaml -n ${NAMESPACE}
```

Set these values in the Job `env` section for your target:

* `ENDPOINT`: the frontend Service of the DynamoGraphDeployment, `${DGD}-frontend:8000` (substitute the selected deployment name).
* `CONCURRENCY`: GB300 uses 32 for vLLM aggregated and 48 for disaggregated. Reported VR200 points use 180 for aggregated and 174 for disaggregated; B200 uses 48.

The benchmark Job uses the same `shared-model-cache` PVC and tokenizer files as the deployment.

```bash
kubectl apply -f recipes/kimi-k3.1-pro/perf/perf.yaml -n ${NAMESPACE}
kubectl logs -f job/kimi-k3-1-aiperf -n ${NAMESPACE}
```

Results are written to `/shared-model-cache/perf/<epoch>_<job-name>/`. Restart the workers between independent runs to reset KV-cache and routing state.

### Optimization Targets

The AgentX benchmark targets use the following interactivity goals:

| Workload | User output tok/s (P50) | TTFT P50 |
| -------- | ----------------------: | -------: |
| AgentX   |                    ≥ 50 |   \< 2 s |

### Performance Results

All results below measure text-only traffic; the selected target is highlighted. B200 and VR200 use synthetic draft acceptance (AL 3.84); GB300 has speculative decoding disabled.

<table>
  <thead>
    <tr><th>Workload</th><th>Recipe</th><th>Framework</th><th>SKU</th><th>GPUs</th><th>Concurrency</th><th>System Output Tokens/s</th><th>System Output Tokens/s/GPU</th><th>User Output Tokens/s (P50)</th><th>TTFT P50 (ms)</th></tr>
  </thead>

  <tbody>
    <tr data-highlight-target="vllm-vr200-agg">
      <td>AgentX</td>

      <td>aggregated, 2 replicas</td>

      <td>vLLM</td>

      <td>VR200</td>

      <td>16</td>

      <td>180</td>

      <td>2,448.08</td>

      <td>153.01</td>

      <td>57.71</td>

      <td>1,534</td>
    </tr>

    <tr data-highlight-target="vllm-vr200-disagg">
      <td>AgentX</td>

      <td>disaggregated 2P1D</td>

      <td>vLLM</td>

      <td>VR200</td>

      <td>24</td>

      <td>174</td>

      <td>2,932.63</td>

      <td>122.19</td>

      <td>73.93</td>

      <td>1,761</td>
    </tr>

    <tr data-highlight-target="vllm-b200-agg">
      <td>AgentX</td>

      <td>aggregated</td>

      <td>vLLM</td>

      <td>B200</td>

      <td>16</td>

      <td>48</td>

      <td>422.50</td>

      <td>≈26.41</td>

      <td>64.02</td>

      <td>550.45</td>
    </tr>

    <tr data-highlight-target="vllm-gb300-agg">
      <td>AgentX</td>

      <td>aggregated TP8</td>

      <td>vLLM</td>

      <td>GB300</td>

      <td>8 (2 nodes)</td>

      <td>32</td>

      <td>≈319.12</td>

      <td>39.89</td>

      <td>76.7</td>

      <td>428</td>
    </tr>

    <tr data-highlight-target="vllm-gb300-disagg">
      <td>AgentX</td>

      <td>disaggregated 2P1D</td>

      <td>vLLM</td>

      <td>GB300</td>

      <td>24</td>

      <td>48</td>

      <td>≈553.92</td>

      <td>23.08</td>

      <td>87.1</td>

      <td>656</td>
    </tr>

    <tr data-highlight-target="vllm-gb300-agg">
      <td>AgentX</td>

      <td>aggregated multimodal overlay (pre-release)</td>

      <td>vLLM</td>

      <td>GB300</td>

      <td>8 (2 nodes)</td>

      <td>32</td>

      <td>≈316.48</td>

      <td>39.56</td>

      <td>70.9</td>

      <td>440</td>
    </tr>

    <tr data-highlight-target="vllm-gb300-disagg">
      <td>AgentX</td>

      <td>disaggregated multimodal overlay (pre-release)</td>

      <td>vLLM</td>

      <td>GB300</td>

      <td>24</td>

      <td>48</td>

      <td>≈561.12</td>

      <td>23.38</td>

      <td>85.7</td>

      <td>693</td>
    </tr>
  </tbody>
</table>

## Compare All Targets

GPU counts below describe the checked-in manifests. P counts prefill workers; D counts decode workers.

<table>
  <thead>
    <tr><th>Framework</th><th>GPU</th><th>Topology</th><th>Total GPUs</th><th>Parallelism</th><th>Speculative Decoding</th><th>Storage</th></tr>
  </thead>

  <tbody>
    <tr data-highlight-target="vllm-gb300-agg">
      <td>vLLM</td>

      <td>GB300</td>

      <td>1 worker</td>

      <td>8</td>

      <td>TP8 across 2 nodes per worker</td>

      <td>Disabled</td>

      <td>Shared PVC</td>
    </tr>

    <tr data-highlight-target="vllm-gb300-disagg">
      <td>vLLM</td>

      <td>GB300</td>

      <td>2P1D</td>

      <td>24</td>

      <td>TP8 across 2 nodes per worker</td>

      <td>Disabled</td>

      <td>Shared PVC</td>
    </tr>

    <tr data-highlight-target="sglang-gb300-agg">
      <td>SGLang</td>

      <td>GB300</td>

      <td>1 worker</td>

      <td>8</td>

      <td>TP8 across 2 nodes per worker</td>

      <td>Disabled</td>

      <td>Shared PVC</td>
    </tr>

    <tr data-highlight-target="sglang-gb300-disagg">
      <td>SGLang</td>

      <td>GB300</td>

      <td>1P1D</td>

      <td>16</td>

      <td>TP8 across 2 nodes per worker</td>

      <td>Disabled</td>

      <td>Shared PVC</td>
    </tr>

    <tr data-highlight-target="vllm-b200-agg">
      <td>vLLM</td>

      <td>B200</td>

      <td>1 worker</td>

      <td>16</td>

      <td>TP8 × PP2 across 2 nodes</td>

      <td>DSpark (7 draft tokens)</td>

      <td>Shared PVC</td>
    </tr>

    <tr data-highlight-target="vllm-vr200-agg">
      <td>vLLM</td>

      <td>VR200</td>

      <td>2 workers</td>

      <td>16</td>

      <td>TP8 across 2 nodes per worker</td>

      <td>DSpark (7 draft tokens)</td>

      <td>Shared PVC</td>
    </tr>

    <tr data-highlight-target="vllm-vr200-disagg">
      <td>vLLM</td>

      <td>VR200</td>

      <td>2P1D</td>

      <td>24</td>

      <td>TP8 across 2 nodes per worker</td>

      <td>DSpark (7 draft tokens)</td>

      <td>Shared PVC</td>
    </tr>
  </tbody>
</table>

## Notes

* SGLang targets are experimental, text-only, and functional-only. No SGLang performance claims are made. Use the vLLM multimodal overlays for image and video input.
  The SGLang frontends run the Dynamo chat processor (`--dyn-chat-processor dynamo`, `DYN_SGL_FRONTEND_DECODING=true`). With the SGLang chat processor, strict `json_schema` output with thinking enabled was returned in `reasoning_content`; if you switch processors, disable thinking for `json_schema` requests (`"chat_template_kwargs": {"thinking": false}`).
* Synthetic draft acceptance is for benchmarking. Use the natural-acceptance variants for normal serving.
* Disaggregated performance depends on your fabric; RDMA tuning is cluster-specific.

## Source

* Deployment assets: [Kimi-K3.1-Pro recipes](https://github.com/ai-dynamo/dynamo/tree/main/recipes/kimi-k3.1-pro)

* Setup assets: [model-cache.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/model-cache/model-cache.yaml) and [model-download.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/model-cache/model-download.yaml)

* Benchmark manifest: [perf.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/perf/perf.yaml)

* B200 aggregated: [deployment manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200.yaml)

* VR200 aggregated: [deployment manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200.yaml)

* VR200 disaggregated: [deployment manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yaml)