Model Recipes

Kimi-K3

Serve Kimi-K3 with Dynamo and vLLM on GB200 or GB300, aggregated or disaggregated over NVLink.

View as Markdown

Each target below is a Dynamo + vLLM deployment of Moonshot AI’s Kimi-K3 — a multimodal MoE model serving up to 1M-token context — with KV-aware routing, MXFP4-packed routed experts, and FP8 KV cache, running aggregated or with prefill/decode disaggregation on GB200 and GB300 NVL72 racks. KV transfer and tensor parallelism run over NVLink (MNNVL) using Kubernetes ComputeDomains. Pick your GPU architecture and serving topology; every command on this page updates to match.

Choose your deployment target

GPU
Topology
Checkpoint moonshotai/Kimi-K3Precision MXFP4 experts + BF16 dense, FP8 KVGPUs 16x GB200 (4 nodes), one workerParallelism TP16 over MNNVLRouting KV-aware
Checkpoint moonshotai/Kimi-K3Precision MXFP4 experts + BF16 dense, FP8 KVGPUs 16x GB200 prefill + 16x GB200 decode (1P1D)Parallelism TP16 / TP16 over MNNVLRouting KV-aware, NIXL KV transfer over RDMA
Checkpoint moonshotai/Kimi-K3Precision MXFP4 experts + BF16 dense, FP8 KVGPUs 16x GB300 — two 8-GPU replicasParallelism TP8 per replica, KV-routedRouting KV-aware across replicas
Checkpoint moonshotai/Kimi-K3Precision MXFP4 experts + BF16 dense, FP8 KVGPUs 8x GB300 prefill + 16x GB300 decode (1P2D)Parallelism TP8 per replica over MNNVLRouting KV-aware, NIXL/MNNVL KV transfer

Prerequisites

  • A Kubernetes cluster with the Dynamo platform installed and GB200 GPUs available — 16x (4 nodes) for aggregated, 32x (8 nodes, 4 prefill + 4 decode) for disaggregated. See the Kubernetes Deployment Guide.
  • The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
  • A Hugging Face token with access to moonshotai/Kimi-K3.
  • A Kubernetes cluster with the Dynamo platform installed and GB300 GPUs available — 16x (4 nodes) for aggregated, 24x (6 nodes) for disaggregated. See the Kubernetes Deployment Guide.
  • The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
  • A Hugging Face token with access to moonshotai/Kimi-K3.

Create the namespace and token secret:

$export NAMESPACE=your-namespace
$kubectl create namespace ${NAMESPACE}
$kubectl create secret generic hf-token-secret \
>  --from-literal=HF_TOKEN="your-token" \
>  -n ${NAMESPACE}

If you use the model-cache PVC (required on GB300, optional on GB200), edit storageClassName in model-cache/model-cache.yaml to a ReadWriteMany storage class on your cluster (kubectl get storageclass) before applying it. Review namespace, image tags, node selectors, and resource claims in the manifests as well.

Deploy

The two SKUs source the checkpoint differently, so follow the block for your target.

The GB300 manifests read the checkpoint from a model-cache PVC (HF_HOME=/model-cache, --model moonshotai/Kimi-K3). Create the cache and download the weights:

$# Edit storageClassName in model-cache/model-cache.yaml first.
$kubectl apply -f recipes/kimi-k3/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/kimi-k3/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s

The GB200 manifests read the checkpoint from node-local storage, not the model-cache PVC: they mount the host path /mnt/stateful_partition/kube-ephemeral-ssd/models (a GKE layout) at /models and serve --model /models/model_weight with HF_HUB_OFFLINE=1. Pick one of:

  • Pre-stage the weights on every GB200 node in the deployment at <host-path>/model_weight, then edit the models hostPath in the deploy manifest to match your cluster’s path.
  • Switch the manifest to the PVC flow used by the GB300 recipes: create the cache and run the download job below, then in the deploy manifest replace the models hostPath volume with a persistentVolumeClaim (claimName: model-cache) mounted at /model-cache, add HF_HOME=/model-cache, and set MODEL_PATH to moonshotai/Kimi-K3.
$# PVC flow only. Edit storageClassName in model-cache/model-cache.yaml first.
$kubectl apply -f recipes/kimi-k3/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/kimi-k3/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s

Then deploy the target DGD:

$kubectl apply -f recipes/kimi-k3/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/kimi-k3/vllm/disagg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/kimi-k3/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/kimi-k3/vllm/disagg-gb300-agentic/deploy.yaml -n ${NAMESPACE}

First worker launch loads weights and captures CUDA graphs and can take tens of minutes.

Smoke Test

Send a test request to verify the deployment serves traffic. First forward the frontend port for your target:

$kubectl port-forward svc/kimi-k3-agg-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/kimi-k3-disagg-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/kimi-k3-agg-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/kimi-k3-disagg-frontend 8000:8000 -n ${NAMESPACE}

All targets serve under the same model name, moonshotai/Kimi-K3:

$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"moonshotai/Kimi-K3","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'

Kimi-K3 reasons before answering, and tool calling is supported — the deployment uses the kimi_k3 reasoning and tool-call parsers at the frontend.

Compare All Targets

All four targets serve Kimi-K3 on vLLM with FP8 KV cache, KV-aware routing, prefix caching, and the FlashInfer TRT-LLM MoE backend, with tensor parallelism over NVLink (MNNVL):

GB200 aggregatedGB200 disaggregatedGB300 aggregatedGB300 disaggregated
GPUs16x GB200 (4 nodes)16x prefill + 16x decode16x GB300 (2 replicas)8x prefill + 16x decode
Workers11P + 1D2 (KV-routed)1P + 2D
ParallelismTP16TP16 / TP16TP8 per replicaTP8 per replica
KV transfer—NIXL over RDMA (IB)—NIXL over MNNVL
AttentionFlashInfer MLAFlashInfer MLAFlashInfer MLAFlashInfer MLA
PrecisionMXFP4 + BF16, FP8 KVMXFP4 + BF16, FP8 KVMXFP4 + BF16, FP8 KVMXFP4 + BF16, FP8 KV
Max context1,048,5761,048,5761,048,5761,048,576

Notes

  • All targets use KV-aware routing at the Dynamo frontend with prefix caching and KV events enabled.
  • Kimi-K3 is multimodal — the deployments enable the multimodal encoder (--enable-multimodal) with data-parallel encoder sharding.
  • Multi-node tensor parallelism runs over NVLink (MNNVL) via Kubernetes ComputeDomains (DRA). Disaggregated KV transfer uses NIXL — over NVLink on GB300, over RDMA InfiniBand on GB200 (UCX_TLS=^cuda_ipc, GKE rdma-* network resources).
  • Kimi-K3 support requires the updated Dynamo frontend (tokenizer, kimi_k3 reasoning and tool-call parsers) that ships with this release.

Source