Kimi-K3.1-Pro

Serve moonshotai/Kimi-K3.1-Pro with NVIDIA Dynamo using vLLM or SGLang. Pick your framework, GPU, and serving topology; deployment commands update to match. SGLang targets are experimental, text-only, and functional-only; no performance claims are made.

Prerequisites

• A Hugging Face token with access to moonshotai/Kimi-K3.1-Pro.
• Access to the runtime images referenced by the selected manifest.

Create the namespace and token secret:

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}

Deploy

kubectl apply -f recipes/kimi-k3.1-pro/model-cache/model-cache.yaml -n ${NAMESPACE}
kubectl apply -f recipes/kimi-k3.1-pro/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s

Deploy the selected DynamoGraphDeployment (DGD):

export DGD=kimi-k3-1-pro-vllm-aggregate-gb300
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-gb300-agentic/deploy-gb300-multimodal.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-disagg-gb300
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-gb300-agentic/deploy-gb300-multimodal.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-sglang-aggregate-gb300
kubectl apply -f recipes/kimi-k3.1-pro/sglang/agg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-sglang-disagg-gb300
kubectl apply -f recipes/kimi-k3.1-pro/sglang/disagg-gb300-agentic/deploy-gb300.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-aggregate-b200
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-aggregate-vr200
kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}

export DGD=kimi-k3-1-pro-vllm-disagg-vr200
kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200.yaml -n ${NAMESPACE}

Each target also has a generic Kustomize overlay for customization. It inherits the base model paths and hardware requirements. B200 and VR200 targets enable DSpark; separate *-synthetic-al overlays change draft acceptance for benchmarking.

Smoke Test

The deploy command sets DGD to the selected resource name. Wait for readiness and forward the frontend port:

kubectl wait --for=condition=Ready dynamographdeployment/${DGD} \
  -n ${NAMESPACE} --timeout=7200s
kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}

Send a chat completion with the served model name moonshotai/Kimi-K3.1-Pro:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"moonshotai/Kimi-K3.1-Pro","messages":[{"role":"user","content":"Reply with exactly: READY"}],"temperature":0,"max_tokens":256}'

Benchmark

Use perf.yaml to run AIPerf 0.13.0 against the deployment. It runs aiperf profile with the chat endpoint type and streaming on the AgentX agentic trace (semianalysis_cc_traces_weka_062126), with an agentic warmup and a 1200 s profiling phase. The tokenizer is read from the model path on the shared-model-cache PVC.

The performance results use text-only AgentX traffic. Apply the selected benchmark variant before running AIPerf; normal serving commands above use natural acceptance.

kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-b200-agentx/deploy-b200-synthetic-al.yaml -n ${NAMESPACE}

kubectl apply -f recipes/kimi-k3.1-pro/vllm/agg-vr200-agentx/deploy-vr200-synthetic-al.yaml -n ${NAMESPACE}

kubectl apply -f recipes/kimi-k3.1-pro/vllm/disagg-vr200-agentx/deploy-vr200-synthetic-al.yaml -n ${NAMESPACE}

Set these values in the Job env section for your target:

• ENDPOINT: the frontend Service of the DynamoGraphDeployment, ${DGD}-frontend:8000 (substitute the selected deployment name).
• CONCURRENCY: GB300 uses 32 for vLLM aggregated and 48 for disaggregated. Reported VR200 points use 180 for aggregated and 174 for disaggregated; B200 uses 48.

The benchmark Job uses the same shared-model-cache PVC and tokenizer files as the deployment.

kubectl apply -f recipes/kimi-k3.1-pro/perf/perf.yaml -n ${NAMESPACE}
kubectl logs -f job/kimi-k3-1-aiperf -n ${NAMESPACE}

Results are written to /shared-model-cache/perf/<epoch>_<job-name>/. Restart the workers between independent runs to reset KV-cache and routing state.

Optimization Targets

The AgentX benchmark targets use the following interactivity goals:

Performance Results

All results below measure text-only traffic; the selected target is highlighted. B200 and VR200 use synthetic draft acceptance (AL 3.84); GB300 has speculative decoding disabled.

Compare All Targets

GPU counts below describe the checked-in manifests. P counts prefill workers; D counts decode workers.

Notes

• SGLang targets are experimental, text-only, and functional-only. No SGLang performance claims are made. Use the vLLM multimodal overlays for image and video input. The SGLang frontends run the Dynamo chat processor (--dyn-chat-processor dynamo, DYN_SGL_FRONTEND_DECODING=true). With the SGLang chat processor, strict json_schema output with thinking enabled was returned in reasoning_content; if you switch processors, disable thinking for json_schema requests ("chat_template_kwargs": {"thinking": false}).
• Synthetic draft acceptance is for benchmarking. Use the natural-acceptance variants for normal serving.
• Disaggregated performance depends on your fabric; RDMA tuning is cluster-specific.

Source

• Deployment assets: Kimi-K3.1-Pro recipes

• Setup assets: model-cache.yaml and model-download.yaml

• Benchmark manifest: perf.yaml

• B200 aggregated: deployment manifest

• VR200 aggregated: deployment manifest

• VR200 disaggregated: deployment manifest