Model Recipes

DeepSeek-V4-Pro-0813

Serve DeepSeek-V4-Pro-0813 with Dynamo and vLLM on GB200 or H200, aggregated or disaggregated, with 1M context.

View as Markdown

Each target below is a Dynamo + vLLM deployment of DeepSeek-V4-Pro-0813 — a 1.6T-parameter MoE model with MXFP4-packed routed experts and hybrid sparse attention — with KV-aware routing, FP8 KV cache and prefix caching, running aggregated or with prefill/decode disaggregation. Every target serves the model’s full 1,048,576-token context without CPU KV offload. Pick your GPU architecture and serving topology; every command on this page updates to match.

Choose your deployment target

GPU
Topology
Checkpoint deepseek-ai/DeepSeek-V4-Pro-0813Precision MXFP4 experts, FP8 KVGPUs 8x GB200 (2 nodes)Context 1,048,576 tokensRouting KV-aware
Checkpoint deepseek-ai/DeepSeek-V4-Pro-0813Precision MXFP4 experts, FP8 KVGPUs 8x GB200 prefill + 8x GB200 decode (1P1D)Context 1,048,576 tokensRouting KV-aware, NIXL KV transfer
Checkpoint deepseek-ai/DeepSeek-V4-Pro-0813Precision MXFP4 experts, FP8 KVGPUs 8x H200, one workerContext 1,048,576 tokensRouting KV-aware
Checkpoint deepseek-ai/DeepSeek-V4-Pro-0813Precision MXFP4 experts, FP8 KVGPUs 8x H200 prefill + 8x H200 decode (1P1D)Context 1,048,576 tokensRouting KV-aware, NIXL KV transfer

Prerequisites

  • A Dynamo Platform install on your Kubernetes cluster — see the Kubernetes quickstart.
  • A Hugging Face token with access to deepseek-ai/DeepSeek-V4-Pro-0813.
  • A ReadWriteMany storage class. The checkpoint is ~832 GiB; the PVC in this recipe requests 1200 Gi.

Eight H200 GPUs on a single node.

Sixteen H200 GPUs across two nodes — one prefill worker and one decode worker, eight GPUs each.

Eight GB200 GPUs across two nodes, connected over NVLink (MNNVL) using a Kubernetes ComputeDomain.

Sixteen GB200 GPUs — eight for prefill and eight for decode, each spanning two nodes over NVLink (MNNVL) using Kubernetes ComputeDomains.

Deploy

Create the namespace and Hugging Face secret:

$export NAMESPACE=your-namespace
$kubectl create namespace ${NAMESPACE}
$kubectl create secret generic hf-token-secret \
>  --from-literal=HF_TOKEN="your-token" \
>  -n ${NAMESPACE}

Create the model cache and download the checkpoint. Edit storageClassName in model-cache.yaml to a ReadWriteMany class available on your cluster:

$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro-0813/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro-0813/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s

Deploy the graph:

$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro-0813/vllm/agg-h200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Ready dgd/dsv4-pro-0813-agg-h200-agentic -n ${NAMESPACE} --timeout=3600s
$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro-0813/vllm/disagg-h200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Ready dgd/dsv4-pro-0813-disagg-h200-agentic -n ${NAMESPACE} --timeout=3600s
$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro-0813/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Ready dgd/dsv4-pro-0813-agg-gb200-agentic -n ${NAMESPACE} --timeout=21600s
$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro-0813/vllm/disagg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Ready dgd/dsv4-pro-0813-disagg-gb200-agentic -n ${NAMESPACE} --timeout=21600s

Loading the 832 GiB checkpoint takes roughly 15–20 minutes on first start on H200. GB200 is slower — allow several hours, which is why the GB200 manifests set VLLM_ENGINE_READY_TIMEOUT_S: 21600 and the waits above use a six-hour timeout.

Smoke Test

Port-forward the frontend and send one request:

$kubectl port-forward svc/dsv4-pro-0813-agg-h200-agentic-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/dsv4-pro-0813-disagg-h200-agentic-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/dsv4-pro-0813-agg-gb200-agentic-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/dsv4-pro-0813-disagg-gb200-agentic-frontend 8000:8000 -n ${NAMESPACE}
$curl -s localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{
>    "model": "deepseek-ai/DeepSeek-V4-Pro-0813",
>    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
>    "max_tokens": 2048
>  }' | jq -r '.choices[0].message | .content // .reasoning_content'

This is a reasoning model. The chain of thought is returned in reasoning_content and the answer in content; a small max_tokens can leave content empty because the budget is consumed by reasoning.

Benchmark

See perf/README.md for the full workflow — staging the Mooncake trace on the PVC, running the AIPerf trace-replay Job, sweeping concurrency, and fetching artifacts.

Expected Performance

Agentic Mooncake trace, 64K median ISL / 400 median OSL, ~90% KV reuse. SLA is user output ≥ 50 tok/s.

ConcurrencySystem tok/s/GPUUser output tok/sTTFT p50
869.4551.85403 ms
ConcurrencySystem tok/s/GPUUser output tok/sTTFT p50
1050.5853.421,237 ms
ConcurrencySystem tok/s/GPUUser output tok/sTTFT p50
421.451.8322 ms
ConcurrencySystem tok/s/GPUUser output tok/sTTFT p50
413.157.2441 ms

Compare All Targets

TargetGPUsTopologyContextStatus
agg-gb200-agentic8x GB200Aggregated1,048,576Trace-validated
disagg-gb200-agentic16x GB2001P1D1,048,576Trace-validated
agg-h200-agentic8x H200Aggregated1,048,576Trace-validated
disagg-h200-agentic16x H2001P1D1,048,576Trace-validated

Notes

  • One config per topology serves both workloads. The 64K agentic workload and 1M-context requests use the same deployment; GPU KV holds roughly 3.1M tokens, about 3x a full 1M request, so no CPU KV offload is needed. At 1M context, time-to-first-token is minutes — treat it as a batch capability, not an interactive one.
  • FP8 KV cache is required, not a tuning choice. vLLM asserts DeepseekV4 fp8_ds_mla layout only supports fp8 kv-cache; the routed experts ship MXFP4 in the checkpoint. Serving precision is fixed by the model.
  • Speculative decoding is not enabled on the H200 targets. It measured a large regression on this checkpoint.
  • Different checkpoint from DeepSeek-V4-Pro. See the DeepSeek-V4-Pro recipe for that model. The two use different Hugging Face repositories and must not share a model cache.

Source