DeepSeek-V4-Pro

Serve DeepSeek-V4-Pro with Dynamo and vLLM — aggregated or disaggregated, on B200 or H200, tuned for the agentic workload.

View as Markdown

Each target below is an agentic-workload vLLM deployment of DeepSeek-V4-Pro — a MoE model (1.6T total / 49B active) with hybrid CSA + HCA attention, a Blackwell FP4 indexer cache, and mHC residual connections. B200 serves the nvidia/DeepSeek-V4-Pro-NVFP4 checkpoint at the full 1M context; H200 serves the public deepseek-ai/DeepSeek-V4-Pro (FP8), capped at max_model_len=86016 by HBM. Pick your GPU and topology; every command updates to match. Text only; reasoning + tool calling supported.

Choose your deployment target

GPU
Topology
Checkpoint nvidia/DeepSeek-V4-Pro-NVFP4GPUs 8x B200, TP8 + EPMoE / spec FLASHINFER_TRTLLM, MTP-2KV / routing FP8 KV (block 256), KV-awareContext 1,048,576
Checkpoint nvidia/DeepSeek-V4-Pro-NVFP4 RecommendedGPUs 16x B200 — 1P1D (TP8 + EP per worker)MoE / spec FLASHINFER_TRTLLM, decode MTP-2Transfer NIXL over RDMA/GDRContext 1,048,576
Checkpoint deepseek-ai/DeepSeek-V4-Pro (FP8) Recommended on H200GPUs 8x H200, TP8 + EPMoE MARLINKV / routing FP8 KV (block 256), KV-awareContext 86,016 (HBM cap)
Checkpoint deepseek-ai/DeepSeek-V4-Pro (FP8)GPUs 32x H200 — 1P3D (TP8 + EP per worker)MoE MARLINTransfer NIXL over RDMA/GDRContext 86,016 (HBM cap)

Recommended picks differ by SKU: B200 → Disaggregated (1P1D) (highest tok/s/GPU); H200 → Aggregated (AGG beats the 1P3D disagg lane). H200 targets are a secondary/best-effort lane on the public FP8 checkpoint.

Prerequisites

  • A Kubernetes cluster with the Dynamo Platform (operator) installed.
  • 8x B200 (aggregated) or 16x B200 (disaggregated, 1P1D) available.
  • A Hugging Face token with access to nvidia/DeepSeek-V4-Pro-NVFP4.
  • 8x H200 (aggregated) or 32x H200 (disaggregated, 1P3D) available.
  • A Hugging Face token with access to deepseek-ai/DeepSeek-V4-Pro.
  • The nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0 image (pinned in each deploy.yaml).

Create the namespace and token secret:

$export NAMESPACE=your-namespace
$kubectl create namespace ${NAMESPACE}
$kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN="your-token" -n ${NAMESPACE}

Edit cluster-specific values before applying: storageClassName in model-cache/model-cache.yaml (a RWX class). For disaggregated targets, also set VLLM_GPU_NIC_PCIE_MAPPING in the manifest — see Per-rank NIC mapping.

Deploy

Create storage, download the checkpoint (~865 GB, 1.5–3 h first apply), then apply the target’s deploy.yaml:

$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/deepseek-v4/deepseek-v4-pro/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s
$RECIPE=recipes/deepseek-v4/deepseek-v4-pro/vllm/agg-b200-agentic
$RECIPE=recipes/deepseek-v4/deepseek-v4-pro/vllm/disagg-b200-agentic
$RECIPE=recipes/deepseek-v4/deepseek-v4-pro/vllm/agg-h200-agentic
$RECIPE=recipes/deepseek-v4/deepseek-v4-pro/vllm/disagg-h200-agentic
$kubectl apply -f ${RECIPE}/deploy.yaml -n ${NAMESPACE}
$
$# DGD name = metadata.name in that deploy.yaml:
$DGD=$(awk '/^metadata:/{m=1} m && /name:/{print $2; exit}' ${RECIPE}/deploy.yaml)
$kubectl wait --for=condition=Ready pod \
>  -l nvidia.com/dynamo-graph-deployment-name=${DGD} -n ${NAMESPACE} --timeout=5400s

First launch loads weights across 8 TP ranks and warms CUDA graphs — up to ~90 min; the startup probes allow for it.

Smoke Test

The model field must be the served model name for your SKU (the --served-model-name in the deploy): B200 serves nvidia/DeepSeek-V4-Pro-NVFP4, H200 serves deepseek-ai/DeepSeek-V4-Pro.

$kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}
$
$curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
>  -d '{"model":"nvidia/DeepSeek-V4-Pro-NVFP4","messages":[{"role":"user","content":"Hello!"}],"max_tokens":512}'
$kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}
$
$curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
>  -d '{"model":"deepseek-ai/DeepSeek-V4-Pro","messages":[{"role":"user","content":"Hello!"}],"max_tokens":512}'

Pro emits chain-of-thought into message.reasoning_content and the answer into message.content. Reasoning effort is selected via chat_template_kwargs ({"thinking":true,"reasoning_effort":"high"|"max"}; Think Max needs --max-model-len >= 393216).

Benchmark

The shared perf.yaml replays the agentic MoonTrace (64K avg ISL / 400 OSL, 90% KV reuse) with AIPerf. Point ENDPOINT at ${DGD}-frontend:8000 and apply it:

$kubectl apply -f recipes/deepseek-v4/perf/perf.yaml -n ${NAMESPACE}

Compare All Targets

TargetGPUsParallelismMoE backendSpec. decodemax_model_lenTransfer
agg-b2008x B200TP8 + EPFLASHINFER_TRTLLMMTP-21,048,576—
disagg-b200 (1P1D)16x B200TP8 + EPFLASHINFER_TRTLLMdecode MTP-21,048,576NIXL GDR
agg-h2008x H200TP8 + EPMARLINnone86,016—
disagg-h200 (1P3D)32x H200TP8 + EPMARLINnone86,016NIXL GDR

Notes

  • Recommended per SKU: B200 ships disaggregated 1P1D (+6% tok/s/GPU over AGG); H200 ships aggregated (AGG > the 1P3D disagg lane). H200 is a secondary/best-effort lane on the public FP8 checkpoint at max_model_len=86016.
  • Reasoning / tool parsers are wired via --dyn-reasoning-parser deepseek_v4 / --dyn-tool-call-parser deepseek_v4.
  • Disaggregated targets require the per-rank NIC map (VLLM_GPU_NIC_PCIE_MAPPING); B200 ships rdma/shared_ib, H200 ships rdma/ib. See the top-level README.
  • Day-0 / GB200 (vLLM + SGLang, ComputeDomain/MNNVL) variants exist in the recipe tree but are experimental (dev images) and not shown here.

Source