DeepSeek-V4-Flash

Serve DeepSeek-V4-Flash with Dynamo and vLLM — aggregated or disaggregated, on B200 or H200, tuned for the agentic workload.

View as Markdown

Each target below is an agentic-workload vLLM deployment of DeepSeek-V4-Flash — a MoE model (284B total / 13B active) with hybrid CSA + HCA attention, FP8 KV cache, and KV-aware routing. B200 serves the nvidia/DeepSeek-V4-Flash-NVFP4 checkpoint; H200 serves the public deepseek-ai/DeepSeek-V4-Flash. Pick your GPU and topology; every command on this page updates to match. Text only; reasoning + tool calling supported.

Choose your deployment target

GPU
Topology
Checkpoint nvidia/DeepSeek-V4-Flash-NVFP4GPUs 4x B200, TP4 + EPMoE / KV FLASHINFER_TRTLLM, FP8 KV (block 256)Routing KV-aware, prefix cachingContext 1,048,576
Checkpoint nvidia/DeepSeek-V4-Flash-NVFP4GPUs 12x B200 — 2P1D (2 prefill + 1 decode, TP4 each)MoE / KV FLASHINFER_TRTLLM, FP8 KV (block 256)Transfer NIXL over RDMA/GDRContext 1,048,576
Checkpoint deepseek-ai/DeepSeek-V4-FlashGPUs 4x H200, DP4 + TP1 + EPMoE / spec MARLIN (FLASHINFER_MLA attn), MTP-1Routing KV-aware, prefix cachingContext 1,048,576
Checkpoint deepseek-ai/DeepSeek-V4-FlashGPUs 28x H200 — 4P3D (DP4+TP1+EP per worker)MoE / spec MARLIN, decode MTP-1Transfer NIXL over RDMA/GDRContext 1,048,576

Prerequisites

  • A Kubernetes cluster with the Dynamo Platform (operator) installed.
  • 4x B200 (aggregated) or 12x B200 (disaggregated, 2P1D) available.
  • A Hugging Face token with access to nvidia/DeepSeek-V4-Flash-NVFP4.
  • 4x H200 (aggregated) or 28x H200 (disaggregated, 4P3D) available.
  • A Hugging Face token with access to deepseek-ai/DeepSeek-V4-Flash.
  • The nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0 image (pinned in each deploy.yaml).

Create the namespace and token secret:

$export NAMESPACE=your-namespace
$kubectl create namespace ${NAMESPACE}
$kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN="your-token" -n ${NAMESPACE}

Edit cluster-specific values before applying: storageClassName in model-cache/model-cache.yaml (a RWX class). For disaggregated targets, also set VLLM_GPU_NIC_PCIE_MAPPING in the manifest to your node’s per-rank affine NICs — see Per-rank NIC mapping.

Deploy

Create storage, download the checkpoint into the PVC, then apply the target’s deploy.yaml:

$kubectl apply -f recipes/deepseek-v4/deepseek-v4-flash/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/deepseek-v4/deepseek-v4-flash/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=7200s
$RECIPE=recipes/deepseek-v4/deepseek-v4-flash/vllm/agg-b200-agentic
$RECIPE=recipes/deepseek-v4/deepseek-v4-flash/vllm/disagg-b200-agentic
$RECIPE=recipes/deepseek-v4/deepseek-v4-flash/vllm/agg-h200-agentic
$RECIPE=recipes/deepseek-v4/deepseek-v4-flash/vllm/disagg-h200-agentic
$kubectl apply -f ${RECIPE}/deploy.yaml -n ${NAMESPACE}
$
$# DGD name = metadata.name in that deploy.yaml:
$DGD=$(awk '/^metadata:/{m=1} m && /name:/{print $2; exit}' ${RECIPE}/deploy.yaml)
$kubectl wait --for=condition=Ready pod \
>  -l nvidia.com/dynamo-graph-deployment-name=${DGD} -n ${NAMESPACE} --timeout=3600s

First launch loads weights and warms CUDA graphs; the startup probes allow for it.

Smoke Test

The model field must be the served model name for your SKU (the --served-model-name in the deploy): B200 serves nvidia/DeepSeek-V4-Flash-NVFP4, H200 serves deepseek-ai/DeepSeek-V4-Flash.

$kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}
$
$curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
>  -d '{"model":"nvidia/DeepSeek-V4-Flash-NVFP4","messages":[{"role":"user","content":"Hello!"}],"max_tokens":512}'
$kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}
$
$curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
>  -d '{"model":"deepseek-ai/DeepSeek-V4-Flash","messages":[{"role":"user","content":"Hello!"}],"max_tokens":512}'

Flash reasons by default: the chain-of-thought fills message.reasoning_content and the answer fills message.content. With too small a max_tokens the budget is spent on reasoning before any content (content: null, finish_reason: "length") — expected; raise max_tokens.

Benchmark

The shared perf.yaml replays the agentic MoonTrace (64K avg ISL / 400 OSL, 90% KV reuse) with AIPerf against the deployed frontend. Point ENDPOINT at ${DGD}-frontend:8000 and apply it:

$kubectl apply -f recipes/deepseek-v4/perf/perf.yaml -n ${NAMESPACE}

Compare All Targets

TargetGPUsParallelismMoE backendSpec. decodeTransfer
agg-b2004x B200TP4 + EPFLASHINFER_TRTLLMnone—
disagg-b200 (2P1D)12x B200TP4 / TP4FLASHINFER_TRTLLMnoneNIXL GDR
agg-h2004x H200DP4 + TP1 + EPMARLIN (FLASHINFER_MLA attn)MTP-1—
disagg-h200 (4P3D)28x H200DP4+TP1+EP / DP4+TP1+EPMARLINdecode MTP-1NIXL GDR

Notes

  • Reasoning / tool parsers are wired via the Dynamo variants (--dyn-reasoning-parser deepseek_v4, --dyn-tool-call-parser deepseek_v4); the engine-native parsers do not feed the Dynamo OpenAI renderer.
  • Disaggregated targets require the per-rank NIC map (VLLM_GPU_NIC_PCIE_MAPPING) for GPU-Direct RDMA KV transfer; B200 ships rdma/shared_ib, H200 ships rdma/ib. See the top-level recipe README.
  • AGG scales by independent single-replica DGDs — KV-routed multi-replica AGG does not improve per-GPU throughput.
  • Day-0 / GB200 SGLang variants exist in the recipe tree but are experimental (dev images) and not shown here.

Source