Choose your deployment target
GPU B200 H100 H200 GB200 Topology vLLM aggregated vLLM disaggregated TensorRT-LLM aggregated Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Precision BF16 GPUs 1x B200, or 4x B200 with KV routing Runtime vLLM aggregate worker Spec decode DSpark or MTP Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 Precision NVFP4 (W4A16) GPUs 1x H100, or 4x H100 with KV routing Runtime vLLM aggregate worker Spec decode DSpark, DFlash, or MTP Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 Precision NVFP4 (W4A16) GPUs 1x H200, or 4x H200 with KV routing Runtime vLLM aggregate worker Spec decode DSpark, DFlash, or MTP Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Precision BF16 GPUs 1x GB200 Runtime vLLM, aggregate worker Spec decode DSpark or MTP Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Precision BF16 GPUs 2x B200 Runtime vLLM, 1 prefill + 1 decode Transport IB/RDMA with UCX Spec decode DSpark Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 Precision NVFP4 (W4A16) GPUs 2x H100 Runtime vLLM, 1 prefill + 1 decode Transport AWS EFA/LIBFABRIC Spec decode DFlash or DSpark Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 Precision NVFP4 (W4A16) GPUs 2x H200 Runtime vLLM, 1 prefill + 1 decode Transport IB/RDMA with UCX Spec decode DFlash or DSpark Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Precision BF16 GPUs 2x GB200 Runtime vLLM, 1 prefill + 1 decode Transport UCX/NIXL with cluster-specific fabric resources Spec decode DFlash or DSpark Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Precision BF16 GPUs 1x B200 Runtime TensorRT-LLM, aggregate worker Spec decode MTP or no-spec fallback Status Experimental Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 Precision NVFP4 (W4A16) GPUs 1x H100 Runtime TensorRT-LLM, aggregate worker Spec decode MTP Status Experimental Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 Precision NVFP4 GPUs 1x H200 Runtime TensorRT-LLM, aggregate worker Spec decode MTP Status Experimental Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Precision BF16 GPUs 1x GB200 Runtime TensorRT-LLM, aggregate worker Spec decode MTP or no-spec fallback Status Experimental; review-stage ## Prerequisites * A Kubernetes cluster with the Dynamo Platform installed and DGD CRDs served. * A namespace with access to the selected GPU target. * A Hugging Face token with access to the base model and draft models. * An image pull secret if your cluster requires authenticated image pulls. - AWS EFA/LIBFABRIC resources exposed as `vpc.amazonaws.com/efa` for the H100 disaggregated manifests. - AWS EFA userspace available in the runtime image or installed by your cluster bootstrap flow before pod startup. The manifests do not download or install EFA packages. Use the [EFA check helper](https://github.com/ai-dynamo/dynamo/blob/main/recipes/nemotron-3.5-lightning/vllm/check-efa-userspace.sh) to validate a prepared image or running pod; its install mode requires a pinned `EFA_INSTALLER_VERSION` and `EFA_INSTALLER_SHA256`. * IB/RDMA resources exposed as `rdma/ib` for the H200 and B200 disaggregated manifests. - GB200 disaggregated manifests leave fabric resources cluster-specific. Add the UCX/NIXL resource claims or network resources required by your platform. Create the namespace and Hugging Face token secret: ```bash export NAMESPACE=your-namespace kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic hf-token-secret \ --from-literal=HF_TOKEN="$HF_TOKEN" \ -n ${NAMESPACE} ``` Edit storage class, image pull secret names, node selectors, fabric resources, and cluster-specific placement before applying these manifests. ## Deploy Create the shared model cache, then download the checkpoints for the selected GPU family: ```bash # Edit storageClassName in model-cache/model-cache.yaml first (kubectl get storageclass). kubectl apply -f recipes/nemotron-3.5-lightning/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/nemotron-3.5-lightning/model-cache/model-download.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=3600s ``` ```bash # Edit storageClassName in model-cache/model-cache.yaml first (kubectl get storageclass). kubectl apply -f recipes/nemotron-3.5-lightning/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/nemotron-3.5-lightning/model-cache/model-download-bf16.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download-bf16 -n ${NAMESPACE} --timeout=3600s ``` Then deploy the selected target. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-b200-dspark-bf16/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-agg-b200-dspark-bf16 -n ${NAMESPACE} -w ``` DSpark is the promoted single-worker B200 aggregate path from the checked-in validation table. Use `agg-b200-mtp-bf16` or `agg-b200-dspark-kv-router-bf16` to compare the other aggregate options. For this validation trace, KV-aware routing did not materially change the result because KV-cache pressure was low; it can help workloads with enough reusable KV state to pressure a single worker. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-h100-dspark/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-agg-h100-dspark -n ${NAMESPACE} -w ``` DSpark is the highest-throughput H100 single-worker aggregate row in the checked-in validation table. Use `agg-h100-dflash`, `agg-h100-mtp`, or `agg-h100-dspark-kv-router` to compare the other aggregate options. For this validation trace, KV-aware routing did not materially change the result because KV-cache pressure was low; it can help workloads with enough reusable KV state to pressure a single worker. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-h200-dspark/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-agg-h200-dspark -n ${NAMESPACE} -w ``` DSpark is the highest-throughput H200 single-worker aggregate row in the checked-in validation table. Use `agg-h200-dflash`, `agg-h200-mtp`, or `agg-h200-dspark-kv-router` to compare the other aggregate options. For this validation trace, KV-aware routing did not materially change the result because KV-cache pressure was low; it can help workloads with enough reusable KV state to pressure a single worker. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-gb200-dspark-bf16/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-agg-gb200-dspark-bf16 -n ${NAMESPACE} -w ``` DSpark is the highest-throughput GB200 aggregate row in the checked-in validation table. Use `agg-gb200-mtp-bf16` for the other passing aggregate row. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-b200-dspark-bf16/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-disagg-b200-dspark-bf16 -n ${NAMESPACE} -w ``` Use this target for B200 disaggregated vLLM validation with DSpark speculative decoding. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-h100-dspark/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-disagg-h100-dspark -n ${NAMESPACE} -w ``` DSpark is the highest-output-throughput H100 disaggregated row in the source inventory. Use `disagg-h100-dflash` to compare the other speculative decoding mode. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-h200-dspark/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-disagg-h200-dspark -n ${NAMESPACE} -w ``` DSpark is the highest-throughput H200 disaggregated row in the source inventory. Use `disagg-h200-dflash` to compare the other speculative decoding mode. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-gb200-dspark-bf16/deploy.yaml -n ${NAMESPACE} kubectl get dgd vllm-disagg-gb200-dspark-bf16 -n ${NAMESPACE} -w ``` The delivered GB200 disaggregated recipes are DSpark and DFlash. The command above deploys DSpark; use `disagg-gb200-dflash-bf16` for the DFlash recipe. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-b200-mtp-bf16/deploy.yaml -n ${NAMESPACE} kubectl get dgd trtllm-agg-b200-mtp-bf16 -n ${NAMESPACE} -w ``` Use `trtllm/agg-b200-bf16/deploy.yaml` for the no-spec fallback. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-h100-mtp/deploy.yaml -n ${NAMESPACE} kubectl get dgd trtllm-agg-h100-mtp -n ${NAMESPACE} -w ``` The H100 TensorRT-LLM target uses the MTP manifest. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-h200-mtp/deploy.yaml -n ${NAMESPACE} kubectl get dgd trtllm-agg-h200-mtp -n ${NAMESPACE} -w ``` The H200 TensorRT-LLM target uses the MTP manifest. ```bash kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-gb200-mtp-bf16/deploy.yaml -n ${NAMESPACE} kubectl get dgd trtllm-agg-gb200-mtp-bf16 -n ${NAMESPACE} -w ``` Use `trtllm/agg-gb200-bf16/deploy.yaml` for the no-spec fallback. ## Smoke Test Port-forward the frontend service for the deployment you applied: ```bash kubectl port-forward svc/vllm-agg-b200-dspark-bf16-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-agg-h100-dspark-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-agg-h200-dspark-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-agg-gb200-dspark-bf16-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-disagg-b200-dspark-bf16-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-disagg-h100-dspark-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-disagg-h200-dspark-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/vllm-disagg-gb200-dspark-bf16-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/trtllm-agg-b200-mtp-bf16-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/trtllm-agg-h100-mtp-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/trtllm-agg-h200-mtp-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/trtllm-agg-gb200-mtp-bf16-frontend 8000:8000 -n ${NAMESPACE} ``` Then send a chat completion request: ```bash curl http://localhost:8000/v1/models curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 32, "temperature": 0 }' ``` ```bash curl http://localhost:8000/v1/models curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 32, "temperature": 0 }' ``` ## Benchmark The checked-in recipe includes benchmark results in [`recipes/nemotron-3.5-lightning/perf/README.md`](https://github.com/ai-dynamo/dynamo/blob/main/recipes/nemotron-3.5-lightning/perf/README.md). The pass criteria are `tok/s/user >= 50` and TTFT p50 under 5 seconds. To reproduce the shape, run AIPerf against the selected frontend with a Mooncake trace replay and the target concurrency from the table. ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-agg-b200-dspark-bf16-frontend:8000 \ --concurrency 50 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-agg-h100-dspark-frontend:8000 \ --concurrency 20 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-agg-h200-dspark-frontend:8000 \ --concurrency 20 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-agg-gb200-dspark-bf16-frontend:8000 \ --concurrency 58 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-disagg-b200-dspark-bf16-frontend:8000 \ --concurrency 80 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-disagg-h100-dspark-frontend:8000 \ --concurrency 24 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-disagg-h200-dspark-frontend:8000 \ --concurrency 24 ``` ```bash aiperf profile \ -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --endpoint-type chat \ --streaming \ --use-server-token-count \ --extra-inputs ignore_eos:true \ --request-timeout-seconds 1200 \ --url http://vllm-disagg-gb200-dspark-bf16-frontend:8000 \ --concurrency 8 ``` TensorRT-LLM rows are experimental smoke or inventory results in the source README. Use the same AIPerf shape after replacing the `--url` and concurrency with the selected TensorRT-LLM row below. ## Expected Performance| Recipe | Spec decode | Routing | Concurrency | TTFT p50 (ms) | Output tok/s/GPU | Tok/s/user |
|---|---|---|---|---|---|---|
vllm/agg-b200-dspark-kv-router-bf16/deploy.yaml |
DSpark | KV router | 192 | 254.26 | 4162.09 | 86.71 |
vllm/agg-b200-dspark-bf16/deploy.yaml |
DSpark | Single worker | 50 | 221.49 | 4005.48 | 80.11 |
vllm/agg-b200-mtp-bf16/deploy.yaml |
MTP | Single worker | 18 | 165.43 | 2454.72 | 136.37 |
vllm/agg-h100-dspark-kv-router/deploy.yaml |
DSpark | KV router | 80 | 303.84 | 2037.43 | 101.87 |
vllm/agg-h100-dspark/deploy.yaml |
DSpark | Single worker | 20 | 238.64 | 1878.11 | 93.91 |
vllm/agg-h100-dflash/deploy.yaml |
DFlash | Single worker | 16 | 216.44 | 1757.75 | 109.86 |
vllm/agg-h100-mtp/deploy.yaml |
MTP | Single worker | 12 | 241.21 | 1275.37 | 106.28 |
vllm/agg-h200-dspark-kv-router/deploy.yaml |
DSpark | KV router | 80 | 286.67 | 2171.00 | 108.55 |
vllm/agg-h200-dspark/deploy.yaml |
DSpark | Single worker | 20 | 243.97 | 2193.59 | 109.68 |
vllm/agg-h200-dflash/deploy.yaml |
DFlash | Single worker | 16 | 243.30 | 1744.04 | 109.00 |
vllm/agg-h200-mtp/deploy.yaml |
MTP | Single worker | 12 | 250.66 | 1478.67 | 123.22 |
vllm/agg-gb200-dspark-bf16/deploy.yaml |
DSpark | Single worker | 58 | 307.73 | 4327.90 | 74.62 |
vllm/agg-gb200-mtp-bf16/deploy.yaml |
MTP | Single worker | 18 | 228.77 | 2289.75 | 127.21 |
vllm/disagg-b200-dspark-bf16/deploy.yaml |
DSpark | 1P1D | 80 | 1059.12 | 3711.37 | 92.78 |
vllm/disagg-h100-dflash/deploy.yaml |
DFlash | 1P1D | 12 | 542.71 | 1117.29 | 186.21 |
vllm/disagg-h100-dspark/deploy.yaml |
DSpark | 1P1D | 24 | 2028.23 | 1445.29 | 120.44 |
vllm/disagg-h200-dflash/deploy.yaml |
DFlash | 1P1D | 12 | 408.30 | 1277.86 | 212.98 |
vllm/disagg-h200-dspark/deploy.yaml |
DSpark | 1P1D | 24 | 834.80 | 1656.90 | 138.08 |
vllm/disagg-gb200-dflash-bf16/deploy.yaml |
DFlash | 1P1D | 24 | 10641.10 | 1148.33 | 95.69 |
vllm/disagg-gb200-dspark-bf16/deploy.yaml |
DSpark | 1P1D | 8 | 569.70 | 802.30 | 200.58 |
trtllm/agg-b200-bf16/deploy.yaml |
None | Single worker | 18 | 219.18 | 1869.54 | 103.86 |
trtllm/agg-b200-mtp-bf16/deploy.yaml |
MTP | Single worker | 18 | 244.95 | 3295.50 | 183.08 |
trtllm/agg-h100-mtp/deploy.yaml |
MTP | Single worker | 12 | 1757.00 | 264.87 | 22.07 |
trtllm/agg-h200-mtp/deploy.yaml |
MTP | Single worker | 12 | 1692.98 | 250.05 | 20.84 |
trtllm/agg-gb200-bf16/deploy.yaml |
None | Single worker | 18 | 216.77 | 1985.84 | 110.32 |
trtllm/agg-gb200-mtp-bf16/deploy.yaml |
MTP | Single worker | 18 | 231.34 | 3556.40 | 197.58 |
| Runtime | Topology | GPU | Spec decode options | Checked-in manifests |
|---|---|---|---|---|
| vLLM | Aggregated | B200 | MTP, DSpark | vllm/agg-b200-dspark-kv-router-bf16/deploy.yaml (DSpark + KV router)vllm/agg-b200-dspark-bf16/deploy.yamlvllm/agg-b200-mtp-bf16/deploy.yaml |
| vLLM | Aggregated | H100 | MTP, DFlash, DSpark | vllm/agg-h100-dspark-kv-router/deploy.yaml (DSpark + KV router)vllm/agg-h100-dspark/deploy.yamlvllm/agg-h100-dflash/deploy.yamlvllm/agg-h100-mtp/deploy.yaml |
| vLLM | Aggregated | H200 | MTP, DFlash, DSpark | vllm/agg-h200-dspark-kv-router/deploy.yaml (DSpark + KV router)vllm/agg-h200-dspark/deploy.yamlvllm/agg-h200-dflash/deploy.yamlvllm/agg-h200-mtp/deploy.yaml |
| vLLM | Aggregated | GB200 | MTP, DSpark | vllm/agg-gb200-dspark-bf16/deploy.yamlvllm/agg-gb200-mtp-bf16/deploy.yaml |
| vLLM | Disaggregated 1P1D | B200 | DSpark | vllm/disagg-b200-dspark-bf16/deploy.yaml |
| vLLM | Disaggregated 1P1D | H100 | DFlash, DSpark | vllm/disagg-h100-dspark/deploy.yamlvllm/disagg-h100-dflash/deploy.yaml |
| vLLM | Disaggregated 1P1D | H200 | DFlash, DSpark | vllm/disagg-h200-dspark/deploy.yamlvllm/disagg-h200-dflash/deploy.yaml |
| vLLM | Disaggregated 1P1D | GB200 | DFlash, DSpark | vllm/disagg-gb200-dflash-bf16/deploy.yamlvllm/disagg-gb200-dspark-bf16/deploy.yaml |
| TensorRT-LLM | Aggregated | B200 | MTP, no-spec fallback | trtllm/agg-b200-mtp-bf16/deploy.yamltrtllm/agg-b200-bf16/deploy.yaml |
| TensorRT-LLM | Aggregated | H100 | MTP | trtllm/agg-h100-mtp/deploy.yaml |
| TensorRT-LLM | Aggregated | H200 | MTP | trtllm/agg-h200-mtp/deploy.yaml |
| TensorRT-LLM | Aggregated | GB200 | MTP, no-spec fallback | trtllm/agg-gb200-mtp-bf16/deploy.yamltrtllm/agg-gb200-bf16/deploy.yaml |