Nemotron-3.5-Lightning | NVIDIA Dynamo Documentation

This recipe deploys nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, a 30B hybrid Mamba/Attention/MoE model with about 3B active parameters per token. The vLLM targets cover H100, H200, B200, and GB200 with MTP, DFlash, or DSpark speculative decoding. The benchmark tables use a Mooncake-format agentic trace with 64K input tokens, 400 output tokens, and 90% KV reuse.

Choose your deployment target

GPUB200H100H200GB200

TopologyvLLM aggregatedvLLM disaggregatedTensorRT-LLM aggregated

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x B200, or 4x B200 with KV routingRuntime vLLM aggregate workerSpec decode DSpark, DFlash, or MTP

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x H100, or 4x H100 with KV routingRuntime vLLM aggregate workerSpec decode DSpark, DFlash, or MTP

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x H200, or 4x H200 with KV routingRuntime vLLM aggregate workerSpec decode DSpark, DFlash, or MTP

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x GB200Runtime vLLM, aggregate workerSpec decode DSpark, DFlash, or MTP

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 2x B200Runtime vLLM, 1 prefill + 1 decodeTransport IB/RDMA with UCXSpec decode DSpark

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 2x H100Runtime vLLM, 1 prefill + 1 decodeTransport AWS EFA/LIBFABRICSpec decode MTP, DFlash, or DSpark

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 2x H200Runtime vLLM, 1 prefill + 1 decodeTransport IB/RDMA with UCXSpec decode MTP, DFlash, or DSpark

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 2x GB200Runtime vLLM, 1 prefill + 1 decodeTransport UCX/NIXL with cluster-specific fabric resourcesSpec decode DFlash or DSpark

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x B200Runtime TensorRT-LLM, aggregate workerSpec decode MTP or no-spec fallbackStatus Experimental

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x H100Runtime TensorRT-LLM, aggregate workerSpec decode MTPStatus Experimental

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x H200Runtime TensorRT-LLM, aggregate workerSpec decode MTPStatus Experimental

Checkpoint nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4GPUs 1x GB200Runtime TensorRT-LLM, aggregate workerSpec decode MTP or no-spec fallbackStatus Experimental; review-stage

Prerequisites

A Kubernetes cluster with the Dynamo Platform installed and DGD CRDs served.
A namespace with access to the selected GPU target.
A Hugging Face token with access to the base model and draft models.
An image pull secret if your cluster requires authenticated image pulls.

AWS EFA/LIBFABRIC resources exposed as vpc.amazonaws.com/efa for the H100 disaggregated manifests.
AWS EFA userspace available in the runtime image or installed by your cluster bootstrap flow before pod startup. The manifests do not download or install EFA packages. Use the EFA check helper to validate a prepared image or running pod; its install mode requires a pinned EFA_INSTALLER_VERSION and EFA_INSTALLER_SHA256.

IB/RDMA resources exposed as rdma/ib for the H200 and B200 disaggregated manifests.

GB200 disaggregated manifests leave fabric resources cluster-specific. Add the UCX/NIXL resource claims or network resources required by your platform.

Create the namespace and Hugging Face token secret:

$ export NAMESPACE=your-namespace
$ kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -
$ kubectl create secret generic hf-token-secret \
>   --from-literal=HF_TOKEN="$HF_TOKEN" \
>   -n ${NAMESPACE}

Edit storage class, image pull secret names, node selectors, fabric resources, and cluster-specific placement before applying these manifests.

Deploy

Create the shared model cache and download the base and draft checkpoints. This step is common to every target:

$ # Edit storageClassName in model-cache/model-cache.yaml first (kubectl get storageclass).
$ kubectl apply -f recipes/nemotron-3.5-lightning/model-cache/model-cache.yaml -n ${NAMESPACE}
$ kubectl apply -f recipes/nemotron-3.5-lightning/model-cache/model-download.yaml -n ${NAMESPACE}
$ kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=3600s

Then deploy the selected target.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-b200-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-agg-b200-dspark -n ${NAMESPACE} -w

DSpark is the promoted single-worker B200 aggregate path from the checked-in validation table. Use agg-b200-dflash, agg-b200-mtp, or agg-b200-dspark-kv-router to compare the other aggregate options. For this validation trace, KV-aware routing did not materially change the result because KV-cache pressure was low; it can help workloads with enough reusable KV state to pressure a single worker.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-h100-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-agg-h100-dspark -n ${NAMESPACE} -w

DSpark is the highest-throughput H100 single-worker aggregate row in the checked-in validation table. Use agg-h100-dflash, agg-h100-mtp, or agg-h100-dspark-kv-router to compare the other aggregate options. For this validation trace, KV-aware routing did not materially change the result because KV-cache pressure was low; it can help workloads with enough reusable KV state to pressure a single worker.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-h200-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-agg-h200-dspark -n ${NAMESPACE} -w

DSpark is the highest-throughput H200 single-worker aggregate row in the checked-in validation table. Use agg-h200-dflash, agg-h200-mtp, or agg-h200-dspark-kv-router to compare the other aggregate options. For this validation trace, KV-aware routing did not materially change the result because KV-cache pressure was low; it can help workloads with enough reusable KV state to pressure a single worker.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/agg-gb200-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-agg-gb200-dspark -n ${NAMESPACE} -w

DSpark is the highest-throughput GB200 aggregate row in the checked-in validation table. Use agg-gb200-mtp for the other passing aggregate row; agg-gb200-dflash missed the tok/s/user objective in the source results.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-b200-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-disagg-b200-dspark -n ${NAMESPACE} -w

Use this target for B200 disaggregated vLLM validation with DSpark speculative decoding.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-h100-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-disagg-h100-dspark -n ${NAMESPACE} -w

DSpark is the highest-output-throughput H100 disaggregated row in the source inventory. Use disagg-h100-dflash or disagg-h100-mtp to compare the other speculative decoding modes.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-h200-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-disagg-h200-dspark -n ${NAMESPACE} -w

DSpark is the highest-throughput H200 disaggregated row in the source inventory. Use disagg-h200-dflash or disagg-h200-mtp to compare the other speculative decoding modes.

$ kubectl apply -f recipes/nemotron-3.5-lightning/vllm/disagg-gb200-dspark/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd vllm-disagg-gb200-dspark -n ${NAMESPACE} -w

DSpark is the highest-throughput GB200 disaggregated row in the source inventory. Use disagg-gb200-dflash for the lower-throughput passing row.

$ kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-b200-mtp/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd trtllm-agg-b200-mtp -n ${NAMESPACE} -w

Use trtllm/agg-b200/deploy.yaml for the no-spec fallback.

$ kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-h100-mtp/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd trtllm-agg-h100-mtp -n ${NAMESPACE} -w

The H100 TensorRT-LLM target uses the MTP manifest.

$ kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-h200-mtp/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd trtllm-agg-h200-mtp -n ${NAMESPACE} -w

The H200 TensorRT-LLM target uses the MTP manifest.

$ kubectl apply -f recipes/nemotron-3.5-lightning/trtllm/agg-gb200-mtp/deploy.yaml -n ${NAMESPACE}
$ kubectl get dgd trtllm-agg-gb200-mtp -n ${NAMESPACE} -w

Use trtllm/agg-gb200/deploy.yaml for the no-spec fallback.

Smoke Test

Port-forward the frontend service for the deployment you applied:

$ kubectl port-forward svc/vllm-agg-b200-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-agg-h100-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-agg-h200-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-agg-gb200-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-disagg-b200-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-disagg-h100-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-disagg-h200-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/vllm-disagg-gb200-dspark-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/trtllm-agg-b200-mtp-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/trtllm-agg-h100-mtp-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/trtllm-agg-h200-mtp-frontend 8000:8000 -n ${NAMESPACE}

$ kubectl port-forward svc/trtllm-agg-gb200-mtp-frontend 8000:8000 -n ${NAMESPACE}

Then send a chat completion request:

$ curl http://localhost:8000/v1/models
$ curl http://localhost:8000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
>     "messages": [{"role": "user", "content": "Hello!"}],
>     "max_tokens": 32,
>     "temperature": 0
>   }'

Benchmark

The checked-in recipe includes benchmark results in recipes/nemotron-3.5-lightning/perf/README.md. The pass criteria are tok/s/user >= 50 and TTFT p50 under 5 seconds.

To reproduce the shape, run AIPerf against the selected frontend with a Mooncake trace replay and the target concurrency from the table.

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-agg-b200-dspark-frontend:8000 \
>   --concurrency 24

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-agg-h100-dspark-frontend:8000 \
>   --concurrency 20

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-agg-h200-dspark-frontend:8000 \
>   --concurrency 20

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-agg-gb200-dspark-frontend:8000 \
>   --concurrency 24

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-disagg-b200-dspark-frontend:8000 \
>   --concurrency 8

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-disagg-h100-dspark-frontend:8000 \
>   --concurrency 24

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-disagg-h200-dspark-frontend:8000 \
>   --concurrency 24

$ aiperf profile \
>   -m nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
>   --custom-dataset-type mooncake_trace \
>   --num-requests 3541 \
>   --endpoint-type chat \
>   --streaming \
>   --use-server-token-count \
>   --extra-inputs ignore_eos:true \
>   --request-timeout-seconds 1200 \
>   --url http://vllm-disagg-gb200-dspark-frontend:8000 \
>   --concurrency 8

TensorRT-LLM rows are experimental smoke or inventory results in the source README. Use the same AIPerf shape after replacing the --url and concurrency with the selected TensorRT-LLM row below.

Expected Performance

Recipe	Spec decode	Routing	Concurrency	TTFT p50 (ms)	Output tok/s/GPU	Tok/s/user
`vllm/agg-b200-dspark-kv-router/deploy.yaml`	DSpark	KV router	96	295.03	2507.70	104.49
`vllm/agg-b200-dspark/deploy.yaml`	DSpark	Single worker	24	207.44	2471.12	102.96
`vllm/agg-b200-dflash/deploy.yaml`	DFlash	Single worker	20	212.57	2004.06	100.20
`vllm/agg-b200-mtp/deploy.yaml`	MTP	Single worker	18	217.12	1829.94	101.66
`vllm/agg-h100-dspark-kv-router/deploy.yaml`	DSpark	KV router	80	303.84	2037.43	101.87
`vllm/agg-h100-dspark/deploy.yaml`	DSpark	Single worker	20	238.64	1878.11	93.91
`vllm/agg-h100-dflash/deploy.yaml`	DFlash	Single worker	16	216.44	1757.75	109.86
`vllm/agg-h100-mtp/deploy.yaml`	MTP	Single worker	12	241.21	1275.37	106.28
`vllm/agg-h200-dspark-kv-router/deploy.yaml`	DSpark	KV router	80	286.67	2171.00	108.55
`vllm/agg-h200-dspark/deploy.yaml`	DSpark	Single worker	20	243.97	2193.59	109.68
`vllm/agg-h200-dflash/deploy.yaml`	DFlash	Single worker	16	243.30	1744.04	109.00
`vllm/agg-h200-mtp/deploy.yaml`	MTP	Single worker	12	250.66	1478.67	123.22
`vllm/agg-gb200-dspark/deploy.yaml`	DSpark	Single worker	24	296.82	2860.20	119.18
`vllm/agg-gb200-mtp/deploy.yaml`	MTP	Single worker	18	402.99	1758.22	97.68
`vllm/agg-gb200-dflash/deploy.yaml`	DFlash	Single worker	20	429.97	842.42	42.12
`vllm/disagg-b200-dspark/deploy.yaml`	DSpark	1P1D	8	236.43	904.03	226.01
`vllm/disagg-h100-dflash/deploy.yaml`	DFlash	1P1D	12	542.71	1117.29	186.21
`vllm/disagg-h100-dspark/deploy.yaml`	DSpark	1P1D	24	2028.23	1445.29	120.44
`vllm/disagg-h100-mtp/deploy.yaml`	MTP	1P1D	12	493.52	1055.68	175.95
`vllm/disagg-h200-dflash/deploy.yaml`	DFlash	1P1D	12	408.30	1277.86	212.98
`vllm/disagg-h200-dspark/deploy.yaml`	DSpark	1P1D	24	834.80	1656.90	138.08
`vllm/disagg-h200-mtp/deploy.yaml`	MTP	1P1D	12	424.66	1203.97	200.66
`vllm/disagg-gb200-dflash/deploy.yaml`	DFlash	1P1D	4	491.40	584.85	292.43
`vllm/disagg-gb200-dspark/deploy.yaml`	DSpark	1P1D	8	599.17	939.28	234.82
`trtllm/agg-b200/deploy.yaml`	None	Single worker	18	377.68	386.998	21.50
`trtllm/agg-b200-mtp/deploy.yaml`	MTP	Single worker	18	446.88	427.82	23.77
`trtllm/agg-h100-mtp/deploy.yaml`	MTP	Single worker	12	1757.00	264.87	22.07
`trtllm/agg-h200-mtp/deploy.yaml`	MTP	Single worker	12	1630.96	269.26	22.44
`trtllm/agg-gb200/deploy.yaml`	None	Single worker	18	318.88	280.10	15.56
`trtllm/agg-gb200-mtp/deploy.yaml`	MTP	Single worker	18	400.88	580.09	32.23

Compare All Targets

Runtime	Topology	GPU	Spec decode options	Checked-in manifests
vLLM	Aggregated	B200	MTP, DFlash, DSpark	`vllm/agg-b200-dspark-kv-router/deploy.yaml` (DSpark + KV router) `vllm/agg-b200-dspark/deploy.yaml` `vllm/agg-b200-dflash/deploy.yaml` `vllm/agg-b200-mtp/deploy.yaml`
vLLM	Aggregated	H100	MTP, DFlash, DSpark	`vllm/agg-h100-dspark-kv-router/deploy.yaml` (DSpark + KV router) `vllm/agg-h100-dspark/deploy.yaml` `vllm/agg-h100-dflash/deploy.yaml` `vllm/agg-h100-mtp/deploy.yaml`
vLLM	Aggregated	H200	MTP, DFlash, DSpark	`vllm/agg-h200-dspark-kv-router/deploy.yaml` (DSpark + KV router) `vllm/agg-h200-dspark/deploy.yaml` `vllm/agg-h200-dflash/deploy.yaml` `vllm/agg-h200-mtp/deploy.yaml`
vLLM	Aggregated	GB200	MTP, DFlash, DSpark	`vllm/agg-gb200-dspark/deploy.yaml` `vllm/agg-gb200-dflash/deploy.yaml` `vllm/agg-gb200-mtp/deploy.yaml`
vLLM	Disaggregated 1P1D	B200	DSpark	`vllm/disagg-b200-dspark/deploy.yaml`
vLLM	Disaggregated 1P1D	H100	MTP, DFlash, DSpark	`vllm/disagg-h100-dspark/deploy.yaml` `vllm/disagg-h100-dflash/deploy.yaml` `vllm/disagg-h100-mtp/deploy.yaml`
vLLM	Disaggregated 1P1D	H200	MTP, DFlash, DSpark	`vllm/disagg-h200-dspark/deploy.yaml` `vllm/disagg-h200-dflash/deploy.yaml` `vllm/disagg-h200-mtp/deploy.yaml`
vLLM	Disaggregated 1P1D	GB200	DFlash, DSpark	`vllm/disagg-gb200-dspark/deploy.yaml` `vllm/disagg-gb200-dflash/deploy.yaml`
TensorRT-LLM	Aggregated	B200	MTP, no-spec fallback	`trtllm/agg-b200-mtp/deploy.yaml` `trtllm/agg-b200/deploy.yaml`
TensorRT-LLM	Aggregated	H100	MTP	`trtllm/agg-h100-mtp/deploy.yaml`
TensorRT-LLM	Aggregated	H200	MTP	`trtllm/agg-h200-mtp/deploy.yaml`
TensorRT-LLM	Aggregated	GB200	MTP, no-spec fallback	`trtllm/agg-gb200-mtp/deploy.yaml` `trtllm/agg-gb200/deploy.yaml`

Notes

H100 disaggregated manifests request vpc.amazonaws.com/efa: 4; H200 and B200 disaggregated manifests request rdma/ib: 1. Change those resources when your cluster exposes different RDMA devices.

TensorRT-LLM manifests are experimental in this recipe.

Source

Source README: recipes/nemotron-3.5-lightning/README.md
vLLM README: recipes/nemotron-3.5-lightning/vllm/README.md
EFA check helper: recipes/nemotron-3.5-lightning/vllm/check-efa-userspace.sh
TensorRT-LLM README: recipes/nemotron-3.5-lightning/trtllm/README.md
Performance notes: recipes/nemotron-3.5-lightning/perf/README.md
Model cache manifests: model-cache