Nemotron 3.5 Super VL | NVIDIA Dynamo Documentation

Each target below is a Dynamo + vLLM or TensorRT-LLM deployment of NVIDIA Nemotron 3.5 Super VL, a multimodal MoE model serving up to 1M-token context. Pick your framework, GPU architecture, and serving topology. Every command on this page updates to match.

Prerequisites

A Kubernetes cluster with the Dynamo platform installed and 4 H100 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 H100 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 2 H200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 2 H200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 B200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 5 B200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 GB200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 5 GB200 GPUs available. See the Kubernetes Deployment Guide.
The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 B300 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 GB300 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 5 B300 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 5 GB300 GPUs available. See the Kubernetes Deployment Guide.
The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 11 VR200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 11 VR200 GPUs available. See the Kubernetes Deployment Guide.
The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 B200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

A Kubernetes cluster with the Dynamo platform installed and 4 B200 GPUs available. See the Kubernetes Deployment Guide.
A Hugging Face token with access to the checkpoint nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.

Create the namespace and token secret:

export NAMESPACE=your-namespace
kubectl create namespace ${NAMESPACE}
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN="your-token" \
  -n ${NAMESPACE}

The recipes require the shared-model-cache PVC. Edit storageClassName in model-cache/model-cache.yaml to a ReadWriteMany storage class on your cluster (kubectl get storageclass) before applying it. Review namespace, image tags, and resource claims in the manifests as well.

Deploy

Create the shared-model-cache PVC and download the checkpoint:

# Edit storageClassName in model-cache/model-cache.yaml first.
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/model-cache/model-cache.yaml -n ${NAMESPACE}
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/model-cache/model-download.yaml -n ${NAMESPACE}
kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s

The workers mount the PVC at /shared-model-cache and resolve the checkpoint from the local Hugging Face cache with HF_HUB_OFFLINE=1.

Deploy the selected DynamoGraphDeployment (DGD):

export DGD=super-3-5-vllm-h100-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-h100-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-h100-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-h100-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-h200-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-h200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-h200-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-h200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-b200-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-b200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-b200-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-b200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-gb200-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-gb200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-gb200-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-gb200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-b300-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-b300-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-gb300-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-gb300-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-b300-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-b300-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-gb300-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-gb300-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-vr200-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/agg-vr200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-vllm-vr200-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/vllm/disagg-vr200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-trtllm-b200-agg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/trtllm/agg-b200-swebenchpro/deploy.yaml -n ${NAMESPACE}

export DGD=super-3-5-trtllm-b200-disagg-swebenchpro
kubectl apply -f recipes/nemotron-3.5-super-vl-120b-a12b/trtllm/disagg-b200-swebenchpro/deploy.yaml -n ${NAMESPACE}

The published deploy.yaml recipes are generic base recipes agnostic to cluster-specific settings. Adapt to your cluster scheduling, tolerations, and networking by using kustomizations.

Smoke Test

Send a test request to verify the selected deployment serves traffic. The deploy command sets DGD to its resource name. Forward the frontend port:

kubectl port-forward svc/${DGD}-frontend 8000:8000 -n ${NAMESPACE}

All Nemotron 3.5 Super VL recipes serve under the same model name, nvidia/nemotron-3.5-super-vl-120b-a12b:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"nvidia/nemotron-3.5-super-vl-120b-a12b","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'

Nemotron 3.5 Super VL reasons before answering, and tool calling is supported — the deployment uses the nemotron_v3 reasoning and qwen3_coder tool-call parsers at the frontend.

Benchmark

See perf/README.md for the full benchmark workflow: staging the trace on the PVC, running the AIPerf trace-replay Job, running a concurrency sweep, and fetching artifacts.

Optimization Targets

Recipes are optimized for the following agentic workload and target user interactivity / TTFT:

Workload	Median ISL	Median OSL	KV cache hit rate	User output tok/s	TTFT P50
SWEBench-Pro	222,076	188	99%	> 50	< 5 s

The benchmark replays the Weka-format agentic trace described in perf/README.md.

Performance Results

Workload	Recipe	Framework	SKU	Concurrency	System Output Tokens/s/GPU	User Output Tokens/s (P50)	TTFT P50 (ms)
SWEBench-Pro	Aggregated (2 workers)	vLLM	H100	16	270	102.2	490
SWEBench-Pro	Disaggregated (1P1D)	vLLM	H100	16	259	106	1,300
SWEBench-Pro	Aggregated (2 workers)	vLLM	H200	16	376	61	710
SWEBench-Pro	Disaggregated (1P1D)	vLLM	H200	16	361	61.6	1,870
SWEBench-Pro	Aggregated (4 workers)	vLLM	B200	128	1,378.9	68.86	396
SWEBench-Pro	Disaggregated (3P2D)	vLLM	B200	128	1,050.44	60.84	1,073
SWEBench-Pro	Aggregated (4 workers)	vLLM	GB200	128	1,503.57	76.15	403
SWEBench-Pro	Disaggregated (3P2D)	vLLM	GB200	128	1,269.93	82.09	836
SWEBench-Pro	Aggregated (4 workers)	vLLM	B300	128	1,536	78.5	438
SWEBench-Pro	Aggregated (4 workers)	vLLM	GB300	160	1,719.51	67.9	381
SWEBench-Pro	Disaggregated (3P2D)	vLLM	GB300	128	1,356.51	82.76	649
SWEBench-Pro	Aggregated (4 workers)	TensorRT-LLM	B200	80	995	78.98	655
SWEBench-Pro	Disaggregated (2P2D)	TensorRT-LLM	B200	80	1,119.13	144.61	2,366
SWEBench-Pro	Aggregated (11 workers)	vLLM	VR200	616	2,969	96.6	490
SWEBench-Pro	Disaggregated (7P4D)	vLLM	VR200	528	3,276	155.9	1,120
SWEBench-Pro	Disaggregated (3P2D)	vLLM	B300	160	1,317	68.1	1,390

The VR200 rows replay SWE-bench Pro agentic multi-turn traces. Throughput is raw output tokens. Each VR200 row is the highest output per GPU with at least 50 output tokens/s per user (P50) and TTFT P50 under 5 s. The VR200 results used a synthetic MTP acceptance length of 4.063 with five draft tokens. The VR200 recipes default to natural acceptance; the synthetic setting is a commented line. VR200 artifacts are not published. At the same GPU count, 7P4D gives +10.3% output per GPU (observed across separate runs). The 7P4D row ran on the previous image with FlashInfer #5267 and vLLM #58231 added, plus a runtime patch for the KV split that FlashInfer #6331 and vLLM #61077 provide. The aggregated row ran on the image in the manifest.

Compare All Targets

GPU counts below use the standalone manifests. P counts prefill workers. D counts decode workers.

Framework	GPU	Topology	Total GPUs	Parallelism	Precision	Routing	Speculative Decoding	Context length
vLLM	H100	2 workers	4	TP2 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	H100	1P1D	4	TP2 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	H200	2 workers	2	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	H200	1P1D	2	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	B200	4 workers	4	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	B200	3P2D	5	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	GB200	4 workers	4	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	GB200	3P2D	5	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP3	1M
vLLM	B300	4 workers	4	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	B300	3P2D	5	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP3	1M
vLLM	GB300	4 workers	4	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	GB300	3P2D	5	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP3	1M
vLLM	VR200	11 workers	11	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
vLLM	VR200	7P4D	11	TP1 per worker	NVFP4 weights, FP8 KV	KV-aware	MTP5	1M
TensorRT-LLM	B200	4 workers	4	TP1 per worker	NVFP4 weights, FP8 KV	Load + session affinity	MTP3	1M
TensorRT-LLM	B200	2P2D	4	TP1 per worker	NVFP4 weights, FP8 KV	Load + session affinity	MTP3	1M