> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Qwen3.6-35B-A3B These aggregated recipes serve the agentic workload with `nvidia/Qwen3.6-35B-A3B-NVFP4` on B200 or GB200 and `Qwen/Qwen3.6-35B-A3B-FP8` on H200. Choose your deployment target GPU B200 GB200 H200 **Checkpoint** nvidia/Qwen3.6-35B-A3B-NVFP4 **Precision** NVFP4 weights + FP8 KV cache **GPUs** 1x B200 **Runtime** SGLang aggregate worker **Spec decode** MTP, 3 tokens **Checkpoint** Qwen/Qwen3.6-35B-A3B-FP8 **Precision** FP8 weights, BF16 KV **GPUs** 1x H200 **Runtime** SGLang aggregate worker **Spec decode** MTP, 3 tokens **Checkpoint** nvidia/Qwen3.6-35B-A3B-NVFP4 **Precision** NVFP4 weights + FP8 KV cache **GPUs** 1x GB200 **Runtime** SGLang aggregate worker **Spec decode** MTP, 3 tokens ## Prerequisites * A Kubernetes cluster with the Dynamo platform and DGD CRDs installed. * A namespace with access to the selected GPU target. * A ReadWriteMany `model-cache` PVC. Set `storageClassName` in the model-cache manifest before deployment. The checkpoint is downloaded from Hugging Face by the selected model-download Job. Add a Hugging Face token only if your environment requires authenticated access. Edit namespace, storage class, image tags, node selectors, resource claims, and cluster-specific placement in the manifests before applying them. ## Deploy Create the model cache and download the selected checkpoint: ```bash export NAMESPACE=your-namespace kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-download-nvfp4.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download-nvfp4 -n ${NAMESPACE} --timeout=7200s ``` ```bash export NAMESPACE=your-namespace kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-download-nvfp4.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download-nvfp4 -n ${NAMESPACE} --timeout=7200s ``` ```bash export NAMESPACE=your-namespace kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-download-fp8.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download-fp8 -n ${NAMESPACE} --timeout=7200s ``` Deploy the selected target: ```bash kubectl apply -f recipes/qwen3.6-35b-a3b/sglang/agg-b200-mtp/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.6-35b-a3b/sglang/agg-gb200-mtp/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.6-35b-a3b/sglang/agg-h200-mtp/deploy.yaml -n ${NAMESPACE} ``` ## Smoke Test Forward the frontend service and verify the model: ```bash kubectl port-forward svc/-frontend 8000:8000 -n ${NAMESPACE} curl http://localhost:8000/v1/models ``` Send a chat request: ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}' ``` Forward the frontend service and verify the model: ```bash kubectl port-forward svc/-frontend 8000:8000 -n ${NAMESPACE} curl http://localhost:8000/v1/models ``` Send a chat request: ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}' ``` Qwen3.6 also accepts image input. Send an `image_url` content part: ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image."},{"type":"image_url","image_url":{"url":"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/fern/assets/img/frontpage-banner.png"}}]}],"max_tokens":1024}' ``` ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image."},{"type":"image_url","image_url":{"url":"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/fern/assets/img/frontpage-banner.png"}}]}],"max_tokens":1024}' ``` ## Benchmark These results use [AIPerf](https://github.com/ai-dynamo/aiperf) 0.10.0 and the [`64k_400_90kv_agent_new_noschedule_short_15perc.jsonl`](https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k2.6/perf/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl) trace. The workload uses 64K input tokens, 400 output tokens, 90% KV reuse, concurrency 32, and 3,541 trace requests. The trace contains 131 requests expected to exceed the model context length and fail. You can fetch the Git LFS-backed trace before running the benchmark: ```bash # Required to pull the 64K trace used by the benchmark. git lfs install && git lfs pull ``` All measurements use one GPU. Run one separate 32-request pass using the same trace and options before the measured phase for warmup. ```bash aiperf profile \ -m nvidia/Qwen3.6-35B-A3B-NVFP4 \ --tokenizer nvidia/Qwen3.6-35B-A3B-NVFP4 \ --input-file /data/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --url http://-frontend:8000 \ --endpoint-type chat \ --streaming --use-server-token-count \ --extra-inputs ignore_eos:true \ --concurrency 32 --workers-max 32 \ --random-seed 42 --ui none \ --tokenizer-trust-remote-code \ --request-timeout-seconds 1200 ``` ```bash aiperf profile \ -m Qwen/Qwen3.6-35B-A3B-FP8 \ --tokenizer Qwen/Qwen3.6-35B-A3B-FP8 \ --input-file /data/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl \ --custom-dataset-type mooncake_trace \ --num-requests 3541 \ --url http://-frontend:8000 \ --endpoint-type chat \ --streaming --use-server-token-count \ --extra-inputs ignore_eos:true \ --concurrency 32 --workers-max 32 \ --random-seed 42 --ui none \ --tokenizer-trust-remote-code \ --request-timeout-seconds 1200 ``` The MTP results use three speculative tokens with a synthetic acceptance length of 3.3153. This value was calculated for the coding workload in SpeedBench. The full trace produced 131 expected context-length errors. When benchmarking with synthetic acceptance, uncomment the `SGLANG_SIMULATE_ACC_*` variables in the manifest. `SGLANG_SIMULATE_ACC_TOKEN_MODE` must be `real-draft-token`: with the default fixed fake token id SGLang emits id 100, which is not a valid token in the Qwen tokenizer, so the frontend drops the response and inter-token latency reports 0. When benchmarking SGLang with synthetic acceptance, set `SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token` together with `SGLANG_SIMULATE_ACC_LEN` and `SGLANG_SIMULATE_ACC_METHOD`. SGLang 0.5.16 uses fixed token ID 100 when the token mode is omitted. That token decodes to the Unicode replacement character for the Qwen tokenizer, so Dynamo buffers the generated text until the response finishes and AIPerf cannot calculate inter-token latency or output throughput per user. ## Expected Performance Measured with 64K input tokens, 400 output tokens, and 90% KV reuse at concurrency 32 with 3,541 trace requests. MTP runs with simulated acceptance, `acc_len=3.3153` from SpeedBench coding, not acceptance measured on this agentic trace. The 32-request warmup is a separate pass and is not included in these figures:
Recipe Model GPU Passed Failed OUTPUT TOK/S/GPU Average tok/s/user P50 tok/s/user P50 ITL P50 TTFT
`sglang/agg-b200-mtp/deploy.yaml` NVFP4 B200 3,410 131 2,988.50 115.81 111.98 8.93 ms 286.90 ms
`sglang/agg-gb200-mtp/deploy.yaml` NVFP4 GB200 3,410 131 3,035.98 114.03 111.47 8.97 ms 156.07 ms
`sglang/agg-h200-mtp/deploy.yaml` FP8 H200 3,410 131 1,332.03 61.90 57.75 17.32 ms 360.31 ms
## Expected Accuracy GPQA Diamond (`gpqa_diamond_aa_v3`) via NVIDIA Eval Factory
Recipe Model GPU GPQA Diamond Baseline
`sglang/agg-b200-mtp/deploy.yaml` NVFP4 B200 79.23% 79.10%
`sglang/agg-gb200-mtp/deploy.yaml` NVFP4 GB200 79.23% 79.48%
`sglang/agg-h200-mtp/deploy.yaml` FP8 H200 81.44% 80.24%
## Source * [B200 SGLang deployment manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.6-35b-a3b/sglang/agg-b200-mtp/deploy.yaml) * [GB200 SGLang deployment manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.6-35b-a3b/sglang/agg-gb200-mtp/deploy.yaml) * [H200 SGLang deployment manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.6-35b-a3b/sglang/agg-h200-mtp/deploy.yaml) * [NVFP4 model-download manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.6-35b-a3b/model-cache/model-download-nvfp4.yaml) * [FP8 model-download manifest](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.6-35b-a3b/model-cache/model-download-fp8.yaml) * [Model-cache manifests](https://github.com/ai-dynamo/dynamo/tree/main/recipes/qwen3.6-35b-a3b/model-cache)