Model Recipes

Qwen3.6-35B-A3B

Serve Qwen3.6-35B-A3B with Dynamo on B200, GB200, or H200.

View as Markdown

These aggregated recipes serve the agentic workload with nvidia/Qwen3.6-35B-A3B-NVFP4 on B200 or GB200 and Qwen/Qwen3.6-35B-A3B-FP8 on H200.

Choose your deployment target

GPU
Checkpoint nvidia/Qwen3.6-35B-A3B-NVFP4Precision NVFP4 weights + FP8 KV cacheGPUs 1x B200Runtime SGLang aggregate workerSpec decode MTP, 3 tokens
Checkpoint Qwen/Qwen3.6-35B-A3B-FP8Precision FP8 weights, BF16 KVGPUs 1x H200Runtime SGLang aggregate workerSpec decode MTP, 3 tokens
Checkpoint nvidia/Qwen3.6-35B-A3B-NVFP4Precision NVFP4 weights + FP8 KV cacheGPUs 1x GB200Runtime SGLang aggregate workerSpec decode MTP, 3 tokens

Prerequisites

  • A Kubernetes cluster with the Dynamo platform and DGD CRDs installed.
  • A namespace with access to the selected GPU target.
  • A ReadWriteMany model-cache PVC. Set storageClassName in the model-cache manifest before deployment.

The checkpoint is downloaded from Hugging Face by the selected model-download Job. Add a Hugging Face token only if your environment requires authenticated access.

Edit namespace, storage class, image tags, node selectors, resource claims, and cluster-specific placement in the manifests before applying them.

Deploy

Create the model cache and download the selected checkpoint:

$export NAMESPACE=your-namespace
$kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-download-nvfp4.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download-nvfp4 -n ${NAMESPACE} --timeout=7200s
$export NAMESPACE=your-namespace
$kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-download-nvfp4.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download-nvfp4 -n ${NAMESPACE} --timeout=7200s
$export NAMESPACE=your-namespace
$kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.6-35b-a3b/model-cache/model-download-fp8.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download-fp8 -n ${NAMESPACE} --timeout=7200s

Deploy the selected target:

$kubectl apply -f recipes/qwen3.6-35b-a3b/sglang/agg-b200-mtp/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.6-35b-a3b/sglang/agg-gb200-mtp/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.6-35b-a3b/sglang/agg-h200-mtp/deploy.yaml -n ${NAMESPACE}

Smoke Test

Forward the frontend service and verify the model:

$kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}
$curl http://localhost:8000/v1/models

Send a chat request:

$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Forward the frontend service and verify the model:

$kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}
$curl http://localhost:8000/v1/models

Send a chat request:

$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Qwen3.6 also accepts image input. Send an image_url content part:

$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image."},{"type":"image_url","image_url":{"url":"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/fern/assets/img/frontpage-banner.png"}}]}],"max_tokens":1024}'
$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image."},{"type":"image_url","image_url":{"url":"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/fern/assets/img/frontpage-banner.png"}}]}],"max_tokens":1024}'

Benchmark

These results use AIPerf 0.10.0 and the 64k_400_90kv_agent_new_noschedule_short_15perc.jsonl trace. The workload uses 64K input tokens, 400 output tokens, 90% KV reuse, concurrency 32, and 3,541 trace requests. The trace contains 131 requests expected to exceed the model context length and fail. You can fetch the Git LFS-backed trace before running the benchmark:

$# Required to pull the 64K trace used by the benchmark.
$git lfs install && git lfs pull

All measurements use one GPU.

Run one separate 32-request pass using the same trace and options before the measured phase for warmup.

$aiperf profile \
>  -m nvidia/Qwen3.6-35B-A3B-NVFP4 \
>  --tokenizer nvidia/Qwen3.6-35B-A3B-NVFP4 \
>  --input-file /data/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl \
>  --custom-dataset-type mooncake_trace \
>  --num-requests 3541 \
>  --url http://<deployment>-frontend:8000 \
>  --endpoint-type chat \
>  --streaming --use-server-token-count \
>  --extra-inputs ignore_eos:true \
>  --concurrency 32 --workers-max 32 \
>  --random-seed 42 --ui none \
>  --tokenizer-trust-remote-code \
>  --request-timeout-seconds 1200
$aiperf profile \
>  -m Qwen/Qwen3.6-35B-A3B-FP8 \
>  --tokenizer Qwen/Qwen3.6-35B-A3B-FP8 \
>  --input-file /data/traces/64k_400_90kv_agent_new_noschedule_short_15perc.jsonl \
>  --custom-dataset-type mooncake_trace \
>  --num-requests 3541 \
>  --url http://<deployment>-frontend:8000 \
>  --endpoint-type chat \
>  --streaming --use-server-token-count \
>  --extra-inputs ignore_eos:true \
>  --concurrency 32 --workers-max 32 \
>  --random-seed 42 --ui none \
>  --tokenizer-trust-remote-code \
>  --request-timeout-seconds 1200

The MTP results use three speculative tokens with a synthetic acceptance length of 3.3153. This value was calculated for the coding workload in SpeedBench. The full trace produced 131 expected context-length errors.

When benchmarking with synthetic acceptance, uncomment the SGLANG_SIMULATE_ACC_* variables in the manifest. SGLANG_SIMULATE_ACC_TOKEN_MODE must be real-draft-token: with the default fixed fake token id SGLang emits id 100, which is not a valid token in the Qwen tokenizer, so the frontend drops the response and inter-token latency reports 0.

When benchmarking SGLang with synthetic acceptance, set SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token together with SGLANG_SIMULATE_ACC_LEN and SGLANG_SIMULATE_ACC_METHOD. SGLang 0.5.16 uses fixed token ID 100 when the token mode is omitted. That token decodes to the Unicode replacement character for the Qwen tokenizer, so Dynamo buffers the generated text until the response finishes and AIPerf cannot calculate inter-token latency or output throughput per user.

Expected Performance

Measured with 64K input tokens, 400 output tokens, and 90% KV reuse at concurrency 32 with 3,541 trace requests. MTP runs with simulated acceptance, acc_len=3.3153 from SpeedBench coding, not acceptance measured on this agentic trace. The 32-request warmup is a separate pass and is not included in these figures:

RecipeModelGPUPassedFailedOUTPUT TOK/S/GPUAverage tok/s/userP50 tok/s/userP50 ITLP50 TTFT
sglang/agg-b200-mtp/deploy.yamlNVFP4B2003,4101312,988.50115.81111.988.93 ms286.90 ms
sglang/agg-gb200-mtp/deploy.yamlNVFP4GB2003,4101313,035.98114.03111.478.97 ms156.07 ms
sglang/agg-h200-mtp/deploy.yamlFP8H2003,4101311,332.0361.9057.7517.32 ms360.31 ms

Expected Accuracy

GPQA Diamond (gpqa_diamond_aa_v3) via NVIDIA Eval Factory

RecipeModelGPUGPQA DiamondBaseline
sglang/agg-b200-mtp/deploy.yamlNVFP4B20079.23%79.10%
sglang/agg-gb200-mtp/deploy.yamlNVFP4GB20079.23%79.48%
sglang/agg-h200-mtp/deploy.yamlFP8H20081.44%80.24%

Source