Model Recipes

Qwen3.8-2.4T-A95B

Serve Qwen3.8-2.4T-A95B with Dynamo and vLLM or SGLang on GB300 or GB200, aggregated or disaggregated.

View as Markdown

Each target below is a Dynamo + vLLM or SGLang deployment of Qwen3.8-2.4T-A95B — a hybrid gated-delta-net + MoE model with 512 experts and a 262k-token context — with KV-aware routing and FP8 weights/KV cache. Tensor parallelism runs over MNNVL through Kubernetes ComputeDomains. Pick your GPU architecture, serving topology, and inference engine; every command on this page updates to match.

Choose your deployment target

GPU
Topology
Engine
Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8)Precision FP8 weights, FP8 KVGPUs 16x GB300 (4 nodes), 1 replicaParallelism TP16 over MNNVLRouting event-driven KV-aware, prefix caching
Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8)Precision FP8 weights, FP8 KVGPUs 16x GB300 prefill (4 nodes) + 16x GB300 decode (4 nodes)Parallelism TP16 / TP16 over MNNVLKV transfer NIXL over cuda_ipc + MNNVLRouting KV-aware, load-balanced prefill; NIXL KV transfer
Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8)Precision FP8 weights, FP8 KVGPUs 16x GB200 (4 nodes), 1 replicaParallelism TP16 over MNNVLMoE backend flashinfer_trtllmRouting KV-aware (ZMQ events)
Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8)Precision FP8 weights, FP8 KVGPUs 16x GB200 (4 nodes), 1 replicaParallelism TP16 over MNNVLMoE backend flashinfer_trtllm; trtllm_mha attentionRouting KV-aware (ZMQ events)
Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8)Precision FP8 weights, FP8 KVGPUs 16x GB200 × 2 prefill + 16x GB200 decode (2P1D, 12 nodes)Parallelism TP16 / TP16 over MNNVLMoE backend flashinfer_trtllm; trtllm_mha attentionRouting KV-aware, load-balanced across prefill replicas; NIXL KV transfer over cuda_ipc + MNNVL

Prerequisites

  • A Kubernetes cluster with the Dynamo platform installed and GB300 GPUs available — 16x (4 nodes). See the Kubernetes Deployment Guide.
  • The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
  • A Hugging Face token with access to Qwen/Qwen3.8-2.4T-A95B.
  • A Kubernetes cluster with the Dynamo platform installed and GB200 GPUs available — 16x (4 nodes) for aggregated; 48x (12 nodes, 2×4 prefill + 4 decode) for 2P1D disaggregated. See the Kubernetes Deployment Guide.
  • The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink).
  • A Hugging Face token with access to Qwen/Qwen3.8-2.4T-A95B.
  • The model-cache PVC, populated by the model-download Job (see step 3 below).

Create the namespace:

$export NAMESPACE=your-namespace
$kubectl create namespace ${NAMESPACE}

Create the Hugging Face token secret:

$kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN="your-token" -n ${NAMESPACE}

Edit storageClassName in model-cache/model-cache.yaml to a ReadWriteMany storage class on your cluster (kubectl get storageclass) before applying it. Review namespace, image tag, node selectors, and resource claims in the manifests as well.

Deploy

Stage the weights into the cache, then deploy:

$kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s

Stage the weights into the model-cache PVC, then deploy:

$kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-cache.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-download.yaml -n ${NAMESPACE}
$kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s

Then deploy the target DGD:

Workload
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb300-chat/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/agg-gb300-chat/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb300-chat/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb200-chat/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/agg-gb200-chat/deploy.yaml -n ${NAMESPACE}
$kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb200-chat/deploy.yaml -n ${NAMESPACE}

First worker launch loads ~2.4 TB of FP8 weights across the TP ranks and warms CUDA graphs. Allow up to 4 hours; the multinode pod set reports ready only once all nodes have joined the TP group.

Smoke Test

Forward the frontend port:

$kubectl port-forward svc/qwen38max-agg-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/sglang-qwen38max-1p1d-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/qwen38max-agg-frontend 8000:8000 -n ${NAMESPACE}
$kubectl port-forward svc/qwen38max-sgl-disagg-frontend 8000:8000 -n ${NAMESPACE}

Send a test request:

$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"Qwen/Qwen3.8-2.4T-A95B","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'
$curl http://localhost:8000/v1/chat/completions \
>  -H 'Content-Type: application/json' \
>  -d '{"model":"Qwen/Qwen3.8-2.4T-A95B","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}'

Qwen3.8-2.4T-A95B reasons before answering; chain-of-thought lands in choices[0].message.reasoning_content and the answer in choices[0].message.content. The deployment wires the qwen3 reasoning parser and qwen3_coder tool-call parser at the frontend.

Benchmark

Standard chat workload: 8k total context / 1k output, 70% KV cache reuse.

$BASE_URL="http://localhost:8000"
$SALT=$(openssl rand -base64 4 | tr -d '=+/')
$for C in 2 4 8 16 32 64 128; do
$  aiperf profile \
>    --model Qwen/Qwen3.8-2.4T-A95B \
>    --url "${BASE_URL}" \
>    --endpoint-type chat --streaming \
>    --shared-system-prompt-length 5734 \
>    --isl 2458 --osl 1024 \
>    --concurrency ${C} --workers-max ${C} \
>    --num-requests $((C*3)) --num-prompts $((C*3)) \
>    --extra-inputs ignore_eos:true \
>    --extra-inputs "cache_salt:salt_${SALT}_c${C}" \
>    --request-timeout-seconds 300 \
>    --warmup-request-count 1 \
>    --tokenizer builtin --use-server-token-count
$done

Compare All Targets

Aggregated targets support both vLLM and SGLang. Disaggregated targets are SGLang only. GB300 disagg is 1P1D; GB200 disagg is 2P1D.

GB300 aggGB300 disagg (SGLang)GB200 aggGB200 disagg (SGLang, 2P1D)
GPUs16× GB300, 4 nodes16× GB300 prefill + 16× GB300 decode (1P1D, 8 nodes)16× GB200, 4 nodes2× 16× GB200 prefill + 16× GB200 decode (2P1D, 12 nodes)
ParallelismTP16TP16 / TP16TP16TP16 / TP16
KV transfer—NIXL over cuda_ipc + MNNVL—NIXL over cuda_ipc + MNNVL
MoE backendflashinfer_trtllmflashinfer_trtllmflashinfer_trtllmflashinfer_trtllm
PrecisionFP8 weights, FP8 KVFP8 weights, FP8 KVFP8 weights, FP8 KVFP8 weights, FP8 KV
Context length262 144262 144262 144262 144

Notes

  • Event-driven KV-aware routing. Both workers publish KV events (--kv-events-config over ZMQ), and the frontend enables --router-kv-events; otherwise the disaggregated route cannot be verified.
  • --no-async-scheduling is required for the shipped fixed shapes.
  • MNNVL all-reduce. VLLM_ALLREDUCE_USE_FLASHINFER=1 with VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl. Keep VLLM_USE_NCCL_SYMM_MEM=0 alongside it.
  • GB300 P-to-D fabric. The disaggregated workers select the DRA-injected RoCE HCA and use UCX_TLS=rc_x,rc,cuda_copy,cuda_ipc with Ethernet GIDs. This is RC/GDR for NIXL payloads; MNNVL remains the TP all-reduce fabric.

Source