Choose your deployment target
GPU GB300 GB200 Topology Aggregated Disaggregated Engine vLLM SGLang Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8) Precision FP8 weights, FP8 KV GPUs 16x GB300 (4 nodes), 1 replica Parallelism TP16 over MNNVL Routing event-driven KV-aware, prefix caching Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8) Precision FP8 weights, FP8 KV GPUs 16x GB300 prefill (4 nodes) + 16x GB300 decode (4 nodes) Parallelism TP16 / TP16 over MNNVL KV transfer NIXL over cuda\_ipc + MNNVL Routing KV-aware, load-balanced prefill; NIXL KV transfer Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8) Precision FP8 weights, FP8 KV GPUs 16x GB200 (4 nodes), 1 replica Parallelism TP16 over MNNVL MoE backend flashinfer\_trtllm Routing KV-aware (ZMQ events) Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8) Precision FP8 weights, FP8 KV GPUs 16x GB200 (4 nodes), 1 replica Parallelism TP16 over MNNVL MoE backend flashinfer\_trtllm; trtllm\_mha attention Routing KV-aware (ZMQ events) Checkpoint Qwen/Qwen3.8-2.4T-A95B (FP8) Precision FP8 weights, FP8 KV GPUs 16x GB200 × 2 prefill + 16x GB200 decode (2P1D, 12 nodes) Parallelism TP16 / TP16 over MNNVL MoE backend flashinfer\_trtllm; trtllm\_mha attention Routing KV-aware, load-balanced across prefill replicas; NIXL KV transfer over cuda\_ipc + MNNVL ## Prerequisites * A Kubernetes cluster with the Dynamo platform installed and GB300 GPUs available — 16x (4 nodes). See the [Kubernetes Deployment Guide](../kubernetes/quickstart.mdx). * The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). * A Hugging Face token with access to `Qwen/Qwen3.8-2.4T-A95B`. - A Kubernetes cluster with the Dynamo platform installed and GB200 GPUs available — 16x (4 nodes) for aggregated; 48x (12 nodes, 2×4 prefill + 4 decode) for 2P1D disaggregated. See the [Kubernetes Deployment Guide](../kubernetes/quickstart.mdx). - The NVIDIA DRA driver with ComputeDomain support installed (required for multi-node NVLink). - A Hugging Face token with access to `Qwen/Qwen3.8-2.4T-A95B`. - The `model-cache` PVC, populated by the model-download Job (see step 3 below). Create the namespace: ```bash export NAMESPACE=your-namespace kubectl create namespace ${NAMESPACE} ``` Create the Hugging Face token secret: ```bash kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN="your-token" -n ${NAMESPACE} ``` Edit `storageClassName` in `model-cache/model-cache.yaml` to a ReadWriteMany storage class on your cluster (`kubectl get storageclass`) before applying it. Review namespace, image tag, node selectors, and resource claims in the manifests as well. ## Deploy Stage the weights into the cache, then deploy: ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-download.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s ``` Stage the weights into the model-cache PVC, then deploy: ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/qwen3.8-2.4t-a95b/model-cache/model-download.yaml -n ${NAMESPACE} kubectl wait --for=condition=Complete job/model-download -n ${NAMESPACE} --timeout=14400s ``` Then deploy the target DGD: Workload Chat Agentic ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb300-chat/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb300-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/agg-gb300-chat/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb300-chat/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb200-chat/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/vllm/agg-gb200-agentic/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/agg-gb200-chat/deploy.yaml -n ${NAMESPACE} ``` ```bash kubectl apply -f recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb200-chat/deploy.yaml -n ${NAMESPACE} ``` First worker launch loads \~2.4 TB of FP8 weights across the TP ranks and warms CUDA graphs. Allow up to 4 hours; the multinode pod set reports ready only once all nodes have joined the TP group. ## Smoke Test Forward the frontend port: ```bash kubectl port-forward svc/qwen38max-agg-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/sglang-qwen38max-1p1d-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/qwen38max-agg-frontend 8000:8000 -n ${NAMESPACE} ``` ```bash kubectl port-forward svc/qwen38max-sgl-disagg-frontend 8000:8000 -n ${NAMESPACE} ``` Send a test request: ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen/Qwen3.8-2.4T-A95B","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}' ``` ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"Qwen/Qwen3.8-2.4T-A95B","messages":[{"role":"user","content":"Write a one-sentence readiness check."}],"max_tokens":64}' ``` Qwen3.8-2.4T-A95B reasons before answering; chain-of-thought lands in `choices[0].message.reasoning_content` and the answer in `choices[0].message.content`. The deployment wires the `qwen3` reasoning parser and `qwen3_coder` tool-call parser at the frontend. ## Benchmark Standard chat workload: 8k total context / 1k output, 70% KV cache reuse. ```bash BASE_URL="http://localhost:8000" SALT=$(openssl rand -base64 4 | tr -d '=+/') for C in 2 4 8 16 32 64 128; do aiperf profile \ --model Qwen/Qwen3.8-2.4T-A95B \ --url "${BASE_URL}" \ --endpoint-type chat --streaming \ --shared-system-prompt-length 5734 \ --isl 2458 --osl 1024 \ --concurrency ${C} --workers-max ${C} \ --num-requests $((C*3)) --num-prompts $((C*3)) \ --extra-inputs ignore_eos:true \ --extra-inputs "cache_salt:salt_${SALT}_c${C}" \ --request-timeout-seconds 300 \ --warmup-request-count 1 \ --tokenizer builtin --use-server-token-count done ``` ## Compare All Targets Aggregated targets support both vLLM and SGLang. Disaggregated targets are SGLang only. GB300 disagg is 1P1D; GB200 disagg is 2P1D. | | GB300 agg | GB300 disagg (SGLang) | GB200 agg | GB200 disagg (SGLang, 2P1D) | | ------------------ | ------------------- | ---------------------------------------------------- | ------------------- | -------------------------------------------------------- | | **GPUs** | 16× GB300, 4 nodes | 16× GB300 prefill + 16× GB300 decode (1P1D, 8 nodes) | 16× GB200, 4 nodes | 2× 16× GB200 prefill + 16× GB200 decode (2P1D, 12 nodes) | | **Parallelism** | TP16 | TP16 / TP16 | TP16 | TP16 / TP16 | | **KV transfer** | — | NIXL over cuda\_ipc + MNNVL | — | NIXL over cuda\_ipc + MNNVL | | **MoE backend** | flashinfer\_trtllm | flashinfer\_trtllm | flashinfer\_trtllm | flashinfer\_trtllm | | **Precision** | FP8 weights, FP8 KV | FP8 weights, FP8 KV | FP8 weights, FP8 KV | FP8 weights, FP8 KV | | **Context length** | 262 144 | 262 144 | 262 144 | 262 144 | ## Notes * **Event-driven KV-aware routing.** Both workers publish KV events (`--kv-events-config` over ZMQ), and the frontend enables `--router-kv-events`; otherwise the disaggregated route cannot be verified. * **`--no-async-scheduling`** is required for the shipped fixed shapes. * **MNNVL all-reduce.** `VLLM_ALLREDUCE_USE_FLASHINFER=1` with `VLLM_FLASHINFER_ALLREDUCE_BACKEND=mnnvl`. Keep `VLLM_USE_NCCL_SYMM_MEM=0` alongside it. * **GB300 P-to-D fabric.** The disaggregated workers select the DRA-injected RoCE HCA and use `UCX_TLS=rc_x,rc,cuda_copy,cuda_ipc` with Ethernet GIDs. This is RC/GDR for NIXL payloads; MNNVL remains the TP all-reduce fabric. ## Source * Source README: [recipes/qwen3.8-2.4t-a95b/README.md](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/README.md) * GB300 aggregated chat (vLLM): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/vllm/agg-gb300-chat/deploy.yaml) * GB300 aggregated agentic (vLLM): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/vllm/agg-gb300-agentic/deploy.yaml) * GB200 aggregated chat (vLLM): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/vllm/agg-gb200-chat/deploy.yaml) * GB200 aggregated agentic (vLLM): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/vllm/agg-gb200-agentic/deploy.yaml) * GB300 aggregated chat (SGLang): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/sglang/agg-gb300-chat/deploy.yaml) * GB300 disaggregated chat (SGLang): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb300-chat/deploy.yaml) * GB200 aggregated chat (SGLang): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/sglang/agg-gb200-chat/deploy.yaml) * GB200 disaggregated chat (SGLang): [deploy.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/sglang/disagg-gb200-chat/deploy.yaml) * Setup assets: [model-cache.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/model-cache/model-cache.yaml) and [model-download.yaml](https://github.com/ai-dynamo/dynamo/blob/main/recipes/qwen3.8-2.4t-a95b/model-cache/model-download.yaml)