Deploy with DGD

Choose your model

Pick the model you want to serve from HuggingFace. This guide uses Qwen3-32B as the running example; substitute your own anywhere you see it.

A few values change per environment — the target namespace, the runtime image, and your HuggingFace token. Rather than hard-code them, keep them in shell variables and fill them into the spec at apply time with envsubst (from the gettext package). Export the two you know now — you pick the image in the next step:

$
export NAMESPACE=your-namespace
$
export HF_TOKEN=hf_xxxxxxxx                 # your HuggingFace token

Gated or rate-limited models need the token as a Secret in your namespace. Create it once from $HF_TOKEN:

$
kubectl create secret generic hf-token-secret \
>
  --from-literal=HF_TOKEN=${HF_TOKEN} \
>
  -n ${NAMESPACE}

Start the spec with a Frontend and one worker. The ${NAMESPACE} and ${RUNTIME_IMAGE} placeholders are expanded by envsubst when you apply (see the Apply step); the worker pulls the token from the Secret via envFrom, and the model name is passed to the engine as an argument. The worker’s launch command is backend-specific — this guide defaults to vLLM; select your backend and the choice syncs across every code block on this page (you set the matching image in the next step):

1
apiVersion: nvidia.com/v1beta1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: qwen3-32b-agg
5
  namespace: ${NAMESPACE}
6
spec:
7
  components:
8
  - name: Frontend
9
    type: frontend
10
    replicas: 1
11
    podTemplate:
12
      spec:
13
        containers:
14
        - name: main
15
          image: ${RUNTIME_IMAGE}
16
  - name: VllmWorker
17
    type: worker
18
    replicas: 1
19
    podTemplate:
20
      spec:
21
        containers:
22
        - name: main
23
          image: ${RUNTIME_IMAGE}
24
          envFrom:
25
          - secretRef:
26
              name: hf-token-secret
27
          command:
28
          - python3
29
          - -m
30
          - dynamo.vllm
31
          args:
32
          - --model
33
          - Qwen/Qwen3-32B

Substitute Qwen/Qwen3-32B with your model, hf-token-secret with your Secret name if you changed it. Keeping the namespace, image, and token in the environment means the same YAML file works across clusters — you change the exports, not the spec.

Choose your backend

The worker in the previous step runs vLLM. Dynamo also supports SGLang and TensorRT-LLM, and the backend you picked there already synced every code block on this page. The backend sets three things: the runtime image, the launch module (dynamo.vllm / dynamo.sglang / dynamo.trtllm), and how you pass the model and parallelism. Export the matching image so envsubst fills the ${RUNTIME_IMAGE} placeholder on both the Frontend and the worker:

$
export RUNTIME_IMAGE=nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1

The launch module and the way each engine takes the model differ, which is why the worker block changed when you switched tabs above:

vLLM and SGLang take tensor/pipeline/data parallelism as CLI flags (covered in the topology step). TensorRT-LLM is the exception: it reads all parallelism from the engine-config YAML referenced by --extra-engine-args, not from worker flags.

You normally do not need to set the top-level spec.backendFramework field — the operator infers the backend from the worker command. Set it explicitly (vllm, sglang, or trtllm) only when a feature needs the framework known up front, such as GMS failover or multinode TensorRT-LLM.

For per-backend setup and tuning, see vLLM, SGLang, and TensorRT-LLM.

Choose aggregated or disaggregated serving

Before sizing hardware, decide how prefill (prompt processing) and decode (token generation) are placed. Both patterns work with the same DGD spec — you are choosing how many worker roles it has.

• Aggregated — the single worker you have been building does both phases. Simplest to deploy and debug; a good default for smaller models and uniform traffic.
• Disaggregated — separate prefill and decode worker pools, each sized, scaled, and parallelized independently, with the KV cache transferred between them over the network. Reach for it when prefill and decode have different bottlenecks — long prompts saturating prefill while decode sits idle, or high concurrency saturating decode.

If you are staying aggregated, keep the single worker and continue to the next step. To go disaggregated, replace the one worker with two, tagged by role. The tabs below use vLLM as a concrete example to highlight the structural difference — one worker versus a prefill/decode pair; SGLang and TensorRT-LLM follow the same pattern with their own flags:

1
  components:
2
  - name: Frontend
3
    type: frontend
4
  - name: VllmWorker
5
    type: worker
6
    podTemplate:
7
      spec:
8
        containers:
9
        - name: main
10
          command: [python3, -m, dynamo.vllm]
11
          args:
12
          - --model
13
          - Qwen/Qwen3-32B

In the disaggregated spec, type: prefill and type: decode tag the roles, the KV cache moves over NIXL (--kv-transfer-config), and sharedMemorySize is raised for the transfer. When you disaggregate, the next step’s sizing applies to prefill and decode separately — you pick TP/PP for each role.

The fastest way to start is to copy a ready-to-apply Kubernetes template rather than type the spec from scratch. Choose the vLLM, SGLang, or TensorRT-LLM templates in the Recipes tab, then adapt the model, image, and parallelism to your case.

For disaggregation-specific configuration — RDMA resources, UCX environment variables, and prefill/decode scaling — see Disaggregated Serving.

Determine topology and parallelism

Now decide how many GPUs each worker needs and how to split the model across them. (If you disaggregated in the previous step, answer this once for the prefill worker and once for the decode worker — they size independently.)

Will it fit? A rough lower bound for the weights is parameters × bytes-per-parameter: 2 bytes for BF16/FP16, 1 for FP8. Qwen3-32B in BF16 is ~64 GB of weights, so it needs at least two 40 GB GPUs or one 80 GB GPU — and that is before the KV cache, which grows with context length and concurrency. Leave headroom.

The parallelism knobs. A worker’s total GPUs equals the product of its parallel sizes; when it spans machines, that product also equals multinode.nodeCount × per-node GPUs. Five knobs do different jobs:

• Tensor parallelism (TP) splits every layer’s weight matrices across GPUs so one model copy spans them, cutting memory per GPU. The split is communication-heavy (an all-reduce every layer), so TP wants fast intra-node interconnect (NVLink) — keep it within a node.
• Pipeline parallelism (PP) splits the model into sequential stages — contiguous blocks of layers, one block per GPU or node. It also cuts memory per GPU, but only passes activations at stage boundaries, so it tolerates slower links and is how you span across nodes.
• Data parallelism (DP) / replicas runs multiple independent model copies for throughput. It does not help a model fit — each copy still needs enough GPUs to hold the model. Scale whole pods with replicas, or use the engine’s own --data-parallel-size (common with MoE attention).
• Expert parallelism (EP) applies to Mixture-of-Experts models only: it spreads a layer’s experts across GPUs so no single GPU holds them all.
• multinode.nodeCount is infrastructure, not a model split: it tells the operator how many physical nodes one worker spans. You set it when TP×PP exceeds the GPUs on a single node.

The exact flag differs per backend. vLLM and SGLang take them as CLI args on the worker; TensorRT-LLM reads them from the engine-config YAML instead:

How to choose. Work from where the model fits:

• Fits on one GPU — no parallelism needed. Leave TP/PP/DP at 1, request one GPU, and skip to the Apply step. Add replicas later only for throughput.
• Doesn’t fit on one GPU, fits on one node — raise TP to the number of GPUs the weights (plus KV cache headroom) require, and set the worker’s nvidia.com/gpu limit to match. TP keeps one model copy on the fast intra-node fabric.
• Doesn’t fit on one node — combine PP with multinode.nodeCount so contiguous layer blocks land on separate nodes, typically with TP within each node. Total GPUs = TP × PP = nodeCount × per-node GPUs.
• Need more throughput — the model already fits; add replicas (or --data-parallel-size) to run more copies. This scales requests-per-second, not model size.
• Mixture-of-Experts — add EP on top of the above (see the MoE note below).

To pick actual numbers, start from a recipe: if one matches your model, backend, and hardware, copy its TP/PP/DP and adapt. If none fits, use AIConfigurator, which sweeps and ranks TP/PP/DP + replica configs for your model, GPU SKU, and latency target — see Sizing with AIConfigurator.

Worked example. Qwen3-32B in BF16 doesn’t fit on one 80 GB GPU with room for KV cache, but TP-2 does. Set the parallel size on the worker and match the GPU limit — for TensorRT-LLM the size lives in the engine-config YAML instead, so only the GPU limit changes on the worker:

1
  - name: VllmWorker
2
    type: worker
3
    podTemplate:
4
      spec:
5
        containers:
6
        - name: main
7
          command: [python3, -m, dynamo.vllm]
8
          args:
9
          - --model
10
          - Qwen/Qwen3-32B
11
          - --tensor-parallel-size
12
          - "2"
13
          resources:
14
            limits:
15
              nvidia.com/gpu: "2"            # must equal TP × PP

Context length. There is no uniform DGD field for maximum context length — it is an engine flag: --max-model-len (vLLM), --context-length (SGLang), or max_seq_len in the engine config (TensorRT-LLM). Longer context needs more free GPU memory for the KV cache, so it trades off against parallel size and concurrency. Set it when the default does not match your workload, for example --max-model-len 32000.

Mixture-of-Experts. MoE models — DeepSeek-R1, Qwen3-235B, Kimi-K2 — add expert parallelism (EP) on top of the knobs above, and expose it as two sizes: how experts are split (EP) and how the non-expert (attention) weights are split (a tensor-parallel size for the MoE layers). In vLLM, enable EP with --enable-expert-parallel, typically alongside --data-parallel-size for the attention layers. In SGLang, use --ep-size with --enable-dp-attention. In TensorRT-LLM, set both moe_expert_parallel_size and moe_tensor_parallel_size in the engine config. Wide-EP deployments usually pair this with multinode and a fast all-to-all backend (for example VLLM_ALL2ALL_BACKEND=deepep_low_latency). For a worked multinode wide-EP example, see the DeepSeek-R1 recipe.

Multinode. When TP × PP exceeds the GPUs on one node, set multinode.nodeCount on the worker so it spans machines; the operator schedules the pods and wires the engine’s cross-node communication (Ray for vLLM, --dist-init-addr/--nnodes for SGLang, MPI for TensorRT-LLM). This needs Grove or LWS installed. See Multinode Deployments and the disagg-multinode.yaml templates under each backend’s deploy/ folder.

Apply the deployment and check status

You have exported NAMESPACE and RUNTIME_IMAGE (and created the token Secret). Expand the placeholders and validate the spec with a server-side dry run, then apply it — pipe the file through envsubst both times:

$
envsubst < qwen3-32b-agg.yaml | kubectl apply -f - -n ${NAMESPACE} --dry-run=server
$
envsubst < qwen3-32b-agg.yaml | kubectl apply -f - -n ${NAMESPACE}

The DGD is the live serving resource. It becomes ready once the Frontend and workers are up — the Ready condition reports True. Watch it reconcile:

$
kubectl get dynamographdeployment -n ${NAMESPACE} -w

Or block until it is ready:

$
kubectl wait --for=condition=Ready dynamographdeployment/qwen3-32b-agg \
>
  -n ${NAMESPACE} --timeout=600s

If it stalls, kubectl describe dynamographdeployment <name> -n ${NAMESPACE} and the operator events surface the cause. Common rejections: a multinode component with neither Grove nor LWS installed, an nvidia.com/gpu limit that does not equal tensor-parallel × pipeline-parallel size, or a missing envFrom Secret.

Send a request

The operator creates a ClusterIP Service named <name>-frontend on port 8000. Port-forward it and send requests to the OpenAI-compatible API:

$
kubectl port-forward svc/qwen3-32b-agg-frontend 8000:8000 -n ${NAMESPACE}

List the served models:

$
curl -s http://localhost:8000/v1/models | python3 -m json.tool

Send a chat completion — the model field must match the model you deployed:

$
curl -s http://localhost:8000/v1/chat/completions \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen3-32B",
>
    "messages": [{"role": "user", "content": "What is NVIDIA Dynamo?"}],
>
    "max_tokens": 200
>
  }' | python3 -m json.tool

For a stable external address instead of port-forward, see Expose the Frontend.