Disaggregated Serving

Check prerequisites

For the single-node example in this guide you need:

1. A node with multiple GPUs — one per prefill and decode worker (this example uses 8×H100 with gpu: "1" per worker).
2. ETCD and NATS deployed for Dynamo coordination.
3. A HuggingFace token secret (hf-token-secret) if the model is gated.

Model caching (the PVC and HF_HOME wiring the workers mount) is configured separately — see Model caching. This guide omits those fields for clarity; without them the model downloads to the container’s default cache.

Scaling across multiple nodes additionally requires an RDMA-capable network and device plugin — covered in Enable multi-node transfer.

Start the spec: the frontend

Begin with the DGD skeleton — just a frontend. This is identical to an aggregated deployment; nothing about disaggregation changes it.

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: dynamo-disagg
5
  namespace: your-namespace
6
spec:
7
  backendFramework: vllm
8
  services:
9
    Frontend:
10
      componentType: frontend
11
      replicas: 1
12
      extraPodSpec:
13
        mainContainer:
14
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1
15
          imagePullPolicy: IfNotPresent

Add the prefill worker

The prefill worker runs the prompt phase. Two things make it a prefill worker rather than a plain worker:

• subComponentType: prefill tags the role for the operator and router.
• --disaggregation-mode prefill tells the engine to stop after producing the KV cache instead of generating tokens.

1
    VLLMPrefillWorker:
2
      envFromSecret: hf-token-secret
3
      componentType: worker
4
      subComponentType: prefill
5
      replicas: 3
6
      resources:
7
        limits:
8
          gpu: "1"
9
      sharedMemory:
10
        size: 16Gi
11
      extraPodSpec:
12
        mainContainer:
13
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1
14
          imagePullPolicy: IfNotPresent
15
          command: ["python3", "-m", "dynamo.vllm"]
16
          args:
17
            - --model
18
            - Qwen/Qwen3-32B
19
            - --disaggregation-mode
20
            - prefill

Qwen3-32B fits on a single H100, so each worker takes gpu: "1" with no tensor parallelism. replicas: 3 gives three prefill workers; size the count for your prompt load independently of decode. (sharedMemory is explained in the KV transfer step.)

Add the decode worker

The decode worker continues generation after prefill hands off the KV cache. It mirrors the prefill worker with subComponentType: decode and --disaggregation-mode decode, and its replica count is sized for concurrency rather than prompt load.

1
    VLLMDecodeWorker:
2
      envFromSecret: hf-token-secret
3
      componentType: worker
4
      subComponentType: decode
5
      replicas: 4
6
      resources:
7
        limits:
8
          gpu: "1"
9
      sharedMemory:
10
        size: 16Gi
11
      extraPodSpec:
12
        mainContainer:
13
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1
14
          imagePullPolicy: IfNotPresent
15
          command: ["python3", "-m", "dynamo.vllm"]
16
          args:
17
            - --model
18
            - Qwen/Qwen3-32B
19
            - --disaggregation-mode
20
            - decode

Enable KV transfer

Because prefill and decode run on different GPUs, the KV cache prefill produces has to be copied into the decode worker’s GPU memory before decode can generate. Dynamo’s transfer layer, NIXL, does this automatically. On a single node it selects CUDA IPC and moves the cache GPU-to-GPU over NVLink — no network, no RDMA, nothing to configure for the transport itself.

The one field you do set is shared memory, already in the worker specs above:

That is the entire KV-transfer configuration for a single node: two workers tagged prefill/decode, matching --model, and enough shared memory for NIXL.

Apply and verify the transfer path

Apply the assembled spec:

$
kubectl apply -f dynamo-disagg.yaml -n your-namespace

Then confirm NIXL initialized its transfer backend — the key check for a disaggregated deployment. Grep a prefill worker’s logs:

$
kubectl logs <prefill-worker-pod> | grep -i "NIXL"

You want to see NIXL bring up a backend, for example:

NIXL INFO Backend UCX was instantiated

If NIXL fails to initialize, the usual cause is sharedMemory.size left at the default. Symptoms of a broken transfer path include high TTFT despite free prefill capacity, decode workers sitting idle while prefill is busy, or disaggregated throughput falling below your aggregated baseline.

Enable multi-node transfer

Everything above keeps prefill and decode on one node, where the KV cache moves over NVLink. When a worker’s parallelism exceeds the GPUs on a node, or you want prefill and decode pools on separate machines, the KV cache travels over the network instead — and that path needs RDMA (InfiniBand or RoCE). Without it, transfers fall back to TCP and KV movement can dominate TTFT and throughput.

Multi-node adds RDMA fields to each worker (rdma/ib resource requests, the IPC_LOCK capability, and UCX_* transport env vars) plus an RDMA device plugin on the cluster. That setup is out of scope here — see the Disaggregated Communication Guide for the transport configuration and Multinode Deployments for spanning workers across machines.