Disaggregated Serving

Dynamo supports disaggregated serving where prefill (prompt processing) and decode (token generation) are handled by separate worker pools. The frontend activates an internal prefill router when it discovers compatible typed prefill and decode services.

For the high-level deployment matrix, see Router Guide. For the router flags used in this setup, see Configuration and Tuning.

If prefill and decode workers span topology domains such as zones or racks, use Topology-Aware KV Transfer to constrain or bias decode routing toward workers in the selected prefill worker’s transfer domain.

Automatic Prefill Router Activation

The prefill router is automatically created when:

1. A decode worker is registered with WorkerType.Decode, for example via register_model() with ModelType.Chat | ModelType.Completions.
2. A prefill service in the same Runtime namespace is registered with the same model name and WorkerType.Prefill.

Key characteristics of the prefill router:

• Always disables active block tracking (track_active_blocks=false) since prefill workers do not perform decode.
• Runs between preprocessing and decode routing and returns handoff metadata for the selected decode worker.
• Uses decode-only routing when no prefill router is active. Once a request has been dispatched to prefill, a prefill or handoff failure is returned to the request; it does not retry through an aggregated decode-only path.

Key characteristics of the decode routing stage in disaggregated mode:

• Disables overlap scoring (overlap_score_credit=0) because decode routing should not chase prefix reuse.
• Disables KV reuse assumption (assume_kv_reuse=false) unless the backend can truly deduplicate transferred blocks.
• Disables prefill-token tracking (track_prefill_tokens=false) so decode-side load reflects decode work rather than already-completed prompt work.

Setup Example

When both workers are registered, requests are automatically routed.

1
# Decode worker registration (in your decode worker)
2
decode_endpoint = runtime.endpoint("dynamo.decode.generate")
3
4
await register_model(
5
    model_input=ModelInput.Tokens,
6
    model_type=ModelType.Chat | ModelType.Completions,
7
    endpoint=decode_endpoint,
8
    model_name="meta-llama/Llama-2-7b-hf",
9
    worker_type=WorkerType.Decode,
10
    needs=[[WorkerType.Prefill]],
11
    # ... other parameters
12
)
13
14
await decode_endpoint.serve_endpoint(decode_handler.generate)
15
16
# Prefill worker registration (in your prefill worker)
17
prefill_endpoint = runtime.endpoint("dynamo.prefill.generate")
18
19
await register_model(
20
    model_input=ModelInput.Tokens,
21
    model_type=ModelType.Empty,  # prefill workers expose no OpenAI surface
22
    endpoint=prefill_endpoint,
23
    model_name="meta-llama/Llama-2-7b-hf",
24
    worker_type=WorkerType.Prefill,
25
    needs=[[WorkerType.Decode]],
26
    # ... other parameters
27
)
28
29
await prefill_endpoint.serve_endpoint(prefill_handler.generate)

Request Flow

The following diagram shows an overview of the major components in disaggregated serving:

When topology-aware KV transfer is enabled, the prefill router also derives decode RoutingConstraints from the selected prefill worker’s runtime topology metadata before the request enters the decode router.