Model DeploymentKV-Aware Routing

KV-Aware Routing with the Dynamo Frontend

Configure the Dynamo Frontend to select the worker most likely to have the prompt prefix cached.
View as Markdown

In this topology, the Dynamo Frontend receives each request and selects the worker most likely to already hold the prompt’s KV cache prefix. Use it when clients send requests directly to a Dynamo Frontend Service. If a Kubernetes Gateway receives requests first, use Gateway API routing instead.

Turning it on in a DynamoGraphDeployment (DGD) takes two steps: switch the Frontend into KV mode, and have the workers publish KV cache events so the router knows what each worker has cached. This is a how-to for an existing deployment. For the routing cost model and concepts, see Routing Concepts; for the full flag and env reference, see the Router Guide.

1

Put the Frontend in KV mode

Set --router-mode kv on the Frontend container, or the equivalent DYN_ROUTER_MODE=kv environment variable:

1spec:
2  components:
3  - name: Frontend
4    type: frontend
5    podTemplate:
6      spec:
7        containers:
8        - name: main
9          command:
10          - python3
11          - -m
12          - dynamo.frontend
13          args:
14          - --router-mode
15          - kv

That alone gives you cache-aware routing using load signals. To make routing decisions from real cache contents, complete the next step.

2

Publish KV events from the workers

For the router to track which blocks each worker holds, workers must publish KV cache events. On a vLLM worker, add --kv-events-config:

1  - name: VllmPrefillWorker
2    type: prefill
3    podTemplate:
4      spec:
5        containers:
6        - name: main
7          command:
8          - python3
9          - -m
10          - dynamo.vllm
11          args:
12          - --model
13          - Qwen/Qwen3-32B
14          - --kv-events-config
15          - '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20080","enable_kv_cache_events":true}'

This is the half that the Router Guide does not show. Without it, the router falls back to load-only decisions even in kv mode.

The Frontend and worker snippets above are drawn from the disagg-kv-router recipe, where six prefill workers publish KV events and the Frontend routes across them.

Tuning Knobs

The KV router scores each worker as prefill_load_scale * adjusted_prefill_blocks + decode_blocks, where cache-overlap credit subtracts from the prefill load. Two knobs shift that balance; set them as Frontend args (or the DYN_* env equivalents). Start with the defaults and adjust only if you have a measured TTFT or ITL problem.

For the flag/env/default reference, see the Frontend Configuration Reference; for the full cost-model detail and every related flag, see Configuration and Tuning.

Cache-Overlap Credit

--router-kv-overlap-score-credit (env DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT) is the primary cache-reuse knob. It credits device-local prefix overlap against a worker’s prefill load, biasing requests toward workers that already hold the prompt’s prefix.

  • Range: 0.0 to 1.0. Default: 1.0.
  • Raise toward 1.0 to prioritize cache reuse and lower TTFT — the router more aggressively co-locates requests that share a prefix.
  • Lower toward 0.0 to spread load more evenly and lower ITL, at the cost of more redundant prefills. 0.0 ignores prefix caches entirely and skips building the local indexer (equivalent to load-only routing).

Most deployments should leave this at 1.0. Lower it only when cache-rich workers are getting overloaded while others sit idle.

Prompt-Side Load Weight

--router-prefill-load-scale (env DYN_ROUTER_PREFILL_LOAD_SCALE) scales the prompt-side prefill load after overlap credit is applied, setting how much prompt work counts relative to decode-side block load.

  • Minimum: 0.0 (ignore prompt-side load). Default: 1.0. No hard maximum — values above 1.0 weight prefill more heavily.
  • Raise above 1.0 when long prompts are saturating workers and you want the router to steer new requests away from workers already doing heavy prefill.
  • Lower below 1.0 when decode-side pressure dominates and you want routing driven mainly by active decode blocks.

Route on Load Only

--no-router-kv-events (env DYN_ROUTER_USE_KV_EVENTS=false) disables event tracking; the router predicts cache state from its own routing decisions with TTL-based expiration instead of consuming real KV events. Use it only when you are not confident the backend emits KV events correctly. --router-reset-states clears cached routing state on Frontend restart.

Routing with Disaggregated Serving

In a disaggregated graph, the router operates over prefill and decode workers separately. The prefill workers publish KV events (the second step above) and the router selects among them; the internal prefill router activates automatically. See Router with Disaggregated Serving.