KV-Aware Routing

KV-Aware Routing

以 Markdown 格式查看

KV-aware routing places multiple workers behind the frontend and routes each request to the worker most likely to already hold its KV cache, eliminating redundant prefill computation. It is the local equivalent of the routing you configure in the Kubernetes KV Routing guide — same router, driven by CLI flags instead of a DynamoGraphDeployment.

For how routing decisions are made, see Routing Concepts and the Router Guide.

How it works locally

A KV-routed deployment is still just a frontend plus workers (see the Overview), with two differences:

  • The frontend runs in KV routing mode so it tracks cache state across workers.
  • Each worker publishes KV cache events (over ZMQ by default) so the frontend knows what each worker holds.

Use two or more workers when you want to evaluate worker selection. The vLLM and SGLang launch/agg_router.sh presets wire up two workers; the TensorRT-LLM preset is a one-worker router-path smoke setup.

Aggregated serving with KV routing

The vLLM and SGLang presets start two workers and require 2 GPUs. The TensorRT-LLM preset starts one worker and requires 1 GPU; use it to verify the router path, then add workers when you need worker selection.

$cd $DYNAMO_HOME/examples/backends/vllm
$bash launch/agg_router.sh

This launches the frontend in KV routing mode with two workers publishing KV events over ZMQ.

Disaggregated serving with KV routing

The vLLM and SGLang presets start 2 prefill and 2 decode workers and require 4 GPUs. The TensorRT-LLM preset starts 1 prefill and 1 decode worker and requires 2 GPUs. The frontend runs in KV routing mode and activates its internal prefill router after discovering compatible typed prefill and decode services for the same model and namespace.

$cd $DYNAMO_HOME/examples/backends/vllm
$bash launch/disagg_router.sh

Once a routed deployment is running, try adding another worker — the frontend discovers it automatically and starts routing to it.

Troubleshooting

Router not routing correctly (vLLM). Ensure PYTHONHASHSEED=0 is set for all vLLM processes when using KV-aware routing, so cache-block hashes are consistent across workers. See Hashing Consistency.

See also