KV-Aware Routing
KV-Aware Routing
KV-aware routing places multiple workers behind the frontend and routes each request to the worker most likely to already hold its KV cache, eliminating redundant prefill computation. It is the local equivalent of the routing you configure in the Kubernetes KV Routing guide — same router, driven by CLI flags instead of a DynamoGraphDeployment.
For how routing decisions are made, see Routing Concepts and the Router Guide.
How it works locally
A KV-routed deployment is still just a frontend plus workers (see the Introduction), with two differences:
- The frontend runs in KV routing mode so it tracks cache state across workers.
- Each worker publishes KV cache events (over ZMQ by default) so the frontend knows what each worker holds.
You run two or more workers so the router has somewhere to choose between. The pre-built launch/agg_router.sh presets wire this up for you.
Aggregated serving with KV routing
Two workers behind the KV-aware router, maximizing cache reuse. Requires 2 GPUs.
vLLM
SGLang
TensorRT-LLM
This launches the frontend in KV routing mode with two workers publishing KV events over ZMQ.
Disaggregated serving with KV routing
Scales to 2 prefill + 2 decode workers with KV-aware routing on both pools. The frontend runs in KV routing mode and automatically detects prefill workers to activate an internal prefill router. Requires 4 GPUs.
vLLM
SGLang
TensorRT-LLM
Once a routed deployment is running, try adding another worker — the frontend discovers it automatically and starts routing to it.
Troubleshooting
Router not routing correctly (vLLM). Ensure PYTHONHASHSEED=0 is set for all vLLM processes when using KV-aware routing, so cache-block hashes are consistent across workers. See Hashing Consistency.
See also
- Router Guide — router modes and configuration
- Disaggregated Serving — split prefill and decode across workers
- vLLM local deployment examples — copyable commands for KV-routing launch scripts