> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# KV-Aware Routing

> Route requests across workers based on KV cache overlap

KV-aware routing places multiple workers behind the frontend and routes each request to the worker most likely to already hold its KV cache, eliminating redundant prefill computation. It is the local equivalent of the routing you configure in the [Kubernetes KV Routing guide](/dynamo/dev/kubernetes/model-deployment/kv-aware-routing/dynamo-frontend) — same router, driven by CLI flags instead of a DynamoGraphDeployment.

For how routing decisions are made, see [Routing Concepts](/dynamo/dev/knowledge-base/modular-components/router/routing-concepts) and the [Router Guide](/dynamo/dev/knowledge-base/modular-components/router/router-guide).

## How it works locally

A KV-routed deployment is still just a frontend plus workers (see the [Introduction](/dynamo/dev/cli/model-deployment/introduction)), with two differences:

* The **frontend** runs in KV routing mode so it tracks cache state across workers.
* Each **worker** publishes KV cache events (over ZMQ by default) so the frontend knows what each worker holds.

You run **two or more workers** so the router has somewhere to choose between. The pre-built `launch/agg_router.sh` presets wire this up for you.

## Aggregated serving with KV routing

Two workers behind the KV-aware router, maximizing cache reuse. Requires 2 GPUs.

#### vLLM

```bash
cd $DYNAMO_HOME/examples/backends/vllm
bash launch/agg_router.sh
```

This launches the frontend in KV routing mode with two workers publishing KV events over ZMQ.

#### SGLang

```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/agg_router.sh
```

This launches the frontend with `--router-mode kv` and two workers with ZMQ-based KV event publishing.

#### TensorRT-LLM

```bash
cd $DYNAMO_HOME/examples/backends/trtllm
./launch/agg_router.sh
```

## Disaggregated serving with KV routing

Scales to 2 prefill + 2 decode workers with KV-aware routing on both pools. The frontend runs in KV routing mode and automatically detects prefill workers to activate an internal prefill router. Requires 4 GPUs.

#### vLLM

```bash
cd $DYNAMO_HOME/examples/backends/vllm
bash launch/disagg_router.sh
```

#### SGLang

```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/disagg_router.sh
```

Each worker publishes KV events over ZMQ on unique ports.

#### TensorRT-LLM

```bash
cd $DYNAMO_HOME/examples/backends/trtllm
./launch/disagg_router.sh
```

In the disaggregated workflow, requests are routed to the prefill worker to maximize KV cache reuse.

Once a routed deployment is running, try adding another worker — the frontend discovers it automatically and starts routing to it.

## Troubleshooting

**Router not routing correctly (vLLM).** Ensure `PYTHONHASHSEED=0` is set for all vLLM processes when using KV-aware routing, so cache-block hashes are consistent across workers. See [Hashing Consistency](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/reference-guide#hashing-consistency-for-kv-events).

## See also

* **[Router Guide](/dynamo/dev/knowledge-base/modular-components/router/router-guide)** — router modes and configuration
* **[Disaggregated Serving](/dynamo/dev/cli/model-deployment/disaggregated-serving)** — split prefill and decode across workers
* **[vLLM local deployment examples](/dynamo/dev/recipes/cli-templates/v-llm)** — copyable commands for KV-routing launch scripts