> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# KV-Aware Routing on Kubernetes

Dynamo supports two KV-aware request-routing topologies on Kubernetes. They use the same routing
concepts but place worker selection in different components. Choose based on how your platform accepts
and governs traffic, not because one topology uses a different KV-routing algorithm.

## Recommendation

Use **Dynamo Frontend routing** unless your platform already needs Kubernetes Gateway API. It has
fewer components, requires no Gateway API installation, and works with a standard
`DynamoGraphDeployment`.

Use **Gateway API routing** when the Gateway must own the external request path—for example, when a
platform team standardizes ingress, authentication, rate limiting, traffic policy, or gateway-level
telemetry through Gateway API.

| Choose | When it fits | Additional infrastructure | Where routing is configured |
|---|---|---|---|
| Dynamo Frontend | Clients can call one Dynamo Frontend Service directly and you want the simplest deployment. | None beyond the normal Dynamo platform. | Frontend with `--router-mode kv`. |
| Gateway API | Traffic must enter through a Kubernetes Gateway or share platform-managed gateway policy. | Gateway API, GAIE, a compatible Gateway implementation, and a Dynamo EPP. | EPP; worker Frontend sidecars use `--router-mode direct`. |

The two topologies are alternatives for a request path. Do not configure the Frontend to select a
worker after the EPP has already selected one.

## Dynamo Frontend Routing

The Dynamo Frontend receives the client request, tokenizes it, scores eligible workers, and forwards
the request to the selected worker. Router flags and environment variables belong on the Frontend
component.

Choose this topology when:

- Each deployment can expose its own Frontend Service.
- You do not need Gateway API policy in the request path.
- You want the fewest routing components to install and operate.
- Dynamo should own both HTTP request handling and worker selection.

This topology works out of the box with the normal Dynamo platform; no Gateway API components are
required. Use [KV-Aware Routing with the Dynamo Frontend](/dynamo/dev/kubernetes/model-deployment/kv-aware-routing/dynamo-frontend) to enable KV-aware worker
selection in a DGD.

## Gateway API Routing

Gateway API routing separates traffic entry from worker selection. A Kubernetes `Gateway` receives
the request and asks the Dynamo Endpoint Picker Plugin (EPP) to select an endpoint before forwarding
the request to a worker pod.

Choose this topology when:

- Your organization already uses Gateway API as the standard ingress layer.
- Authentication, authorization, rate limiting, traffic policy, or telemetry belongs at the Gateway.
- Multiple model routes should share a platform-managed address and listener.
- A platform team manages the Gateway while an application team manages the DGD and `HTTPRoute`.

Gateway API routing requires a separate installation. See
[Install Gateway API Routing](/dynamo/dev/kubernetes/installation/gateway-api-routing), then
[Route Requests with Gateway API](/dynamo/dev/kubernetes/model-deployment/kv-aware-routing/gateway-api).

## What Is the Dynamo EPP?

The Dynamo **Endpoint Picker Plugin (EPP)** is a service that implements the GAIE endpoint-selection
protocol. The Gateway calls it before forwarding an inference request. The EPP tokenizes the request,
reads Dynamo worker and KV cache state, scores eligible endpoints, and returns the selected worker to
the Gateway.

The EPP is not a public inference endpoint and does not run the model. The Dynamo operator deploys it
when a DGD contains a component with `type: epp`. In this topology, worker Frontend sidecars run with
`--router-mode direct` because the EPP has already made the routing decision.

For the full request and state flow, see
[Gateway API Routing Architecture](/dynamo/dev/knowledge-base/design-documents/gateway-api-routing).

## Shared Routing Behavior

Both topologies can use worker-published KV cache events to account for prefix overlap and active
load. Both can also operate with predicted state when precise KV events are unavailable. The
selection algorithm and tuning concepts are shared; only the component hosting them changes.

For the cost model, see [Routing Concepts](/dynamo/dev/knowledge-base/modular-components/router/routing-concepts). For shared tuning
guidance, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning).