KV-Aware Routing on Kubernetes | NVIDIA Dynamo Documentation

Dynamo supports two KV-aware request-routing topologies on Kubernetes. They use the same routing concepts but place worker selection in different components. Choose based on how your platform accepts and governs traffic, not because one topology uses a different KV-routing algorithm.

Recommendation

Use Dynamo Frontend routing unless your platform already needs Kubernetes Gateway API. It has fewer components, requires no Gateway API installation, and works with a standard DynamoGraphDeployment.

Use Gateway API routing when the Gateway must own the external request path—for example, when a platform team standardizes ingress, authentication, rate limiting, traffic policy, or gateway-level telemetry through Gateway API.

Choose	When it fits	Additional infrastructure	Where routing is configured
Dynamo Frontend	Clients can call one Dynamo Frontend Service directly and you want the simplest deployment.	None beyond the normal Dynamo platform.	Frontend with `--router-mode kv`.
Gateway API	Traffic must enter through a Kubernetes Gateway or share platform-managed gateway policy.	Gateway API, GAIE, a compatible Gateway implementation, and a Dynamo EPP.	EPP; worker Frontend sidecars use `--router-mode direct`.

The two topologies are alternatives for a request path. Do not configure the Frontend to select a worker after the EPP has already selected one.

Dynamo Frontend Routing

The Dynamo Frontend receives the client request, tokenizes it, scores eligible workers, and forwards the request to the selected worker. Router flags and environment variables belong on the Frontend component.

Choose this topology when:

Each deployment can expose its own Frontend Service.
You do not need Gateway API policy in the request path.
You want the fewest routing components to install and operate.
Dynamo should own both HTTP request handling and worker selection.

This topology works out of the box with the normal Dynamo platform; no Gateway API components are required. Use KV-Aware Routing with the Dynamo Frontend to enable KV-aware worker selection in a DGD.

Gateway API Routing

Gateway API routing separates traffic entry from worker selection. A Kubernetes Gateway receives the request and asks the Dynamo Endpoint Picker Plugin (EPP) to select an endpoint before forwarding the request to a worker pod.

Choose this topology when:

Your organization already uses Gateway API as the standard ingress layer.
Authentication, authorization, rate limiting, traffic policy, or telemetry belongs at the Gateway.
Multiple model routes should share a platform-managed address and listener.
A platform team manages the Gateway while an application team manages the DGD and HTTPRoute.

Gateway API routing requires a separate installation. See Install Gateway API Routing, then Route Requests with Gateway API.

What Is the Dynamo EPP?

The Dynamo Endpoint Picker Plugin (EPP) is a service that implements the GAIE endpoint-selection protocol. The Gateway calls it before forwarding an inference request. The EPP tokenizes the request, reads Dynamo worker and KV cache state, scores eligible endpoints, and returns the selected worker to the Gateway.

The EPP is not a public inference endpoint and does not run the model. The Dynamo operator deploys it when a DGD contains a component with type: epp. In this topology, worker Frontend sidecars run with --router-mode direct because the EPP has already made the routing decision.

For the full request and state flow, see Gateway API Routing Architecture.

Shared Routing Behavior

Both topologies can use worker-published KV cache events to account for prefix overlap and active load. Both can also operate with predicted state when precise KV events are unavailable. The selection algorithm and tuning concepts are shared; only the component hosting them changes.

For the cost model, see Routing Concepts. For shared tuning guidance, see Configuration and Tuning.