Gateway API Routing Reference

Use this reference for a DynamoGraphDeployment (DGD) that routes requests through the Gateway API Inference Extension (GAIE) and the Dynamo Endpoint Picker Plugin (EPP). For the procedure and topology, see Using GAIE with Dynamo.

EPP Modes

The native Rust EPP runs in one of two modes, selected by the DYN_EPP_MODE environment variable. This reference covers Dynamo mode only. The deprecated Go EPP, kept by eppConfig with the image pinned to 1.4, does not read DYN_EPP_MODE and has no standalone mode.

A standalone EPP is not managed by the Dynamo operator, so the operator generates no resources for it. It uses neither the EPP ServiceAccount nor the dynamo-platform chart’s EPP RBAC. Standalone mode targets clusters without the Dynamo platform chart; do not install it for this path. A standalone EPP carries its own ServiceAccount and Role, which grant the permissions that mode needs. Do not bind a standalone EPP to the chart’s EPP ClusterRole; its rules cover only the resources a Dynamo-mode EPP reads. For the standalone path, see Using GAIE with vanilla vLLM.

DGD EPP Fields

The EPP is a DGD component with type: epp. The Dynamo operator creates the EPP workload, Service, and InferencePool from these fields. The Dynamo EPP runs the full Dynamo KV-aware router natively and is configured through DYN_* environment variables. It implements the GAIE Lightweight Endpoint Picker (LW-EPP) ext_proc interface and does not use a plugin or scheduling-profile config.

For the complete component schema, see the DynamoComponentDeployment Reference.

EPP Runtime Environment

Set these values on the EPP component’s main container. Shared KV router settings use the same semantics as Frontend-hosted routing; this section identifies the settings most relevant to the Gateway topology.

For the full shared configuration surface and tuning guidance, see Configuration and Tuning.

EPP Replication and Routing Accuracy

Set DYN_ROUTER_REPLICA_SYNC to true when running more than one EPP replica.

The EPP embeds the same KV router as the Frontend, so replicated EPPs behave like replicated Frontends:

• Prefix-cache state is the same on every replica. Each replica consumes worker KV events directly, so replication does not affect cache-overlap scoring.
• Active-load state is where replicas differ. Without DYN_ROUTER_REPLICA_SYNC, each replica sees only the requests it routed. With it, replicas share that state on a best-effort basis and can briefly disagree, for example when two replicas pick the same worker at the same time.
• Router queueing, when enabled, is per replica. A replica does not see requests queued on its peers, and its queued requests are lost if it stops.

For which active-load events are synchronized and which stay local, see Active Block Replica Synchronization.

Generated Resource Contract

For a DGD named <dgd-name>, the operator generates an InferencePool named <dgd-name>-pool in the DGD namespace. Its endpointPickerRef points to the generated EPP Service on gRPC port 9002, and its target port points to worker Frontend sidecars on port 8000.

spec:
  selector:
    matchLabels:
      nvidia.com/dynamo-component-class: worker
      nvidia.com/dynamo-namespace: <dynamo-namespace>
  endpointPickerRef:
    kind: Service
    name: <epp-service-name>
    port:
      number: 9002
  targetPorts:
  - number: 8000

Treat the generated InferencePool, EPP Service, and Deployment as operator-owned resources. Make persistent changes in the DGD.

Endpoint Picker Failure Mode

InferencePool.spec.endpointPickerRef.failureMode decides what the gateway does when the EPP does not answer, such as when its pod is restarting or has failed:

• FailClose drops the request. The client gets an error, in practice HTTP 500.
• FailOpen forwards the request to an endpoint the gateway picks. The request is served, but without KV-aware routing.

The operator does not set this field, so the Gateway API default of FailClose applies. Setting it by hand on the generated InferencePool does not hold: the operator restores its own rendering on the next reconcile, which leaves the field unset and lets the default apply again. No DGD field selects it.

A standalone EPP is unaffected. Its InferencePool is user-applied, so the value you write is the value you keep. The on-ramp example leaves the field out, so it also gets FailClose.

HTTPRoute Contract

Request Mutation Contract

The EPP returns the selected endpoint and routing metadata to the Gateway. The Gateway forwards the mutated request to the selected Frontend sidecar.

For supported body-bearing OpenAI requests, the EPP can inject precomputed token data into nvext.token_data so the Frontend sidecar does not repeat tokenization.

Service Mesh Configuration

When service-mesh integration is enabled, the operator creates a DestinationRule for each EPP Service. If the operator does not manage the mesh integration, create an equivalent rule so the Gateway proxy can connect to EPP gRPC port 9002.

agentgateway and Istio Injection

When namespace-level Istio injection is enabled, an injected istio-proxy in an agentgateway data plane pod can intercept the EPP external-processing gRPC connection and cause HTTP 500 responses. Use an AgentgatewayParameters resource in the same namespace as the Gateway:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: inference-gateway-params
spec:
  deployment:
    spec:
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"

Reference it from the Gateway:

spec:
  gatewayClassName: agentgateway
  infrastructure:
    parametersRef:
      group: agentgateway.dev
      kind: AgentgatewayParameters
      name: inference-gateway-params

AgentgatewayParameters is a local reference and must be in the Gateway namespace. Use a per-Gateway resource unless disabling injection is an intentional cluster-wide policy.

Developer References

Image build commands belong with the component source, not in this user reference. Use this source location when developing or replacing the standard EPP image:

• Dynamo EPP source
• GAIE Lightweight Endpoint Picker (LW-EPP) reference