Gateway API Routing Reference | NVIDIA Dynamo Documentation

Use this reference for a DynamoGraphDeployment (DGD) that routes requests through the Gateway API Inference Extension (GAIE) and the Dynamo Endpoint Picker Plugin (EPP). For the procedure and topology, see Using GAIE with Dynamo.

DGD EPP Fields

The EPP is a DGD component with type: epp. The Dynamo operator creates the EPP workload, Service, and InferencePool from these fields. The Dynamo EPP runs the full Dynamo KV-aware router natively and is configured through DYN_* environment variables. It implements the GAIE Lightweight Endpoint Picker (LW-EPP) ext_proc interface and does not use a plugin or scheduling-profile config.

spec.components[].type

stringRequired

Set to epp for the Endpoint Picker Plugin component. A DGD can contain at most one EPP component.

Value: epp

spec.components[].eppConfig

object

Deprecated Go-EPP configuration. When present, keep the EPP image pinned to 1.4; upgrading only the Dynamo Operator does not require migration. To select the native Rust EPP, remove this field and switch the image to 1.5 or later in the same update. Admission rejects mixed combinations. See Route Requests with Gateway API.

spec.components[].replicas

integerDefaults to 1

Number of EPP pods. Increase this value when endpoint-picker request volume requires more EPP capacity.

spec.components[].frontendSidecar

stringRequired

Name of the Frontend container in each routable worker pod. The named container must exist in podTemplate.spec.containers and run with --router-mode direct.

For the complete component schema, see the DynamoComponentDeployment Reference.

EPP Runtime Environment

Set these values on the EPP component’s main container. Shared KV router settings use the same semantics as Frontend-hosted routing; this section identifies the settings most relevant to the Gateway topology.

DYN_MODEL_NAME

string

Model identifier used to load tokenizer and model configuration. Set it when discovery does not provide the model name. It must match the model served by the workers.

DYN_KV_CACHE_BLOCK_SIZE

integer

Backend KV cache block size. Set it when discovery does not provide the value. It must match the worker backend’s block size; a mismatch changes prefix block hashes and produces incorrect overlap scores.

DYN_USE_KV_EVENTS

booleanDefaults to true

Enables worker-published KV event consumption in the EPP router. When false, the router relies on predicted local state instead of precise worker cache events.

DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT

numberDefaults to 1.0

Credit applied to device-local prompt-prefix overlap. Higher values prefer workers that already hold more of the prompt prefix. Set to 0 to remove device-local cache overlap from endpoint scoring.

DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY

numberDefaults to 0.0

Reduces overlap credit as active prefill load rises above the least-loaded eligible worker. 0 disables decay.

DYN_ROUTER_PREFILL_LOAD_SCALE

numberDefaults to 1.0

Scales prompt-side prefill load after cache-hit credits are applied.

DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT

numberDefaults to 0.0

Adds block-equivalent routing cost for each active request on a candidate worker. Tune this value only when decode step latency depends materially on active batch size.

DYN_ROUTER_TEMPERATURE

numberDefaults to 0.0

Controls worker exploration through normalized softmax sampling. 0 selects deterministically.

DYN_ROUTER_REPLICA_SYNC

booleanDefaults to false

Publishes and consumes best-effort active-sequence state across EPP replicas through the Dynamo event plane.

DYN_ROUTER_TRACK_ACTIVE_BLOCKS

booleanDefaults to true

Includes blocks used by active generation in worker load accounting.

DYN_ROUTER_TRACK_OUTPUT_BLOCKS

booleanDefaults to false

Predicts output blocks during generation and decays them as requests make progress.

DYN_ROUTER_TRACK_PREFILL_TOKENS

booleanDefaults to true

Includes active prompt-side prefill tokens in worker load accounting.

For the full shared configuration surface and tuning guidance, see Configuration and Tuning.

Generated Resource Contract

For a DGD named <dgd-name>, the operator generates an InferencePool named <dgd-name>-pool in the DGD namespace. Its endpointPickerRef points to the generated EPP Service on gRPC port 9002, and its target port points to worker Frontend sidecars on port 8000.

spec:
  selector:
    matchLabels:
      nvidia.com/dynamo-component-class: worker
      nvidia.com/dynamo-namespace: <dynamo-namespace>
  endpointPickerRef:
    kind: Service
    name: <epp-service-name>
    port:
      number: 9002
  targetPorts:
  - number: 8000

Treat the generated InferencePool, EPP Service, and Deployment as operator-owned resources. Make persistent changes in the DGD.

HTTPRoute Contract

spec.parentRefs[]

ParentReferenceRequired

Gateway that receives model traffic. The Gateway and route can be in different namespaces only when the Gateway listener allows the route namespace.

spec.rules[].backendRefs[]

BackendRefRequired

Reference to the operator-generated InferencePool.

group

stringRequired

Set to inference.networking.k8s.io.

kind

stringRequired

Set to InferencePool.

name

stringRequired

Generated pool name, normally <dgd-name>-pool.

port

integerRequired

Worker Frontend target port. The operator-generated pool uses 8000.

Request Mutation Contract

The EPP returns the selected endpoint and routing metadata to the Gateway. The Gateway forwards the mutated request to the selected Frontend sidecar.

x-dynamo-worker-instance-id

HTTP header

Aggregated or decode worker selected for the request.

x-dynamo-dp-rank

HTTP header

Data-parallel rank selected for the aggregated or decode worker.

x-dynamo-routing-mode

HTTP header

Routing topology selected for the request.

Values: aggregated disaggregated

x-dynamo-prefill-instance-id

HTTP header

Prefill worker selected for a disaggregated request.

x-dynamo-prefill-dp-rank

HTTP header

Data-parallel rank selected for the prefill worker.

For supported body-bearing OpenAI requests, the EPP can inject precomputed token data into nvext.token_data so the Frontend sidecar does not repeat tokenization.

Service Mesh Configuration

dynamo.serviceMesh.enabled

booleanDefaults to false

Generates service-mesh resources for EPP Services when the provider CRDs are installed.

dynamo.serviceMesh.provider

stringDefaults to istio

Service-mesh provider. Only Istio is supported.

Value: istio

dynamo.serviceMesh.istio.tlsMode

stringDefaults to SIMPLE

TLS mode in generated Istio DestinationRule resources.

dynamo.serviceMesh.istio.insecureSkipVerify

booleanDefaults to true

Skips server certificate verification for the EPP’s self-signed serving certificate.

dynamo.serviceMesh.istio.clientCertificate

string

Client certificate path for MUTUAL TLS mode.

dynamo.serviceMesh.istio.privateKey

string

Client private-key path for MUTUAL TLS mode.

dynamo.serviceMesh.istio.caCertificates

string

Certificate authority path for MUTUAL TLS mode.

When service-mesh integration is enabled, the operator creates a DestinationRule for each EPP Service. If the operator does not manage the mesh integration, create an equivalent rule so the Gateway proxy can connect to EPP gRPC port 9002.

agentgateway and Istio Injection

When namespace-level Istio injection is enabled, an injected istio-proxy in an agentgateway data plane pod can intercept the EPP external-processing gRPC connection and cause HTTP 500 responses. Use an AgentgatewayParameters resource in the same namespace as the Gateway:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: inference-gateway-params
spec:
  deployment:
    spec:
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"

Reference it from the Gateway:

spec:
  gatewayClassName: agentgateway
  infrastructure:
    parametersRef:
      group: agentgateway.dev
      kind: AgentgatewayParameters
      name: inference-gateway-params

AgentgatewayParameters is a local reference and must be in the Gateway namespace. Use a per-Gateway resource unless disabling injection is an intentional cluster-wide policy.

Developer References

Image build commands belong with the component source, not in this user reference. Use this source location when developing or replacing the standard EPP image: