> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Gateway API Routing Reference Use this reference for a `DynamoGraphDeployment` (DGD) that routes requests through the Gateway API Inference Extension (GAIE) and the Dynamo Endpoint Picker Plugin (EPP). For the procedure and topology, see [Using GAIE with Dynamo](/dynamo/kubernetes/kv-aware-routing/using-gaie-with-dynamo). ## DGD EPP Fields The EPP is a DGD component with `type: epp`. The Dynamo operator creates the EPP workload, Service, and `InferencePool` from these fields. The Dynamo EPP runs the full Dynamo KV-aware router natively and is configured through `DYN_*` environment variables. It implements the GAIE [Lightweight Endpoint Picker (LW-EPP)](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/README.md) `ext_proc` interface and does not use a plugin or scheduling-profile config. **`spec.components[].type`** `string` — required Set to `epp` for the Endpoint Picker Plugin component. A DGD can contain at most one EPP component. Value: epp --- **`spec.components[].eppConfig`** `object` Deprecated Go-EPP configuration. When present, keep the EPP image pinned to 1.4; upgrading only the Dynamo Operator does not require migration. To select the native Rust EPP, remove this field and switch the image to 1.5 or later in the same update. Admission rejects mixed combinations. See [Route Requests with Gateway API](/dynamo/kubernetes/kv-aware-routing/using-gaie-with-dynamo). --- **`spec.components[].replicas`** `integer` — default: 1 Number of EPP pods. Increase this value when endpoint-picker request volume requires more EPP capacity. --- **`spec.components[].frontendSidecar`** `string` — required Name of the Frontend container in each routable worker pod. The named container must exist in `podTemplate.spec.containers` and run with `--router-mode direct`. --- For the complete component schema, see the [DynamoComponentDeployment Reference](/dynamo/reference/api/kubernetes/dynamo-component-deployment). ## EPP Runtime Environment Set these values on the EPP component's `main` container. Shared KV router settings use the same semantics as Frontend-hosted routing; this section identifies the settings most relevant to the Gateway topology. **`DYN_MODEL_NAME`** `string` Model identifier used to load tokenizer and model configuration. Set it when discovery does not provide the model name. It must match the model served by the workers. --- **`DYN_KV_CACHE_BLOCK_SIZE`** `integer` Backend KV cache block size. Set it when discovery does not provide the value. It must match the worker backend's block size; a mismatch changes prefix block hashes and produces incorrect overlap scores. --- **`DYN_USE_KV_EVENTS`** `boolean` — default: true Enables worker-published KV event consumption in the EPP router. When `false`, the router relies on predicted local state instead of precise worker cache events. --- **`DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT`** `number` — default: 1.0 Credit applied to device-local prompt-prefix overlap. Higher values prefer workers that already hold more of the prompt prefix. Set to `0` to remove device-local cache overlap from endpoint scoring. --- **`DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY`** `number` — default: 0.0 Reduces overlap credit as active prefill load rises above the least-loaded eligible worker. `0` disables decay. --- **`DYN_ROUTER_PREFILL_LOAD_SCALE`** `number` — default: 1.0 Scales prompt-side prefill load after cache-hit credits are applied. --- **`DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT`** `number` — default: 0.0 Adds block-equivalent routing cost for each active request on a candidate worker. Tune this value only when decode step latency depends materially on active batch size. --- **`DYN_ROUTER_TEMPERATURE`** `number` — default: 0.0 Controls worker exploration through normalized softmax sampling. `0` selects deterministically. --- **`DYN_ROUTER_REPLICA_SYNC`** `boolean` — default: false Publishes and consumes best-effort active-sequence state across EPP replicas through the Dynamo event plane. --- **`DYN_ROUTER_TRACK_ACTIVE_BLOCKS`** `boolean` — default: true Includes blocks used by active generation in worker load accounting. --- **`DYN_ROUTER_TRACK_OUTPUT_BLOCKS`** `boolean` — default: false Predicts output blocks during generation and decays them as requests make progress. --- **`DYN_ROUTER_TRACK_PREFILL_TOKENS`** `boolean` — default: true Includes active prompt-side prefill tokens in worker load accounting. --- For the full shared configuration surface and tuning guidance, see [Configuration and Tuning](/dynamo/knowledge-base/modular-components/router/configuration-and-tuning). ## Generated Resource Contract For a DGD named ``, the operator generates an `InferencePool` named `-pool` in the DGD namespace. Its `endpointPickerRef` points to the generated EPP Service on gRPC port `9002`, and its target port points to worker Frontend sidecars on port `8000`. ```yaml spec: selector: matchLabels: nvidia.com/dynamo-component-class: worker nvidia.com/dynamo-namespace: endpointPickerRef: kind: Service name: port: number: 9002 targetPorts: - number: 8000 ``` Treat the generated `InferencePool`, EPP Service, and Deployment as operator-owned resources. Make persistent changes in the DGD. ## HTTPRoute Contract **`spec.parentRefs[]`** `ParentReference` — required Gateway that receives model traffic. The Gateway and route can be in different namespaces only when the Gateway listener allows the route namespace. --- **`spec.rules[].backendRefs[]`** `BackendRef` — required Reference to the operator-generated `InferencePool`. --- **`group`** `string` — required Set to `inference.networking.k8s.io`. --- **`kind`** `string` — required Set to `InferencePool`. --- **`name`** `string` — required Generated pool name, normally `-pool`. --- **`port`** `integer` — required Worker Frontend target port. The operator-generated pool uses `8000`. --- ## Request Mutation Contract The EPP returns the selected endpoint and routing metadata to the Gateway. The Gateway forwards the mutated request to the selected Frontend sidecar. **`x-dynamo-worker-instance-id`** `HTTP header` Aggregated or decode worker selected for the request. --- **`x-dynamo-dp-rank`** `HTTP header` Data-parallel rank selected for the aggregated or decode worker. --- **`x-dynamo-routing-mode`** `HTTP header` Routing topology selected for the request. Values: aggregated disaggregated --- **`x-dynamo-prefill-instance-id`** `HTTP header` Prefill worker selected for a disaggregated request. --- **`x-dynamo-prefill-dp-rank`** `HTTP header` Data-parallel rank selected for the prefill worker. --- For supported body-bearing OpenAI requests, the EPP can inject precomputed token data into `nvext.token_data` so the Frontend sidecar does not repeat tokenization. ## Service Mesh Configuration **`dynamo.serviceMesh.enabled`** `boolean` — default: false Generates service-mesh resources for EPP Services when the provider CRDs are installed. --- **`dynamo.serviceMesh.provider`** `string` — default: istio Service-mesh provider. Only Istio is supported. Value: istio --- **`dynamo.serviceMesh.istio.tlsMode`** `string` — default: SIMPLE TLS mode in generated Istio `DestinationRule` resources. --- **`dynamo.serviceMesh.istio.insecureSkipVerify`** `boolean` — default: true Skips server certificate verification for the EPP's self-signed serving certificate. --- **`dynamo.serviceMesh.istio.clientCertificate`** `string` Client certificate path for `MUTUAL` TLS mode. --- **`dynamo.serviceMesh.istio.privateKey`** `string` Client private-key path for `MUTUAL` TLS mode. --- **`dynamo.serviceMesh.istio.caCertificates`** `string` Certificate authority path for `MUTUAL` TLS mode. --- When service-mesh integration is enabled, the operator creates a `DestinationRule` for each EPP Service. If the operator does not manage the mesh integration, create an equivalent rule so the Gateway proxy can connect to EPP gRPC port `9002`. ## agentgateway and Istio Injection When namespace-level Istio injection is enabled, an injected `istio-proxy` in an agentgateway data plane pod can intercept the EPP external-processing gRPC connection and cause HTTP 500 responses. Use an `AgentgatewayParameters` resource in the same namespace as the `Gateway`: ```yaml apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayParameters metadata: name: inference-gateway-params spec: deployment: spec: template: metadata: annotations: sidecar.istio.io/inject: "false" ``` Reference it from the Gateway: ```yaml spec: gatewayClassName: agentgateway infrastructure: parametersRef: group: agentgateway.dev kind: AgentgatewayParameters name: inference-gateway-params ``` `AgentgatewayParameters` is a local reference and must be in the Gateway namespace. Use a per-Gateway resource unless disabling injection is an intentional cluster-wide policy. ## Developer References Image build commands belong with the component source, not in this user reference. Use this source location when developing or replacing the standard EPP image: * [Dynamo EPP source](https://github.com/ai-dynamo/dynamo/tree/v1.5.0/deploy/inference-gateway/ext-proc) * [GAIE Lightweight Endpoint Picker (LW-EPP) reference](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/README.md)