> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Gateway API Routing Reference

Use this reference for a `DynamoGraphDeployment` (DGD) that routes requests through the Gateway API
Inference Extension (GAIE) and the Dynamo Endpoint Picker Plugin (EPP). For the procedure, see
[Using GAIE with Dynamo](/dynamo/dev/kubernetes/kv-aware-routing/using-gaie-with-dynamo). For architecture, see
[Gateway API Routing Architecture](/dynamo/dev/knowledge-base/kubernetes/kubernetes-operator/gateway-api-routing).

## DGD EPP Fields

The EPP is a DGD component with `type: epp`. The Dynamo operator creates the EPP workload, Service,
configuration, and `InferencePool` from these fields.

**`spec.components[].type`** `string` — required

Set to `epp` for the Endpoint Picker Plugin component. A DGD can contain at most one EPP component.

Value:

epp

---

**`spec.components[].replicas`** `integer` — default: 1

Number of EPP pods. Increase this value when endpoint-picker request volume requires more EPP capacity.

---

**`spec.components[].eppConfig`** `EPPConfig` — required

EPP configuration source. Set exactly one of `config` or `configMapRef`.

---

**`config`** `EndpointPickerConfig`

Inline endpoint-picker configuration. The operator serializes the object into a generated `ConfigMap`.
Mutually exclusive with `configMapRef`.

---

**`configMapRef`** `ConfigMapKeySelector`

Reference to a user-managed `ConfigMap` containing an `EndpointPickerConfig`. Mutually exclusive with
`config`.

---

**`spec.components[].frontendSidecar`** `string` — required

Name of the Frontend container in each routable worker pod. The named container must exist in
`podTemplate.spec.containers` and run with `--router-mode direct`.

---

For the complete component schema, see the
[DynamoComponentDeployment Reference](/dynamo/dev/reference/api/kubernetes/dynamo-component-deployment).

## EndpointPickerConfig Fields

**`plugins`** `[]Plugin` — required

Plugin instances available to scheduling profiles. A plugin can filter endpoints, score endpoints,
choose a result, or establish aggregated and disaggregated profiles.

---

**`name`** `string`

Local name used by `schedulingProfiles[].plugins[].pluginRef`. Omit only for a plugin that does not
need to be referenced directly, such as `disagg-profile-handler`.

---

**`type`** `string` — required

Plugin implementation.

Common Dynamo values: disagg-profile-handler label-filter dyn-prefill-scorer dyn-decode-scorer max-score-picker

---

**`parameters`** `object`

Plugin-specific configuration. The operator preserves this object without pruning unknown fields.

---

**`schedulingProfiles`** `[]SchedulingProfile` — required

Ordered plugin chains for endpoint selection. Aggregated deployments use a `decode` profile.
Disaggregated deployments use both `prefill` and `decode` profiles.

---

**`name`** `string` — required

Worker role handled by the profile.

Dynamo values: prefill decode

---

**`plugins`** `[]PluginRef` — required

Plugins applied by the profile.

---

**`pluginRef`** `string` — required

Name of a plugin declared in the top-level `plugins` list.

---

**`weight`** `integer` — default: 1

Relative influence of the plugin in the scheduling profile. Keep required filters and the final
picker enabled. Validate scorer-weight changes with production-like traffic.

---

## Dynamo Plugin Parameters

### Label Filter

**`parameters.label`** `string` — required

Pod label used to identify eligible workers. For DGD v1beta1 components, use
`nvidia.com/dynamo-component-type`.

---

**`parameters.validValues`** `[]string` — required

Accepted values for the selected label. Use `prefill` or `decode` to separate worker roles.

---

**`parameters.allowsNoLabel`** `boolean` — default: false

Whether pods without the selected label remain eligible. Aggregated profiles can set this to `true`
when worker pods do not carry a separate decode role. Disaggregated profiles should use `false`.

---

## EPP Runtime Environment

Set these values on the EPP component's `main` container. Shared KV router settings use the same
semantics as Frontend-hosted routing; this section identifies the settings most relevant to the
Gateway topology.

**`DYN_MODEL_NAME`** `string`

Model identifier used to load tokenizer and model configuration. Set it when discovery does not
provide the model name. It must match the model served by the workers.

---

**`DYN_KV_CACHE_BLOCK_SIZE`** `integer`

Backend KV cache block size. Set it when discovery does not provide the value. It must match the
worker backend's block size; a mismatch changes prefix block hashes and produces incorrect overlap
scores.

---

**`DYN_USE_KV_EVENTS`** `boolean` — default: true

Enables worker-published KV event consumption in the EPP router. When `false`, the router relies on
predicted local state instead of precise worker cache events.

---

**`DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT`** `number` — default: 1.0

Credit applied to device-local prompt-prefix overlap. Higher values prefer workers that already hold
more of the prompt prefix. Set to `0` to remove device-local cache overlap from endpoint scoring.

---

**`DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY`** `number` — default: 0.0

Reduces overlap credit as active prefill load rises above the least-loaded eligible worker. `0`
disables decay.

---

**`DYN_ROUTER_PREFILL_LOAD_SCALE`** `number` — default: 1.0

Scales prompt-side prefill load after cache-hit credits are applied.

---

**`DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT`** `number` — default: 0.0

Adds block-equivalent routing cost for each active request on a candidate worker. Tune this value
only when decode step latency depends materially on active batch size.

---

**`DYN_ROUTER_TEMPERATURE`** `number` — default: 0.0

Controls worker exploration through normalized softmax sampling. `0` selects deterministically.

---

**`DYN_ROUTER_REPLICA_SYNC`** `boolean` — default: false

Publishes and consumes best-effort active-sequence state across EPP replicas through the Dynamo event
plane.

---

**`DYN_ROUTER_TRACK_ACTIVE_BLOCKS`** `boolean` — default: true

Includes blocks used by active generation in worker load accounting.

---

**`DYN_ROUTER_TRACK_OUTPUT_BLOCKS`** `boolean` — default: false

Predicts output blocks during generation and decays them as requests make progress.

---

**`DYN_ROUTER_TRACK_PREFILL_TOKENS`** `boolean` — default: true

Includes active prompt-side prefill tokens in worker load accounting.

---

For the full shared configuration surface and tuning guidance, see
[Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning).

## Generated Resource Contract

For a DGD named `<dgd-name>`, the operator generates an `InferencePool` named `<dgd-name>-pool` in
the DGD namespace. Its `endpointPickerRef` points to the generated EPP Service on gRPC port `9002`,
and its target port points to worker Frontend sidecars on port `8000`.

```yaml
spec:
  selector:
    matchLabels:
      nvidia.com/dynamo-component-class: worker
      nvidia.com/dynamo-namespace: <dynamo-namespace>
  endpointPickerRef:
    kind: Service
    name: <epp-service-name>
    port:
      number: 9002
  targetPorts:
  - number: 8000
```

Treat the generated `InferencePool`, EPP Service, Deployment, and inline configuration `ConfigMap` as
operator-owned resources. Make persistent changes in the DGD.

## HTTPRoute Contract

**`spec.parentRefs[]`** `ParentReference` — required

Gateway that receives model traffic. The Gateway and route can be in different namespaces only when
the Gateway listener allows the route namespace.

---

**`spec.rules[].backendRefs[]`** `BackendRef` — required

Reference to the operator-generated `InferencePool`.

---

**`group`** `string` — required

Set to `inference.networking.k8s.io`.

---

**`kind`** `string` — required

Set to `InferencePool`.

---

**`name`** `string` — required

Generated pool name, normally `<dgd-name>-pool`.

---

**`port`** `integer` — required

Worker Frontend target port. The operator-generated pool uses `8000`.

---

## Request Mutation Contract

The EPP returns the selected endpoint and routing metadata to the Gateway. The Gateway forwards the
mutated request to the selected Frontend sidecar.

**`x-dynamo-worker-instance-id`** `HTTP header`

Aggregated or decode worker selected for the request.

---

**`x-dynamo-dp-rank`** `HTTP header`

Data-parallel rank selected for the aggregated or decode worker.

---

**`x-dynamo-routing-mode`** `HTTP header`

Routing topology selected for the request.

Values:

aggregated

disaggregated

---

**`x-dynamo-prefill-instance-id`** `HTTP header`

Prefill worker selected for a disaggregated request.

---

**`x-dynamo-prefill-dp-rank`** `HTTP header`

Data-parallel rank selected for the prefill worker.

---

For supported body-bearing OpenAI requests, the EPP can inject precomputed token data into
`nvext.token_data` so the Frontend sidecar does not repeat tokenization.

## Service Mesh Configuration

**`dynamo.serviceMesh.enabled`** `boolean` — default: false

Generates service-mesh resources for EPP Services when the provider CRDs are installed.

---

**`dynamo.serviceMesh.provider`** `string` — default: istio

Service-mesh provider. Only Istio is supported.

Value:

istio

---

**`dynamo.serviceMesh.istio.tlsMode`** `string` — default: SIMPLE

TLS mode in generated Istio `DestinationRule` resources.

---

**`dynamo.serviceMesh.istio.insecureSkipVerify`** `boolean` — default: true

Skips server certificate verification for the EPP's self-signed serving certificate.

---

**`dynamo.serviceMesh.istio.clientCertificate`** `string`

Client certificate path for `MUTUAL` TLS mode.

---

**`dynamo.serviceMesh.istio.privateKey`** `string`

Client private-key path for `MUTUAL` TLS mode.

---

**`dynamo.serviceMesh.istio.caCertificates`** `string`

Certificate authority path for `MUTUAL` TLS mode.

---

When service-mesh integration is enabled, the operator creates a `DestinationRule` for each EPP
Service. If the operator does not manage the mesh integration, create an equivalent rule so the
Gateway proxy can connect to EPP gRPC port `9002`.

## agentgateway and Istio Injection

When namespace-level Istio injection is enabled, an injected `istio-proxy` in an agentgateway data
plane pod can intercept the EPP external-processing gRPC connection and cause HTTP 500 responses.
Use an `AgentgatewayParameters` resource in the same namespace as the `Gateway`:

```yaml
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayParameters
metadata:
  name: inference-gateway-params
spec:
  deployment:
    spec:
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
```

Reference it from the Gateway:

```yaml
spec:
  gatewayClassName: agentgateway
  infrastructure:
    parametersRef:
      group: agentgateway.dev
      kind: AgentgatewayParameters
      name: inference-gateway-params
```

`AgentgatewayParameters` is a local reference and must be in the Gateway namespace. Use a
per-Gateway resource unless disabling injection is an intentional cluster-wide policy.