Gateway API Routing Reference | NVIDIA Dynamo Documentation

Use this reference for a DynamoGraphDeployment (DGD) that routes requests through the Gateway API Inference Extension (GAIE) and the Dynamo Endpoint Picker Plugin (EPP). For the procedure, see Route Requests with Gateway API. For architecture, see Gateway API Routing Architecture.

DGD EPP Fields

The EPP is a DGD component with type: epp. The Dynamo operator creates the EPP workload, Service, configuration, and InferencePool from these fields.

spec.components[].type

stringRequired

Set to epp for the Endpoint Picker Plugin component. A DGD can contain at most one EPP component.

Value: epp

spec.components[].replicas

integerDefaults to 1

Number of EPP pods. Increase this value when endpoint-picker request volume requires more EPP capacity.

spec.components[].eppConfig

EPPConfigRequired

EPP configuration source. Set exactly one of config or configMapRef.

config

EndpointPickerConfig

Inline endpoint-picker configuration. The operator serializes the object into a generated ConfigMap. Mutually exclusive with configMapRef.

configMapRef

ConfigMapKeySelector

Reference to a user-managed ConfigMap containing an EndpointPickerConfig. Mutually exclusive with config.

spec.components[].frontendSidecar

stringRequired

Name of the Frontend container in each routable worker pod. The named container must exist in podTemplate.spec.containers and run with --router-mode direct.

For the complete component schema, see the DynamoComponentDeployment Reference.

EndpointPickerConfig Fields

plugins

[]PluginRequired

Plugin instances available to scheduling profiles. A plugin can filter endpoints, score endpoints, choose a result, or establish aggregated and disaggregated profiles.

name

string

Local name used by schedulingProfiles[].plugins[].pluginRef. Omit only for a plugin that does not need to be referenced directly, such as disagg-profile-handler.

type

stringRequired

Plugin implementation.

Common Dynamo values: disagg-profile-handler label-filter dyn-prefill-scorer dyn-decode-scorer max-score-picker

parameters

object

Plugin-specific configuration. The operator preserves this object without pruning unknown fields.

schedulingProfiles

[]SchedulingProfileRequired

Ordered plugin chains for endpoint selection. Aggregated deployments use a decode profile. Disaggregated deployments use both prefill and decode profiles.

name

stringRequired

Worker role handled by the profile.

Dynamo values: prefill decode

plugins

[]PluginRefRequired

Plugins applied by the profile.

pluginRef

stringRequired

Name of a plugin declared in the top-level plugins list.

weight

integerDefaults to 1

Relative influence of the plugin in the scheduling profile. Keep required filters and the final picker enabled. Validate scorer-weight changes with production-like traffic.

Dynamo Plugin Parameters

Label Filter

parameters.label

stringRequired

Pod label used to identify eligible workers. For DGD v1beta1 components, use nvidia.com/dynamo-component-type.

parameters.validValues

[]stringRequired

Accepted values for the selected label. Use prefill or decode to separate worker roles.

parameters.allowsNoLabel

booleanDefaults to false

Whether pods without the selected label remain eligible. Aggregated profiles can set this to true when worker pods do not carry a separate decode role. Disaggregated profiles should use false.

EPP Runtime Environment

Set these values on the EPP component’s main container. Shared KV router settings use the same semantics as Frontend-hosted routing; this section identifies the settings most relevant to the Gateway topology.

DYN_MODEL_NAME

string

Model identifier used to load tokenizer and model configuration. Set it when discovery does not provide the model name. It must match the model served by the workers.

DYN_KV_CACHE_BLOCK_SIZE

integer

Backend KV cache block size. Set it when discovery does not provide the value. It must match the worker backend’s block size; a mismatch changes prefix block hashes and produces incorrect overlap scores.

DYN_USE_KV_EVENTS

booleanDefaults to true

Enables worker-published KV event consumption in the EPP router. When false, the router relies on predicted local state instead of precise worker cache events.

DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT

numberDefaults to 1.0

Credit applied to device-local prompt-prefix overlap. Higher values prefer workers that already hold more of the prompt prefix. Set to 0 to remove device-local cache overlap from endpoint scoring.

DYN_ROUTER_KV_OVERLAP_SCORE_CREDIT_DECAY

numberDefaults to 0.0

Reduces overlap credit as active prefill load rises above the least-loaded eligible worker. 0 disables decay.

DYN_ROUTER_PREFILL_LOAD_SCALE

numberDefaults to 1.0

Scales prompt-side prefill load after cache-hit credits are applied.

DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT

numberDefaults to 0.0

Adds block-equivalent routing cost for each active request on a candidate worker. Tune this value only when decode step latency depends materially on active batch size.

DYN_ROUTER_TEMPERATURE

numberDefaults to 0.0

Controls worker exploration through normalized softmax sampling. 0 selects deterministically.

DYN_ROUTER_REPLICA_SYNC

booleanDefaults to false

Publishes and consumes best-effort active-sequence state across EPP replicas through the Dynamo event plane.

DYN_ROUTER_TRACK_ACTIVE_BLOCKS

booleanDefaults to true

Includes blocks used by active generation in worker load accounting.

DYN_ROUTER_TRACK_OUTPUT_BLOCKS

booleanDefaults to false

Predicts output blocks during generation and decays them as requests make progress.

DYN_ROUTER_TRACK_PREFILL_TOKENS

booleanDefaults to true

Includes active prompt-side prefill tokens in worker load accounting.

For the full shared configuration surface and tuning guidance, see Configuration and Tuning.

Generated Resource Contract

For a DGD named <dgd-name>, the operator generates an InferencePool named <dgd-name>-pool in the DGD namespace. Its endpointPickerRef points to the generated EPP Service on gRPC port 9002, and its target port points to worker Frontend sidecars on port 8000.

1 spec:
2   selector:
3     matchLabels:
4       nvidia.com/dynamo-component-class: worker
5       nvidia.com/dynamo-namespace: <dynamo-namespace>
6   endpointPickerRef:
7     kind: Service
8     name: <epp-service-name>
9     port:
10       number: 9002
11   targetPorts:
12   - number: 8000

Treat the generated InferencePool, EPP Service, Deployment, and inline configuration ConfigMap as operator-owned resources. Make persistent changes in the DGD.

HTTPRoute Contract

spec.parentRefs[]

ParentReferenceRequired

Gateway that receives model traffic. The Gateway and route can be in different namespaces only when the Gateway listener allows the route namespace.

spec.rules[].backendRefs[]

BackendRefRequired

Reference to the operator-generated InferencePool.

group

stringRequired

Set to inference.networking.k8s.io.

kind

stringRequired

Set to InferencePool.

name

stringRequired

Generated pool name, normally <dgd-name>-pool.

port

integerRequired

Worker Frontend target port. The operator-generated pool uses 8000.

Request Mutation Contract

The EPP returns the selected endpoint and routing metadata to the Gateway. The Gateway forwards the mutated request to the selected Frontend sidecar.

x-dynamo-worker-instance-id

HTTP header

Aggregated or decode worker selected for the request.

x-dynamo-dp-rank

HTTP header

Data-parallel rank selected for the aggregated or decode worker.

x-dynamo-routing-mode

HTTP header

Routing topology selected for the request.

Values: aggregated disaggregated

x-dynamo-prefill-instance-id

HTTP header

Prefill worker selected for a disaggregated request.

x-dynamo-prefill-dp-rank

HTTP header

Data-parallel rank selected for the prefill worker.

For supported body-bearing OpenAI requests, the EPP can inject precomputed token data into nvext.token_data so the Frontend sidecar does not repeat tokenization.

Service Mesh Configuration

dynamo.serviceMesh.enabled

booleanDefaults to false

Generates service-mesh resources for EPP Services when the provider CRDs are installed.

dynamo.serviceMesh.provider

stringDefaults to istio

Service-mesh provider. Only Istio is supported.

Value: istio

dynamo.serviceMesh.istio.tlsMode

stringDefaults to SIMPLE

TLS mode in generated Istio DestinationRule resources.

dynamo.serviceMesh.istio.insecureSkipVerify

booleanDefaults to true

Skips server certificate verification for the EPP’s self-signed serving certificate.

dynamo.serviceMesh.istio.clientCertificate

string

Client certificate path for MUTUAL TLS mode.

dynamo.serviceMesh.istio.privateKey

string

Client private-key path for MUTUAL TLS mode.

dynamo.serviceMesh.istio.caCertificates

string

Certificate authority path for MUTUAL TLS mode.

When service-mesh integration is enabled, the operator creates a DestinationRule for each EPP Service. If the operator does not manage the mesh integration, create an equivalent rule so the Gateway proxy can connect to EPP gRPC port 9002.

agentgateway and Istio Injection

When namespace-level Istio injection is enabled, an injected istio-proxy in an agentgateway data plane pod can intercept the EPP external-processing gRPC connection and cause HTTP 500 responses. Use an AgentgatewayParameters resource in the same namespace as the Gateway:

1 apiVersion: agentgateway.dev/v1alpha1
2 kind: AgentgatewayParameters
3 metadata:
4   name: inference-gateway-params
5 spec:
6   deployment:
7     spec:
8       template:
9         metadata:
10           annotations:
11             sidecar.istio.io/inject: "false"

Reference it from the Gateway:

1 spec:
2   gatewayClassName: agentgateway
3   infrastructure:
4     parametersRef:
5       group: agentgateway.dev
6       kind: AgentgatewayParameters
7       name: inference-gateway-params

AgentgatewayParameters is a local reference and must be in the Gateway namespace. Use a per-Gateway resource unless disabling injection is an intentional cluster-wide policy.