Installation

Install Gateway API Routing

Install Gateway API, the Inference Extension, and a compatible Gateway implementation for Dynamo EPP routing.
View as Markdown

Complete this installation only if you plan to use the Gateway API routing topology. The default Dynamo Frontend topology works without Gateway API, the Gateway API Inference Extension (GAIE), or an Endpoint Picker Plugin (EPP). To use the default topology, continue to KV-Aware Routing with the Dynamo Frontend.

Install this integration after the Dynamo platform. The Dynamo operator needs the GAIE InferencePool API before it can reconcile a DynamoGraphDeployment with an EPP component.

1

Choose a Gateway implementation

Dynamo requires a Gateway implementation that supports GAIE InferencePool resources and calls the configured EPP before forwarding a request.

  • Use agentgateway when the cluster does not already standardize on Istio or when you want a focused AI gateway deployment.
  • Use Istio when the cluster already relies on Istio for ingress, service-mesh policy, or telemetry.

Other Gateway implementations can work when they support the same GAIE endpoint-picker contract.

2

Set the installation variables

Set the namespace where the Gateway and model routes will live. Keep the Dynamo version aligned with the platform installation.

$export NAMESPACE=my-model
$export DYNAMO_SYSTEM_NAMESPACE=dynamo-system
$export DYNAMO_VERSION=1.3.0
$export AGW_NAMESPACE=agentgateway-system
3

Install the Gateway API layer

From the root of a Dynamo source checkout, run the repository installation script:

$./deploy/inference-gateway/scripts/install_gaie_crd_agentgateway.sh

The script installs:

  • Gateway API CRDs.
  • GAIE CRDs.
  • agentgateway with the inference extension enabled.
  • An AgentgatewayParameters resource that prevents Istio sidecar injection into Gateway proxy pods.
  • A Gateway named inference-gateway in $NAMESPACE.

The script pins compatible dependency versions. Review it before running it when your platform team owns Gateway API or agentgateway lifecycle.

4

Verify the required APIs

Confirm that the cluster serves the Gateway, route, and inference-pool resources:

$kubectl api-resources | grep -E 'gateways|httproutes|inferencepools'
$kubectl get gatewayclass

Expected API groups include:

gateway.networking.k8s.io
inference.networking.k8s.io
5

Verify the Gateway

Wait for the inference-gateway resource to become programmed:

$kubectl wait gateway/inference-gateway -n "$NAMESPACE" \
>  --for=condition=Programmed --timeout=180s
$
$kubectl get gateway inference-gateway -n "$NAMESPACE"

If Programmed remains false, inspect the Gateway and the controller for your implementation:

$kubectl describe gateway inference-gateway -n "$NAMESPACE"
$kubectl get pods --all-namespaces | grep -E 'agentgateway|istio'

Next Step

Add an EPP component and route an existing DGD with Route Requests with Gateway API.