> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Inference Gateway (GAIE) ## Inference Gateway Setup with Dynamo # Inference Gateway (GAIE) Integrate Dynamo with the Gateway API Inference Extension for intelligent KV-aware request routing at the gateway layer. ## Features - EPP's default kv-routing approach is not token-aware because the prompt is not tokenized. But the Dynamo plugin uses a token-aware KV algorithm. It employs the dynamo router which implements kv routing by running your model's tokenizer inline. The EPP plugin configuration lives in [`helm/dynamo-gaie/epp-config-dynamo.yaml`](https://github.com/ai-dynamo/dynamo/blob/v1.0.2/deploy/inference-gateway/standalone/helm/dynamo-gaie/epp-config-dynamo.yaml), following the checked-in GAIE/EPP configuration layout used by this repository. - Dynamo Integration with the Inference Gateway supports Aggregated and Disaggregated Serving. A request only exercises disaggregated routing when the EPP config defines a `prefill` profile and prefill workers are available. The standalone [`epp-config-dynamo.yaml`](https://github.com/ai-dynamo/dynamo/blob/v1.0.2/deploy/inference-gateway/standalone/helm/dynamo-gaie/epp-config-dynamo.yaml) currently only defines a `decode` profile, while the recipe examples use separate aggregated and disaggregated configs under `recipes/llama-3-70b/vllm/agg/gaie/` and `recipes/llama-3-70b/vllm/disagg-single-node/gaie/`. Unless `DYN_ENFORCE_DISAGG=true`, deployments without a `prefill` profile or prefill workers fall back to aggregated serving. - GAIE integration supports Data Parallelism. - If you want to use LoRA deploy Dynamo without the Inference Gateway. - Currently, these setups are only tested with the kGateway Inference Gateway. ## Prerequisites - Kubernetes cluster with kubectl configured - NVIDIA GPU drivers installed on worker nodes ## Installation Steps ### 1. Install Dynamo Platform ### [See Quickstart Guide](/dynamo/v1.0.2/kubernetes-deployment/deployment-guide) to install Dynamo Kubernetes Platform. If you are installing from the source tree rather than a release chart, follow [Path B: Custom Build from Source](/dynamo/v1.0.2/kubernetes-deployment/deployment-guide/detailed-installation-guide#path-b-custom-build-from-source) and run `helm dep build ./platform/` before `helm install` so the vendored subcharts match the local chart contents. ### 2. Deploy Inference Gateway ### First, deploy an inference gateway service. In this example, we'll install `kgateway` based gateway implementation. ```bash cd deploy/inference-gateway export NAMESPACE=my-model # You can put the inference gateway into another namespace and then adjust your http-route.yaml ./scripts/install_gaie_crd_kgateway.sh ``` **Note**: The manifest at `config/manifests/gateway/kgateway/gateway.yaml` uses `gatewayClassName: agentgateway`, but kGateway's helm chart creates a GatewayClass named `kgateway`. The patch command in the script fixes this mismatch. #### f. Verify the Gateway is running ```bash kubectl get gateway inference-gateway # Sample output # NAME CLASS ADDRESS PROGRAMMED AGE # inference-gateway kgateway True 1m ``` ### 3. Setup secrets ### Do not forget docker registry secret if needed. ```bash kubectl create secret docker-registry docker-imagepullsecret \ --docker-server=$DOCKER_SERVER \ --docker-username=$DOCKER_USERNAME \ --docker-password=$DOCKER_PASSWORD \ --namespace=$NAMESPACE ``` Do not forget to include the HuggingFace token. ```bash export HF_TOKEN=your_hf_token kubectl create secret generic hf-token-secret \ --from-literal=HF_TOKEN=${HF_TOKEN} \ -n ${NAMESPACE} ``` ### 4. Build EPP image (Optional) You can either use the provided Dynamo FrontEnd image for the EPP image or you need to build your own Dynamo EPP custom image following the steps below. ```bash # export env vars export DOCKER_SERVER=ghcr.io/nvidia/dynamo # Container registry export IMAGE_TAG=YOUR-TAG # Or auto from git tag cd deploy/inference-gateway/epp make all # Do everything in one command # or make all-push to also push # Or step-by-step make dynamo-lib # Build Dynamo library and copy to project make image-load # Build Docker image and load locally make image-push # Build and push to registry make info # Check image tag ``` #### All-in-one Targets | Target | Description | |--------|-------------| | `make dynamo-lib` | Build Dynamo static library and copy to project | | `make all` | Build Dynamo lib + Docker image + load locally | | `make all-push` | Build Dynamo lib + Docker image + push to registry | ### 5. Deploy We recommend deploying Inference Gateway's Endpoint Picker as a Dynamo operator's managed component. Alternatively, you could deploy it as a standalone pod. Note that when deploying Dynamo with the Inference Gateway Extension each worker must have the FrontEnd as a sidecar. #### 5.a. Deploy as a DGD component (recommended) We provide an example for the Qwen vLLM below. You have to deploy the Dynamo Graph and the HttpRoute service. For the HttpRoute service make sure to specify the namespace where your gateway (i.e. kGateway was deployed) as shown below. ```bash parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: inference-gateway namespace: my-model # the namespace where your gateway is deployed. ``` ```bash cd # kubectl get httproutes -n my-model # Make sure you do not have an incompatible HttpRoute running, delete if so. # Choose disagg or agg example kubectl apply -f examples/backends/vllm/deploy/gaie/disagg.yaml -n my-model # or kubectl apply -f examples/backends/vllm/deploy/gaie/agg.yaml -n my-model # make sure to apply the route kubectl apply -f examples/backends/vllm/deploy/gaie/http-route.yaml -n my-model ``` Examples for other models can be found in the recipes folder. ```bash # Deploy PVC, having first Update `storageClassName` in recipes/llama-3-70b/model-cache/model-cache.yaml to match your cluster before deploying kubectl apply -f recipes/llama-3-70b/model-cache/model-cache.yaml -n ${NAMESPACE} kubectl apply -f recipes/llama-3-70b/model-cache/model-download.yaml -n ${NAMESPACE} ``` We provide examples for llama-3-70b vLLM under the `recipes/llama-3-70b/vllm/agg/gaie/` for aggregated and `recipes/llama-3-70b/vllm/disagg-single-node/gaie/` for disaggregated serving. Note for the aggregated serving you need to disable DYN_ENFORCE_DISAGG in epp config. ```bash - name: DYN_ENFORCE_DISAGG value: "false" ``` Use the proper folder in commands below. ```bash # Deploy your Dynamo Graph. # agg kubectl apply -f recipes/llama-3-70b/vllm/agg/gaie/deploy.yaml -n ${NAMESPACE} # Deploy the GAIE http-route CR. Adjust parentRefs.namespace in this file first to point where your gateway is. kubectl apply -f recipes/llama-3-70b/vllm/agg/gaie/http-route.yaml -n ${NAMESPACE} # or disagg kubectl apply -f recipes/llama-3-70b/vllm/disagg-single-node/gaie/deploy.yaml -n ${NAMESPACE} kubectl apply -f recipes/llama-3-70b/vllm/disagg-single-node/gaie/http-route.yaml -n ${NAMESPACE} ``` - When using GAIE the FrontEnd does not choose the workers. The routing is determined in the EPP. - The FrontEnd must run with `--router-mode direct` so that it respects the EPP's routing decisions passed via request headers. - Use the `frontendSidecar` field on a worker service to have the operator automatically inject a fully configured frontend sidecar container with all required Dynamo env vars, probes, and ports: ```yaml frontendSidecar: image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:my-tag args: - --router-mode - direct envFromSecret: hf-token-secret ``` - The pre-selected worker (decode and prefill in case of the disaggregated serving) are passed in the request headers. - The `--router-mode direct` flag ensures the routing respects this selection. **Startup Probe Timeout:** The EPP has a default startup probe timeout of 30 minutes (10s × 180 failures). If your model takes longer to load, increase the `failureThreshold` in the EPP's `startupProbe`. For example, to allow 60 minutes for startup: ```yaml extraPodSpec: mainContainer: startupProbe: failureThreshold: 360 # 10s × 360 = 60 minutes ``` **Gateway Namespace** Note that this assumes your gateway is installed into `NAMESPACE=my-model` (examples' default) If you installed it into a different namespace, you need to adjust the HttpRoute entry in `http-route.yaml`. #### 5.b. Deploy as a standalone pod We do not recommend this method but there are hints on how to do this here. ##### 5.b.1 Deploy Your Model ### ##### 5.b.2 Install Dynamo GIE helm chart ### ```bash cd deploy/inference-gateway/standalone # Export the EPP image - use the Dynamo FrontEnd image or build your own EPP image (see section 4) export EPP_IMAGE= ``` Create a model configuration file similar to the vllm_agg_qwen.yaml for your model. ```bash helm upgrade --install dynamo-gaie ./helm/dynamo-gaie -n my-model -f ./vllm_agg_qwen.yaml --set-string extension.image=$EPP_IMAGE ``` By default, the Kubernetes discovery mechanism is used. If you prefer etcd, please use the `--set epp.dynamo.useEtcd=true` flag below. ```bash helm upgrade --install dynamo-gaie ./helm/dynamo-gaie -n my-model -f ./vllm_agg_qwen.yaml --set-string extension.image=$EPP_IMAGE --set epp.dynamo.useEtcd=true ``` Key configurations include: - An InferenceModel resource for the Qwen model - A service for the inference gateway - Required RBAC roles and bindings - RBAC permissions - dynamoGraphDeploymentName - the name of the Dynamo Graph where your model is deployed. **Configuration** You can configure the plugin by setting environment variables in the EPP component of your DGD in case of the operator-managed installation or in your [values.yaml](https://github.com/ai-dynamo/dynamo/blob/v1.0.2/deploy/inference-gateway/standalone/helm/dynamo-gaie/values.yaml). Common Vars for Routing Configuration: **Enabling KV-Aware Routing (most precise)** KV-aware routing uses live KV cache block events from workers so the EPP can route requests to the worker with the best prefix cache overlap. To enable it (default): 1. **Workers — enable prefix caching and KV event publishing.** Each worker must publish KV cache events to event plane (NATS/ZMQ) so the EPP's router can track per-worker cache state. - **vLLM:** Pass `--enable-prefix-caching` and `--kv-events-config '{"enable_kv_cache_events":true}'`. - **SGLang:** Pass `--kv-events-config` with the appropriate endpoint. - **TRT-LLM:** Pass `--publish-events-and-metrics`. 2. **EPP — leave `DYN_USE_KV_EVENTS` at its default (`true`).** The EPP subscribes to worker KV events via event plane (NATS/ZMQ) and uses them for prefix-overlap scoring. 3. **Block size — must be consistent.** The `--block-size` on all workers must match `DYN_KV_CACHE_BLOCK_SIZE` on the EPP (default: 128). Mismatched block sizes cause incorrect block hash computation. **Disabling KV-Aware Routing** To disable the EPP from listening for KV events (e.g., when prefix caching is off on workers, or for simpler load-balanced routing): 1. **EPP:** Set `DYN_USE_KV_EVENTS=false`. The router falls back to approximate mode (routing decisions are tracked locally with TTL decay instead of live KV events from workers). 2. **Workers:** Pass `--no-enable-prefix-caching` to disable prefix caching entirely. Without prefix caching, no KV events are generated regardless of other flags. 3. **Optionally** set `DYN_OVERLAP_SCORE_WEIGHT=0` on the EPP to skip prefix-overlap scoring altogether, making the router select workers based on load only. - Set `DYN_BUSY_THRESHOLD` to configure the upper bound on how "full" a worker can be (often derived from kv_active_blocks or other load metrics) before the router skips it. If the selected worker exceeds this value, routing falls back to the next best candidate. By default the value is negative meaning this is not enabled. - Set `DYN_ENFORCE_DISAGG=true` (default: `false`) to control per-request behavior when prefill workers are unavailable: - **`true` (recommended for disaggregated serving):** Requests fail with an error if prefill workers are not available. Use this when disaggregated serving is required and aggregated fallback is not acceptable. - **`false` (default):** Requests gracefully fall back to aggregated mode (skip prefill, route directly to decode) when prefill workers are not available. When prefill workers appear later, subsequent requests automatically use disaggregated routing. - Set `DYN_OVERLAP_SCORE_WEIGHT` to weigh how heavily the score uses token overlap (predicted KV cache hits) versus other factors (load, historical hit rate). Higher weight biases toward reusing workers with similar cached prefixes. (default: 1) - Set `DYN_ROUTER_TEMPERATURE` to soften or sharpen the selection curve when combining scores. Low temperature makes the router pick the top candidate deterministically; higher temperature lets lower-scoring workers through more often (exploration). - `DYN_ROUTER_TEMPERATURE` — Temperature for worker sampling via softmax (default: 0.0) - `DYN_ROUTER_REPLICA_SYNC` — Enable replica synchronization (default: false) - `DYN_ROUTER_TRACK_ACTIVE_BLOCKS` — Track active blocks (default: true) - `DYN_ROUTER_TRACK_OUTPUT_BLOCKS` — Track output blocks during generation (default: false) - See the [KV cache routing design](/dynamo/v1.0.2/design-docs/component-design/router-design) for details. Stand-Alone installation only: - Overwrite the `DYN_NAMESPACE` env var if needed to match your model's dynamo namespace. ### 6. Verify Installation ### Check that all resources are properly deployed: ```bash kubectl get inferencepool kubectl get httproute kubectl get service kubectl get gateway ``` Sample output: ```bash # kubectl get inferencepool NAME AGE qwen-pool 33m # kubectl get httproute NAME HOSTNAMES AGE qwen-route 33m ``` ### 7. Usage ### The Inference Gateway provides HTTP endpoints for model inference. #### 1: Populate gateway URL for your k8s cluster #### a. To test the integration in minikube, proceed as below: Use minikube tunnel to expose the gateway to the host. This requires `sudo` access to the host machine. Alternatively, you can use port-forward to expose the gateway to the host as shown in alternative (b). ```bash # in first terminal ps aux | grep "minikube tunnel" | grep -v grep # make sure minikube tunnel is not already running. minikube tunnel # start the tunnel # in second terminal where you want to send inference requests GATEWAY_URL=$(kubectl get svc inference-gateway -n my-model -o jsonpath='{.spec.clusterIP}') && echo $GATEWAY_URL ``` b. To test on a cluster use commands below: use port-forward to expose the gateway to the host ```bash # in first terminal kubectl port-forward svc/inference-gateway 8000:80 -n ${NAMESPACE} # for NAMESPACE put wherever you installed the gateway i.e. kgateway-system or my-model # in second terminal where you want to send inference requests GATEWAY_URL=http://localhost:8000 ``` #### 2: Check models deployed to inference gateway #### a. Query models: ```bash # in the second terminal where you GATEWAY_URL is set curl $GATEWAY_URL/v1/models | jq . # or if you added the host name to http route: curl -H "Host: llama3-70b-disagg.example.com" $GATEWAY_URL/v1/models | jq . ``` Sample output: ```json { "data": [ { "created": 1753768323, "id": "Qwen/Qwen3-0.6B", "object": "object", "owned_by": "nvidia" } ], "object": "list" } ``` b. Send inference request to gateway: ```bash MODEL_NAME="Qwen/Qwen3-0.6B" curl $GATEWAY_URL/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "'"${MODEL_NAME}"'", "messages": [ { "role": "user", "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden." } ], "stream":false, "max_tokens": 30, "temperature": 0.0 }' ``` or ```bash MODEL_NAME="RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic" curl -H "Host: llama3-70b-disagg.example.com" http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "'"${MODEL_NAME}"'", "messages": [ { "role": "user", "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden." } ], "stream":false, "max_tokens": 30, "temperature": 0.0 }' ``` Sample inference output: ```json { "choices": [ { "finish_reason": "stop", "index": 0, "logprobs": null, "message": { "audio": null, "content": "\nOkay, I need to develop a character background for the user's query. Let me start by understanding the requirements. The character is an", "function_call": null, "refusal": null, "role": "assistant", "tool_calls": null } } ], "created": 1753768682, "id": "chatcmpl-772289b8-5998-4f6d-bd61-3659b684b347", "model": "Qwen/Qwen3-0.6B", "object": "chat.completion", "service_tier": null, "system_fingerprint": null, "usage": { "completion_tokens": 29, "completion_tokens_details": null, "prompt_tokens": 196, "prompt_tokens_details": null, "total_tokens": 225 } } ``` ***If you have more than one HttpRoute running on the cluster*** Add the host to your HttpRoute.yaml and add the header `curl -H "Host: llama3-70b-agg.example.com" ...` or `curl -H "Host: llama3-70b-disagg.example.com" http://localhost:8000/v1/models` ```bash spec: hostnames: - llama3-70b-agg.example.com ``` ### 8. Deleting the installation ### If you need to uninstall run: ```bash kubectl delete dynamoGraphDeployment vllm-agg helm uninstall dynamo-gaie -n my-model # To uninstall GAIE # 1. Delete the inference-gateway kubectl delete gateway inference-gateway --ignore-not-found # 2. Uninstall kgateway helm releases helm uninstall kgateway -n kgateway-system helm uninstall kgateway-crds -n kgateway-system # 3. Delete the kgateway-system namespace (optional, cleans up everything in it) helm uninstall kgateway --namespace kgateway-system kubectl delete namespace kgateway-system --ignore-not-found # 4. Delete the Inference Extension CRDs IGW_LATEST_RELEASE=v1.5.0-rc.2 kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${IGW_LATEST_RELEASE}/manifests.yaml --ignore-not-found # 5. Delete the Gateway API CRDs GATEWAY_API_VERSION=v1.4.1 kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/$GATEWAY_API_VERSION/standard-install.yaml --ignore-not-found ``` ## Gateway API Inference Extension Integration This section documents the updated plugin implementation for Gateway API Inference Extension **v1.5.0-rc.2**. ### Router bookkeeping operations EPP performs Dynamo router book keeping operations so the FrontEnd's Router does not have to sync its state. ### Header Routing Hints Since v1.5.0-rc.1, the EPP uses **headers and body mutations** for communicating routing decisions. The plugins set HTTP headers for worker targeting and inject pre-computed token IDs into the request body (`nvext.token_data`) so the frontend sidecar can skip redundant tokenization. #### Headers Set by Dynamo Plugins | Header | Description | Set By | |--------|-------------|--------| | `x-worker-instance-id` | Primary worker ID (decode worker in disagg mode) | kv-aware-scorer | | `x-prefill-instance-id` | Prefill worker ID (disaggregated mode only) | kv-aware-scorer |