Vanilla vLLM GAIE On-ramp

Use this on-ramp to add Dynamo’s advanced KV-aware routing to Gateway API Inference Extension (GAIE) deployments that run stock vllm serve workers. Stock vLLM means the upstream vLLM container images. You do not need to replace them with Dynamo vLLM images, install the Dynamo operator, or migrate the rest of the deployment to Dynamo. The standalone Dynamo Endpoint Picker Plugin (EPP) adds the routing layer while your Gateway, HTTPRoute, and InferencePool remain standard GAIE resources and your workers keep running the upstream vLLM images.

The EPP runs the same Dynamo worker selection service used by a full Dynamo deployment. In standalone mode, it runs in-process without a DynamoGraphDeployment (DGD) or the Dynamo distributed runtime and event plane.

This is still the Gateway API routing topology described in KV-Aware Routing on Kubernetes. Standalone mode changes who creates the resources and how the EPP discovers worker state; it does not add a third request-entry topology.

Architecture

The InferencePool is the source of truth for the worker labels and HTTP port. The EPP watches that pool and Ready pods in its namespace, subscribes directly to each worker’s KV event socket, and calls vLLM’s /v1/chat/completions/render endpoint to tokenize each routing request. The Gateway forwards the original request to the selected pod. The render Service is not a separate renderer workload; it is a stable Kubernetes Service address for the HTTP endpoint served by the same vLLM pods.

For the operator-managed architecture and generated-resource boundary, see Gateway API Routing Architecture.

Before You Begin

You need:

• A Kubernetes cluster with GPU nodes.
• kubectl and Helm configured for the cluster.
• A Dynamo source checkout.
• A container registry that every cluster node can pull from.
• Two schedulable GPUs for the example’s two vLLM replicas. You can scale the worker Deployment to one replica for a smoke test.

The example serves the public Qwen/Qwen3-0.6B model, which does not require a Hugging Face token.

Build the EPP Image

Standalone mode is not available in a published Dynamo image yet. From the repository root, build the Rust EPP from the current source revision and push it to your container registry:

$
export EPP_IMAGE=registry.example.com/your-project/dynamo-rust-epp:standalone
$
$
make -C deploy/inference-gateway/ext-proc \
>
  IMAGE_TAG="$EPP_IMAGE" \
>
  image-push

Choose a Deployment Path

Choose the path that matches your starting point:

• Deploy from Scratch installs the Gateway API layer and applies the complete example, including vLLM workers, the EPP, an InferencePool, and an HTTPRoute.
• Use the Dynamo EPP in an Existing GAIE Deployment keeps your current Gateway, HTTPRoute, InferencePool, and vLLM workers. Deploy a replacement EPP as shown below, or adapt the existing EPP Deployment in place with the same configuration.

Both paths use the same example names so the verification commands apply to either path:

Set the shared namespace variables:

$
export NAMESPACE=gaie-vllm-onramp
$
export AGW_NAMESPACE=agentgateway-system

$
./deploy/inference-gateway/scripts/install_gaie_crd_agentgateway.sh

$
kubectl wait gateway/inference-gateway -n "$AGW_NAMESPACE" \
>
  --for=condition=Programmed --timeout=180s

$
read -rsp "Hugging Face token: " HF_TOKEN
$
echo
$
kubectl create secret generic hf-token-secret -n "$NAMESPACE" \
>
  --from-literal=HF_TOKEN="$HF_TOKEN"
$
unset HF_TOKEN

1
env:
2
  - name: HF_TOKEN
3
    valueFrom:
4
      secretKeyRef:
5
        name: hf-token-secret
6
        key: HF_TOKEN

$
sed "s#dynamo/dynamo-rust-epp:dev#$EPP_IMAGE#" \
>
  deploy/inference-gateway/ext-proc/examples/onramp/agg.yaml |
$
  kubectl apply -n "$NAMESPACE" -f -

Verify the Deployment

These commands apply to both paths because they use the same example names. Wait for the vLLM workers and EPP:

$
kubectl wait deployment/vllm-qwen -n "$NAMESPACE" \
>
  --for=condition=Available --timeout=600s
$
$
kubectl wait deployment/dynamo-epp -n "$NAMESPACE" \
>
  --for=condition=Available --timeout=180s

Confirm that the route attached to the intended Gateway:

$
kubectl get httproute/vllm-qwen-route -n "$NAMESPACE" \
>
  -o jsonpath='{range .status.parents[*]}{.parentRef.name}{"\\t"}{range .conditions[*]}{.type}={.status}{" "}{end}{"\\n"}{end}'

The output should identify inference-gateway and report Accepted=True and ResolvedRefs=True.

Find the Service generated for the Gateway and start a port-forward:

$
export GATEWAY_SERVICE=$(kubectl get service -n "$AGW_NAMESPACE" \
>
  -l gateway.networking.k8s.io/gateway-name=inference-gateway \
>
  -o jsonpath='{.items[0].metadata.name}')
$
$
kubectl port-forward -n "$AGW_NAMESPACE" "service/$GATEWAY_SERVICE" 8000:80

In another terminal, send an OpenAI-compatible request:

$
curl --max-time 120 -sS http://localhost:8000/v1/chat/completions \
>
  -H 'content-type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "messages": [
>
      {"role": "user", "content": "Write one sentence about prefix caching."}
>
    ],
>
    "max_tokens": 64
>
  }'

Inspect the EPP logs near the request:

$
kubectl logs -n "$NAMESPACE" deployment/dynamo-epp --tail=200

The logs should show worker discovery and endpoint selection. If the model responds but the EPP logs show no selection, the route is bypassing the InferencePool.

EPP Replication

Each EPP replica has an in-process selector and KV index. When DYN_EPP_PEER_SERVICE is set, replicas watch their own Service’s EndpointSlice resources and discover its named TCP replica-agg port. They synchronize admission, prefill-complete, and free events so active-load accounting converges across replicas.

Replica synchronization does not copy the full KV index to a new EPP. A new replica warms its index from live worker events and optional replay. For consistency details, see Standalone Selection Service.

For a single EPP replica, set spec.replicas: 1 and remove DYN_EPP_PEER_SERVICE, POD_IP, and the replica-agg Service port.

Standalone EPP Configuration Reference

Scope and Limitations

The standalone path currently targets aggregated vLLM serving with data-parallel size 1 per pod. It does not provide the DGD/operator lifecycle, Dynamo runtime discovery, disaggregated prefill/decode orchestration, request migration, topology-aware routing, or full operator-managed observability.

Use Route Requests with Gateway API when you want the supported operator-managed lifecycle, Dynamo workers, or disaggregated serving. Use Gateway API Routing Reference for the DGD and operator-generated resource contract; that contract does not configure standalone mode.

Troubleshoot

Follow the full request path when debugging:

$
kubectl describe gateway/inference-gateway -n "$AGW_NAMESPACE"
$
kubectl describe httproute/vllm-qwen-route -n "$NAMESPACE"
$
kubectl get inferencepool/vllm-qwen-pool -n "$NAMESPACE" -o yaml
$
kubectl get pods -n "$NAMESPACE" -l app=vllm-qwen -o wide
$
kubectl logs -n "$NAMESPACE" deployment/dynamo-epp --tail=200

Clean Up the From-Scratch Example

If you used the Deploy from Scratch path, delete the example resources:

$
kubectl delete -n "$NAMESPACE" \
>
  -f deploy/inference-gateway/ext-proc/examples/onramp/agg.yaml

Delete the namespace only if it contains no resources you want to keep:

$
kubectl delete namespace "$NAMESPACE"