Using GAIE with Dynamo

Set the deployment variables

Set the namespace, resource names, model, and local filenames used throughout the procedure:

export NAMESPACE=my-model
export DYNAMO_VERSION=1.4.0
export DGD_NAME=qwen
export DGD_MANIFEST=qwen-gateway.yaml
export ROUTE_NAME=qwen
export ROUTE_MANIFEST=qwen-gateway-route.yaml
export MODEL_NAME=Qwen/Qwen3-0.6B

Copy your working DGD manifest to $DGD_MANIFEST, then make the changes in the next three steps.

Add the EPP component

Add one component with type: epp to qwen-gateway.yaml. The EPP configuration below selects aggregated or decode workers and scores them with the Dynamo decode scorer.

spec:
  components:
  - name: Epp
    type: epp
    replicas: 1
    eppConfig:
      config:
        plugins:
        - type: disagg-profile-handler
        - name: decode-filter
          type: label-filter
          parameters:
            label: nvidia.com/dynamo-component-type
            validValues: [decode]
            allowsNoLabel: true
        - name: dyn-decode
          type: dyn-decode-scorer
        - name: picker
          type: max-score-picker
        schedulingProfiles:
        - name: decode
          plugins:
          - pluginRef: decode-filter
            weight: 1
          - pluginRef: dyn-decode
            weight: 1
          - pluginRef: picker
            weight: 1
    podTemplate:
      spec:
        containers:
        - name: main
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.4.2
          envFrom:
          - secretRef:
              name: hf-token-secret
          env:
          - name: DYN_MODEL_NAME
            value: Qwen/Qwen3-0.6B
          - name: DYN_KV_CACHE_BLOCK_SIZE
            value: "16"

If your deployment uses another Dynamo version, model, secret name, or backend block size, update those values in the manifest. Keep the platform, EPP, Frontend sidecars, and workers on the same Dynamo release line. The EPP block size must match the backend block size.

For a disaggregated graph, add separate prefill and decode filters, scorers, and scheduling profiles. Start from the repository’s disaggregated GAIE example instead of extending the aggregated profile by hand.

Put worker sidecars in direct mode

The EPP selects the worker before the request reaches its pod. In each routable worker component, add a Frontend sidecar and run it in direct mode so it forwards the request without making another worker selection.

spec:
  components:
  - name: VllmDecodeWorker
    type: decode
    frontendSidecar: sidecar-frontend
    podTemplate:
      spec:
        containers:
        - name: main
          # Keep the existing worker configuration.
        - name: sidecar-frontend
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.2
          args:
          - -m
          - dynamo.frontend
          - --router-mode
          - direct
          envFrom:
          - secretRef:
              name: hf-token-secret

Keep the existing worker container in the component. If your worker uses another runtime image or credential source, apply the equivalent values to the sidecar.

Publish KV cache events

To route from actual cache contents, enable prefix caching and KV event publication in each routable worker. For the vLLM worker in qwen-gateway.yaml, include settings equivalent to:

args:
- >-
  python3 -m dynamo.vllm
  --model $MODEL_PATH
  --enable-prefix-caching
  --block-size 16
  --kv-events-config '{"enable_kv_cache_events":true}'

The operator-managed EPP receives these events through the Dynamo event plane. Do not configure a direct EPP-to-vLLM ZMQ subscription for this topology.

Apply the updated DGD

Apply qwen-gateway.yaml, wait for the DGD, and inspect the generated qwen-pool InferencePool:

kubectl apply -n "$NAMESPACE" -f "$DGD_MANIFEST"
kubectl wait -n "$NAMESPACE" \
  dynamographdeployment/$DGD_NAME \
  --for=condition=Ready \
  --timeout=1800s
kubectl get inferencepool "${DGD_NAME}-pool" -n "$NAMESPACE"

The operator also creates the EPP Deployment and Service.

Create the HTTPRoute

Create qwen-gateway-route.yaml. The route matches the model header and points to the generated qwen-pool resource:

cat > "$ROUTE_MANIFEST" <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ${ROUTE_NAME}
spec:
  parentRefs:
  - name: inference-gateway
  rules:
  - matches:
    - headers:
      - name: X-Gateway-Model-Name
        type: Exact
        value: ${MODEL_NAME}
      path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: inference.networking.k8s.io
      kind: InferencePool
      name: ${DGD_NAME}-pool
      port: 8000
    timeouts:
      request: 300s
EOF
kubectl apply -n "$NAMESPACE" -f "$ROUTE_MANIFEST"
kubectl get httproute "$ROUTE_NAME" -n "$NAMESPACE"

Verify the request path

Port-forward the Service created for the Gateway implementation:

export GATEWAY_SERVICE=$(kubectl get service -n "$NAMESPACE" \
  -l gateway.networking.k8s.io/gateway-name=inference-gateway \
  -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n "$NAMESPACE" "service/$GATEWAY_SERVICE" 8000:80

In another terminal, restore the variables and send a request through the Gateway:

export NAMESPACE=my-model
export MODEL_NAME=Qwen/Qwen3-0.6B
curl --max-time 180 -sS http://localhost:8000/v1/chat/completions \
  -H "X-Gateway-Model-Name: $MODEL_NAME" \
  -H 'content-type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Explain KV-aware routing."}],
    "max_tokens": 64
  }' | jq .

Confirm that the EPP handled endpoint selection:

kubectl logs -n "$NAMESPACE" \
  -l nvidia.com/dynamo-component-type=epp \
  --tail=200