Route Requests with Gateway API

Set the deployment variables

Set the namespace, resource names, model, and local filenames used throughout the procedure:

$
export NAMESPACE=my-model
$
export DYNAMO_VERSION=1.3.0
$
export DGD_NAME=qwen
$
export DGD_MANIFEST=qwen-gateway.yaml
$
export ROUTE_NAME=qwen
$
export ROUTE_MANIFEST=qwen-gateway-route.yaml
$
export MODEL_NAME=Qwen/Qwen3-0.6B

Copy your working DGD manifest to $DGD_MANIFEST, then make the changes in the next three steps.

Add the EPP component

Add one component with type: epp to qwen-gateway.yaml. The EPP configuration below selects aggregated or decode workers and scores them with the Dynamo decode scorer.

1
spec:
2
  components:
3
  - name: Epp
4
    type: epp
5
    replicas: 1
6
    eppConfig:
7
      config:
8
        plugins:
9
        - type: disagg-profile-handler
10
        - name: decode-filter
11
          type: label-filter
12
          parameters:
13
            label: nvidia.com/dynamo-component-type
14
            validValues: [decode]
15
            allowsNoLabel: true
16
        - name: dyn-decode
17
          type: dyn-decode-scorer
18
        - name: picker
19
          type: max-score-picker
20
        schedulingProfiles:
21
        - name: decode
22
          plugins:
23
          - pluginRef: decode-filter
24
            weight: 1
25
          - pluginRef: dyn-decode
26
            weight: 1
27
          - pluginRef: picker
28
            weight: 1
29
    podTemplate:
30
      spec:
31
        containers:
32
        - name: main
33
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.3.0
34
          envFrom:
35
          - secretRef:
36
              name: hf-token-secret
37
          env:
38
          - name: DYN_MODEL_NAME
39
            value: Qwen/Qwen3-0.6B
40
          - name: DYN_KV_CACHE_BLOCK_SIZE
41
            value: "16"

If your deployment uses another Dynamo version, model, secret name, or backend block size, update those values in the manifest. Keep the platform, EPP, Frontend sidecars, and workers on the same Dynamo release line. The EPP block size must match the backend block size.

For a disaggregated graph, add separate prefill and decode filters, scorers, and scheduling profiles. Start from the repository’s disaggregated GAIE example instead of extending the aggregated profile by hand.

Put worker sidecars in direct mode

The EPP selects the worker before the request reaches its pod. In each routable worker component, add a Frontend sidecar and run it in direct mode so it forwards the request without making another worker selection.

1
spec:
2
  components:
3
  - name: VllmDecodeWorker
4
    type: decode
5
    frontendSidecar: sidecar-frontend
6
    podTemplate:
7
      spec:
8
        containers:
9
        - name: main
10
          # Keep the existing worker configuration.
11
        - name: sidecar-frontend
12
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0
13
          args:
14
          - -m
15
          - dynamo.frontend
16
          - --router-mode
17
          - direct
18
          envFrom:
19
          - secretRef:
20
              name: hf-token-secret

Keep the existing worker container in the component. If your worker uses another runtime image or credential source, apply the equivalent values to the sidecar.

Publish KV cache events

To route from actual cache contents, enable prefix caching and KV event publication in each routable worker. For the vLLM worker in qwen-gateway.yaml, include settings equivalent to:

1
args:
2
- >-
3
  python3 -m dynamo.vllm
4
  --model $MODEL_PATH
5
  --enable-prefix-caching
6
  --block-size 16
7
  --kv-events-config '{"enable_kv_cache_events":true}'

The operator-managed EPP receives these events through the Dynamo event plane. Do not configure a direct EPP-to-vLLM ZMQ subscription for this topology.

Apply the updated DGD

Apply qwen-gateway.yaml, wait for the DGD, and inspect the generated qwen-pool InferencePool:

$
kubectl apply -n "$NAMESPACE" -f "$DGD_MANIFEST"
$
$
kubectl wait -n "$NAMESPACE" \
>
  dynamographdeployment/$DGD_NAME \
>
  --for=condition=Ready \
>
  --timeout=1800s
$
$
kubectl get inferencepool "${DGD_NAME}-pool" -n "$NAMESPACE"

The operator also creates the EPP Deployment and Service.

Create the HTTPRoute

Create qwen-gateway-route.yaml. The route matches the model header and points to the generated qwen-pool resource:

$
cat > "$ROUTE_MANIFEST" <<EOF
$
apiVersion: gateway.networking.k8s.io/v1
$
kind: HTTPRoute
$
metadata:
$
  name: ${ROUTE_NAME}
$
spec:
$
  parentRefs:
$
  - name: inference-gateway
$
  rules:
$
  - matches:
$
    - headers:
$
      - name: X-Gateway-Model-Name
$
        type: Exact
$
        value: ${MODEL_NAME}
$
      path:
$
        type: PathPrefix
$
        value: /
$
    backendRefs:
$
    - group: inference.networking.k8s.io
$
      kind: InferencePool
$
      name: ${DGD_NAME}-pool
$
      port: 8000
$
    timeouts:
$
      request: 300s
$
EOF
$
$
kubectl apply -n "$NAMESPACE" -f "$ROUTE_MANIFEST"
$
kubectl get httproute "$ROUTE_NAME" -n "$NAMESPACE"

Verify the request path

Port-forward the Service created for the Gateway implementation:

$
export GATEWAY_SERVICE=$(kubectl get service -n "$NAMESPACE" \
>
  -l gateway.networking.k8s.io/gateway-name=inference-gateway \
>
  -o jsonpath='{.items[0].metadata.name}')
$
$
kubectl port-forward -n "$NAMESPACE" "service/$GATEWAY_SERVICE" 8000:80

In another terminal, restore the variables and send a request through the Gateway:

$
export NAMESPACE=my-model
$
export MODEL_NAME=Qwen/Qwen3-0.6B
$
$
curl --max-time 180 -sS http://localhost:8000/v1/chat/completions \
>
  -H "X-Gateway-Model-Name: $MODEL_NAME" \
>
  -H 'content-type: application/json' \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "messages": [{"role": "user", "content": "Explain KV-aware routing."}],
>
    "max_tokens": 64
>
  }' | jq .

Confirm that the EPP handled endpoint selection:

$
kubectl logs -n "$NAMESPACE" \
>
  -l nvidia.com/dynamo-component-type=epp \
>
  --tail=200