Model Deployment

Expose the Frontend

Route external traffic to a DynamoGraphDeployment's Frontend with a Kubernetes Ingress, a LoadBalancer Service, or the Inference Gateway.
View as Markdown

The DGD Guide reaches the Frontend with kubectl port-forward, which is fine for a smoke test but not for production traffic. This page shows how to give the Frontend a stable external address. Whatever the API version, the operator creates a ClusterIP Service named <name>-frontend (where <name> is metadata.name) on port 8000 for the Frontend component — the options below route external traffic to that Service. Choose one of three paths depending on what your cluster runs.

PathUse when
Kubernetes IngressYou have an ingress controller (NGINX, etc.) and want a host-routed HTTP(S) entry point
LoadBalancer ServiceYou want a cloud load balancer and manage routing yourself
Inference Gateway (GAIE)You want Gateway API model-aware routing across multiple deployments

Option 1: A Kubernetes Ingress

Point a standard Kubernetes Ingress at the operator-created <name>-frontend Service. This requires an ingress controller (NGINX, etc.) already installed in the cluster:

1apiVersion: networking.k8s.io/v1
2kind: Ingress
3metadata:
4  name: dynamo-frontend
5  annotations:
6    nginx.ingress.kubernetes.io/backend-protocol: HTTP
7spec:
8  ingressClassName: nginx
9  rules:
10  - host: dynamo.example.com
11    http:
12      paths:
13      - path: /
14        pathType: Prefix
15        backend:
16          service:
17            name: vllm-agg-frontend       # <name>-frontend from your DGD
18            port:
19              number: 8000
20  tls:
21  - hosts:
22    - dynamo.example.com
23    secretName: dynamo-tls

The operator creates the Ingress object; the controller provisions the actual address.

The nvidia.com/v1alpha1 API had a convenience ingress field on the Frontend component that generated this Ingress (or a service-mesh VirtualService) for you. That field is not part of nvidia.com/v1beta1 — author the Ingress directly as shown above, or use a LoadBalancer Service (Option 2). See the API Reference for the v1alpha1 field.

Option 2: A LoadBalancer Service

To skip an Ingress and use a cloud load balancer directly, expose the Frontend Service as type: LoadBalancer:

$kubectl expose deployment <name>-frontend \
>  --type=LoadBalancer --port=8000 --target-port=8000 \
>  -n <namespace>
$kubectl get svc <name>-frontend -n <namespace> -w   # wait for EXTERNAL-IP

Send requests to the external IP on port 8000.

Option 3: The Inference Gateway (GAIE)

The Gateway API Inference Extension (GAIE) is a different mechanism: instead of exposing one Frontend, it puts a Gateway in front of one or more deployments and makes model-aware routing decisions in an Endpoint Picker Plugin (EPP). The Frontend runs with --router-mode direct and respects the EPP’s routing. Use GAIE when you serve multiple models or deployments behind one address. See Inference Gateway (GAIE).

A Kubernetes Ingress and GAIE are independent. An Ingress (Option 1) exposes a single Frontend Service; GAIE routes across deployments through the Gateway API and does not use an Ingress.