Kubernetes Gateway API

NVIDIA Run:ai supports the Kubernetes Gateway API as an alternative to Ingress for routing external traffic. Gateway API provides a flexible and extensible model for defining how traffic is exposed and routed within the cluster.

This page builds on the concepts described in Routing Traffic to and from NVIDIA Run:ai Services, including FQDN configuration, TLS certificates, and routing behavior.

Choose Your Installation Path

The networking concepts and Gateway API requirements on this page apply to both installation paths. Where configuration differs, follow the instructions for your installation method:

• Guided Installation - Configure NVIDIA Run:ai using runai-installer and config.yaml.
• Manual Helm - Configure NVIDIA Run:ai directly using Helm values and Kubernetes resources.

Scope and Prerequisites

This guide assumes a self-hosted deployment where the NVIDIA Run:ai control plane and cluster services are installed on the same Kubernetes cluster.

Before proceeding, ensure that the following prerequisites are already configured:

• Fully Qualified Domain Names (FQDN) for:
  • Control plane access
  • Development workspaces, training, and distributed inference workloads
  • Standard inference (Knative) workloads
• TLS certificates associated with the configured FQDNs

These configurations are described in the Routing Traffic to and from NVIDIA Run:ai Services section.

Routing Modes in NVIDIA Run:ai

NVIDIA Run:ai supports two routing approaches for exposing services: host-based routing and path-based routing.

Different NVIDIA Run:ai services use these routing approaches as follows:

Installing a Gateway Controller

NVIDIA Run:ai supports any conformant Gateway API implementation. The example below uses KGateway. If you are using a different conformant controller, follow its installation documentation and then proceed to the migration steps.

1. Install the Gateway API CRDs:

   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

2. Install the KGateway CRDs:

   helm upgrade -i --create-namespace \
     --namespace kgateway-system \
     --version v2.2.1 \
     kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds

3. Install the KGateway controller:

   helm upgrade -i -n kgateway-system kgateway \
     oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
     --version v2.2.1

Ensure that the Gateway controller is running before proceeding.

Routing Traffic for Workspaces, Training, and Distributed Inference Workloads

The following sections describe how development workspaces, training, and distributed inference workloads are exposed using Gateway API.

Depending on the selected routing mode, these workloads are accessed differently:

• In host-based routing, each development workspace, training, and distributed inference workload is exposed using its own subdomain:

  https://<project>-<workload>.<CLUSTER_URL>

• In path-based routing, development workspaces, training, and distributed inference workloads are exposed under a shared domain using URL paths:

  https://runai.mycorp.local/<project>/<workload>

This distinction determines the required FQDN structure, TLS certificates, and Gateway configuration described in the sections below.

Gateway API with Host-Based Routing

In this configuration:

• Development workspaces, training, and distributed inference workloads are exposed using subdomains
• A wildcard FQDN is required (e.g., *.runai.mycorp.local)
• A wildcard TLS certificate is required for those workloads
• The Gateway includes listeners for:
  • The cluster domain (control plane access)
  • Wildcard subdomains (workspaces, training, and distributed inference workloads)

Gateway Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: runai-gateway
  namespace: runai
spec:
  gatewayClassName: kgateway
  listeners:
  - name: https-cluster-domain
    protocol: HTTPS
    port: 443
    hostname: "<CLUSTER_DOMAIN>"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: runai-cluster-domain-tls-secret
  - name: https-workloads
    protocol: HTTPS
    port: 443
    hostname: "*.<CLUSTER_DOMAIN>"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: runai-cluster-domain-star-tls-secret

Gateway API with Path-Based Routing

In this configuration:

• Development workspaces, training, and distributed inference workloads are exposed under a single domain using URL paths
• A wildcard FQDN is not required for workspaces, training, and distributed inference workloads
• A wildcard TLS certificate is not required for workspaces, training, and distributed inference workloads
• The Gateway uses a single domain listener for these services

Ensure that host-based routing is disabled. For more details, see Advanced cluster configurations.

cluster:
  workloadRouting: path

Gateway Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: runai-gateway
  namespace: runai
spec:
  gatewayClassName: kgateway
  listeners:
  - name: https-cluster-domain
    protocol: HTTPS
    port: 443
    hostname: "<CLUSTER_DOMAIN>"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: runai-cluster-domain-tls-secret

Standard Inference Routing with Gateway API

Standard inference (Knative) workloads are always exposed using host-based routing via a dedicated inference subdomain, regardless of the selected routing mode for development workspaces, training, and distributed inference workloads.

Standard inference endpoints are accessed via dedicated subdomains, for example:

https://<service>.runai-inference.mycorp.local

To enable standard inference routing with Gateway API:

• A wildcard FQDN must be configured for inference (e.g., *.runai-inference.mycorp.local)
• A wildcard TLS certificate must be configured for that domain
• The Gateway must include a listener for the inference subdomain
• Knative Serving must be configured to route inference traffic from the Gateway to inference workloads

TLS Certificate

Ensure that a wildcard TLS certificate for standard inference is created:

• Secret name: runai-cluster-inference-tls-secret
• Namespace: runai

Replace /path/to/inference-fullchain.pem and /path/to/inference-private.pem with the actual paths to your certificate and private key:

kubectl create secret tls runai-cluster-inference-tls-secret -n runai \
    --cert /path/to/inference-fullchain.pem \
    --key /path/to/inference-private.pem

Gateway Configuration

Add a standard inference listener to the existing Gateway resource:

- name: https-inference
  protocol: HTTPS
  port: 443
  hostname: "*.runai-inference.<CLUSTER_DOMAIN>"
  tls:
    mode: Terminate
    certificateRefs:
    - kind: Secret
      name: runai-cluster-inference-tls-secret

Configure Knative Serving

NVIDIA Run:ai uses Knative Serving to support standard inference workloads. Inference traffic arrives at the Gateway and is forwarded to Knative Serving through Kourier, which routes it to the individual inference endpoints.

cluster:
  inference:
    wildcardDomain: "*.<INFERENCE_WILDCARD_DOMAIN>"
features:
  knative:
    mode: install
externalConnection:
  type: gatewayApi
  gatewayApi:
    name: <GATEWAY_NAME>
    namespace: <GATEWAY_NAMESPACE>

Introduce the Gateway API to NVIDIA Run:ai Services

To enable NVIDIA Run:ai to route traffic through the configured Gateway, both the control plane and the cluster must be updated to reference the Gateway.

At this stage, traffic is still served through the existing Ingress. The Gateway is introduced alongside the current setup and will become active only after DNS is updated in a later step.

externalConnection:
  type: gatewayApi
  gatewayApi:
    name: <GATEWAY_NAME>
    namespace: <GATEWAY_NAMESPACE>

Verify Gateway Configuration

kubectl get gateway -n runai

kubectl get httproute -A

Test connectivity:

curl -H "Host: <CLUSTER_DOMAIN>" https://<gateway-ip>/

Switch Traffic to Gateway

Before switching traffic, ensure that the NVIDIA Run:ai control plane and cluster are already configured to use the Gateway API.

Update DNS:

• <CLUSTER_DOMAIN> -> Gateway IP
• *.<CLUSTER_DOMAIN> -> Gateway IP

Traffic is routed based on DNS configuration. Until DNS records are updated to point to the Gateway IP, all traffic continues to be served through the existing Ingress.

nslookup <CLUSTER_DOMAIN>

curl https://<CLUSTER_DOMAIN>/

Disable Ingress on both the cluster and the control plane. There is no dedicated externalConnection field for this - Ingress and Gateway API can be enabled together, so disabling Ingress is a separate step.

cluster:
  customHelmValues:
    clusterConfig:
      global:
        ingress:
          enabled: false
controlPlane:
  customHelmValues:
    global:
      ingress:
        enabled: false

Rollback to Ingress

To roll back to Ingress, re-enable Ingress and disable Gateway API on both the cluster and the control plane. Then update DNS records to point back to the Ingress IP.

externalConnection:
  type: ingress
  ingress:
    className: <ingress-class>

Update DNS records for <CLUSTER_DOMAIN> and *.<CLUSTER_DOMAIN> to point back to the Ingress IP.