Upgrade NVIDIA Run:ai

This page walks through the NVIDIA Run:ai upgrade procedure: preparing software artifacts, upgrading the control plane, upgrading the cluster, and verifying the result.

Before You Begin

Complete the following before starting the upgrade:

• Plan the Upgrade
• Back Up Before Upgrading
• Migrate GPU Fractions, if your tenant clusters use GPU fractions

Helm

NVIDIA Run:ai requires Helm 3.14 or later. Before you continue, validate your installed helm client version. To install or upgrade Helm, see Installing Helm.

Prepare Software Artifacts

Prepare the artifacts according to the release type you are upgrading to.

Open Source

Open Source artifacts are publicly available from NGC and do not require authentication.

helm repo add runai https://helm.ngc.nvidia.com/nvidia/runai-oss --force-update
helm repo update
helm search repo -l runai/control-plane

Support-Entitled

Before upgrading, make sure an image pull secret is already configured, as described in Software Artifacts.

helm repo add runai https://helm.ngc.nvidia.com/nvidia/runai-oss --force-update
helm repo update
helm search repo -l runai/control-plane

Upgrade the Control Plane

Before You Upgrade

Before upgrading the NVIDIA Run:ai control plane, review the System Requirements and complete the applicable steps in Preparations.

Upgrade

Run the following command for the release type you are upgrading to and update the values as described below.

Open Source

Replace <VERSION> with the NVIDIA Run:ai version you are upgrading to.

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai/control-plane \
  --version "<VERSION>" \
  -f runai_control_plane_values.yaml \
  --reset-then-reuse-values

Support-Entitled

• Replace <VERSION> with the NVIDIA Run:ai version you are upgrading to.
• Set supportEntitled=true to configure the installation as Support-Entitled.
• Set telemetryService.enabled=true to enable NVIDIA Run:ai’s telemetry.
• Set global.imagePullSecrets[0].name=runai-reg-creds to use the NGC image pull secret prepared in Software Artifacts.
• Optional: associate NVIDIA Run:ai’s telemetry with your NCA ID by adding one of the following to the helm upgrade command. See Registry Credentials for how to generate the NGC Service API Key. By default, telemetry is not associated with your organization.
  • --set telemetryService.telemetryAuth.token=<SERVICE_API_KEY> - Passes the Service API Key directly. Visible in shell history and process listings.
  • --set telemetryService.telemetryAuth.existingSecret=<SECRET_NAME> --set telemetryService.telemetryAuth.secretKey=<KEY_NAME> - References a secret you create yourself beforehand - the chart does not create or manage it.

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai/control-plane \
  --version "<VERSION>" \
  -f runai_control_plane_values.yaml \
  --reset-then-reuse-values \
  --set supportEntitled=true \
  --set telemetryService.enabled=true \
  --set global.imagePullSecrets[0].name=runai-reg-creds \
  --set telemetryService.telemetryAuth.token=<SERVICE_API_KEY>

Migrate from NGINX to HAProxy Kubernetes Ingress Controller

NVIDIA Run:ai recommends using HAProxy Kubernetes Ingress Controller as the ingress controller. This change aligns with the announced retirement of the upstream NGINX Ingress Controller project.

Clusters upgraded from earlier versions typically already have NGINX installed. After upgrading to <VERSION>, follow the steps below to migrate ingress traffic from NGINX Ingress Controller to HAProxy Kubernetes Ingress Controller.

Check the Service Type of the Existing Ingress Controller

Before installing the HAProxy ingress controller, identify which ingress controller is currently in use. If your cluster already has an ingress controller installed, verify how it is exposed to avoid port address conflicts.

kubectl get svc -n <nginx-namespace>

If the existing ingress controller uses NodePort, note the HTTP/HTTPS NodePort values to ensure HAProxy is configured with non-overlapping ports.

Install and Configure HAProxy Kubernetes Ingress Controller

If your cluster already has an ingress controller installed (for example, NGINX) and it is exposed via NodePort, configure HAProxy to use different NodePort values so both controllers can run simultaneously.

Ensure the selected NodePort values do not overlap with ports already used by the existing ingress controller.

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
 --create-namespace \
 --namespace haproxy-controller \
 --set controller.ingressClassResource.enabled=true \
 --set controller.service.type=NodePort \
 --set controller.service.nodePorts.http=32080 \
 --set controller.service.nodePorts.https=32443

Verify HAProxy Ingress

After installing the HAProxy Kubernetes Ingress Controller, verify that HAProxy ingresses are reachable before switching NVIDIA Run:ai components to use it. You can do this by deploying a simple hello-world application.

To run the test, identify the IP address that should reach the cluster’s nodes in your environment.

1. Create a local haproxy-test.yml file:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hello
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hello
     template:
       metadata:
         labels:
           app: hello
       spec:
         containers:
         - name: hello
           image: hashicorp/http-echo:1.0
           args:
             - "-text=hello from haproxy-ingress"
           ports:
             - containerPort: 5678
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: hello
   spec:
     selector:
       app: hello
     ports:
     - port: 80
       targetPort: 5678
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello
   spec:
     ingressClassName: haproxy
     rules:
     - http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hello
               port:
                 number: 80

2. Run the following command:

   kubectl apply -f ha-proxy-test.yml

Once the application is deployed, access the cluster’s IP address in a browser. If the page displays “hello from haproxy-ingress”, HAProxy is functioning correctly and you can proceed with upgrading NVIDIA Run:ai.

Upgrade the Control Plane

Run the following Helm command to update the NVIDIA Run:ai control plane to use HAProxy instead of NGINX.

helm upgrade runai-backend -n runai-backend runai/control-plane \
  --version "<VERSION>" \
  --reuse-values \
  --set-string global.ingress.ingressClass=haproxy