Upgrade NVIDIA Run:ai

This page walks through the NVIDIA Run:ai upgrade procedure: preparing software artifacts, upgrading the control plane, upgrading the cluster, and verifying the result.

Before You Begin

Complete the following before starting the upgrade:

• Plan the Upgrade
• Back Up Before Upgrading
• Migrate GPU Fractions, if your cluster uses GPU fractions

Helm

NVIDIA Run:ai requires Helm 3.14 or later. Before you continue, validate your installed Helm client version. To install or upgrade Helm, see Installing Helm.

Prepare Software Artifacts

helm repo add runai https://helm.ngc.nvidia.com/nvidia/runai-oss --force-update
helm repo update
helm search repo -l runai/control-plane

helm repo add runai https://helm.ngc.nvidia.com/nvidia/runai-oss --force-update
helm repo update
helm search repo -l runai/control-plane

Upgrade the Control Plane

Before You Upgrade

Before upgrading the NVIDIA Run:ai control plane, review the System Requirements and complete the applicable steps in Preparations.

Upgrade

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai/control-plane \
  --version "<VERSION>" \
  -f runai_control_plane_values.yaml \
  --reset-then-reuse-values

helm get values runai-backend -n runai-backend > runai_control_plane_values.yaml
helm upgrade runai-backend -n runai-backend runai/control-plane \
  --version "<VERSION>" \
  -f runai_control_plane_values.yaml \
  --reset-then-reuse-values \
  --set supportEntitled=true \
  --set telemetryService.enabled=true \
  --set global.imagePullSecrets[0].name=runai-reg-creds \
  --set telemetryService.telemetryAuth.token=<SERVICE_API_KEY>

helm upgrade runai-backend ./charts/control-plane-<VERSION>.tgz -n runai-backend \
  -f <VALUES_DIR>/control-plane-airgap-values.yaml \
  --reset-then-reuse-values

helm upgrade runai-backend ./charts/control-plane-<VERSION>.tgz -n runai-backend \
  -f <VALUES_DIR>/control-plane-airgap-values.yaml \
  --reset-then-reuse-values \
  --set supportEntitled=true \
  --set telemetryService.enabled=true

Upgrade the Cluster

Before You Upgrade

Before upgrading the NVIDIA Run:ai cluster, review the System Requirements and complete the applicable steps in Preparations and Prerequisites and Features.

Setup

1. In the NVIDIA Run:ai UI, go to Resources -> Clusters
2. Select the cluster you want to upgrade
3. Click INSTALLATION INSTRUCTIONS
4. Optional: Select the NVIDIA Run:ai cluster version (latest, by default)
5. Under Settings, review the cluster location (Same as control plane or Remote control plane) and Cluster URL
6. Click CONTINUE

Run the Upgrade Command

The command below is the same for both Open Source and Support-Entitled Release upgrades.

Once installation is complete, validate the cluster is Connected and listed with the new cluster version. Once you have done this, the cluster is upgraded to the latest version. If the cluster does not appear as expected, see the cluster troubleshooting scenarios.

Migrate from NGINX to HAProxy Kubernetes Ingress Controller

NVIDIA Run:ai recommends using HAProxy Kubernetes Ingress Controller as the ingress controller. This change aligns with the announced retirement of the upstream NGINX Ingress Controller project.

Clusters upgraded from earlier versions typically already have NGINX installed. After upgrading to <VERSION>, follow the steps below to migrate ingress traffic from NGINX Ingress Controller to HAProxy Kubernetes Ingress Controller.

Check the Service Type of the Existing Ingress Controller

Before installing the HAProxy Kubernetes Ingress Controller, identify which ingress controller is currently in use. If your cluster already has an ingress controller installed, verify how it is exposed to avoid port address conflicts.

kubectl get svc -n <nginx-namespace>

• If the existing ingress controller uses NodePort, note the HTTP/HTTPS NodePort values to ensure HAProxy is configured with non-overlapping ports.
• If the existing ingress controller uses LoadBalancer, no additional action is required.

When running more than one ingress controller in the same cluster, port conflicts are relevant only for NodePort-based setups. LoadBalancer-based controllers automatically receive separate external IP addresses.

Install and Configure HAProxy Kubernetes Ingress Controller

Ingress controllers can be installed and configured in different ways depending on your Kubernetes distribution and how you expose services (for example, NodePort vs. LoadBalancer).

The sections below provide environment-specific Helm installation examples. Select the option that matches your deployment environment.

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
 --create-namespace \
 --namespace haproxy-controller \
 --set controller.ingressClassResource.enabled=true \
 --set controller.service.type=NodePort \
 --set controller.service.nodePorts.http=32080 \
 --set controller.service.nodePorts.https=32443

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
 --create-namespace \
 --namespace haproxy-controller \
 --set controller.service.type=LoadBalancer

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
 --create-namespace \
 --namespace haproxy-controller \
 --set controller.kind=DaemonSet \
 --set controller.service.type=LoadBalancer \
 --set controller.service.externalTrafficPolicy=Local \
 --set controller.service.annotations."oci-network-load-balancer\.oraclecloud\.com/is-preserve-source"="True" \
 --set controller.service.annotations."oci-network-load-balancer\.oraclecloud\.com/security-list-management-mode"=All \
 --set controller.service.annotations."oci\.oraclecloud\.com/load-balancer-type"=nlb

Verify HAProxy Ingress

After installing the HAProxy Kubernetes Ingress Controller, verify that HAProxy ingresses are reachable before switching NVIDIA Run:ai components to use it. You can do this by deploying a simple hello-world application.

To run the test, identify the IP address that should reach the cluster’s nodes in your environment.

1. Create a local haproxy-test.yml file:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: hello
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: hello
     template:
       metadata:
         labels:
           app: hello
       spec:
         containers:
         - name: hello
           image: hashicorp/http-echo:1.0
           args:
             - "-text=hello from haproxy-ingress"
           ports:
             - containerPort: 5678
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: hello
   spec:
     selector:
       app: hello
     ports:
     - port: 80
       targetPort: 5678
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello
   spec:
     ingressClassName: haproxy
     rules:
     - http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hello
               port:
                 number: 80

2. Run the following command:

   kubectl apply -f haproxy-test.yml

Once the application is deployed, access the cluster’s IP address in a browser. If the page displays “hello from haproxy-ingress”, HAProxy is functioning correctly and you can proceed with upgrading NVIDIA Run:ai.

Upgrade the Control Plane

Run the following Helm command to update the NVIDIA Run:ai control plane to use HAProxy instead of NGINX.

helm upgrade runai-backend -n runai-backend runai/control-plane \
  --version "<VERSION>" \
  --reuse-values \
  --set-string global.ingress.ingressClass=haproxy

Upgrade the Cluster

Setup

1. In the NVIDIA Run:ai UI, go to Resources -> Clusters
2. Select the cluster you want to upgrade
3. Click INSTALLATION INSTRUCTIONS
4. Optional: Select the NVIDIA Run:ai cluster version (latest, by default)
5. Under Settings, review the cluster location (Same as control plane or Remote control plane) and Cluster URL
6. Click CONTINUE

Installation Instructions

1. Follow the installation instructions. Run the Helm commands provided on your Kubernetes cluster.

2. If not present, add the following flag to the helm install command:

   --set clusterConfig.global.ingress.ingressClass=haproxy

3. Click DONE

4. Once installation is complete, validate the cluster is Connected and listed with the new cluster version (see the cluster troubleshooting scenarios). Once you have done this, the cluster is upgraded and the workloads in this cluster will now use HAProxy instead of NGINX.

Troubleshooting

If you encounter an issue with the cluster upgrade, use the troubleshooting scenarios below.

Installation Fails

If the NVIDIA Run:ai cluster installation failed, check the installation logs to identify the issue. Run the following script to print the installation logs:

#!/bin/bash
# Print logs of helm hooks if installation failed
hook_pods=$(kubectl get pod -n runai -l run.ai/hook 2> /dev/null | wc -l)
if [ $hook_pods -ne 0 ]
then
  kubectl logs -n runai -l run.ai/hook
fi
runaiconfig="$(kubectl get runaiconfig runai -n runai 2> /dev/null; echo $?)"
if [[ $runaiconfig -eq 0 ]]
then
  kubectl logs -n runai -l name=runai | grep ERROR
fi

Cluster Status

If the NVIDIA Run:ai cluster upgrade completes, but the cluster status does not show as Connected, refer to Troubleshooting scenarios.