Skip to navigation
Installation and ConfigurationInstallationManual Helm

Install the Cluster

View as Markdown

This page describes how to install the NVIDIA Run:ai cluster on your Kubernetes environment using Helm. The cluster extends Kubernetes with NVIDIA Run:ai orchestration capabilities - scheduling and workload management - and connects to the previously installed control plane for centralized management.

Once the control plane is installed and you access the NVIDIA Run:ai UI for the first time, an onboarding wizard opens automatically. The wizard guides you through the cluster setup and generates a Helm installation command. Follow the instructions below to modify and run the command based on your artifact source and environment.

This procedure includes:

  • Installing the NVIDIA Run:ai cluster into the runai namespace
  • Registering the NVIDIA Run:ai cluster with the NVIDIA Run:ai control plane using the provided connection details

By completing this process, the NVIDIA Run:ai cluster will be connected to the NVIDIA Run:ai control plane and ready to run training, inference, and other workloads.

Before You Start

Before installing the NVIDIA Run:ai cluster, complete Preparations and Prerequisites and Features, and make sure you have installed the control plane.

Helm

NVIDIA Run:ai requires Helm 3.14 or later. To install Helm, see Installing Helm. If you are installing an air-gapped version of NVIDIA Run:ai, the NVIDIA Run:ai tar file contains the helm binary.

Note

Helm 4 defaults to server-side apply when installing a new chart release, which can conflict with resources managed by the NVIDIA Run:ai operator. Append --server-side=false to your helm upgrade command. NVIDIA Run:ai clusters originally installed with Helm 3.x are unaffected.

Permissions

Using a Kubernetes user with the cluster-admin role to ensure a successful installation is recommended. For more information, see Using RBAC authorization.

Installation

Note

  • To customize the installation based on your environment, see Advanced cluster configurations.
  • You can store the clientSecret as a Kubernetes secret within the cluster instead of using plain text. You can then configure the installation to use it by setting the controlPlane.existingSecret and controlPlane.secretKeys.clientSecret parameters as described in Advanced cluster configurations.
  • The bootstrapToken in the generated Helm command is a short-lived token that expires 2 hours after generation. If you do not complete the installation within that window, return to the cluster wizard in the NVIDIA Run:ai UI to generate a new command before running it.

Kubernetes

Connected

When adding a cluster for the first time, the onboarding wizard opens automatically when you log in to the NVIDIA Run:ai platform. You cannot perform other actions in the platform until the cluster is created. For additional clusters, use the Clusters form instead. See Clusters.

  1. Enter a unique name for your cluster.
  2. Set the cluster location. Choose where the NVIDIA Run:ai cluster will be installed:

    • Same as the control plane - Install the NVIDIA Run:ai cluster on the same Kubernetes cluster as the NVIDIA Run:ai control plane.

    • Remote control plane - Install the NVIDIA Run:ai cluster on a different Kubernetes cluster than the NVIDIA Run:ai control plane.

      Note

      This selection depends on your deployment model - the NVIDIA Run:ai cluster system requirements differ depending on whether the NVIDIA Run:ai cluster is installed on the same Kubernetes cluster as the NVIDIA Run:ai control plane (same-cluster deployment) or on a separate one (multi-cluster deployment).

  3. If you selected Remote control plane, enter the Cluster URL. For more information, see FQDN and DNS requirement.

Install NVIDIA Run:ai on Your Cluster

In the next section, the NVIDIA Run:ai cluster installation steps will be presented.

  1. Before installing the NVIDIA Run:ai cluster, see Preparations and follow the instructions there.
  2. The NVIDIA Run:ai platform displays the Helm installation command in the cluster wizard.
  3. Modify the UI-generated command as follows:
    • If you are using a local certificate authority, add --set global.customCA.enabled=true to the Helm command as described in the Cluster Local CA section.
    • The recommended ingress controller is HAProxy. If you are using a different ingress controller, update the ingress class to match the ingress controller configured during the control plane installation.
    • Save the image pull secrets and the hardened image values of the cluster third-party services in a values file and add -f <GOV_READY_VALUES_FILE> to the Helm command. See Gov-Ready Third-Party Services for the values. Gov-ready
    • Add --set global.fipsMode=only to the Helm command to enable FIPS mode, so that the NVIDIA Run:ai services use only FIPS-validated cryptography. Gov-ready
helm repo add runai https://helm.ngc.nvidia.com/nvidia/runai-oss --force-update
helm repo update
helm upgrade -i runai-cluster runai/runai-cluster -n runai \
    --set controlPlane.url=... \
    --set controlPlane.bootstrapToken=... \
    --set cluster.uid=... \
    --set cluster.url=... --version="<VERSION>" --create-namespace \
    --set clusterConfig.global.ingress.ingressClass=haproxy

The wizard displays Waiting for cluster to connect while the cluster is being installed and connected to the control plane. Once the installation completes successfully and the cluster establishes communication with the control plane, the wizard updates to Cluster connected. After completing the wizard flow, the cluster is added to the Clusters table.

Tip

Use the dry-run flag --dry-run=client to gain an understanding of what is being installed before the actual installation.

Air-gapped

When adding a cluster for the first time, the onboarding wizard opens automatically when you log in to the NVIDIA Run:ai platform. You cannot perform other actions in the platform until the cluster is created. For additional clusters, use the Clusters form instead. See Clusters.

  1. Enter a unique name for your cluster.
  2. Set the cluster location. Choose where the NVIDIA Run:ai cluster will be installed:

    • Same as the control plane - Install the NVIDIA Run:ai cluster on the same Kubernetes cluster as the NVIDIA Run:ai control plane.

    • Remote control plane - Install the NVIDIA Run:ai cluster on a different Kubernetes cluster than the NVIDIA Run:ai control plane.

      Note

      This selection depends on your deployment model - the NVIDIA Run:ai cluster system requirements differ depending on whether the NVIDIA Run:ai cluster is installed on the same Kubernetes cluster as the NVIDIA Run:ai control plane (same-cluster deployment) or on a separate one (multi-cluster deployment).

  3. If you selected Remote control plane, enter the Cluster URL. For more information, see FQDN and DNS requirement.

Install NVIDIA Run:ai on Your Cluster

In the next section, the NVIDIA Run:ai cluster installation steps will be presented.

  1. Before installing the NVIDIA Run:ai cluster, see Preparations and follow the instructions there.

  2. The NVIDIA Run:ai platform displays the Helm installation command in the cluster wizard.

    - Do not run the command exactly as shown in the UI -

  3. Update the UI-generated Helm command as follows (see example command below) and use the cluster-airgap-values.yaml file written by the airgap load step in Preparations, instead of using helm repositories:

    • Do not add the helm repository, helm repo add, and do not run helm repo update.
    • Instead, edit the helm upgrade command:
      • Replace runai/runai-cluster with ./charts/runai-cluster-<VERSION>.tgz, where <VERSION> is the full version number (e.g., runai-cluster-3.0.2.tgz). This file is located in the charts folder of the working directory you specified to airgap load.
      • Add -f <VALUES_DIR>/cluster-airgap-values.yaml. Replace <VALUES_DIR> with the valuesDir path you set in airgap-load.yaml during Preparations - this is where cluster-airgap-values.yaml was written.
      • Add --set global.customCA.enabled=true as described in the Cluster Local CA section.
      • The recommended ingress controller is HAProxy. If you are using a different ingress controller, update the ingress class to match the ingress controller configured during the control plane installation.
      • Add --set global.fipsMode=only to the Helm command to enable FIPS mode, so that the NVIDIA Run:ai services use only FIPS-validated cryptography. Gov-ready
      • Keep the remaining --set values exactly as generated by the UI.
helm upgrade -i runai-cluster ./charts/runai-cluster-<VERSION>.tgz \
    --set controlPlane.url=... \
    --set controlPlane.bootstrapToken=... \
    --set cluster.uid=... \
    --set cluster.url=... --create-namespace \
    --set global.customCA.enabled=true \
    --set clusterConfig.global.ingress.ingressClass=haproxy \
    -f <VALUES_DIR>/cluster-airgap-values.yaml

The wizard displays Waiting for cluster to connect while the cluster is being installed and connected to the control plane. Once the installation completes successfully and the cluster establishes communication with the control plane, the wizard updates to Cluster connected. After completing the wizard flow, the cluster is added to the Clusters table.

Tip

Use the dry-run flag --dry-run=client to gain an understanding of what is being installed before the actual installation.

OpenShift

Connected

When adding a cluster for the first time, the onboarding wizard opens automatically when you log in to the NVIDIA Run:ai platform. You cannot perform other actions in the platform until the cluster is created. For additional clusters, use the Clusters form instead. See Clusters.

  1. Enter a unique name for your cluster.
  2. Set the cluster location. Choose where the NVIDIA Run:ai cluster will be installed:

    • Same as the control plane - Install the NVIDIA Run:ai cluster on the same Kubernetes cluster as the NVIDIA Run:ai control plane.

    • Remote control plane - Install the NVIDIA Run:ai cluster on a different Kubernetes cluster than the NVIDIA Run:ai control plane.

      Note

      This selection depends on your deployment model - the NVIDIA Run:ai cluster system requirements differ depending on whether the NVIDIA Run:ai cluster is installed on the same Kubernetes cluster as the NVIDIA Run:ai control plane (same-cluster deployment) or on a separate one (multi-cluster deployment).

  3. If you selected Remote control plane, enter the Cluster URL. For more information, see FQDN and DNS requirement.

Install NVIDIA Run:ai on Your Cluster

In the next section, the NVIDIA Run:ai cluster installation steps will be presented.

  1. Before installing the NVIDIA Run:ai cluster, see Preparations and follow the instructions there.
  2. The NVIDIA Run:ai platform displays the Helm installation command in the cluster wizard.
  3. Modify the UI-generated command as follows:
    • If you are using a local certificate authority, add --set global.customCA.enabled=true to the Helm command as described in the Cluster Local CA section.
    • Save the image pull secrets and the hardened image values of the cluster third-party services in a values file and add -f <GOV_READY_VALUES_FILE> to the Helm command. See Gov-Ready Third-Party Services for the values. Gov-ready
    • Add --set global.fipsMode=only to the Helm command to enable FIPS mode, so that the NVIDIA Run:ai services use only FIPS-validated cryptography. Gov-ready
helm repo add runai https://helm.ngc.nvidia.com/nvidia/runai-oss --force-update
helm repo update
helm upgrade -i runai-cluster runai/runai-cluster -n runai \
  --set controlPlane.url=... \
  --set controlPlane.bootstrapToken=... \
  --set cluster.uid=... \
  --set cluster.url=... --version="<VERSION>" --create-namespace

The wizard displays Waiting for cluster to connect while the cluster is being installed and connected to the control plane. Once the installation completes successfully and the cluster establishes communication with the control plane, the wizard updates to Cluster connected. After completing the wizard flow, the cluster is added to the Clusters table.

Tip

Use the dry-run flag --dry-run=client to gain an understanding of what is being installed before the actual installation.

Air-gapped

When adding a cluster for the first time, the onboarding wizard opens automatically when you log in to the NVIDIA Run:ai platform. You cannot perform other actions in the platform until the cluster is created. For additional clusters, use the Clusters form instead. See Clusters.

  1. Enter a unique name for your cluster.
  2. Set the cluster location. Choose where the NVIDIA Run:ai cluster will be installed:

    • Same as the control plane - Install the NVIDIA Run:ai cluster on the same Kubernetes cluster as the NVIDIA Run:ai control plane.

    • Remote control plane - Install the NVIDIA Run:ai cluster on a different Kubernetes cluster than the NVIDIA Run:ai control plane.

      Note

      This selection depends on your deployment model - the NVIDIA Run:ai cluster system requirements differ depending on whether the NVIDIA Run:ai cluster is installed on the same Kubernetes cluster as the NVIDIA Run:ai control plane (same-cluster deployment) or on a separate one (multi-cluster deployment).

  3. If you selected Remote control plane, enter the Cluster URL. For more information, see FQDN and DNS requirement.

Install NVIDIA Run:ai on Your Cluster

In the next section, the NVIDIA Run:ai cluster installation steps will be presented.

  1. Before installing the NVIDIA Run:ai cluster, see Preparations and follow the instructions there.

  2. The NVIDIA Run:ai platform displays the Helm installation command in the cluster wizard.

    - Do not run the command exactly as shown in the UI -

  3. Update the UI-generated Helm command as follows (see example command below) and use the cluster-airgap-values.yaml file written by the airgap load step in Preparations, instead of using helm repositories:

    • Do not add the helm repository, helm repo add, and do not run helm repo update.
    • Instead, edit the helm upgrade command:
      • Replace runai/runai-cluster with ./charts/runai-cluster-<VERSION>.tgz, where <VERSION> is the full version number (e.g., runai-cluster-3.0.2.tgz). This file is located in the charts folder of the working directory you specified to airgap load.
      • Add -f <VALUES_DIR>/cluster-airgap-values.yaml. Replace <VALUES_DIR> with the valuesDir path you set in airgap-load.yaml during Preparations - this is where cluster-airgap-values.yaml was written.
      • Add --set global.customCA.enabled=true as described in the Cluster Local CA section.
      • Add --set global.fipsMode=only to the Helm command to enable FIPS mode, so that the NVIDIA Run:ai services use only FIPS-validated cryptography. Gov-ready
      • Keep the remaining --set values exactly as generated by the UI.
    helm upgrade -i runai-cluster ./charts/runai-cluster-<VERSION>.tgz \
        --set controlPlane.url=... \
        --set controlPlane.bootstrapToken=... \
        --set cluster.uid=... \
        --set cluster.url=... --create-namespace \
        --set global.customCA.enabled=true \
        -f <VALUES_DIR>/cluster-airgap-values.yaml

The wizard displays Waiting for cluster to connect while the cluster is being installed and connected to the control plane. Once the installation completes successfully and the cluster establishes communication with the control plane, the wizard updates to Cluster connected. After completing the wizard flow, the cluster is added to the Clusters table.

Tip

Use the dry-run flag --dry-run=client to gain an understanding of what is being installed before the actual installation.

Troubleshooting

If you encounter an issue with the installation, try the troubleshooting scenario below.

Installation

If the NVIDIA Run:ai cluster installation failed, check the installation logs to identify the issue. Run the following script to print the installation logs:

#!/bin/bash
# Print logs of helm hooks if installation failed
hook_pods=$(kubectl get pod -n runai -l run.ai/hook 2> /dev/null | wc -l)
if [ $hook_pods -ne 0 ]
then
  kubectl logs -n runai -l run.ai/hook
fi
runaiconfig="$(kubectl get runaiconfig runai -n runai 2> /dev/null; echo $?)"
if [[ $runaiconfig -eq 0 ]]
then
  kubectl logs -n runai -l name=runai | grep ERROR
fi

Cluster Status

If the NVIDIA Run:ai cluster installation completed, but the cluster status did not change its status to Connected, refer to the cluster Troubleshooting scenarios section.

Next Steps

Once the cluster is installed and connected, the NVIDIA Run:ai UI guides you through optional post-installation configurations. These steps are optional but recommended for a production setup:

  • SSO (Single Sign-On) - Configure SSO to allow users to log in with your organization’s identity provider. See SSO for setup instructions using SAML or OpenID Connect.
  • Email server - Configure an SMTP server to enable email notifications for workload events and system alerts. See Email notifications for configuration details.
  • Create your first research team - Set up projects and permissions to organize your AI practitioners and allocate GPU resources.