NVIDIA Run:ai Installation

Use the BCM 11 setup tools to create the user Kubernetes cluster and install NVIDIA Run:ai.

Important

Run:ai installs on the k8s-user cluster (k8s-system-user nodes), not the k8s-admin cluster. Before running the installation wizard, ensure the k8s-system-user node category and the GPU worker node category (for example, dgx-gb300-k8s) are created in BCM.

If you plan to use Run:ai with Mission Control autonomous job recovery, install the Kubeflow Training Operator on the user Kubernetes cluster, and then install Run:ai with cm-runai-setup. Leave the Run:ai advice restart controller disabled. After you install Mission Control autonomous job recovery, enable the controller in action mode. The controller requires the Mission Control autonomous job recovery Advice custom resource definition. Follow Run:ai mode deployment for the complete procedure and the manual enablement command.

Detailed instructions are available in the official Run:ai documentation.

Run:ai Configuration

Configure Run:ai Placement After Installation

Apply the following steps immediately after cm-runai-setup completes and before submitting user workloads. This updates the existing Helm releases; it does not perform a second Run:ai installation. Run the commands on the active BCM head node in the same shell session.

The updates restart Run:ai system services. Retain their existing PVCs and PVs. Their storage must be accessible from every selected control-plane node. If a PV uses node-local storage or has node affinity that excludes those nodes, resolve its storage placement before continuing. This procedure does not migrate persistent data.

1. Switch to the user cluster and assign the existing Run:ai label set to the control-plane configuration overlay. Clear its category and explicit node selections so they cannot label GPU workers:

   module switch kubernetes/k8s-user
   cmsh -c "kubernetes; use k8s-user; labelsets; use runai-control-plane; clear categories; clear nodes; set overlays kube-k8s-user-master; set labels node-role.kubernetes.io/runai-system=true; commit"
   

2. Allow approximately 60–90 seconds for BCM to apply the labels, then verify the live assignment:

   kubectl get nodes -l node-role.kubernetes.io/runai-system=true -o wide
   kubectl get nodes -l 'node-role.kubernetes.io/runai-system=true,node-role.kubernetes.io/control-plane' -o wide
   

   Both queries must return the same intended control-plane nodes, and every selected node must be Ready. If the labels have not appeared, wait and repeat the queries before continuing.

3. Create one placement values file for the existing backend and cluster releases:

   cat > /root/runai-service-placement.yaml <<'YAML'
   global:
     affinity: &runai_system_affinity
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: node-role.kubernetes.io/runai-system
               operator: In
               values: ["true"]
             - key: node-role.kubernetes.io/control-plane
               operator: Exists
     tolerations: &runai_system_tolerations
     - key: node-role.kubernetes.io/control-plane
       operator: Exists
       effect: NoSchedule
   nats:
     nodeSelector: &runai_system_selector
       node-role.kubernetes.io/runai-system: "true"
       node-role.kubernetes.io/control-plane: ""
     tolerations: *runai_system_tolerations
   grafana:
     nodeSelector: *runai_system_selector
   keycloakx:
     nodeSelector: *runai_system_selector
   notificationsService:
     nodeSelector: *runai_system_selector
   postgresql:
     primary:
       nodeSelector: *runai_system_selector
   redisQueue:
     master:
       nodeSelector: *runai_system_selector
   thanos:
     query:
       nodeSelector: *runai_system_selector
     receive:
       nodeSelector: *runai_system_selector
   runai-engine:
     global:
       affinity: *runai_system_affinity
       tolerations: *runai_system_tolerations
   clusterConfig:
     global:
       affinity: *runai_system_affinity
       tolerations: *runai_system_tolerations
   YAML
   

   The required affinity constrains system services to nodes with both labels. The component selectors cover backend dependencies that use their own scheduling settings. The toleration permits these services on the control-plane nodes already tainted during Kubernetes setup. GPU workloads and required node agents retain their existing placement.

4. Read the versions of the installed releases. The placement updates use the Helm repository configured by cm-runai-setup and retain those versions:

   backend_version=$(helm get metadata runai-backend -n runai-backend -o json | jq -er '.version')
   cluster_version=$(helm get metadata runai-cluster -n runai -o json | jq -er '.version')
   printf 'Backend chart: %s\nCluster chart: %s\n' "$backend_version" "$cluster_version"
   

   Download the installed backend chart version to prepare the hook patch. The cluster release continues to use its configured repository reference:

   mkdir -p /root/runai-placement
   helm pull runai-backend/control-plane \
     --version "$backend_version" \
     --destination /root/runai-placement
   backend_chart="/root/runai-placement/control-plane-$backend_version.tgz"
   

5. Prepare a local backend chart copy so its PostgreSQL guard hook uses the control-plane placement settings. Extract the backend package into /root/runai-placement:

   tar -xzf "$backend_chart" -C /root/runai-placement
   backend_chart_dir=/root/runai-placement/control-plane
   guard_template="$backend_chart_dir/templates/helm-hooks/scram-md5-guard/job.yaml"
   

   Create the hook placement patch:

   cat > /root/runai-hook-placement.patch <<'PATCH'
   --- job.yaml
   +++ job.yaml
   @@ -38,4 +38,6 @@
        spec:
          restartPolicy: Never
   +      affinity: {{- toYaml .Values.global.affinity | nindent 8 }}
   +      tolerations: {{- toYaml .Values.global.tolerations | nindent 8 }}
          {{- with .Values.global.imagePullSecrets }}
          imagePullSecrets:
   PATCH
   

   Check the patch against the extracted hook template, if the chart contains that hook:

   if [ -f "$guard_template" ]; then
     patch --dry-run --batch --forward --fuzz=0 \
       "$guard_template" /root/runai-hook-placement.patch
   fi
   

   If the dry run succeeds, apply the patch. If the hook template is absent, continue to the release updates. If the dry run fails, stop and review the template before continuing:

   if [ -f "$guard_template" ]; then
     patch --batch --forward --fuzz=0 \
       "$guard_template" /root/runai-hook-placement.patch
   fi
   

   This adds the existing global required affinity and control-plane toleration to the guard pod. The guard’s database check and deadline remain unchanged. The original chart archive and installer package remain intact.

6. Update the existing backend release using the patched local chart. Retain its registry, certificates, credentials, storage settings, and other installed values:

   helm upgrade runai-backend "$backend_chart_dir" \
     --namespace runai-backend \
     --reuse-values \
     --values /root/runai-service-placement.yaml \
     --wait --timeout 40m
   

   Wait for the backend update to succeed before updating the cluster release:

   helm upgrade runai-cluster runai-backend/runai-cluster \
     --version "$cluster_version" \
     --namespace runai \
     --reuse-values \
     --values /root/runai-service-placement.yaml \
     --wait --timeout 20m
   

   Keep --reuse-values so the placement file merges with the installed settings, including the air-gapped registry configuration where applicable. The patched backend hook and the cluster upgrade hooks use the control-plane nodes.

7. Pin the Gateway proxies through the GatewayParameters referenced by the kgateway GatewayClass:

   kubectl -n kgateway-system patch gatewayparameters kgateway \
     --type merge \
     -p '{"spec":{"kube":{"podTemplate":{"nodeSelector":{"node-role.kubernetes.io/control-plane":""}}}}}'
   kubectl -n runai-backend rollout status deployment/runai-gateway --timeout=300s
   

   This applies to all proxies that use the kgateway class, including runai-backend/runai-gateway. The merge preserves the existing load-balancer exclusion affinity, toleration, and replica count.

   Installers that create the separate kgateway-knative GatewayClass require the same placement on its GatewayParameters. Run the following command to patch that resource and wait for both Knative gateway rollouts. If the class is absent, the command skips this part:

   knative_class=$(kubectl get gatewayclass kgateway-knative --ignore-not-found -o name) &&
   if [ -n "$knative_class" ]; then
     kubectl -n kgateway-system patch gatewayparameters kgateway-knative \
       --type merge \
       -p '{"spec":{"kube":{"podTemplate":{"nodeSelector":{"node-role.kubernetes.io/control-plane":""}}}}}' &&
     kubectl -n knative-serving rollout status deployment/knative-gateway --timeout=300s &&
     kubectl -n knative-serving rollout status deployment/knative-local-gateway --timeout=300s
   fi
   

   Both Knative gateway proxies inherit this class’s control-plane selector and toleration. The local gateway’s separate GatewayParameters changes only its Service type to ClusterIP; it retains the class’s placement settings. The merge preserves existing service, affinity, and replica settings. If the class exists but its parameters or either Deployment is missing, resolve the incomplete installation before continuing.

8. Confirm that active Run:ai service pods and any kgateway-knative proxy pods are Ready on nodes with the control-plane label:

   kubectl get nodes,pods -A -o json | jq -er '[.items[] | select(.kind == "Node" and (.metadata.labels | has("node-role.kubernetes.io/control-plane"))) | .metadata.name] as $cp | [.items[] | select(.kind == "Pod" and (.metadata.namespace == "runai" or .metadata.namespace == "runai-backend" or (.metadata.namespace == "knative-serving" and (.metadata.labels["gateway.networking.k8s.io/gateway-class-name"] == "kgateway-knative")))) | select(.status.phase != "Succeeded" and .status.phase != "Failed") | select(any(.metadata.ownerReferences[]?; .kind == "DaemonSet") | not)] as $pods | if ($pods | length) == 0 then error("No Run:ai service pods found") else [$pods[] | select((.spec.nodeName as $n | $cp | index($n)) == null or (any(.status.conditions[]?; .type == "Ready" and .status == "True") | not)) | "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName // "<unassigned>")"] | if length == 0 then "PASS: All Run:ai service pods are Ready on control-plane nodes" else error("Not Ready or not on control-plane nodes: " + join(", ")) end end'
   

   The command prints PASS when all checked pods are Ready on control-plane nodes. Otherwise, it lists pods that are not Ready or have incorrect placement and exits with an error. It excludes completed pods, failed pods, and DaemonSets, which retain their intended worker coverage. This checks current pod placement and readiness. If a pod fails the check, inspect it with kubectl -n <namespace> describe pod <pod>.

Retain /root/runai-service-placement.yaml and the patched chart in /root/runai-placement/control-plane for subsequent placement updates. Apply the hook placement patch to a fresh chart copy when repeating this procedure. If you rerun cm-runai-setup, repeat this post-installation procedure because the installer can replace the label set and release values.

NIM Operator

The NVIDIA NIM Operator automates the deployment and management of NVIDIA NIM microservices.

Installation

You can install the operator directly through the Cluster Manager setup wizard:

1. Run the cm-kubernetes-setup wizard.

2. When you reach the Helm chart selection prompt, select NIM Operator.

For detailed configuration and usage instructions, refer to the official documentation.

NVIDIA Dynamo

Run:ai 2.23 introduces support for distributed inference via NVIDIA Dynamo. This enables optimized, multi-node scheduling for large-scale LLM inference.

See also

For a practical example of this integration, refer to the guide on Smart Multi-Node Scheduling with Run:ai and Dynamo.

Installation

NVIDIA Mission Control installs Dynamo for you. The nvidiaDynamo component deploys the dynamo-platform chart onto the k8s-user cluster after bcmUserK8s, in both connected and air-gapped deployments. No manual helm install is required.

deployctl apply -f <cluster-spec>.yaml nvidiaDynamo

The component installs the Dynamo operator together with the etcd, NATS, KAI scheduler, and Grove subcharts that the platform depends on. Verify and remove it the same way:

deployctl check  -f <cluster-spec>.yaml nvidiaDynamo
deployctl delete -f <cluster-spec>.yaml nvidiaDynamo

Configuration

The chart version is pinned by the blueprint and is not set in the cluster spec. The following settings may be supplied under the nvidiaDynamo component:

Parameter	Default	Description
dynamoClusterName	k8s-user	Kubernetes cluster the release targets.
dynamoNamespace	dynamo	Namespace for the release. Created if absent.
dynamoInstallEtcd	true	Install the bundled etcd subchart.
dynamoInstallKaiScheduler	true	Install the bundled KAI scheduler subchart.
dynamoInstallGrove	true	Install the bundled Grove subchart.
dynamoRemoveNamespace	false	Delete the namespace on removal.

Note

Disabling dynamoInstallKaiScheduler is supported only on a cluster where another component has already installed a compatible queues.scheduling.run.ai CRD. The chart applies Queue resources as post-install hooks whenever the KAI scheduler is in use, and that CRD ships inside the KAI subchart, so the release fails without it.

For further details on deployment and usage, refer to the Dynamo Documentation.

Node Categories

In the NVIDIA Base Command Manager (BCM), a node category is a way to group nodes that share the same hardware profile and intended role. Defining node categories allows the system to assign the appropriate software image and configurations to each group during provisioning.

Before installing NVIDIA Run:ai, make sure BCM node categories are created for:

• Kubernetes nodes (for example, k8s-system-user)

• NVIDIA Run:ai GPU worker nodes (for example, dgx-gb300-k8s)

These will be employed when setting up Run:ai for the first time via the BCM setup assistant. More details on node category configuration for Run:ai installation and full instructions are available in the Run:ai BCM Install Getting Started Guide.

Note

Install NVIDIA Run:ai before the Autonomous Recovery Engine components, as shown in the installation sequence. Installing Run:ai first makes its node categories available when you select categories for the Mission Control autonomous hardware recovery (AHR) agent. Its Kubernetes context is also available when you deploy the AHR runbooks. Refer to Runbook Setup and Deployment to configure the Run:ai Kubernetes context for AHR runbooks.

AHR does not enforce this order. With AHR agent version release-29.2.16 or later and cm-setup-ahr version 2.5.1-bcm11 or later, AHR automatically detects Run:ai GPU nodes. You do not need to reinstall the AHR agent if you install Run:ai after AHR. Earlier or mismatched component versions do not include this behavior.

Note

After you uninstall Run:ai from a node, manually remove /var/lib/runai from that node. The uninstaller leaves this path in place, and AHR continues to tag the node scheduler_system_type=runai while the path exists.

NVIDIA Run:ai Validation

To validate Run:ai, refer to the Run:ai usage guides for deploying training & inference jobs:

• Run Your First Run:ai Distributed Training

• Run Your First Run:ai Custom Inference Workload

The distributed inference tutorials can also be used to validate multi-node inference:

• Hugging Face Distributed Inference Deployment

• NVIDIA NIM Distributed Inference Deployment