Run:ai Installation#
Use the BCM 11 setup tools to install NVIDIA Run:ai on the k8s-user cluster that you deployed earlier.
Important
Run:ai installs on the k8s-user cluster (k8s-system-user nodes), not
the k8s-admin cluster. Before running the installation wizard, ensure the
k8s-system-user node category and the GPU worker node category (for
example, dgx-gb300-k8s) are created in BCM.
If you plan to use Run:ai with Mission Control autonomous job recovery,
install the Kubeflow Training Operator on the user Kubernetes cluster, and
then install Run:ai with cm-runai-setup. Leave the Run:ai advice
restart controller disabled. After you install Mission Control autonomous
job recovery, enable the controller in action mode. The controller
requires the Mission Control autonomous job recovery Advice custom
resource definition. Follow Run:ai mode deployment for the
complete procedure and the manual enablement command.
The earlier NVIDIA Mission Control air-gapped steps deploy both the k8s-admin and k8s-user
clusters. Refer to Staging and Installing Artifacts. This page
therefore does not repeat the Run:ai air-gapped deployment flow. Complete the
Run:ai preparations for the version included in your deployment, then use the
Deploy NVIDIA Run:ai Using the Wizard
procedure in the Run:ai 2.26 BCM air-gapped deployment guide.
Run the deployment wizard from the active BCM head node. The host where you run
the wizard is distinct from the Kubernetes nodes in the k8s-user cluster.
Configure Run:ai Placement After Installation#
Apply the following steps immediately after cm-runai-setup completes and before submitting user workloads. This updates the existing Helm releases; it does not perform a second Run:ai installation. Run the commands on the active BCM head node in the same shell session.
The updates restart Run:ai system services. Retain their existing PVCs and PVs. Their storage must be accessible from every selected control-plane node. If a PV uses node-local storage or has node affinity that excludes those nodes, resolve its storage placement before continuing. This procedure does not migrate persistent data.
Switch to the user cluster and assign the existing Run:ai label set to the control-plane configuration overlay. Clear its category and explicit node selections so they cannot label GPU workers:
module switch kubernetes/k8s-user cmsh -c "kubernetes; use k8s-user; labelsets; use runai-control-plane; clear categories; clear nodes; set overlays kube-k8s-user-master; set labels node-role.kubernetes.io/runai-system=true; commit"
Allow approximately 60–90 seconds for BCM to apply the labels, then verify the live assignment:
kubectl get nodes -l node-role.kubernetes.io/runai-system=true -o wide kubectl get nodes -l 'node-role.kubernetes.io/runai-system=true,node-role.kubernetes.io/control-plane' -o wide
Both queries must return the same intended control-plane nodes, and every selected node must be Ready. If the labels have not appeared, wait and repeat the queries before continuing.
Create one placement values file for the existing backend and cluster releases:
cat > /root/runai-service-placement.yaml <<'YAML' global: affinity: &runai_system_affinity nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/runai-system operator: In values: ["true"] - key: node-role.kubernetes.io/control-plane operator: Exists tolerations: &runai_system_tolerations - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule nats: nodeSelector: &runai_system_selector node-role.kubernetes.io/runai-system: "true" node-role.kubernetes.io/control-plane: "" tolerations: *runai_system_tolerations grafana: nodeSelector: *runai_system_selector keycloakx: nodeSelector: *runai_system_selector notificationsService: nodeSelector: *runai_system_selector postgresql: primary: nodeSelector: *runai_system_selector redisQueue: master: nodeSelector: *runai_system_selector thanos: query: nodeSelector: *runai_system_selector receive: nodeSelector: *runai_system_selector runai-engine: global: affinity: *runai_system_affinity tolerations: *runai_system_tolerations clusterConfig: global: affinity: *runai_system_affinity tolerations: *runai_system_tolerations YAML
The required affinity constrains system services to nodes with both labels. The component selectors cover backend dependencies that use their own scheduling settings. The toleration permits these services on the control-plane nodes already tainted during Kubernetes setup. GPU workloads and required node agents retain their existing placement.
Locate the Run:ai package fetched by
nmc-airgapusing the path saved in/root/cm-runai-setup.conf. If you saved the wizard configuration elsewhere, replace that path in the command. The helper prints the package and chart paths:runai_package=$(python3 -c 'import yaml; print(yaml.safe_load(open("/root/cm-runai-setup.conf"))["modules"]["runai"]["airgap"]["runai_airgapped_tarball"])') runai_chart_dir="$(dirname "$runai_package")/runai-ag/charts" backend_chart="$runai_chart_dir/control-plane.tgz" cluster_chart="$runai_chart_dir/runai-cluster.tgz" printf 'Run:ai package: %s\nBackend chart: %s\nCluster chart: %s\n' "$runai_package" "$backend_chart" "$cluster_chart"
During installation,
cm-runai-setupextracts the charts intorunai-ag/chartsbeside the package. Reuse these local charts for the placement updates. Read the installed release versions:backend_version=$(helm get metadata runai-backend -n runai-backend -o json | jq -er '.version') cluster_version=$(helm get metadata runai-cluster -n runai -o json | jq -er '.version') printf 'Backend chart version: %s\nCluster chart version: %s\n' "$backend_version" "$cluster_version"
Verify both local chart packages against the installed releases:
python3 - "$backend_chart" "$backend_version" "$cluster_chart" "$cluster_version" <<'PYTHON' import subprocess, sys, yaml for path, name, version in [(sys.argv[1], "control-plane", sys.argv[2]), (sys.argv[3], "runai-cluster", sys.argv[4])]: metadata = yaml.safe_load(subprocess.check_output(["helm", "show", "chart", path], text=True)) if metadata["name"] != name or str(metadata["version"]) != version: raise SystemExit(f"Chart mismatch: {path}; expected {name} {version}") print(f"Matched {name} {version}") PYTHON
Continue only after both charts report a match. Keep these versions for the placement update.
Prepare a local backend chart copy so its PostgreSQL guard hook uses the control-plane placement settings. Extract the backend package into
/root/runai-placement:mkdir -p /root/runai-placement tar -xzf "$backend_chart" -C /root/runai-placement backend_chart_dir=/root/runai-placement/control-plane guard_template="$backend_chart_dir/templates/helm-hooks/scram-md5-guard/job.yaml"
Create the hook placement patch:
cat > /root/runai-hook-placement.patch <<'PATCH' --- job.yaml +++ job.yaml @@ -38,4 +38,6 @@ spec: restartPolicy: Never + affinity: {{- toYaml .Values.global.affinity | nindent 8 }} + tolerations: {{- toYaml .Values.global.tolerations | nindent 8 }} {{- with .Values.global.imagePullSecrets }} imagePullSecrets: PATCH
Check the patch against the extracted hook template, if the chart contains that hook:
if [ -f "$guard_template" ]; then patch --dry-run --batch --forward --fuzz=0 \ "$guard_template" /root/runai-hook-placement.patch fi
If the dry run succeeds, apply the patch. If the hook template is absent, continue to the release updates. If the dry run fails, stop and review the template before continuing:
if [ -f "$guard_template" ]; then patch --batch --forward --fuzz=0 \ "$guard_template" /root/runai-hook-placement.patch fi
This adds the existing global required affinity and control-plane toleration to the guard pod. The guard’s database check and deadline remain unchanged. The original chart archive and installer package remain intact.
Update the existing backend release using the patched local chart. Retain its registry, certificates, credentials, storage settings, and other installed values:
helm upgrade runai-backend "$backend_chart_dir" \ --namespace runai-backend \ --reuse-values \ --values /root/runai-service-placement.yaml \ --wait --timeout 40m
Wait for the backend update to succeed before updating the cluster release:
helm upgrade runai-cluster "$cluster_chart" \ --namespace runai \ --reuse-values \ --values /root/runai-service-placement.yaml \ --wait --timeout 20m
Keep
--reuse-valuesso the placement file merges with the installed settings, including the air-gapped registry configuration where applicable. The patched backend hook and the cluster upgrade hooks use the control-plane nodes.Pin the Gateway proxies through the
GatewayParametersreferenced by thekgatewayGatewayClass:kubectl -n kgateway-system patch gatewayparameters kgateway \ --type merge \ -p '{"spec":{"kube":{"podTemplate":{"nodeSelector":{"node-role.kubernetes.io/control-plane":""}}}}}' kubectl -n runai-backend rollout status deployment/runai-gateway --timeout=300s
This applies to all proxies that use the
kgatewayclass, includingrunai-backend/runai-gateway. The merge preserves the existing load-balancer exclusion affinity, toleration, and replica count.Installers that create the separate
kgateway-knativeGatewayClass require the same placement on itsGatewayParameters. Run the following command to patch that resource and wait for both Knative gateway rollouts. If the class is absent, the command skips this part:knative_class=$(kubectl get gatewayclass kgateway-knative --ignore-not-found -o name) && if [ -n "$knative_class" ]; then kubectl -n kgateway-system patch gatewayparameters kgateway-knative \ --type merge \ -p '{"spec":{"kube":{"podTemplate":{"nodeSelector":{"node-role.kubernetes.io/control-plane":""}}}}}' && kubectl -n knative-serving rollout status deployment/knative-gateway --timeout=300s && kubectl -n knative-serving rollout status deployment/knative-local-gateway --timeout=300s fi
Both Knative gateway proxies inherit this class’s control-plane selector and toleration. The local gateway’s separate
GatewayParameterschanges only its Service type toClusterIP; it retains the class’s placement settings. The merge preserves existing service, affinity, and replica settings. If the class exists but its parameters or either Deployment is missing, resolve the incomplete installation before continuing.Confirm that active Run:ai service pods and any
kgateway-knativeproxy pods are Ready on nodes with the control-plane label:kubectl get nodes,pods -A -o json | jq -er '[.items[] | select(.kind == "Node" and (.metadata.labels | has("node-role.kubernetes.io/control-plane"))) | .metadata.name] as $cp | [.items[] | select(.kind == "Pod" and (.metadata.namespace == "runai" or .metadata.namespace == "runai-backend" or (.metadata.namespace == "knative-serving" and (.metadata.labels["gateway.networking.k8s.io/gateway-class-name"] == "kgateway-knative")))) | select(.status.phase != "Succeeded" and .status.phase != "Failed") | select(any(.metadata.ownerReferences[]?; .kind == "DaemonSet") | not)] as $pods | if ($pods | length) == 0 then error("No Run:ai service pods found") else [$pods[] | select((.spec.nodeName as $n | $cp | index($n)) == null or (any(.status.conditions[]?; .type == "Ready" and .status == "True") | not)) | "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName // "<unassigned>")"] | if length == 0 then "PASS: All Run:ai service pods are Ready on control-plane nodes" else error("Not Ready or not on control-plane nodes: " + join(", ")) end end'
The command prints
PASSwhen all checked pods are Ready on control-plane nodes. Otherwise, it lists pods that are not Ready or have incorrect placement and exits with an error. It excludes completed pods, failed pods, and DaemonSets, which retain their intended worker coverage. This checks current pod placement and readiness. If a pod fails the check, inspect it withkubectl -n <namespace> describe pod <pod>.
Retain /root/runai-service-placement.yaml and the patched chart in /root/runai-placement/control-plane for subsequent placement updates. Apply the hook placement patch to a fresh chart copy when repeating this procedure. If you rerun cm-runai-setup, repeat this post-installation procedure because the installer can replace the label set and release values.
Install the NVIDIA DRA Driver#
The NVIDIA Mission Control bundle includes the NVIDIA dynamic resource allocation (DRA) driver
chart and image (version 25.12.0),
and the import steps push them to the BCM registry. After the wizard completes,
install the driver onto k8s-user from that registry. Use this command instead
of the helm repo add command in the Run:ai guide, which requires internet access.
module switch kubernetes/k8s-user
helm install nvidia-dra-driver-gpu \
oci://master.cm.cluster:5000/helm-charts/nvidia-dra-driver-gpu \
--version 25.12.0 \
--ca-file /cm/local/apps/containerd/var/etc/certs.d/master.cm.cluster:5000/ca.crt \
--create-namespace \
--namespace nvidia-dra-driver-gpu \
--set nvidiaDriverRoot=/ \
--set resources.gpus.enabled=false
Prepare an SSH-Enabled nvbandwidth Image#
The Run:ai validation tests
run the MNNVL ComputeDomain and InfiniBand bandwidth tests as MPI jobs. Each
worker pod installs openssh-server with apt-get when it starts, which
fails without internet access. Before you run these tests, build an
nvbandwidth image that already includes openssh-server and push it to the
BCM registry.
The AHR air-gap upload already pushes the nvbandwidth:1.11.2 base image to
the BCM registry. This procedure adds about 3 MB of Ubuntu packages to that
image. You do not need to publish a new image or give the cluster internet
access.
You need the following:
An internet-connected host to download the packages. Use an Ubuntu host with Bash, any release and either architecture, or any host with Podman or Docker.
On the active BCM head node:
skopeo, and Docker installed as described in Staging and Installing Artifacts. The head node must trust the BCM registry certificate. Refer to Trusting the BCM local registry certificate authority.
Download the Packages#
The image is based on Ubuntu 24.04, so the packages must come from Ubuntu 24.04 regardless of what the download host runs. The following commands use a private APT configuration, so they neither read nor change the host’s own APT sources and installed packages.
openssh-server needs exactly five packages on top of the base image. The
list includes openssh-client because the server requires a client of the
same version.
On the internet-connected host, run the following commands in Bash:
mkdir -p nvbandwidth-openssh/debs/{amd64,arm64} apt-noble/{lists/partial,cache,sources.list.d} touch apt-noble/status cat > apt-noble/sources.list.d/noble.sources <<'EOF' Types: deb URIs: http://archive.ubuntu.com/ubuntu Suites: noble noble-updates noble-security Components: main Architectures: amd64 Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg Types: deb URIs: http://ports.ubuntu.com/ubuntu-ports Suites: noble noble-updates noble-security Components: main Architectures: arm64 Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg EOF # Use only the sources above, and ignore what is installed on this host. NOBLE=(-o Dir::Etc::SourceList=/dev/null -o Dir::Etc::SourceParts="$PWD/apt-noble/sources.list.d" -o Dir::State::Lists="$PWD/apt-noble/lists" -o Dir::State::status="$PWD/apt-noble/status" -o Dir::Cache="$PWD/apt-noble/cache" -o APT::Architectures=amd64,arm64) apt-get "${NOBLE[@]}" update for a in amd64 arm64; do (cd nvbandwidth-openssh/debs/$a && apt-get "${NOBLE[@]}" download \ openssh-server:$a openssh-sftp-server:$a openssh-client:$a libwrap0:$a ucf) done
On a host without APT, such as RHEL or macOS, save the commands as
fetch-debs.shand run them in an Ubuntu container from the same directory. Usedocker runwith the same arguments if the host has Docker instead of Podman:podman run --rm -v "$PWD:/work" -w /work docker.io/library/ubuntu:24.04 bash fetch-debs.sh
Warnings that start with
W: Download is performed unsandboxed as rootare harmless.Confirm that the
openssh-*packages show the same version for both architectures:for f in nvbandwidth-openssh/debs/*/*.deb; do dpkg-deb -f "$f" Package Version Architecture | paste -sd' '; done
Create the Dockerfile#
Save the following as nvbandwidth-openssh/Dockerfile. The base image runs
as the ubuntu user, but the Run:ai test manifests start sshd as root
and mount the MPI SSH keys at /root/.ssh, so the image switches to root. It
also removes the SSH host keys that the package installation generates, so
that each pod creates its own at startup.
ARG BASE=master.cm.cluster:5000/nvidia/nv-mission-control/nvbandwidth:1.11.2
FROM ${BASE}
USER root
ARG TARGETARCH
COPY debs/${TARGETARCH}/ /tmp/debs/
ARG SSH_PORT=2222
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends /tmp/debs/*.deb && \
rm -rf /tmp/debs /var/lib/apt/lists/* && \
rm -f /etc/ssh/ssh_host_* && \
mkdir -p /run/sshd && \
sed -i "s/[ #]\(.*StrictHostKeyChecking \).*/ \1no/g" /etc/ssh/ssh_config && \
echo " UserKnownHostsFile /dev/null" >> /etc/ssh/ssh_config && \
sed -i "s/[ #]\(.*Port \).*/ \1${SSH_PORT}/g" /etc/ssh/ssh_config && \
sed -i "s/#\(StrictModes \).*/\1no/g" /etc/ssh/sshd_config && \
sed -i "s/#\(Port \).*/\1${SSH_PORT}/g" /etc/ssh/sshd_config
Copy the nvbandwidth-openssh directory to the active BCM head node. You do
not need the apt-noble directory.
Build and Push the Image#
Run the following steps as root on the active BCM head node.
Load Docker, and set the registry path and the architecture of the
k8s-userGPU nodes:module load docker cd nvbandwidth-openssh R=master.cm.cluster:5000/nvidia/nv-mission-control/nvbandwidth A=arm64 # or amd64
Confirm that the base image is in the registry, and list the architectures it contains:
skopeo inspect --raw docker://$R:1.11.2 \ | python3 -c 'import json,sys; d=json.load(sys.stdin); print([m["platform"]["architecture"] for m in d["manifests"]] if "manifests" in d else "single image")' skopeo inspect --format '{{.Architecture}}' docker://$R:1.11.2
The architecture you set in
Amust be in the output. A bundle that you downloaded for one architecture contains a single-architecture image.Build and push the image:
docker build --platform linux/$A --build-arg BASE=$R:1.11.2 -t $R:1.11.2-openssh . docker push $R:1.11.2-openssh
On an x86 head node, an arm64 build runs through QEMU emulation and takes longer.
Test the image. The command generates host keys first, as the worker pods must:
docker run --rm --runtime runc --platform linux/$A --entrypoint sh $R:1.11.2-openssh -c \ 'uname -m; id -un; ssh-keygen -A >/dev/null && /usr/sbin/sshd -t && echo "sshd OK"; for x in nvbandwidth mpirun ssh; do command -v $x; done'
The output shows the architecture,
root,sshd OK, and three paths. The--runtime runcoption is needed only for this test. BCM makes the NVIDIA container runtime the Docker default, and that runtime can refuse to start a container on a head node without a GPU driver.
Update the Test Manifests#
In the MNNVL ComputeDomain and InfiniBand bandwidth test manifests from the Run:ai guide, make the following changes before you apply them:
Set
image:for both the launcher and the workers tomaster.cm.cluster:5000/nvidia/nv-mission-control/nvbandwidth:1.11.2-openssh.In the worker command, remove the
apt-get updateandapt-get install openssh-servercommands. Keepssh-keygen -Abeforesshdstarts, because the image contains no host keys.In the InfiniBand bandwidth test, prefix each network name in the worker
k8s.v1.cni.cncf.io/networksannotation with the namespace that contains it, for exampledefault/sriovibnet-rdma-default-a-su-1. The tests run in a Run:ai project namespace, and without the prefix the workers look for the networks in that namespace and cannot start. To find the namespace, runkubectl get network-attachment-definitions -A.
Run the tests in the namespace of a Run:ai project, not in default. The
NVIDIA Mission Control deployment applies the Kubernetes security policies to k8s-user
before you reach this step. Those policies block DNS and pod-to-pod traffic in
default, so the launcher cannot resolve the worker host names and the test
fails. Run:ai project namespaces, which carry the runai/queue label, are
allowed.
Use the
runai-ajrnamespace, which the NVIDIA Mission Control deployment creates. Confirm that it exists and has therunai/queuelabel:kubectl get namespace runai-ajr --show-labels
If the namespace does not exist, create a Run:ai project for the tests and use its namespace instead. A project named
validationgets the namespacerunai-validation:runai project create validation kubectl get namespace runai-validation --show-labels
The manifests do not set a namespace, so pass
-nwith the namespace to eachkubectlcommand for the tests, for example:kubectl apply -n runai-ajr -f <test-manifest>.yaml kubectl logs -n runai-ajr <launcher-pod-name>
If the Run:ai guide has you create a secret or other resource for the tests, create it in the same namespace.
After each test, delete its resources before you run the next one. A test that is still running holds the GPUs, and the next test fails with
CUDA-capable device(s) is/are busy or unavailable:kubectl delete -n runai-ajr -f <test-manifest>.yaml
Then continue with the DRA validation and “Enable DRA and Multi-Node NVLink” steps in the Run:ai guide.