NVIDIA Mission Control Prerequisites#

Before installing NVIDIA Mission Control software components, ensure the following prerequisites are in place.

For deployments with a k8s-user cluster, configure Kubernetes and Mission Control autonomous job recovery services for dedicated control-plane nodes before you deploy them. Apply the Run:ai placement steps after Run:ai is installed and before you submit user workloads. The procedures are in NVIDIA Mission Control Connected Installation and NMC Air-Gap Installation.

Shared Storage Requirements#

Before installing Kubernetes, mount the same writable Network File System (NFS) export at /cm/shared on every control and worker node in each cluster. This guide’s storage setup requires NFS for both connected and air-gapped installations, independently of BCM head node high availability (HA).

The wizard’s Local path StorageClass uses this mount for persistent volumes; it does not configure NFS. Without shared storage, workloads such as Grafana, Loki, OpenBao, and the Keycloak database can fail to start or access their data when pods move between nodes.

If NFS is not configured, complete NFS Configuration, Shared Storage Setup, and Post HA and NFS Configuration Steps in the Management Plane and Rack Setup Installation Guide.

Verify Shared Storage#

Run these checks from the active BCM head node before creating each Kubernetes cluster and again after the setup wizard reboots its nodes. Replace <node-list> with the comma-separated host names of all control and worker nodes selected for that cluster, for example admin-01,admin-02,admin-03.

pdsh -w <node-list> \
  'findmnt --target /cm/shared --noheadings --output SOURCE,FSTYPE,TARGET'

Every selected node must respond with the same NFS server and export, filesystem type nfs or nfs4, and mount target /cm/shared. A local filesystem such as /dev/md0 ext4 / fails the check.

After the mount check passes, confirm that root can write to the shared directory. This command creates and removes one temporary file per node:

pdsh -w <node-list> \
  'check_file=$(mktemp /cm/shared/.nmc-storage-check.XXXXXX) &&
   rm -- "$check_file" && echo writable'

Every selected node must report writable. If either check fails on any node, stop the installation. Correct the BCM mounts or NFS export permissions before continuing, but do not change mounts used by running workloads.

Network Planning#

The NVIDIA Mission Control software stack requires several Virtual IP (VIP) addresses. In each case, the VIP must be allocated from the same subnet as the corresponding interface (inband or out-of-band) on the nodes that will hold it. The BCM HA VIPs are held by the active BCM head node; the remaining VIPs are implemented via MetalLB on the k8s-system-admin nodes. In this context, inband refers to the internalnet collection of subnets used for cluster-internal traffic. Coordinate with your network administrator to reserve these addresses before beginning installation.

Table 3 Required VIPs#
Component	VIP	Purpose
BCM	Inband HA VIP	HA VIP on the BCM head node inband subnet, held by the active BCM head node
BCM	Out-of-band HA VIP	HA VIP on the BCM head node out-of-band subnet, held by the active BCM head node
Run:ai	Control-plane VIP (inband)	Single entry point for the Run:ai UI, API, and workspaces
Run:ai	Inference VIP (inband)	Entry point for inference workloads (NIMs, customer inference services)
Autonomous Recovery Engine (AHR)	Control-plane VIP (inband)	Access to the AHR web UI, APIs, and runbook engine
NetQ	Control-plane VIP (inband)	Ingress point for the NetQ UI and APIs

DGX-OS Image Configuration#

The default DGX-OS image, installed by the BCM 11 installer or through the cm-create-image command, contains all the software required to operate a DGX GB200 NVL72 system. However, the default configuration might not be optimized for your specific cluster architecture.

This section describes additional modifications you can apply to the OS image to better support the rest of the software stack.

Use the following steps to further configure your DGX-OS image for use in a GB200 NVL72 cluster:

Locate your DGX-OS image and enter the chroot
```
cm-chroot /cm/images/dgxos-7.2-image
```

Once inside the chrooted image, run this command to create a script within the chroot.

cat > /tmp/dgx-post-install.sh <<EOF
#!/bin/bash

set -e
set -x


setup_cleanup() {
  systemctl disable nvidia-fabricmanager # RC10 issue
  rm -f /etc/apt/sources.list.d/doca.list
}

setup_dcgm() {
  systemctl enable nvidia-dcgm-exporter
  echo "DCGM_EXP_XID_ERRORS_COUNT, gauge, Count of XID Errors within user-specified time window (see xid-count-window-size param)." >> /etc/dcgm-exporter/default-counters.csv
}

setup_ib() {
  systemctl enable openibd
  rm -f /etc/libibverbs.d/vmw_pvrdma.driver
}

setup_slurm() {
  # These modifications are documented in the NMC guide for installing slurm.
  # if the image isn't actually a slurm worker image, this does not hurt.
  cat > /etc/sysconfig/slurmd <<EOT
PMIX_MCA_ptl=^usock
PMIX_MCA_psec=none
PMIX_SYSTEM_TMPDIR=/var/empty
PMIX_MCA_gds=hash
EOT
}

setup_tuning() {
  cat > /etc/sysctl.d/99-sysctl.conf  <<EOT
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.enP6p3s0f0np0.rp_filter=1
net.ipv4.conf.lo.rp_filter=1
EOT
}

disable_nvsm_efi_check() {
  sed -i '/Status of volumes/s/^[[:space:]]*#//' /cm/local/apps/cmd/scripts/healthchecks/configfiles/nvsm_show_health.py
}

setup_cleanup
setup_ib
setup_dcgm
setup_slurm
setup_tuning
disable_nvsm_efi_check

EOF

Make the script executable using the following command
```
chmod +x /tmp/dgx-post-install.sh
```
Run the following script
```
/tmp/dgx-post-install.sh
```
Exit the chroot using the following command:
```
exit
```