Is this page helpful?

Part 4: Installing NVIDIA Software into the OS Image#

The NVIDIA software stack and service configuration in this part is RHEL-specific (dnf, subscription-manager, systemd unit paths). For other distributions, substitute the equivalent package manager and installation commands. The logical sequence — register, update, install DOCA-OFED, install the driver, configure services — is the same across distros.

For full details, refer to the NVIDIA DGX Software for RHEL 9 guide.

  1. Chroot into the image.

    # At the OS/shell level on the BCM head node, run

cm-chroot-sw-img /cm/images/dgx-rhel9u6-x86_64

  2. Register and update.

    # Run within chroot

# 1. Register + pin release
subscription-manager register
subscription-manager release --set 9.6

# 2. Base update
dnf update -y --nobest

  3. Install DOCA-OFED. DOCA-OFED provides the same drivers and tools as MLNX_OFED using the DOCA-Host package, without the broader DOCA functionality.

    # Run within chroot

echo "[doca]
name=DOCA Online Repo
baseurl=https://linux.mellanox.com/public/repo/doca/DGX_latest_DOCA/rhel9/x86_64/
enabled=1
gpgcheck=1
gpgkey=https://linux.mellanox.com/public/repo/doca/DGX_latest_DOCA/rhel9/x86_64/repodata/repomd.xml.key" > /etc/yum.repos.d/doca.repo

dnf clean all && dnf makecache
dnf install -y doca-ofed

  4. Install the NVIDIA driver.

    # Run within chroot

dnf group install -y 'DGX B200 Configurations' --allowerasing
dnf module reset nvidia-driver -y
dnf module install --nobest -y nvidia-driver:580-open/fm --allowerasing

  5. Install supporting NVIDIA packages.

    # Run within chroot

dnf install -y nvlsm nv-persistence-mode nvidia-fm-enable nvidia-peermem-loader

  6. Install the Container Runtime and Build Dynamic Kernel Module Support (DKMS) Modules.

    # Run within chroot

dnf group install -y --allowerasing 'NVIDIA Container Runtime'
KVER=$(ls -1 /lib/modules/ | sort -V | tail -1)
dkms autoinstall -k $KVER

  7. Configure Slurm and system services.

    # Run within chroot

# 1. Create Slurm service drop-in for NVML library path
mkdir -p /etc/systemd/system/slurmd.service.d/
cat > /etc/systemd/system/slurmd.service.d/20-nvml.conf << 'EOF'
[Service]
Environment="LD_LIBRARY_PATH=/lib64:/usr/lib64"
EOF

# 2. Configure cgroup v2 for Slurm
mkdir -p /etc/slurm
cat > /etc/slurm/cgroup.conf << 'EOF'
CgroupPlugin=cgroup/v2
EOF

# 3. Create the libnvidia-ml symlink
ln -sf /lib64/libnvidia-ml.so.1 /lib64/libnvidia-ml.so

# 4. Load nvidia-peermem at boot
echo "nvidia-peermem" > /etc/modules-load.d/nvidia-peermem.conf

# 5. Enable required services
systemctl enable openibd nvidia-persistenced nvidia-fabricmanager nvidia-dcgm slurmd

exit

  8. Rebuild the software image ramdisk.

    # At the OS/shell level on the BCM head node, run

cmsh -c "softwareimage; use dgx-rhel9u6-x86_64; createramdisk"

  9. Reboot the node to apply changes.

    # At the OS/shell level on the BCM head node, run
cmsh -c "device; use dgx-01; reboot; commit"