Appendix A: Installing Software#

RHEL uses the dnf package manager to install, update, and remove packages. The utility can also be used to manage repositories. For more information about using dnf, refer to the Red Hat Managing Software with the DNF tool article.

A.1 NVIDIA DOCA-OFED#

NVIDIA MLNX_OFED has transitioned to DOCA-OFED. Refer to the following resources for more information:

To install NVIDIA DOCA-OFED on RHEL 9:

  1. Add the NVIDIA DOCA-OFED repository to your system.

    sudo dd status=none of=/etc/yum.repos.d/doca.repo << EOF
[doca]
name=DOCA latest
baseurl=https://linux.mellanox.com/public/repo/doca/latest/rhel9.5/arm64-sbsa/
enabled=1
gpgcheck=0
EOF
sudo dnf makecache

  2. To install the dependent software, run the following commands.

    sudo dnf install kernel-64k-devel-$(uname -r| sed 's/+64k//g') -y
sudo dnf install kernel-devel-$(uname -r| sed 's/+64k//g') -y
sudo dnf install kernel-headers-$(uname -r| sed 's/+64k//g') -y
sudo dnf install kernel-modules-extra-$(uname -r| sed 's/+64k//g') -y
sudo dnf install kernel-64k-modules-extra-$(uname -r| sed 's/+64k//g') -y
sudo dnf install kernel-abi-stablelists-$(uname -r| sed 's/aarch64+64k//g') -y
sudo dnf install bison flex gcc make openssl-devel perl lsof rpm-build automake libtool patch kernel-rpm-macros autoconf gcc-gfortran tcl tk createrepo python3-devel -y

    Note

    These steps will install the 4k kernel because NVIDIA DOCA-OFED currently has a dependency on the 4k kernel on RHEL, even when it is not being used.

  3. Install DOCA-Extra and build the DOCA modules.

    sudo dnf install doca-extra -y
sudo /opt/mellanox/doca/tools/doca-kernel-support

  4. Install the DOCA modules and DOCA userspace.

    The output from the previous command will have instructions about installing the DOCA modules. The instructions will look like the following output, but users need to provide the path, package name, and the repository values.

    sudo rpm -ivh /tmp/DOCA.<random value>/doca-kernel-repo-<version value>.rpm
sudo dnf makecache
sudo dnf install doca-ofed-userspace -y
sudo dnf install --disablerepo=doca doca-kernel-<version value>.rpm -y

    For GPUDirect Storage (GDS) functionality, NVIDIA recommends that you install kmod-mlnx-nvme and kmod-mlnx-nfsrdma with disablerepo=doca.

    Note

    These steps are due to DOCA not supporting the 64k kernel by default. Refer to DOCA Extra Package and doca-kernel-support for more information.

  5. When using NVIDIA BlueField-3 SoC Management Interface, enable the Rshim service.

    sudo systemctl daemon-reload
sudo systemctl enable rshim
sudo systemctl start rshim

  6. To confirm that the NVIDIA BlueField-3 SoC Management Interface is on the system, run the following commands.

    # Print the PCI BDF for the BlueField-3 Soc Management Interface
sudo lspci | grep "BlueField-3 SoC Management Interface" | awk '{print $1}'
0006:03:00.2
0016:03:00.2

  7. Update the device firmware.

    sudo dnf install mlnx-fw-updater -y

  8. If the BlueField-3 SoC Management Interface is on the system, install the BF-bundle.

    sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb>

    Where <N> is the rshim device identifier.

  9. Update the boot image to include NVIDIA DOCA OFED support and reboot the system.

    sudo dracut -f
sudo reboot now

A.2 NVIDIA GPU Driver and CUDA Toolkit#

Refer to the NVIDIA CUDA Installation Guide for Linux for information about installing the NVIDIA GPU driver and CUDA support for RHEL. The R535.129.03 driver is the minimum level required for the Hopper GPU.

The following commands can be used to install the minimum levels required for the Hopper GPU:

sudo dnf install kernel-64k-devel-$(uname -r | sed 's/+64k//g') -y
sudo dnf install kernel-headers-$(uname -r | sed 's/+64k//g') -y
sudo dnf install kernel-devel-matched-$(uname -r | sed 's/+64k//g') -y
sudo dnf install kernel-64k-devel-matched-$(uname -r| sed 's/+64k//g') -y
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm -y
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/sbsa/cuda-rhel9.repo
sudo dnf clean expire-cache
sudo dnf install cuda-toolkit-12-8 -y
sudo dnf module install nvidia-driver:570-open -y
sudo systemctl enable nvidia-persistenced
sudo reboot now

Note

The open-source GPU driver is required for Hopper GPUs.