Installing NVIDIA MLNX_OFED#
The DGX software stack for Red Hat Enterprise Linux does not include the NVIDIA MLNX_OFED (OpenFabrics Enterprise Distribution) for Linux. This is to ensure that the MLNX_OFED driver is in sync with the Red Hat distribution kernel. This section describes how to download, install, and upgrade MLNX_OFED on systems that are running Red Hat Enterprise Linux.
Prerequisites#
NVIDIA validates each release of NVIDIA DGX Software for Red Hat Enterprise Linux with a specific MLNX_OFED version. Refer to the NVIDIA DGX Software for Red Hat Enterprise Linux 8 Release Notes for the recommended MLNX_OFED version to install for a particular version of NVIDIA EL8 software.
Installing and Configuring MLNX_OFED#
Perform the following steps to install MLNX_OFED on systems that do not yet have it installed. It is imperative that a validated MLNX_OFED version is used for the RHEL version that the DGX system is running.
Important
Running the dnf update command at any time to install the drivers can update the system to the latest Red Hat Enterprise Linux version.
Determine which version of Red Hat Enterprise Linux is installed on the DGX system.
cat /etc/redhat-releaseAfter referring to the NVIDIA DGX Software for Red Hat Enterprise Linux 8 Release Notes document, download the MLNX_OFED software bundle:
Go to the Linux InfiniBand Drivers page, and scroll down to the MLNX_OFED Download Center matrix.
At the MLNX_OFED Download Center matrix, choose
The version to install (you may need to select Archive Versions),
RHEL/CentOS/Rocky (under OS Distribution), and
The relevant OS Distribution Version and Architecture.
Click the supported ISO or tgz package.
The Mellanox OFED (MLNX_OFED) Software: End-User Agreement page appears.
Accept the End User License Agreement by clicking I Have Read the Above End User License Agreement.
The selected software package starts to download.
Install kernel headers and development packages for your kernel.
These packages are needed when building MLNX_OFED kernel support and for the ensuing DKMS compilation. Install packages matching the target kernel; the following command uses the running kernel.
sudo dnf install -y kernel-headers-$(uname -r) kernel-devel-$(uname -r)
After downloading the correct MLNX_OFED software bundle, proceed with the installation instructions:
Go to the Adapter Software site and select the MLNX_OFED software version you downloaded.
Click the User Manual link, and then go to Installation > Installing MLNX_OFED.
Follow the installation instructions.
Note
The system can report that additional software is required before installation. If this message appears, install the required software, and then retry the MLNX_OFED driver installation.
Some MLNX_OFED releases and kernels require additional prerequisites. The following commands provide examples:
sudo dnf install -y gcc-gfortran "kernel-modules-extra-$(uname -r)" sudo dnf install -y rpm-build createrepo_c sudo dnf install -y kernel-rpm-macros libtool automake autoconf
Package requirements vary by release and kernel. Follow the prerequisites in the selected MLNX_OFED user manual. These example packages do not apply to every EL8 release.
If the installer reports that the bundled kernel module packages (KMP RPMs) are incompatible with your kernel, follow the kernel-support build instructions for the selected MLNX_OFED release. Confirm that your configuration is supported, install the build prerequisites, and then use
--add-kernel-supportto build packages for your kernel.For example:
sudo ./mlnxofedinstall --force --without-fw-update --add-kernel-support
Run this command from the extracted MLNX_OFED bundle.
--add-kernel-supportbuilds kernel support,--forceenables unattended installation, and--without-fw-updateskips adapter firmware updates. Use--forceand--without-fw-updateonly when appropriate for your installation. These options are not required by--add-kernel-support.
If you intend to use NVIDIA GPUDirect Storage (GDS), enable the driver’s GDS support according to the instructions at MLNX_OFED Requirements and Installation.
Install
nvidia-mlnx-config.sudo dnf install -y nvidia-mlnx-config
After installing the MLNX_OFED drivers, install the NVIDIA peer memory module.
sudo dnf install -y nvidia-peer-memory-dkms
Note
If your NVIDIA GPU driver release provides
nvidia-peermemand you installed the GPU driver before MLNX_OFED, reinstall the GPU driver by using the procedure supported for your DGX release. For DKMS-based installations, rebuild and install the NVIDIA modules after installing MLNX_OFED. This process compilesnvidia-peermemagainst the MLNX_OFED RDMA APIs.nvidia-peermemis distinct from the legacynv_peer_memmodule provided bynvidia-peer-memory-dkms. Load only one peer-memory module at a time. Use the implementation supported for your DGX software and GPU driver release.Activate the installed drivers.
Regenerate initramfs when required by the installer or your boot configuration. For the running kernel:
sudo dracut -f
If you installed for a different kernel, regenerate initramfs for that kernel before booting it. Follow the selected MLNX_OFED release’s instructions to activate the drivers. If a live
openibdrestart cannot unload in-use RDMA modules, reboot during a maintenance window.
Note
nvidia-peer-memory-dkms version 1.2 or later, requires MOFED version 5.4-3.0.3.0 or later. Using MOFED versions that do not fit these criteria will result in a build failure of the nv_peer_mem DKMS module. For more information, see:
Note
While in-box drivers might be available, using the in-box drivers is not recommended as they provide lower performance than the official MLNX OFED drivers and they do not support the GPUDirectTM RDMA feature. For more information on configuring the in-box drivers, see the Red Hat Enterprise Linux documentation: Configuring InfiniBand and RDMA Networks
Updating NVIDIA MLNX_OFED#
Perform the following steps to update MOFED on systems that already have it installed. The Mellanox InfiniBand Drivers in RPM packages are precompiled for a specific kernel version.
It is imperative to use the correct MOFED version for the RHEL version that the DGX system has been updated to.
There is no need to uninstall the current MOFED first, because the mlnxofedinstall script automatically uninstalls any previously installed versions.
Upgrade the Red Hat Enterprise Linux release and kernel version.
sudo dnf update --nobest
Determine which version of Red Hat Enterprise Linux is installed on the DGX system.
cat /etc/redhat-releaseAfter referring to the release notes, download the MLNX_OFED software bundle.
Go to the Linux InfiniBand Drivers page, and scroll down to the MLNX_OFED Download Center matrix.
At the MLNX_OFED Download Center matrix, choose
The version to install (you may need to select Archive Versions),
RHEL/CentOS/Rocky (under OS Distribution), and
The relevant OS Distribution Version and Architecture.
Click the supported ISO or tgz package.
The Mellanox OFED (MLNX_OFED) Software: End-User Agreement page appears.
Accept the End User License Agreement by clicking I Have Read the Above End User License Agreement.
The selected software package starts to download.
Mount the downloaded ISO somewhere on the system.
The following example shows the ISO being mounted on the
/mntdirectory.sudo mount MLNX_OFED_LINUX-<version>.iso /mnt
Prepare to install the driver.
Remove
nvidia-mlnx-configandnvidia-peer-memory-dkms.sudo dnf remove -y nvidia-mlnx-config nvidia-peer-memory-dkms
The
mlnxofedinstallstep will remove packages prior to installing new ones, causingnvidia-mlnx-configandnvidia-peer-memory-dkmsto fall out because they depend on some of these removed packages. Removing those components ahead of time avoids issues. These will be reinstalled as a final step.Specify the new kernel version to use when installing the driver.
NEXTKERNEL=$(sudo grubby --default-kernel | sed 's/.*vmlinuz\-//g')
Install the driver with the -
kand -sflags to specify the new kernel version and kernel source path.sudo /mnt/mlnxofedinstall -k ${NEXTKERNEL} -s /lib/modules/${NEXTKERNEL}/build --force
Note
The system might report that additional software must be installed before performing the installation. If the message appears, install the software and then repeat this step.
Reboot.
sudo rebootReinstall
nvidia-mlnx-configandnvidia-peer-memory-dkms.sudo dnf install -y nvidia-mlnx-config nvidia-peer-memory-dkms
If you intend to use NVIDIA GPUDirect Storage (GDS), enable the driver’s GDS support according to the instructions at MLNX_OFED Requirements and Installation.