OFED Software Stack | NVIDIA Base Command Manager

This section explains how OFED (OpenFabrics Enterprise Distribution) packages are installed so that BCM can use InfiniBand for data transfer during regular use—that is for computation runs, rather than for booting up or provisioning. The configuration of PXE booting over InfiniBand is described in section 5.1.3 of the Administrator Manual. The configuration of node provisioning over InfiniBand is described in section 5.3.3 of the Administrator Manual.

10.1 Choosing A Distribution Version, Or A Vendor Version, Ensuring The Kernel Matches, And Logging The Installation

By default, the Linux distribution OFED packages are matched to the distribution kernel version and installed on the cluster. This is the safest option in most cases, and also allows NFS over RDMA.

BCM also packages NVIDIA’s Mellanox OFED (sometimes called MOFED) software. Such vendor OFED packages can be more recent than the distribution packages, which means that they can provide support for more recent hardware and firmware, as well as more features.

For the vendor OFED packages to work, the OFED firmware as provided by the manufacturer should in general be recent, to ensure software driver compatibility.

If NVIDIA AI Enterprise is to be used, then a matching, supported, version of OFED must be selected. A matching, supported OFED version can be found by contacting support via the BCM subscription at the NVIDIA Enterprise Support page at:

https://www.nvidia.com/en-us/support/enterprise/

BCM vendor OFED packages can be selected and installed during the initial cluster installation (figure 3.23), replacing the default distribution OFED stack. The stack can also be installed later on, after the cluster is set up.

If there is no prebuilt OFED kernel modules package available for the kernel in use, then using a supported kernel is recommended.

When updating kernels on the head or the regular nodes, the updated BCM OFED software stack must be reinstalled.

If the BCM OFED software stack is installed during the cluster installation procedure itself (section 3.3.18), then some basic information is logged to /var/log/cmfirstboot.log, which is the general first boot log.

If the BCM OFED software stack is not installed during the cluster installation procedure itself, then it can be installed later when the cluster is up and running.

NVIDIA DOCA OFED is a further non-distribution OFED stack. Unlike the BCM repository OFED packages, it is not packaged by BCM, and is installed from the NVIDIA DOCA package repositories using an installation script supplied with BCM. It is available in BCM 10.30.0 and later, and is covered in section 10.3.

A successful installation of the BCM OFED software stack (section 10.2) onto a running cluster consists of the BCM OFED package installation, as well as then running an installation script. The vendor and version number installed can then be found in /etc/cm-ofed. Further installation details can be found in /var/log/cm-ofed.log.

10.2 Mellanox OFED Stack Installation Using The BCM Package Repository

Package names: mlnx-ofed54, mlnx-ofed56, mlnx-ofed57, mlnx-ofed58, mlnx-ofed59-dgx-h100, mlnx-ofed59, mlnx-ofed23.04, mlnx-ofed23.07, mlnx-ofed23.10, mlnx-ofed24.01, mlnx-ofed24.04, mlnx-ofed24.07

The Mellanox stacks are installed and configured by BCM in an identical way as far as the administrator is concerned. In this section (section 10.2):
<vendor-ofedVersion>
is used to indicate where the administrator must carry out a substitution. For Mellanox, the substitution is one of the following:

mlnx-ofed54 for the Mellanox version 5.4 stack
mlnx-ofed56 for the Mellanox version 5.6 stack
mlnx-ofed57 for the Mellanox version 5.7 stack
mlnx-ofed58 for the Mellanox version 5.8 stack
mlnx-ofed59 for the Mellanox version 5.9 stack
mlnx-ofed59-dgx-h100 for the Mellanox version 5.9 stack for DGX H100 hardware
mlnx-ofed23.04 for the Mellanox stack release of April 2023
mlnx-ofed23.07 for the Mellanox stack release of July 2023
mlnx-ofed23.10 for the Mellanox stack release of October 2023
mlnx-ofed24.01 for the Mellanox stack release of January 2024
mlnx-ofed24.04 for the Mellanox stack release of April 2024
mlnx-ofed24.07 for the Mellanox stack release of July 2024
mlnx-ofed24.10 for the Mellanox stack release of October 2024

These stacks are currently supported by the NVIDIA Base Command Manager 10-supported distributions (RHEL and derivatives, SLES, and Ubuntu), as determined by the compatibility matrices in the downloads pages accessible from https://network.nvidia.com/support/mlnx-ofed-matrix/.

For most use cases it usually makes sense to get the most recent supported stack.

The mlnx-ofed49 stack is an LTS release, aimed mainly at supporting older legacy hardware. The stack may be useful for one of the following:
- ConnectX-3 Pro cards
- ConnectX-3 cards
- Connect-IB cards
- RDMA experimental verbs library (mlnx_lib)
More recent distributions, such as RHEL9, Ubuntu 22.04, SLES15sp4, no longer support the mlnx-ofed49 stack.
Other LTS releases exist. At the time of writing of this paragraph (October 2024), there is a specific mlnx-ofed58 LTS version release aimed at ConnectX-4 cards and beyond, and there is also a specific mlnx-ofed23.10 LTS version. Updated details of the LTS version releases are given at https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/.

Each stack version needs to be matched to a firmware version associated with the OFED device used. OFED devices used, ConnectX, BlueField, and others, must also be matched along with their firmware version via the downloads pages accessible from the URL https://network.nvidia.com/support/mlnx-ofed-matrix/. Deviating from compatible versions is not supported.

Details on the compatibility of older stack versions can also be found via that URL.

Returning back to the subject of OFED package installation via the package manager: For example, a yum install command indicated by:
yum install <vendor-ofedVersion>
means that the installation of the BCM OFED package is executed with one of these corresponding yum install commands, for example:
yum install mlnx-ofed59

10.2.1 Installing The OFED Stack Provided By The BCM Repository Vendor Package

Running the package manager command associated with the distribution (yum install, zypper up, apt install), unpacks and installs or updates several packages and scripts. For example:
yum install <vendor-ofedVersion>
However, it does not carry out the installation and configuration of the driver itself due to the fundamental nature of the changes it would carry out. The script:
<vendor-ofedVersion>-install.sh
can be used after the package manager installation to carry out the installation and configuration of the driver itself. Running it without options displays a usage help text.

10.2.1.1 Head And Software Image Installation With Prebuilt Drivers

The script can be run on the nodes as follows:

On the head node, the default distribution OFED software stack can be replaced with the vendor OFED software stack made available from the BCM repository, by using the script’s head option, -h:
```
[root@basecm10~]# /cm/local/apps/<vendor-ofedVersion>/current/bin/<vendor-ofedVersion>-install.sh -h
```
A reboot is recommended after the script completes the install, to help ensure the new image is cleanly used by the head node.
For a software image, for example default-image, used by the regular nodes, the default distribution OFED software stack can be replaced with the vendor OFED software stack made available from the BCM repository, by using the script’s software image option, -s:
```
[root@basecm10~]# /cm/local/apps/<vendor-ofedVersion>/current/bin/<vendor-ofedVersion>-install.sh -s \
 default-image
```
A reboot updates the software image used on the regular node.

If the distribution kernel is updated on any of these head or regular nodes after the vendor OFED stack has been installed, then the vendor OFED kernel modules made available from the BCM repository must be reinstalled. This can be done by running the installation scripts again, which replaces the kernel modules again, along with all the other OFED packages.

The OFED Stack provided by BCM can be removed by appending the -r option to the appropriate -h or -s option. Removing the packages from a head node or software image can lead to package dependency breakage, and software not working any more. So using the “-r” option should be done with caution.

10.2.2 Upgrading Kernels When The OFED Stack Has Been Provided By The BCM Repository Vendor Package—Reinstallation Of The OFED Stack

For all distributions, as explained in the preceding text, a vendor OFED stack is installed and configured via the script <vendor-ofedVersion>-install.sh.

OFED reinstallation may be needed if the kernel is upgraded.

10.2.2.1 Kernel Upgrades—Blocking And Unblocking

If the OFED stack is installed from the vendor OFED packages, then the script customizes the vendor OFED stack for the existing kernel, and replaces the distribution stack. However, updating the kernel afterwards, without updating the stack along with it, could lead to unexpected behavior due to the customization. Kernel and kernel development updates are therefore prevented from taking place by a package management system block. The block is set up when the <vendor-ofedVersion>-install.sh script is run.

Updating the kernel, kernel development, and OFED stack in such a configuration therefore requires that the administrator manually overrides the block so that the OFED stack can be handled with consideration.

The following procedure can thus be followed to update and install the kernel packages and OFED stack:

Overriding the block:
- In Red Hat-based systems, the /etc/yum.conf file must be edited. In that file, in the line that starts with exclude, the kernel and kernel-devel packages need to be removed, so that they are no longer excluded from updates.
- In SUSE, the kernel-default and kernel-default-devel packages must be unlocked. The command:
```
  zypper removelock kernel-default kernel-default-devel
```
  unlocks them so that they can take part in updates again.
- In Ubuntu, the packages that are blocking the update can be checked with the apt-mark showhold command.
  
  Example
```
apt-mark showhold
linux-generic
linux-headers-generic
linux-image-generic
```
  The apt-mark unhold command is used to unblock them:
  
  Example
```
apt-mark unhold linux-generic linux-headers-generic linux-image-generic
```
Updating the kernel and kernel development packages:
- The packages can be updated on the head node with:
  - yum update for the RHEL family
  - zypper up for SLES
  - apt update; apt upgrade for Ubuntu
- To update the packages on the regular nodes the procedure outlined in section 11.3.3 of the Administrator Manual is followed:
  - The packages on the regular node image (for example, default-image) are updated according to distribution:
    - in Red Hat-based systems as follows:
      yum --installroot=/cm/images/default-image update
    - or in SLES as follows:
      zypper --root=/cm/images/default-image up
    - or in Ubuntu as follows, using the cm-chroot-sw-img tool (here):
      root@basecm10:~# cm-chroot-sw-img /cm/images/default-image
      root@basecm10:~# apt update; apt upgrade #upgrade takes place in image
      ...
      root@basecm10:~# exit #get out of chroot
  - The kernelversion setting for the regular node image, which in this example is the default default-image, can be updated as follows:
    
    Example
```
[root@basecm10 ~]# cmsh
[basecm10]% softwareimage
[basecm10->softwareimage]% use default-image
[basecm10->softwareimage[default-image]]% set kernelversion 5.15.0-113-generic
[basecm10->softwareimage[default-image*]]% commit
```
    This ensures that the updated kernel is used after reboot. Tab-completion in the set kernelversion line prompts for the right kernel from available options.
A reboot of the head and regular nodes installs the new kernel.

Configuring and installing the vendor OFED stack driver for the new kernel is done by running the script <vendor-ofedVersion>-install.sh as before, as follows:

For a stack that is on the head node, the compilation should be done together with the -h option:

[root@basecm10~]# /cm/local/apps/<vendor-ofedVersion>/current/bin/<vendor-ofedVersion>-ins\
tall.sh -h

For a software image used by the regular nodes, for example default-image, the compilation should be done together with the -s option:
```
[root@basecm10~]# /cm/local/apps/<vendor-ofedVersion>/current/bin/<vendor-ofedVersion>-ins\
tall.sh -s default-image
```

10.3 DOCA OFED Stack Installation Using The BCM DOCA OFED Installation Script

NVIDIA DOCA OFED is an OFED stack provided by NVIDIA. Unlike the BCM repository OFED packages of section 10.2, the DOCA OFED stack is not packaged by BCM. It is instead installed directly from the NVIDIA DOCA package repositories, with the help of an installation script that comes with the BCM cluster-tools package. The upstream NVIDIA DOCA OFED stack is a subset of the full DOCA software stack.

The installation script is available in BCM 10.30.0 and later. The BCM repository OFED stacks of section 10.2 remain available, and either stack may be used.

For DOCA OFED, installation is carried out on a running cluster by running the BCM DOCA installation script. This method is covered in this section.

10.3.1 Available DOCA OFED Versions

The set of DOCA OFED versions that the script offers is fixed per BCM release, so different clusters may offer different sets. Rather than relying on a version list reproduced here, the list for a particular cluster should be read from the cluster itself. Running the script with no arguments displays a usage text that includes the available versions. The same list is held in:

/cm/local/apps/cluster-tools/doca/etc/doca-ofed-versions.conf

If a required version is not offered by the cluster, then support should be contacted via the BCM subscription at the NVIDIA Enterprise Support page at:

https://www.nvidia.com/en-us/support/enterprise/

10.3.2 Running The DOCA OFED Installation Script

The installation script is located at:

root@basecm10:~# /cm/local/apps/cluster-tools/bin/cm-doca-ofed-install.sh

Running the script without any arguments shows a usage help text.

The following arguments must be specified:

-v|--version: This chooses which DOCA OFED version is to be used. The versions that can be given here are the ones the script itself lists (section 10.3.1).
-h or -s: these select whether the DOCA OFED installation is carried out on the head node or on a software image

Example

Installing on the head node, where <version> is replaced by one of the versions the script lists:

/cm/local/apps/cluster-tools/bin/cm-doca-ofed-install.sh -v <version> -h

Installing into a software image:

/cm/local/apps/cluster-tools/bin/cm-doca-ofed-install.sh -v <version> -s default-image

The installation log is recorded in:

/var/log/cm-doca-ofed.log

The --cmdvd option makes the script install from a BCM install medium rather than from the NVIDIA DOCA package repositories. BCM 10 install media do not carry DOCA OFED packages, so the option finds no versions when it is pointed at a BCM 10 medium. The packages are present on a BCM 11 install medium for the same distribution and architecture, and such a medium can be used as the source. Supplying a local mirror or a copied repository does not work, because the script relies on the directory layout of a BCM install medium.

If there is no DOCA OFED stack available for the kernel in use, then falling back to a supported kernel is recommended. When updating kernels on the head node or on the regular nodes, the DOCA OFED software stack must be reinstalled.