Skip to navigation

OFED Software Stack

View as Markdown

This section explains how OFED (OpenFabrics Enterprise Distribution) packages are installed so that BCM can use InfiniBand for data transfer during regular use—that is for computation runs, rather than for booting up or provisioning. The configuration of PXE booting over InfiniBand is described in section 5.1.3 of the Administrator Manual. The configuration of node provisioning over InfiniBand is described in section 5.3.3 of the Administrator Manual.

10.1 Choosing A Distribution Version, Or A Non-distribution Version, Ensuring The Kernel Matches, And Logging The Installation

By default, the Linux distribution OFED stack is matched to the kernel version and installed on the cluster. This is the safest option in most cases, and also allows NFS over RDMA.

BCM also allows NVIDIA’s Mellanox OFED (sometimes called MOFED) software to be installed. Such separate, non-distribution OFED stacks can be more recent than the distribution version, which means that they can provide support for more recent hardware and firmware, as well as more features.

For recent OFED stacks to work, the OFED firmware as provided by the manufacturer should in general also be recent, to ensure software driver compatibility.

If NVIDIA AI Enterprise is to be used, then a matching, supported, version of OFED must be selected. A matching, supported OFED version can be found by contacting support via the BCM subscription at the NVIDIA Enterprise Support page at:

https://www.nvidia.com/en-us/support/enterprise/

Since June 2025, BCM no longer supports the non-distribution legacy BCM-packaged Mellanox OFED stack. These were packages that were provided from the BCM repository, with names such as mlnx-ofed58, mlnx-ofed59-dgx-h100, mlnx-ofed59, mlnx-ofed23.04, and mlnx-ofed24.07.

BCM has moved on to installing the non-distribution NVIDIA DOCA OFED stack directly (section 10.2), with the help of an installation script. The upstream NVIDIA DOCA OFED stack is a subset of the full DOCA software stack.

If there is no DOCA OFED stack available for the kernel in use, then falling back to a supported kernel is recommended.

When updating kernels on the head or the regular nodes, the updated NVIDIA DOCA OFED software stack must be reinstalled.

10.2 DOCA OFED Stack Installation Using The BCM DOCA OFED Installation Script

The set of DOCA OFED versions that can be installed can change with each BCM point release, as the DOCA LTS releases that BCM tracks are themselves updated. Running the installation script without arguments displays a usage help text that shows the versions the cluster currently offers. The same list is held in:

/cm/local/apps/cluster-tools/doca/etc/doca-ofed-versions.conf

At the time of writing of this section (August 2026), the file content is:

Example

root@basecm11:~# cat /cm/local/apps/cluster-tools/doca/etc/doca-ofed-versions.conf
# DOCA versions
2.9.4
3.2.2

The versions in the file on the actual cluster are the ones that BCM has install-tested against the supported Linux distribution versions. Restricting the choice in this way avoids a version being installed that is not supported on the cluster, which can leave the cluster in a broken state. An administrator may add a version to the file, but a version added in that way has not been tested by BCM.

If a required version is not offered by the cluster, then support should be contacted via the BCM subscription at the NVIDIA Enterprise Support page at:

https://www.nvidia.com/en-us/support/enterprise/

For DOCA OFED drivers, installation can be carried out for BCM with these methods:

  • during the initial cluster installation (figure 3.23)

  • on a running cluster by running the BCM DOCA installation script, which comes with the BCM cluster-tools package. This method is covered in this section.

  • on a running cluster by creating an new image for nodes using the cm-create-image or cm-image tools with the --extra-pkg-group option. This method is covered in here.

The installation script is located at:

root@basecm11:~# /cm/local/apps/cluster-tools/bin/cm-doca-ofed-install.sh

Running the script without any arguments shows a usage help text.

The following arguments must be specified:

  • -v|--version: This chooses which DOCA OFED version is to be used. The versions that can be given here are the ones the script itself lists.

  • -h or -s: these select whether the DOCA OFED installation is carried out on the head node or on a software image

Example

Installing on the head node, where <version> is replaced by one of the versions the script lists:

/cm/local/apps/cluster-tools/bin/cm-doca-ofed-install.sh -v <version> -h

The following argument for the installation script is optional:

  • --cmdvd: this option makes the script install from a BCM install medium rather than from the NVIDIA DOCA package repositories, which is useful for air-gapped deployments. BCM 11 install media carry DOCA OFED packages, so pointing the option at one finds the packages directly.

The installation log is recorded in:

/var/log/cm-doca-ofed.log

while the version installed is recorded in

/etc/cm-ofed