MLNX_OFED Documentation Rev 4.6-1.0.1.1
MLNX_OFED Documentation Rev 4.6-1.0.1.1

General Support in MLNX_OFED

Operating System

Platform

Debian 8.11-Kernel 4.4

x86_64

Debian 9.4

x86_64

Debian 9.5

x86_64

Debian 9.5-Kernel 4.4

x86_64

Debian 9.6-Kernel 4.4

x86_64

EulerOS2.0 SP3

x86_64/AArch64 (Arm)

EulerOS2.0 SP5

x86_64

Fedora 27

x86_64

Fedora 29

x86_64

Kernel 4.20.14

x86_64

RHEL6.3/CentOS6.3

x86_64

RHEL6.8/CentOS6.8

x86_64

RHEL6.9/CentOS6.9

x86_64/PPC64 (Power8)

RHEL6.10/CentOS6.10

x86_64

RHEL7.2/CentOS7.2

x86_64/PPC64 (Power8)/PPC64LE (Power8)

RHEL7.3/CentOS7.3

x86_64/PPC64 (Power8)/PPC64LE (Power8)

RHEL7.4/CentOS7.4

x86_64/PPC64 (Power8)/PPC64LE (Power8)/PPC64LE (Power9)/ARM64

RHEL7.5/CentOS7.5

x86_64/PPC64 (Power8)/PPC64LE/ARM64

RHEL7.6/CentOS7.6

x86_64/PPC64 (Power8)/PPC64LE

RHEL7.6 ALT (Pegas 1.2 LE for P9)

x86_64/PPC64LE (Power9)

RHEL8.0 (beta)

x86_64/PPC64 (Power8)/PPC64LE (Power8)/PPC64LE (Power9)

SLES11 SP3

x86_64/PPC64 (Power8)

SLES11 SP4

x86_64/PPC64 (Power8)

SLES12 SP2

x86_64/PPC64LE (Power8)

SLES12 SP3

x86_64/PPC64LE (Power8)

SLES12 SP4

x86_64/PPC64LE (Power8)

SLES15

x86_64/PPC64LE (Power8)

SLES15 SP1 (beta)

x86_64

Ubuntu 14.04.05

x86_64

Ubuntu 16.04.04*

x86_64/PPC64LE (Power8)/PPC64LE (Power9)

Ubuntu 18.04.01

x86_64/PPC64LE (Power8)/PPC64LE (Power9)

Ubuntu 18.04.02

x86_64/PPC64LE (Power9)

XenServer 6.5

x86_64

XenServer 7.1

x86_64

* Ubuntu 16.04.05 and 16.04.06 can be used only with Kernels of version 4.4.0-143 or below.

Notes:

  • 32 bit platforms are no longer supported in MLNX_OFED.

  • For RPM based distributions, if you wish to install OFED on a different kernel, you need to create a new ISO image, using mlnx_add_kernel_support.sh script. See the MLNX_OFED User Manual for instructions.

  • Upgrading MLNX_OFED on your cluster requires upgrading all of its nodes to the newest version as well.

The following are the supported non-Linux Virtual Machines in this current MLNX_OFED version:

HCA

Windows Virtual Machine Type

WinOF version

Protocol

ConnectX-3

Windows 2012 R2 DC

MLNX_VPI 5.50

IPoIB, ETH

ConnectX-3 Pro

Windows 2016 DC

MLNX_VPI 5.50

IPoIB, ETH

ConnectX-4

Windows 2012 R2 DC

MLNX_WinOF2 2.10

IB, IPoIB, ETH

ConnectX-4 Lx

Windows 2016 DC

MLNX_WinOF2 2.10

IB, IPoIB, ETH

Tested Hypervisors

HCAs

Operating System

SR-IOV

All

RHEL6.10 KVM

RHEL7.5 KVM

RHEL7.6 KVM

SLES12 SP3 KVM

Ubuntu18.04.1 KVM

Debian 8.11 KVM Kernel 4.4

Debian 9.4 KVM

Debian 9.5 KVM Kernel 4.14

ConnectX-4

XenServer 7.0 (IB only)

XenServer 7.2 (IB only)

ConnectX-4/ConnectX-5

RHEL7.6 KVM

SLES15 KVM

Ubuntu16.04.5 KVM

EulerOS2.0 SP5 KVM (IB only)

Paravirtualized

ConnectX-3/ConnectX-4

RHEL 7.5 KVM

Ubuntu 16.04.4 KVM

Ubuntu 18.04.1 KVM

ConnectX-4/ConnectX-5

RHEL 7.4 KVM

RHEL 7.6 KVM

XenServer 7.0 (IB only)

XenServer 7.2 (IB only)

Below is a list of all the OSs that support Accelerated Switch and Packet Processing (ASAP2) in MLNX_OFED package:

  • RHEL 7.4

  • RHEL 7.5

  • RHEL 7.6

  • CentOS 7.2 (default Kernel + Kernel 4.9)

  • Ubuntu 18.04 (Kernel 4.15)

The following are the hardware and software requirements of the current MLNX_OFED version.

  • Linux operating system

  • Administrator privileges on your machine(s)

  • Disk Space: 1GB

For the OFED Distribution to compile on your machine, some software packages of your operating
system (OS) distribution are required.
To install the additional packages, run the following commands per OS:

Operating System

Required Packages Installation Command

RHEL/OL/Fedora

yum install perl pciutils python gcc-gfortran libxml2-python tcsh libnl.i686 libnl expat glib2 tcl libstdc++ bc tk gtk2 atk cairo numactl pkgconfig ethtool lsof

XenServer

yum install perl pciutils python libxml2-python libnl expat glib2 tcl bc libstdc++ tk pkgconfig ethtool

SLES 11 SP2

zypper install perl pciutils python libnl-32bit libxml2-python tcsh libnl libstdc++46 expat glib2 tcl bc tklibcurl4 gtk2 atk cairo pkg-config ethtool

SLES 11 SP3

zypper install perl pciutils python libnl-32bit libxml2-python tcsh libstdc++43 libnl expat glib2 tcl bc tk libcurl4 gtk2 atk cairo pkg-config ethtool lsof

SLES 12

zypper install pkg-config expat libstdc++6 libglib-2_0-0 lib- gtk-2_0-0 tcl libcairo2 tcsh python bc pciutils libatk-1_0-0 tk python-libxml2 lsof libnl3-200 ethtool lsof

SLESS 15

python ethtool libatk-1_0-0 python2-libxml2-python tcsh lib- stdc++6-devel-gcc7 libgtk-2_0-0 tcl libopenssl1_1 libnl3-200 make libcairo2 expat libmnl0 insserv-compat pciutils lsof lib- glib-2_0-0 pkg-config tk

Ubuntu/Debian

apt-get install perl dpkg autotools-dev autoconf libtool auto- make1.10 automake m4 dkms debhelper tcl tcl8.4 chrpath swig graphviz tcl-dev tcl8.4-dev tk-dev tk8.4-dev bison flex dpatch zlib1g-dev curl libcurl4-gnutls-dev python-libxml2 libvirt-bin libvirt0 libnl-dev libglib2.0-dev libgfortran3 automake m4

pkg-config libnuma logrotate ethtool lsof

Debian 8

apt-get install libnl-3-200 automake debhelper curl dkms logrotate libglib2.0-0 python-libxml2 graphviz tk tcl libvirt- bin coreutils pkg-config autotools-dev flex autoconf pciutils quilt module-init-tools libvirt0 libstdc++6 dpkg libgfortran3 procps lsof libltdl-dev gcc dpatch chrpath grep m4 gfortran bison libnl-route-3-200 swig perl make ethtool lsof

This current MLNX_OFED version supports the following Mellanox network adapter cards firmware versions:

HCA

Recommended Firmware Rev.

Additional Firmware Rev. Supported

ConnectX®-3

2.42.5000

2.40.7000

ConnectX®-3 Pro

2.42.5000

2.40.7000

ConnectX®-4

12.25.1020

12.24.1000

ConnectX®-4 Lx

14.25.1020

14.24.1000

ConnectX®-5

16.25.1020

16.24.1000

ConnectX®-5 Ex

16.25.1020

16.24.1000

ConnectX®-6 (beta)

20.25.1500

N/A

Innova IPsec EN

16.25.1020

14.24.1000

BlueField™

18.25.1020

N/A

Connect-IB®

10.16.1200

10.16.1020

For the official firmware versions, please see:
http://www.mellanox.com/content/pages.php?pg=firmware_download

This MLNX_OFED version is compatible with the following:

Mellanox Product

Description/Version

MLNX-OS®

MSB7xx0 w/w MLNX-OS® version 3.7.1000 (see note 2 below)

Unified Fabric Manager (UFM®)

v6.1

HCOLL (see note 1 below)

v4.2

Open MPI

v4.0.x

MFT

v4.11.0

Notes:

  1. MLNX_OFED was tested with this switch. However, additional switches may be supported as well.

  2. HCOLL is now the default FCA version used in HPC-X, starting from HPC-X v1.8. This version replaces FCA v2.x.

RoCE Modes Matrix

Software Stack / Inbox Distribution

RoCEv1 (IP Based GIDs) Supported as of Version

RoCEv2 Supported as of Version

RoCEv1 & RoCEv2 (RoCE per GID)

Supported as of Version

ConnectX-3/ ConnectX-3 Pro

ConnectX-4/ ConnectX-4 Lx/ ConnectX-5/

ConnectX-5 Ex

ConnectX-3 Pro

ConnectX-4/ ConnectX-4 Lx/

ConnectX-5/ ConnectX-5 Ex

ConnectX-3 Pro/ConnectX-4/ConnectX-4 Lx/

ConnectX-5/ConnectX-5 Ex

MLNX_OFED

2.1-x.x.x

3.0-x.x.x

2.3-x.x.x

3.0-x.x.x

3.0-x.x.x

Kernel.org

3.14

4.4

4.4

4.4

4.4

RHEL

6.6, 7.0

-

-

-

-

SLES

12

-

-

-

-

Ubuntu

14.04.4,

16.04, 15.10

-

-

-

-

Note: Support for ConnectX-5 and ConnectX-5 Ex adapter cards in MLNX_OFED starts from v4.0.

RDMA CM Default RoCE Mode

The default RoCE mode on which RDMA CM runs is RoCEv2 instead of RoCEv1, starting from MLNX_OFED v4.1. RDMA_CM session requires both the client and server sides to support the same RoCE mode. Otherwise, the client will fail to connect to the server.
For further information, refer to RDMA CM and RoCE Version Defaults Community post.

© Copyright 2023, NVIDIA. Last updated on Dec 26, 2023.