Changes and New Features

Operating Systems

Added support for the following operating systems:

• RHEL 9.2

• RHEL 8.8

• SLES15-SP5

VXLAN GBP Options Offload with OVS

[ConnectX-6 Dx and above] Added support of encap and decap of VXLAN tunnel with GBP options offload with OVS.

TC Rules: Additional Actions

[ConnectX-5 and above] Added support for adding TC rules with trap action with additional actions (mirror and pedit).

Multiport E-Switch

[ConnectX-6 Dx] Added support for Multiport E-Switch, a mode where a single E-Switch connects all VPorts and physical ports on the NIC. This allows for scenarios such as sending traffic from a VF created on PF0 to an uplink that is natively associated with the uplink of PF1.

OVS Offload with MACVLAN Interface Above Bond

[ConnectX-6 Dx and above] Added support for OVS offload when MACVLAN interface above bond is attached to OVS bridge.

PCC fwtrace

[ConnectX-6 Dx and BlueField-2] Added support for installing a special user image into the firmware, which can be burned into either of two available slots for such applications, which enables monitoring the image's activities using the fw_tracer located inside the mlx5 driver. To view the output of the tracer, the user can access trace point, but it is important to note that they can only view traces that are generated after enabling the trace point by using the following command: 'echo 1 > sys/kernel/debug/tracing/events/mlx5/mlx5_fw/enable'." Note: Not supported in ConnectX-6 Lx adapter cards.

Relaxed Ordering in VFs/QEMU Assigned VFs

[ConnectX-6 and above] Added support for using Relaxed Ordering in VFs directly and in VFs assigned to QEMU. Relaxed Ordering can significantly improve performance on certain setups and, until now, it could be used only in PFs.

Migratable Bit

[ConnectX-6 Dx and above] Added support for Migratable Bit in Live Migration. Because some features cannot be migrated, such as IPsec, for example, when VF is marked as migratable, those features are disabled. This feature allows the user to configure whether VF can be migrated.

Live Migration Dirty Page Tracking Support in Linux

[ConnectX-7] The dirty pages tracking support enables reducing downtime upon live migration. Once it is used, only the pages that were really dirtied by the device will be marked in QEMU as dirty and will be sent to the target upon stop. Without dirty tracking, all RAM is marked dirty so all RAM is resent upon stop and the downtime is increased.

Configuring Hairpin Queue Size

[ConnectX-5 and above] Added the ability to configure the number and size of hairpin queues through devlink param command.

For example:
devlink dev param set pci/0000:08:00.0 name hairpin_queue_size value 512 cmode driverinit
devlink dev param set pci/0000:08:00.0 name hairpin_num_queues value 1 cmode driverinit

CC - RTT response SL (CNP SL)

[ConnectX-6 Dx and above] Users can now customize the DSCP value of RTT Response packets (in Ethernet, using debugfs). This feature allows for prioritization of RTT Response packets, preventing any delay that might lead to incorrect congestion assumptions on the RTT Requester side.

Expose VF RoCE Statistics on Host Side (Representor)

[All HCAs] Added support for allowing the host to track various statistics for the VFs, specifically all of the Q_counters stats even if he gives the VF to a VM, through the representor Q_counters for the VF which are now exposed over the host.

Fatal QP Error Logging

[All HCAs] With this feature, a kernel error log is now generated when certain fatal QP errors occur.

Enabling Selective Repeat by Default

[ConnectX-7 and above] Added support for making Selective Repeat (SR) protocol to be anabled by default. SR retransmits only the frame that is damaged and not all of the frames that were sent. As such, SR makes more efficient use of network bandwidth.

Matching IB BTH in RoCE Packets

[ConnectX-6 Dx and above] Added support for matching the IB BTH in RoCE packets. One example of this match is that the user can monitor RoCEv2 CNP (Congestion Notification Packet) by matching BTH opcode 0x81.

General

Bug fixes

Netdev Interface Configuration is not Preserved During Reload/Reset/Recovery

As of OFED 23.04, during reset/reload/recovery flows, the netdev interface is destroyed and re-created (rather than just suspended). As a result, the netdev interface configuration is not preserved, and must be re-applied. The way to do this is to use proper network-scripts and/or udev rules files to configure network interface parameters. These are automatically triggered whenever a netdev interface is added, regardless of whether it was added due to a user-initiated operation or an automatic failure recovery operation. Thus, no special processing is required to re-apply the network interface configuration parameters following a reset/reload/recovery operation – it is performed automatically.

Prenotification: ULP Mode Deprecation

Starting from next release, MLNX_OFED will support IPoIB enhanced mode only. The ability to switch back to ULP mode using ipoib_enhanced module param will not be supported.
For more information about the enhanced mode, please refer to OFED user manual, example: Enhanced IP over InfiniBand

Installation, ISO, RedHat

In order to address RHEL kernel symbol changes, ISO images for the following operating systems are built with the updated kernel versions as follows:

            

            
OS Name            Old Kernel                       New Kernel
rhel8.6-aarch64    4.18.0-372.9.1.el8_6.aarch64     4.18.0-372.41.1.el8_6.aarch64 
rhel8.6-ppc64le    4.18.0-372.0.1.el8_6.ppc64le     4.18.0-372.41.1.el8_6.ppc64le 
rhel8.6-x86_64     4.18.0-372.9.1.el8_6.x86_64      4.18.0-372.41.1.el8_6.x86_64 
rhel8.7-aarch64    4.18.0-425.3.1.el8.aarch64       4.18.0-425.14.1.el8_7.aarch64 
rhel8.7-ppc64le    4.18.0-425.3.1.el8.ppc64le       4.18.0-425.14.1.el8_7.ppc64le 
rhel8.7-x86_64     4.18.0-425.3.1.el8.x86_64        4.18.0-425.14.1.el8_7.x86_64 
rhel9.0-aarch64    5.14.0-70.13.1.el9_0.aarch64     5.14.0-70.46.1.el9_0.aarch64 
rhel9.0-ppc64le    5.14.0-70.13.1.el9_0.ppc64le     5.14.0-70.46.1.el9_0.ppc64le 
rhel9.0-x86_64     5.14.0-70.13.1.el9_0.x86_64      5.14.0-70.46.1.el9_0.x86_64 
rhel9.1-aarch64    5.14.0-162.6.1.el9_1.aarch64     5.14.0-162.19.1.el9_1.aarch64 
rhel9.1-ppc64le    5.14.0-162.6.1.el9_1.ppc64le     5.14.0-162.19.1.el9_1.ppc64le 
rhel9.1-x86_64     5.14.0-162.6.1.el9_1.x86_64      5.14.0-162.19.1.el9_1.x86_64 
        

This change comes to support RedHat updated kernels without the need to add --add-kernel-support during OFED installation.

Power Setups on UCX/HPC-X

UCX/HPC-X no longer supports Power setups.

NEO-Host

Starting from this release, OFED will discontinue the provision of NEO-Host.

NEO-Host can be manually downloaded and installed using the following guide: https://docs.nvidia.com/networking/display/NEOSDKv26/Installation+and+Initial+Configuration#InstallationandInitialConfiguration-DownloadingtheMellanoxNEOSDKSoftware

dapl

Starting from this release, OFED will discontinue the provision of dapl.

Signing Key for SLES15 sp4 and sp5

As of version 23.04, the builds for SLES15 sp4 and sp5 are being singed with a newer signing key. The corresponding public key can be downloaded from https://www.mellanox.com/downloads/ofed/nv_nbu_kernel_signing_key_pub.der instead of https://www.mellanox.com/downloads/ofed/mlnx_signing_key_pub.der.

dump_pr SM Plugin

Starting from this release, OFED will discontinue the provision of dump_pr subnet manager plugin.

mpi-selector

Starting from this release, OFED will discontinue the provision of mpi-selector.

OpenSM Init

Starting 23.07 release, opensm init service will move from init.d to systemd.