Changes and New Features History

All HCAs

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-4 / ConnectX -4 Lx / ConnectX-5 / ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-6 Dx and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-6 and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-5 and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-5 / ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-4 and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-4 / ConnectX -4 Lx / ConnectX-5 / ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

23.10-4.0.9.1

General

Embedded Components

Updated the versions of the following embedded component:

• MFT

• ConnectX adapter cards firmware

For further information, see Embedded Components section.

General

Bug fixes

23.10-3.2.2.0

Operating Systems

Added support for the following OSes:

• RHEL 8.10

• RHEL 9.4

• SLES15-SP6

• Debian 12.5

For further information, see Supported Operating Systems section

Embedded Components

Updated the versions of the following embedded component:

• MFT

• ConnectX adapter cards firmware

For further information, see Embedded Components section.

Bridge Offload

Enabled Bridge Offload in mlx5_core by default. A new option was added to the configure of mlnx_ofa_kernel to disable it: "--without-bridge-offload".

To use this option from "mlnxofedinstall --add-kernel-support" (kernel modules rebuilding), add the following to the command line: "--kernel-extra-args --without-bridge-offload".

General

Bug fixesץ

23.10-2.1.3.1

General

General

Bug fixes

23.10-0.5.5.0

ASAP² Features

Rate Limiter Extension for LAG Mode

[ConnectX-5 and above] During rate configuration, this feature considers the aggregated port speed when LAG is configured, so the functions (PF,VF,SF) can utilize the maximum aggregated link speed for transmission.

Core Features

Live Migration Support with IPsec Full Offload

[ConnectX-7 and above] Added an option to perform live migration on a machine with IPsec enabled. This allows having a VF on VM that has a valid IPsec state and migrate it without having to remove the IPsec.

Note: This feature is currently not supported when working in MPV mode (Multi port VHCA/Dual port). Such support will be added in future releases.

Exposing the Max SF Configuration via Devlink Resource

[All HCAs] This feature allows user applications to view the maximum number of SFs (Scalable Functions) configured on the system via general Linux API, Devlink resource.

NetDev Features

NIC Temperature Exposure

[ConnectX-5 and above] Added support for NIC temperature exposure by implementing the hwmon kernel API.

Counters Indicating Packet Drops due to Catastrophic Steering Errors

[All HCAs] Added two counters through the Devlink health reporter - generated_pkt_steering_fail and handled_pkt_steering_fail. These counters indicate packet drops due to catastrophic steering errors.

RDMA Features

Using the RDMA Tool to Allow Controlled QKEYs Use by Non-privileged Users

[All HCAs] To allow using controlled QKEYs by non-privileged users, the following command can now be used in the iproute2 RDMA tool, instead of the "enforce_qkey_check" module parameter, which will be deprecated:

----- $rdma system set privileged-qkey on

IPsec Configuration Support When All IPsec Operations are Offloaded to the Hardware

[ConnectX-7 and above] Added support for configuring IPsec even when all IPsec operations are completely offloaded to the hardware. This feature not only provides a significant performance improvement, but also enables the use of IPsec over RoCE packets, which are outside the network stack and cannot be used without full hardware offload. As a result, users can now leverage the benefits of IPsec protocol with RoCE V2, even when using SR-IOV VFs.

Security Features

Configuration from VM/VF for Full Offload Transport Mode

[ConnectX-6 Lx and above] This feature introduces two new boolean attributes of a port function: ipsec_crypto and ipsec_packet (also known as IPsec Full Offload). The goal is to provide a level of granularity for controlling VF IPsec offload capabilities similar to what is currently offered in the software mode. This allows users to decide if they want both types of IPsec offload enabled for a VF, just one of them, or none (which is the default).

General

General

Bug fixes

23.07-0.5.1.2

RDMA Features

QKEY Mitigation in the Kernel

QKEY creation with the MSB set is available now for non-privileged users as well.

To allow non-privileged users to create QKEY with MSB set, the below new module parameter was added to ib_uverbs module:

• Module Parameter: enforce_qkey_check

• Description: Force QKEY MSB check for non-privileged user on UD QP creation

• Default: 0 (disabled)

Note: In this release, this module parameter is disabled by default to ensure backward compatibility and give customers the opportunity to update their applications accordingly. In the upcoming release, it will be enabled by default, and later on deprecated.

23.07-0.5.0.0

ASAP² Features

Header Rewrite and CT Support

[BlueField-2] Added support for offloading CT rules with header rewrite of L3.

CT UDP Unidirectional Traffic Offload

[All HCAs] Added support for offloading long-running unidirectional UDP connection with Conntrack.

CT Rules with NAT and Mirror Offload

[BlueField-2] Added support for offloading CT rules with NAT and mirror.

Core Features

Embedded CPU Virtual Functions

[Bluefield-2] Added the ability to create virtual functions within the ARM.

Full Chip Reset on BlueField-2

[BlueField-2] Full chip reset in DPU mode is now supported on BlueField-2 when using mlxfwreset with the "--sync 1" option. During this flow, the ARM is rebooted and the firmware is reloaded.

Firmware Page Limit via VHCA_ICM_CTRL

[ConnectX-5 and above] Added the ability to expose page limit via the VHCA_ICM_CTRL FW register. The expected behavior is similar to the previous page limit, except for the firmware size limitation. The new max limit is 2^32-1, which represents 'no limit'.

PRE_COPY State Support in the mlx5-vfio Driver

[ConnectX-6 Dx and above] Added support for PRE_COPY migration. The optional PRE_COPY state opens the saving data transfer FD before reaching STOP_COPY, and thereby helps reducing the downtime of the VM.

Lightweight Local SFs

[ConnectX-5 and above] Added a feature that decreases the amount of time needed to probe and configure SFs (s ub-functions) by reducing the time of Devlink reload of the SF.

QEMU Live Migration Support

[ConnectX-6 Dx and above] Added support for QEMU VM migration with an assigned VF from one source host to another destination host. This is achieved as part of the general QEMU migration flow, which involves suspending the VF on the source host, transferring all its data to the destination host, and resuming the VF on the destination.

Note: This migration feature includes basic functionality only, and does not yet support advanced features such as dirty page tracking or pre-copy.

4 Ports VF LAG Support

[ConnectX-7] Added support for VF LAG over 4 ports HCAs.

Note: Only LAGs with all HCA ports are supported. Meaning, with a 4-port HCA, only 4 port LAG is supported. A 2-port or a 3-port LAG is not supported.

Installation

FIPS Compliance

[All HCAs] Starting from v23.07, OFED is FIPS (Federal Information Processing Standard) compliant for the RH8.8, RH9.2 and SLES15.5 operating systems. The following limitations should be considered:

• OFED Installation with FIPS support requires a manual firmware upgrade.

• OFED Installation with add-kernel-support will result in the removal of the FIPS support.

NetDev Features

Enhanced CQE Compression

[ConnectX-5 and above] Added support for the enhanced version of the RX CQE compression hardware feature. By compressing the RX CQEs, the PCI bandwidth utilization is improved and the load on it is reduced. The enhanced version of this device feature has improved latency and CPU utilization.

256 Bit Keys with kTLS Offload Support

[BlueField-2] Added support for kTLS offload with a 256 bit key size.

Improving Affinity Hints according to Numa Distances

[All HCAs] Updated the binary NUMA preference for the system to consider actual distances. Following the update, remote NUMA nodes with shorter distances are preferred over further ones, rather than relying solely on local/remote distinctions.

Flow Steering Decoupling

[All HCAs] Added the ability to decouple flow steering into a separate module, making it loosely coupled and thus easier to read, maintain and debug.

RDMA Features

IPv6 Address Support

[All HCAs] Reduced the storage IO latency that occurred when establishing a large number of RDMACM (RDMA Connection Manager) connections by setting the RDMACM RoCE static rate to 0.

Reducing Storage IO Latency for a Large Number of RDMACM Connections

[All HCAs] Reduced the storage IO latency that occurred while establishing a large number of RDMACM (RDMA Connection Manager) connections by setting the RDMACM RoCE static rate to 0.

QKEY Mitigation in the Kernel

[All HCAs] Non-privileged users are now blocked from setting controlled/privileged QKEYs (QKEY with MSB set).

To allow non-privileged users to create a QKEY with an MSB set, a new module parameter was added to the ib_uverbs module:

• module parameter: enforce_qkey_check

• Description: Force a QKEY MSB check for non-privileged users on UD QP creation

• Default is 0 (disabled)

In this release, this module parameter is introduced in disabled state in order to ensure backward compatibility and allow the required applications updates. In the upcoming release, this feature will be enabled by default, and later on deprecated.

Security

Fast Update Encryption Key Support

[ConnectX-6 Dx and above] Enhanced the performance of DEK operations by introducing a DEK pool that serves the same key purpose and utilizes bulk allocation, destruction and invalidation provided by the firmware. Users can now retrieve a DEK object from the pool, and update it with a key using the modify_DEK command.

Software Steering

Matching over BTH Acknowledge Bit Support

[Connect-6 DX, Connect-7 and BlueField-2] Added support for matching over BTH acknowledge bit using the mlx5dv_dr API.

General

Bug fixes

23.04-1.1.3.0

Operating Systems

Added support for the following operating systems:

• RHEL 9.2

• RHEL 8.8

• SLES15-SP5

23.04-0.5.3.3

ASAP² Features

VXLAN GBP Options Offload with OVS

[ConnectX-6 Dx and above] Added support of encap and decap of VXLAN tunnel with GBP options offload with OVS.

TC Rules: Additional Actions

[ConnectX-5 and above] Added support for adding TC rules with trap action with additional actions (mirror and pedit).

Multiport E-Switch

[ConnectX-6 Dx] Added support for Multiport E-Switch, a mode where a single E-Switch connects all VPorts and physical ports on the NIC. This allows for scenarios such as sending traffic from a VF created on PF0 to an uplink that is natively associated with the uplink of PF1.

OVS Offload with MACVLAN Interface Above Bond

[ConnectX-6 Dx and above] Added support for OVS offload when MACVLAN interface above bond is attached to OVS bridge.

Core Features

PCC fwtrace

[ConnectX-6 Dx and BlueField-2] Added support for installing a special user image into the firmware, which can be burned into either of two available slots for such applications, which enables monitoring the image's activities using the fw_tracer located inside the mlx5 driver. To view the output of the tracer, the user can access trace point, but it is important to note that they can only view traces that are generated after enabling the trace point by using the following command: 'echo 1 > sys/kernel/debug/tracing/events/mlx5/mlx5_fw/enable'." Note: Not supported in ConnectX-6 Lx adapter cards.

Relaxed Ordering in VFs/QEMU Assigned VFs

[ConnectX-6 and above] Added support for using Relaxed Ordering in VFs directly and in VFs assigned to QEMU. Relaxed Ordering can significantly improve performance on certain setups and, until now, it could be used only in PFs.

Migratable Bit

[ConnectX-6 Dx and above] Added support for Migratable Bit in Live Migration. Because some features cannot be migrated, such as IPsec, for example, when VF is marked as migratable, those features are disabled. This feature allows the user to configure whether VF can be migrated.

Live Migration Dirty Page Tracking Support in Linux

[ConnectX-7] The dirty pages tracking support enables reducing downtime upon live migration. Once it is used, only the pages that were really dirtied by the device will be marked in QEMU as dirty and will be sent to the target upon stop. Without dirty tracking, all RAM is marked dirty so all RAM is resent upon stop and the downtime is increased.

NetDev Features

Configuring Hairpin Queue Size

[ConnectX-5 and above] Added the ability to configure the number and size of hairpin queues through devlink param command.

For example:

devlink dev param set pci/0000:08:00.0 name hairpin_queue_size value 512 cmode driverinit

devlink dev param set pci/0000:08:00.0 name hairpin_num_queues value 1 cmode driverinit

RDMA Features

CC - RTT response SL (CNP SL)

[ConnectX-6 Dx and above] Users can now customize the DSCP value of RTT Response packets (in Ethernet, using debugfs). This feature allows for prioritization of RTT Response packets, preventing any delay that might lead to incorrect congestion assumptions on the RTT Requester side.

Expose VF RoCE Statistics on Host Side (Representor)

[All HCAs] Added support for allowing the host to track various statistics for the VFs, specifically all of the Q_counters stats even if he gives the VF to a VM, through the representor Q_counters for the VF which are now exposed over the host.

Fatal QP Error Logging

[All HCAs] With this feature, a kernel error log is now generated when certain fatal QP errors occur.

Enabling Selective Repeat by Default

[ConnectX-7 and above] Added support for making Selective Repeat (SR) protocol to be anabled by default. SR retransmits only the frame that is damaged and not all of the frames that were sent. As such, SR makes more efficient use of network bandwidth.

Software Steering

Matching IB BTH in RoCE Packets

[ConnectX-6 Dx and above] Added support for matching the IB BTH in RoCE packets. One example of this match is that the user can monitor RoCEv2 CNP (Congestion Notification Packet) by matching BTH opcode 0x81.

General

Bug fixes

5.9-0.5.6.0

ASAP² Features

Linux Bridge VLAN Filtering of 802.1 Q Packets

[ConnectX-6 Dx] Extended mlx5 Linux bridge VLAN offload to support packets tagged with 802.1 Q VLAN ethertype.

Offloading sFlow Sampling Rules

[ConnectX-5 and above] Added support for sFlow sampling rules offloads. sFlow is an industry standard technology for monitoring high speed switched networks. Open vSwitch integrated sFlow to extend the visibility into virtual servers, ensuring data center visibility and control.

Core Features

Configuring Shared Buffer Size

[ConnectX-6 Dx and above] Enabled user to control shared buffer size and configuration, implicitly.

As with each port buffer command the user triggers, the shared buffer configuration will be updated accordingly by the driver.

Control SF Class

[All HCAs] Added support for Control SF Class. Each PCI, PF, VF, SF function, by default, has netdevice, RDMA, and vdpa-net devices always enabled. This feature enables the user to control which device functionality to enable/disable.

Note: Requires kernel 5.18 or higher.

Installation Features

ip2gid Tool

[All HCAs] Added support for ip2gid tool.

This tool does the following:

1. Resolves a destination IP into a destination GID needed when running a rdmacm applications (ip2gid).

2. Resolves a GID into one PR (PathRecord) or multiple PRs if needed (gid2lid).

This tool is needed when rdmacm is used to initiate InfiniBand traffic between nodes on different IP subnets in InfiniBand fabrics.

NetDev Features

Support RSS over XSK Queues

[All HCAs] Use default RSS functionality to spread traffic across different XSK queues instead of having to provide explicit steering rules.

TLS TIS Pool

[TLS-Enabled Devices] Per-connection hardware TIS objects is used to maintain the device TLS TX context. Use a SW TIS pool for recycling the TIS objects instead of destroying/creating them. This reduces the interaction with the device via the FW command interface, which increases the TLS connection rate.

RDMA Features

Expand Rep Counters

[ConnectX-5 and above] Adding RDMA traffic-only counters for rep devices. These counters can now be read from host with ethtool or from sysfs and not only from the cointainer.

UMR QP Recilency

[ConnectX-5 and above] Added a recovery flow for the driver's UMR logic so that other UMR requests can be proccessed after the error UMR was dropped and the UMR QP was reset. Previously, a faulty UMR request would have moved the QP to error state and disable any option to continue issuing UMRs.

General

Bug fixes

5.8- 1.1.2.1

General

Bug fixes

5.8- 1.0.1.1

Remove Dependency Between SR-IOV and eSwitch Mode

[All HCAs] Removed dependency between SR-IOV and eSwitch mode. Currently, there are three eSwitch modes: none, legacy, and switchdev (non of which are the default mode). When disabling SR-IOV, the current eSwitch mode will be changed to none. This feature removes eSwitch mode none and also removes dependency between SR-IOV and eSwitch mode.

DevLink Parallel Command

[All HCAs] Added support for running DevLink commands in parallel on different DevLink devices is possible. For example, burning firmware on a few cards on the same host in parallel using DevLink API is now possible.

Graceful Shutdown of Parent and Page Supplier

[All HCAs] Set default graceful period values for functions based on their type. ECPFs will get graceful period of 3 minutes, PFs get 1 minute, and VFs/SFs get 30 seconds.

N Pulses Per Second (NPPS)

[ConnectX-6 Dx and above] Enhanced NPPS to allow setting a pulse period higher than 1 pulse per second and to allow setting the pulse width.

If the width is unset, the driver implicitly sets it to half the given period (the width should be less than the pulse period). In this release, the pulse duration ranges between 65536 NS–524288 NS.

Remote Invalidate Option for MKeys

[All HCAs] Added support for the option to enable remote invalidation when creating a new mkey. This way the rkey for a memory region can be changed frequently.

GPUDirect Over DMA-BUF

[All HCAs] Added support for GPUDirect support over dma-buf. As such, using the new mechanism nv_peer_mem is no longer required.

The following is required for dma-buf support:

• Linux kernel version 5.12 or later

• OpenRM version 515 or later

Perftest support was added as well:

Default option in perftest is without dmabuf. To run with this option, add --use_cuda_dmabuf in addition to use_cuda flag.

Floating LID

[ConnectX-7] Added support for Floating LID (FLID) which can be used to identify a group of InfiniBand routers that allow communication with another subnet's entity. With this feature, multiple routers can be used per destination so that adaptive routing is supported.

The FLID feature needs support from components such as the host, the subnet manager, the router, and more. This feature is only supported on the host portion of the sysrem.

General

Bug fixes

5.7-1.0.2.0

Support Representor Metering Over SFs

[ConnectX-6 Dx and above and BlueField-2] Extended the support of representor metering from supporting only VFs representor to also supporting SFs representor.

Exposing Error Counters on a VPort Manager

[ConnectX-4 and above] Added support for exposing error counters on a VPort manager function for all other VPorts.

These counters can be used to detect malicious users who are exploiting flows that can slow the device.

The counters are exposed through debugfs under: /sys/kernel/debug/mlx5/esw/<func>/vnic_diag/

Ethtool RX Flow Steering for IPoIB

[All HCAs] Enabled steering of IPoIB packets via Ethtool, in the same way it is done today for Ethernet packets.

Memory Consumption Minimization

[ConnectX-4 and above] Added support for providing knobs which enable users to minimize memory consumption of mlx5 functions (PF/VF/SF).

XDP Support for Uplink Representors

[ConnectX-5 and ConnectX-6 Dx) Added XDP support for uplink representors in switchdev mode.

Resiliency to tx_port_ts

[ConnectX-6 Dx and above] Added resiliency to the tx_port_ts feature. private-flag may be enabled via ethtool tx_port_ts which provides a more accurate time-stamp. In very rare cases, the said time-stamp was lost, leading to losing the synchronization altogether. This feature allows for fast recovery and allows to quickly regain synchronization.

Database of Devlink Health Asserts

[ConnectX-4 and above] Health buffer now contains more debug information like the epoch time in sec of the error and the error's severity. The print to dmesg is done with the debug level corresponding to the error's severity. This allows the user to use dmesg attribute: dmesg --level to focus on different severity levels of firmware errors.

Expose FEC Counters via Ethtool

[ConnectX-5 and above] Exposed the following FEC (forward error detection) counters:

ETHTOOL_A_FEC_STAT_CORRECTED

• fc_fec_corrected_blocks_laneX

• rs_fec_corrected_blocks

ETHTOOL_A_FEC_STAT_UNCORR

• fc_fec_uncorrectable_blocks_laneX

• rs_fec_uncorrectable_blocks

ETHTOOL_A_FEC_STAT_CORR_BITS

• phy_corrected_bits

Command: ethtool -I show-fec <ifc>

Application Device Queues

[ConnectX-4 and above] Added driver-level support for Application Device Queues. This feature allows partition defining over the RX/TX queues into groups and isolates traffic of different applications. This mainly improves predictability and tail latency.

Reinjection of Packets Into Kernel

[All HCAs] Added support for a new software steering action, mlx5dv_dr_action_create_dest_root_table(). This action can be used to forward packets back into a level 0 table.

As a table with level 0 is the kernel owned table, this will result in injecting packets to the kernel steering pipeline.

DCT LAG

[ConnectX-6 Dx and above] Added firmware support to allow explicit port selection based on steering and not QP affinity.

Functionality:

1. Use LAG Hash Mode for the HCA with two ports, if supported.

2. Keep port affinity function in LAG Hash Mode if it supports bypass select flow table in non-SwitchDev mode.

AES-XTS in RDMA

Added support for plaintext AES-XTS DEKs.

General

Bug fixes

23.10-1.1.9.0

Lightweight Local SFs

Following the addition of the Lightweight Local SFs feature in version 23.07, in order to configure the scalable-functions, follow the revised instructions as detailed in the Step-by-Step Guide.

Note: "Step 2.9 - Set all SF specific device parameters" is now mandatory for local SFs.

Customer Affecting Change

Description

Debugfs Directory Path Change

The debugfs directory of each interface can now be found under: /sys/kernel/debug/mlx5//, and not directly under the root of the debugfs filesystem (/sys/debug/kernel).

Deprecation of OFED Public Power PC Installation

Starting from this release, MLNX_OFED releases for Power PC are no longer available for download from the public Download Center web page. Instead, you can find it on the following page: https://network.nvidia.com/support/firmware/ibm-systemp/.

Pre-notification: Deprecation of Older Operating Systems

Starting from next release, MLNX_OFED releases will no longer support operating systems with kernels below v4.18. This includes the following systems:

• Rhel7.x

• Debian9.13

• Sles12.x

• Xenserver7.1

Customer Affecting Change

Description

23.07-0.5.1.2

Creating a QKEY with an MSB Set

To allow non-privileged users to create a QKEY with an MSB set, a new module parameter was added. For details, please see “QKEY Mitigation in the Kernel“ under New Features.

23.07-0.5.0.0

IRQ Naming

IRQ renaming is no longer done when bringing the interface up/down. The IRQ name is now constant and is not affected by the interface state.

RPM Packages Verification Key

RPM_GPG-KEY-Mellanox (or its variants) is no longer the public key that verifies RPM packages of MLNX_OFED. Instead, the RPM_GPG-KEY-Mellanox file on the top-level directory of the ISO should be used.

Hairpin sysfs Support

Hairpin sysfs support was restricted to physical and virtual functions only.

mlx5_core node_guid Module Parameter

Removed a non-functional mlx5_core node_guid module parameter.

OpenSM Init

Starting from this release, the opensm init service moves from init.d (/etc/init.d/opensmd start) to systemd (# service opensmd start).

Apt Signing Key

Starting from this release, the public key that signed the apt repository of MLNX_OFED is included in the ISO in a format that can be used directly by the apt for repository signatures verification.

IPoIB ULP Mode Deprecation

Starting from this release, MLNX_OFED supports IPoIB enhanced mode only. The ability to switch back to ULP mode using ipoib_enhanced module parameter is not supported. For more information about the enhanced mode, please refer to the OFED user manual, example: Enhanced IP over InfiniBand.

Pre-notification: Deprecation of OFED Public Power PC Installation

Starting from next release, MLNX_OFED releases for Power PC will no longer be available for download from the public Download Center web page.

23.07-0.5.0.0

Creating a QKEY with an MSB Set

To allow non-privileged users to create a QKEY with an MSB set, a new module parameter was added. For details, please see “QKEY Mitigation in the Kernel“ under New Features.

IRQ Naming

IRQ renaming is no longer done when bringing the interface up/down. The IRQ name is now constant and is not affected by the interface state.

RPM Packages Verification Key

RPM_GPG-KEY-Mellanox (or its variants) is no longer the public key that verifies RPM packages of MLNX_OFED. Instead, the RPM_GPG-KEY-Mellanox file on the top-level directory of the ISO should be used.

Hairpin sysfs Support

Hairpin sysfs support was restricted to physical and virtual functions only.

mlx5_core node_guid Module Parameter

Removed a non-functional mlx5_core node_guid module parameter.

OpenSM Init

Starting from this release, the opensm init service moves from init.d (/etc/init.d/opensmd start) to systemd (# service opensmd start).

Apt Signing Key

Starting from this release, the public key that signed the apt repository of MLNX_OFED is included in the ISO in a format that can be used directly by the apt for repository signatures verification.

IPoIB ULP Mode Deprecation

Starting from this release, MLNX_OFED supports IPoIB enhanced mode only. The ability to switch back to ULP mode using ipoib_enhanced module parameter is not supported. For more information about the enhanced mode, please refer to the OFED user manual, example: Enhanced IP over InfiniBand.

Pre-notification: Deprecation of OFED Public Power PC Installation

Starting from next release, MLNX_OFED releases for Power PC will no longer be available for download from the public Download Center web page.

23.04-0.5.3.3

Netdev Interface Configuration is not Preserved During Reload/Reset/Recovery

As of OFED 23.04, during reset/reload/recovery flows, the netdev interface is destroyed and re-created (rather than just suspended). As a result, the netdev interface configuration is not preserved, and must be re-applied. The way to do this is to use proper network-scripts and/or udev rules files to configure network interface parameters. These are automatically triggered whenever a netdev interface is added, regardless of whether it was added due to a user-initiated operation or an automatic failure recovery operation. Thus, no special processing is required to re-apply the network interface configuration parameters following a reset/reload/recovery operation – it is performed automatically.

Prenotification: Deprecation of OFED Public Power PC Installation

Starting next release, MLNX_OFED releases for Power PC will no longer be available for download from the public Download Center web page.

Prenotification: ULP Mode Deprecation

Starting next release, MLNX_OFED will support IPoIB enhanced mode only. The ability to switch back to ULP mode using ipoib_enhanced module param will not be supported.

For more information about the enhanced mode, please refer to OFED user manual, example: Enhanced IP over InfiniBand

Installation, ISO, RedHat

In order to address RHEL kernel symbol changes, ISO images for the following operating systems are built with the updated kernel versions as follows:

            

            
OS Name            Old Kernel                       New Kernel
rhel8.6-aarch64    4.18.0-372.9.1.el8_6.aarch64     4.18.0-372.41.1.el8_6.aarch64 
rhel8.6-ppc64le    4.18.0-372.0.1.el8_6.ppc64le     4.18.0-372.41.1.el8_6.ppc64le 
rhel8.6-x86_64     4.18.0-372.9.1.el8_6.x86_64      4.18.0-372.41.1.el8_6.x86_64 
rhel8.7-aarch64    4.18.0-425.3.1.el8.aarch64       4.18.0-425.14.1.el8_7.aarch64 
rhel8.7-ppc64le    4.18.0-425.3.1.el8.ppc64le       4.18.0-425.14.1.el8_7.ppc64le 
rhel8.7-x86_64     4.18.0-425.3.1.el8.x86_64        4.18.0-425.14.1.el8_7.x86_64 
rhel9.0-aarch64    5.14.0-70.13.1.el9_0.aarch64     5.14.0-70.46.1.el9_0.aarch64 
rhel9.0-ppc64le    5.14.0-70.13.1.el9_0.ppc64le     5.14.0-70.46.1.el9_0.ppc64le 
rhel9.0-x86_64     5.14.0-70.13.1.el9_0.x86_64      5.14.0-70.46.1.el9_0.x86_64 
rhel9.1-aarch64    5.14.0-162.6.1.el9_1.aarch64     5.14.0-162.19.1.el9_1.aarch64 
rhel9.1-ppc64le    5.14.0-162.6.1.el9_1.ppc64le     5.14.0-162.19.1.el9_1.ppc64le 
rhel9.1-x86_64     5.14.0-162.6.1.el9_1.x86_64      5.14.0-162.19.1.el9_1.x86_64 
        

This change comes to support RedHat updated kernels without the need to add --add-kernel-support during OFED installation.

Power Setups on UCX/HPC-X

UCX/HPC-X no longer supports Power setups.

NEO-Host

Starting from this release, OFED will discontinue the provision of NEO-Host.

NEO-Host can be manually downloaded and installed using the following guide: https://docs.nvidia.com/networking/display/NEOSDKv26/Installation+and+Initial+Configuration#InstallationandInitialConfiguration-DownloadingtheMellanoxNEOSDKSoftware

dapl

Starting from this release, OFED will discontinue the provision of dapl.

Signing Key for SLES15 sp4 and sp5

As of version 23.04, the builds for SLES15 sp4 and sp5 are being singed with a newer signing key. The corresponding public key can be downloaded from https://www.mellanox.com/downloads/ofed/nv_nbu_kernel_signing_key_pub.der instead of https://www.mellanox.com/downloads/ofed/mlnx_signing_key_pub.der.

dump_pr SM Plugin

Starting from this release, OFED will discontinue the provision of dump_pr subnet manager plugin.

mpi-selector

Starting from this release, OFED will discontinue the provision of mpi-selector.

OpenSM Init

Starting 23.07 release, opensm init service will move from init.d to systemd.

5.9-0.5.6.0

Deprecation, LAG Mode via Sysfs

Setting LAG mode via Sysfs is going to be deprecated in a future release. Instead, LAG Hash mode will be used by default, similar to upstream behavior.

LAG Configuration, PCI Error

From version 5.9, LAG configuration will be lost in case driver incurs a PCI error. Make sure to reconfigure the bond after driver completes the recovery from the PCI error.

In releases prior to 5.9, in case of PCI error (EEH injections on PPC setup), the driver recovers LAG bond and reconfigures it automatically in case it what configured before the appearance of the error.

5.7-1.0.2.0

Multi-Block Encryption

Multi-block encryption is currently unsupported, due to a hardware limitation.

5.6-2.0.9.0

Operating Systems

Added support for the following Operating Systems: RHEL8.6, RHEL9.0, SLES15-SP4.

General

Bug fixes

5.6-1.0.3.3

General

New Adapter Card Support

Added support for ConnectX-7 adapter cards. ConnectX-7 has the same feature set as the ConnectX-6 adapter card.

ASAP² Features

Bridge Spoof Check

[All HCAs] Added support for spoof check with TC flower rules on representors attached to bridge to mirror spoof check SR-IOV functionality.

Setting VF Group Rate Limit

[ConnectX-5 and above] Added support for setting VF group rate limit using Devlink command.

TC Flows on Shared Block

[ConnectX-5 and above] Added support for creation of TC flows on shared block of VF representors.

Flow Metering

[ConnectX-6 Dx and above] Added support for offloading OpenFlow Meters in OVS-DPDK.

Please note the following:

• Meter offload can be applied only on port 0 and it's VFs

• Only one meter per flow is allowed

• Only one meter band per meter is allowed

• Only meter band type drop is supported

• Meter-stats might not be accurate

Core Features

Firmware Reset

[BlueField-2] Added support of firmware reset in DPU NIC mode.

Increased Robustness of mlx5_core Driver Recovery

[All HCAs] Increased the firmware pre-initialization timeout from 2 minutes to 2 hours when waiting for firmware during driver health recovery, allowing the driver to passively recover from a firmware reset, even if the reset takes an unusually long time. Additionally, added an exit clause to the wait for firmware loop, allowing immediate response to a user initiated device removal.

NetDev Features

Ethtool CQE Mode Control

[ConnectX-4 and above] Replaced the vendor-specific Ethtool API (priv-flag) with a standard Ethtool API (replaced 'ethtool --set-priv-flags ethX rx_cqe_moder on/off tx_cqe_moder on/off' with 'ethtool -C ethX cqe-mode-rx on/off cqe-mode-tx on/off'). This decreases the amount of vendor-specific configurations and aligns mlx5 driver with the upstream Ethtool API.

SyncE

[ConnectX-6 Dx] Added an indication in SyncE Daemon that states whether SyncE engine moved to holdover state due to failure (the reason for failure will be displayed). In addition, added indication whether SyncE engines collected enough frequency samples in order to move to holdover.

Note: Not supported in ConnectX-6 Lx adapter cards.

RDMA Features

VFIO, CQ Interrupt Mode

[ConnectX-5 and above] Added support for VFIO applications to listen on and capture completion events via the Event Queue mechanism.

VFIO, Asynchronous Event

[ConnectX-5 and above] Added support for VFIO applications to listen on and capture device asynchronous events via the Event Queue mechanism.

Security

OVS-IPSec Full Offload

[BlueField-2] Added support for configuration of IPsec full offload using OVS by adding VXLAN tunnel to OVS with the PSK option.

Software Steering Features

Full Tunnel Header Matching

[ConnectX-6 Dx and above] Added support for using full-tunnel-header matching along with many other criteria within one matcher. This feature uses the new definer index, defined in the firmware, to build a matcher so that the full tunnel header matching can be used along with all other criteria.

Matching Granularity Change

[ConnectX-5 and above] Added support for matching granularity change. As a result, when creating FDB flow with destination of VPORT, a src_port matching must no longer be added. Now, FDB flow can match all vports and goto a VPORT destination. The new behavior is the same as done on firmware steering.

Installation

Installation

New options were added to the ofed_uninstall.sh script: --only-kernel and --only-user. Those can be used to uninstall only kernel packages or only user-space packages (the equivalent of kernel-only install or user-space-only install, respectively). This may be useful to keep different sets of kernel and user-space installations.

When running the uninstall script with a combination of --only-kerneland --only-userproduced an undefined result.

Customer Affecting Changes 5.6-1.0.3.3

Interface Renaming, PF/VF, Udev

The OFED driver no longer performs Ethernet NetDev interface renaming for PFs and VFs.

The udev rules file which implemented renaming (82-net-setup-link.rules) and its supporting script vf-net-link-name.sh are no longer installed by default.

Renaming is thus performed by underlying mechanisms -- in udev, in the kernel, and in the BIOS.

Users who wish to continue using the OFED driver renaming mechanism must add option --copy-ifnames-udev to the OFED install command.

To install these files at a later time, copy them from one of the following directories:

• /usr/share/doc/mlnx-ofa_kernel (RHEL8 and newer)

• /usr/share/doc/mlnx-ofa_kernel-[1-9]* (RHEL 7.X)

• /usr/share/doc/packages/mlnx-ofa_kernel (SLES)

• /usr/share/doc/mlnx-ofed-kernel-utils/examples (Debian-based releases)

Community Operating Systems

Starting OFED 5.6, NVIDIA is introducing a new support model for OFED used on open source community operating systems. The goal of this new support model is to enable customers to use community-maintained variants of the Linux operating system, without being limited to major distributions that NVIDIA provides primary support for. For more information, see "Installation on Community Operating Systems" section in the user manual. For a list of supported Community OSs, please see "Supported Community Operating Systems" section in the release notes.

ar_mgr Subnet Manager Plugin

ar_mgr subnet manager plugin is no longer supported.

For adaptive routing and SHIELD subnet manager configuration, please see the MLNX_OFED user manual.

Fabric Collector in UFM

Starting UFM v6.7, Fabric Collector is no longer supported. For more information, see the UFM release notes.

OVS-DPDK—Partial Offload

Starting OFED 5.6, OVS-DPDK does not support partial offload.