Release Notes Change Log History

Devlink Health CR-Space Dump

Added the option to dump configuration space via the devlink tool in order to improve debug capabilities.

Multi-packet TX WQE Support for XDP Transmit Flows

The conventional TX descriptor (WQE or Work Queue Element) describes a single packet for transmission. Added driver support for the HW feature of multi-packet TX WQEs in XDP transmit flows. With this, the HW becomes capable of working with a new and improved WQE layout that describes several packets. In effect, this feature saves PCI bandwidth and transactions, and improves transmit packet rate.

GENEVE Encap/Decap Rules Offload

Added support for GENEVE encapsulation/decapsulation rules offload.

Multi Packet Tx WQE Support for XDP Transmit Flows

Added driver support for the hardware feature of multi-packet Tx to work with a new and improved WQE layout that describes several packets instead of a single packet for XDP transmission flows. This saves PCI bandwidth and transactions, and improves transmit packet rate.

Kernel Software Steering for Connection Tracking (CT)

[Beta] Added support for updating CT rules using the software steering mechanism.

Kernel Software Steering Remote Mirroring

[Beta] Added support for updating remote mirroring rules using the software steering mechanism.

Discard Counters

Exposed rx_prio[p]_discards discard counters per priority that count the number of received packets dropped due to lack of buffers on the physical port.

MPLS Traffic

Added support for reporting TSO and CSUM offload capabilities for MPLS tagged traffic and, allowed the kernel stack to use these offloads.

mlx5e Max Combined Channels

Increased the driver’s maximal combined channels value from 64 to 128 (however, note that OOB value will not cross 64).
128 is the upper bound. Lower maximal value can be seen on the host, depending on the number of cores and MSIX's configured by the firmware.

RoCE Accelerator Counters

Added the following RoCE accelerator counters:

• roce_adp_retrans - counts the number of adaptive retransmissions for RoCE traffic

• roce_adp_retrans_to - counts the number of times RoCE traffic reached timeout due to adaptive retransmission

• roce_slow_restart - counts the number of times RoCE slow restart was used

• roce_slow_restart_cnps - counts the number of times RoCE slow restart generated CNP packets

• roce_slow_restart_trans - counts the number of times RoCE slow restart changed state to slow restart

Memory Region

Added support for the user to register memory regions with a relaxed ordering access flag through experimental verbs. This can enhance performance, depending on architecture and scenario.

ibdev2netdev Tool Output

ibdev2netdev tool output was changed such that the bonding device now points at the bond instead of the slave interface.

Devlink Health Reporters

Added support for monitoring and recovering from errors that occur on the RX queue, such as cookie errors and timeout.

GSO Optimization

Improved GSO (Generic Segmentation Offload) workload performance by decreasing doorbells usage to the minimum required.

TX CQE Compression

Added support for TX CQE (Completion Queue Element) compression. Saves on outgoing PCIe bandwidth by compressing CQEs together. Disabled by default. Configurable via private flags of ethtool.

Firmware Versions Query via Devlink

Added the option to query for running and stored firmware versions using the devlink tool.

Firmware Flash Update via Devlink

Added the option to update the firmware image in the flash using the devlink tool.

Usage: devlink dev flash <dev> file <file_name>.mfa2

For further information on how to perform this update, see "Updating Firmware Using ethtool/devlink and .mfa2 File" section in MFT User Manual.

Devlink Health WQE Dump

Added support for WQE (Work Queue Element) dump, triggered by an error on Rx/Tx reporters. In addition, some dumps (not triggered by an error) can be retrieved by the user via devlink health reporters.

GENEVE Tunnel Stateless Offload

Added support for GENEVE tunneled hardware offloads of TSO, CSUM and RSS.

TCP Segmentation and Checksum Offload

Added TCP segmentation and checksum offload support for MPLS-tagged traffic.

NEO-Host SDK

Added support for NEO-Host SDK installation on MLNX_OFED.

Bug Fixes

See Bug Fixes section.

Counters Monitoring

Added support for monitoring selected counters and generating a notification event (Monitor_Counter_Change event) upon changes made to these counters.
The counters to be monitored are selected using the SET_MONITOR_COUNTER command.

EEPROM Device Thresholds via Ethtool

Added support to read additional EEPROM information from high pages of modules such as SFF-8436 and SFF-8636. Such information can be: 1. Application Select table 2. User writable EEPROM 3. Thresholds and alarms - Ethtool dump works on active cables only (e.g. optic), but thresholds and alarms can be read with “offset” and “length” parameters in any cable by running: ethtool -m <DEVNAME> offset X length Y

RDMA_RX RoCE Steering Support

Added the ability to create rules to steer RDMA traffic, with two destinations supported: DevX object and QP. Multiple priorities are also supported.

ASAP²

Incorporated the documentation of Accelerated Switching And Packet Processing (ASAP²): Hardware Offloading for vSwitches into MLNX_OFED Release Notes and User Manual.

MLNX_OFED Installation via Repository

The repository providing legacy verbs has been moved from RPMS or DEBS folders to RPMS/MLNX_LIBS and DEBS/MLNX_LIBS.

In addition, a new repository providing RDMA-Core based userspace has been added to RPMS/UPSTREAM_LIBS and DEBS/UPSTREAM_LIBS.

Devlink Configuration Parameters Tool

Added support for a set of configuration parameters that can be changed by the user through the Devlink user interface.

ODP Pre-fetch

Added support for pre-fetching a range of an on-demand paging (ODP) memory region (MR), this way reducing latency by making pages present with RO/RW permissions before the actual IO is conducted.

DevX Privilege Enforcement

Enforced DevX privilege by firmware. This enables future device functionality without the need to make driver changes unless a new privilege type is introduced.

DevX Interoperability APIs

Added support for modifying and/or querying for a verb object (including CQ, QP, SRQ, WQ, and IND_TBL APIs) via the DevX interface.

This enables interoperability between verbs and DevX.

DevX Asynchronous Query Commands

Added support for running QUERY commands over the DevX interface in an asynchronous mode. This enables applications to issue many commands in parallel while firmware processes the commands.

DevX User-space PRM Handles Exposure

Exposed all PRM handles to user-space so DevX user application can mix verbs objects with DevX objects.

For example: Take the cqn from the created ibv_cq and use it on a devx)create(QP).

Indirect Mkey ODP

Added the ability to create indirect Mkeys with ODP support over DevX interface.

XDP Redirect

Added support for XDP_REDIRECT feature for both ingress and egress sides. Using this feature, incoming packets on one interface can be redirected very quickly into the transmission queue of another capable interface. Typically used for load balancing.

RoCE Disablement

Added the option to disable RoCE traffic handling. This enables forwarding of traffic over UDP port 4791 that is handled as RoCE traffic when RoCE is enabled.

When RoCE is disabled, there is no GID table, only Raw Ethernet QP type is supported and RoCE traffic is handled as regular Ethernet traffic.

Forward Error Correction (FEC) Encoding

Added the ability to query and modify Forward Error Correction (FEC) encoding, as well as disabling it via Ethtool.

RAW Per-Lane Counters Exposure

Exposed RAW error counters per cable-module lane via ethtool stats. The counters show the number of errors before FEC correction (if enabled).

For further information, please see phy_raw_errors_lane[i] under Physical Port Counters section in Understanding mlx5 ethtool Counters Community post.

VF LAG

Added support for High Availability and load balancing for Virtual Functions of different physical ports in SwitchDev SR-IOV mode.

ASAP² Offloading VXLAN Decapsulation with HW LRO

Added support for performing hardware Large Receive Offload (HW LRO) on VFs with HW-decapsulated VXLAN.
For further information on the VXLAN decapsulation feature, please refer to ASAP² User Manual under www.mellanox.com -> Products -> Software -> ASAP².

PCI Atomic Operations

Added the ability to run atomic operations on local memory without involving verbs API or compromising the operation's atomicity.

Virtual Ethernet Port Aggregator (VEPA)

Added support for activating/deactivating Virtual Ethernet Port Aggregator (VEPA) mode on a single virtual function (VF). To turn on VEPA on the second VF, run:
echo ON > /sys/class/net/enp59s0/device/sriov/1/vepa

VFs Rate Limit

Added support for setting a rate limit on groups of Virtual Functions rather on an individual Virtual Function.

ConnectX-6 Support

[Beta] Added support for ConnectX-6 (VPI only) adapter cards.

NOTE: In HDR installations that are built with remotely managed Quantum-based switches, the switch’s firmware must be upgraded to version 27.2000.1142 prior to upgrading the HCA’s (ConnectX-6) firmware to version 20.25.1500. When using ConnectX-6 HCAs with firmware v20.25.1500 and connecting them to Quantum-based switches, make sure the Quantum firmware version is 27.2000.1142 in order to avoid any critical link issues.

Ethtool 200Gbps

ConnectX-6 hardware introduces support for 200Gbps and 50Gbps-per-lane link mode. MLNX_OFED supports full backward compatibility with previous configurations.

Note that in order to advertise newly added link-modes, the full bitmap related to the link modes must be advertised from ethtool man page. For the full bitmap list per link mode, please refer to MLNX_OFED User Manual.

NOTE: This feature is firmware-dependent. Currently, ConnectX-6 Ethernet firmware supports up to 100Gbps only. Thus, this capability may not function properly using the current driver and firmware versions.

PCIe Power State

Added support for the following PCIe power state indications to be printed to dmesg:

1. Info message #1: PCIe slot power capability was not advertised.

2. Warning message: Detected insufficient power on the PCIe slot (xxxW).

3. Info message #2: PCIe slot advertised sufficient power (xxxW).
   When indication #1 or #2 appear in dmesg, user should make sure to use a PCIe slot that is capable of supplying the required power.

Message Signaled
Interrupts-X (MSI-X)
Vectors

Added support for using a single MSI-X vector for all control event queues instead of one MSI-X vector per queue in a virtual function driver. This frees extra MSI-X vectors to be used for completion event queue, allowing for additional traffic channels in the network device.

Send APIs

Introduced a new set of QP Send operations (APIs) which allows extensibility for new Send opcodes.

BlueField Support

BlueField is now fully supported as part of the Mellanox OFED mainstream version sharing the same code baseline with all the adapters product line.

Representor Name Change

In SwitchDev mode:

• Uplink representors are now called p0/p1

• Host PF representors are now called pf0hpf/pf1hpf

• VF representors are now called pf0vfN/pf1vfN

ECPF Net Devices

In SwitchDev mode, net devices enp3s0f0 and enp3s0f1 are no longer created.

Setting Host MAC and Tx Rate Limit from ECPF

Expanded to support VFs as well as the host PFs.

RDMA-CM Application Managed QP

Added support for the RDMA application to manage its own QPs and use RDMA-CM only for exchanging Address information.

RDMA-CM QP Timeout Control

Added a new option to rdma_set_option that allows applications to override the RDMA-CM's QP ACK timeout value.

MLNX_OFED Verbs API

As of MLNX_OFED v5.0 release (Q1 2020) onwards, MLNX_OFED Verbs API will be migrated from the legacy version of the user space verbs libraries (libibervs, libmlx5 ..) to the upstream version rdma-core.
More details are available in MLNX_OFED user manual under Installing Upstream rdma-core Libraries.

VFs per PF

Increased the amount of maximum virtual functions (VF) that can be allocated to a physical function (PF) to 127 VF.

SW-Defined UDP Source Port for RoCE v2

UDP source port for RoCE v2 packets is now calculated by the driver rather than the firmware, achieving better distribution and less congestion. This mechanism works for RDMA- CM QPs only, and ensures that RDMA connection messages and data messages have the same UDP source port value.

Local Loopback Disable

Added the ability to manually disable Local Loopback regardless of the number of open user-space transport domains.

Adapter Cards

Added support for ConnectX-6 Ready. For further information, please contact Mellanox Support.

Bug Fixes

See “Bug Fixes" section.

Operating Systems

Added support for additional OSs. See "General Support in MLNX_EN" section.

Adaptive Interrupt Moderation

Added support for adaptive Tx, which optimizes the moderation values of the Tx CQs on runtime for maximum throughput with minimum CPU overhead.

This mode is enabled by default.

Updated Adaptive Rx to ignore ACK packets so that queues that only handle ACK packets remain with the default moderation.

Docker Containers [Beta]

Added support for Docker containers to run over Virtual RoCE and InfiniBand devices using SR-IOV mode.

Firmware Tracer

Added a new mechanism for the device’s FW/HW to log important events into the event tracing system (/sys/kernel/debug/tracing) without requiring any Mellanox-specific tool.

Note: This feature is enabled by default.

CR-Dump

Accelerated the original cr-dump by optimizing the reading process of the device’s CR-Space snapshot.

VST Q-in-Q

Added support for C-tag (0x8100) VLAN insertion to tagged packets in VST mode.

OVS Offload using ASAP2

Added support for Mellanox Accelerated Switching And Packet Processing (ASAP2) technology, which allows OVS offloading by handling OVS data-plane, while maintaining OVS control-plane unmodified. OVS Offload using ASAP2 technology provides significantly higher OVS performance without the associated CPU load.

For further information, refer to ASAP2 Release Notes under www.mellanox.com -> Products -> Software -> ASAP2

Adaptive Interrupt Moderation

Added support for adaptive Tx, which optimizes the moderation values of the Tx CQs on runtime for maximum throughput with minimum CPU overhead.

This mode is enabled by default.

Updated Adaptive Rx to ignore ACK packets so that queues that only handle ACK packets remain with the default moderation.

Docker Containers [Beta]

Added support for Docker containers to run over Virtual RoCE and InfiniBand devices using SR-IOV mode.

Firmware Tracer

Added a new mechanism for the device’s FW/HW to log important events into the event tracing system (/sys/kernel/debug/tracing) without requiring any Mellanox-specific tool.

Note: This feature is enabled by default.

CR-Dump

Accelerated the original cr-dump by optimizing the reading process of the device’s CR-Space snapshot.

VST Q-in-Q

Added support for C-tag (0x8100) VLAN insertion to tagged packets in VST mode.

OVS Offload using ASAP2

Added support for Mellanox Accelerated Switching And Packet Processing (ASAP2) technology, which allows OVS offloading by handling OVS data-plane, while maintaining OVS control-plane unmodified. OVS Offload using ASAP2 technology provides significantly higher OVS performance without the associated CPU load.

For further information, refer to ASAP2 Release Notes under www.mellanox.com -> Products -> Software -> ASAP2

Erasure Coding
Offload verbs

Added support for erasure coding offload software verbs (encode/decode/update API) supporting a number of redundancy blocks (m) greater than 4.

Virtual MAC

Removed support for Virtual MAC feature.

RoCE LAG

Added out of box RoCE LAG support for RHEL 7.2 and RHEL 6.9.

Dropped Counters

Added a new counter rx_steer_missed_packets which provides the number of packets that were received by the NIC, yet were discarded/dropped since they did not match any flow in the NIC steering flow table.

Added the ability for SR-IOV counter rx_dropped to count the number of packets that were dropped while vport was down.

Reset Flow

Added support for triggering software reset for firmware/driver recovery. When fatal errors occur, firmware can be reset and driver reloaded.

Striding RQ with HW Time-Stamping

Added the option to retrieve the HW timestamp when polling for completions from a completion queue that is attached to a multi-packet RQ (Striding RQ).

Physical Address Memory Allocation

Added support to register a specific physical address range.

Innova IPsec Adapter Cards

Added support for Mellanox Innova IPsec EN adapter card, that provides security acceleration for IPsec-enabled networks.

HCAs: ConnectX-4/ConnectX-4 Lx/ConnectX-5

Precision Time Protocol (PTP)

Added support for PTP feature over PKEY interfaces.

This feature allows for accurate synchronization between the distributed entities over the network. The synchronization is based on symmetric Round Trip Time (RTT) between the master and slave devices, and is enabled by default.

Virtual MAC

Added support for Virtual MAC feature, which allows users to add up to 4 virtual MACs (VMACs) per VF. All traffic that is destined to the VMAC will be forwarded to the relevant VF instead of PF. All traffic going out from the VF with source MAC equal to VMAC will go to the wire also when Spoof Check is enabled.

For further information, please refer to “Virtual MAC” section in MLNX_EN User Manual.

Receive Buffer

Added the option to change receive buffer size and cable length. Changing cable length will adjust the receive buffer's xon and xoff thresholds.

For further information, please refer to “Receive Buffer” section in MLNX_EN User Manual.

GRE Tunnel Offloads

Added support for the following GRE tunnel offloads:

• TSO over GRE tunnels

• Checksum offloads over GRE tunnels

• RSS spread for GRE packets

NVMEoF

Added support for the host side (RDMA initiator) in RedHat 7.2 and above.

Dropless Receive Queue (RQ)

Added support for the driver to notify the FW when SW receive queues are overloaded.

PFC Storm Prevention

Added support for configuring PFC stall prevention in cases where the device unexpectedly becomes unresponsive for a long period of time. PFC stall prevention disables flow control mechanisms when the device is stalled for a period longer than the default pre-configured timeout. Users now have the ability to change the default timeout by moving to auto mode.

For further information, please refer to “PFC Stall Prevention” section in MLNX_EN User Manual.

Q-in-Q

Added support for Q-in-Q VST feature in ConnectX-5 adapter cards family.

Virtual Guest Tagging (VGT+)

Added support for VGT+ in ConnectX-4/ConnectX-5 HCAs. This feature is s an advanced mode of Virtual Guest Tagging (VGT), in which a VF is allowed to tag its own packets as in VGT, but is still subject to an administrative VLAN trunk policy. The policy determines which VLAN IDs are allowed to be transmitted or received. The policy does not determine the user priority, which is left unchanged.

For further information, please refer to “Virtual Guest Tagging (VGT+)” section in MLNX_EN User Manual.

Tag Matching Offload

Added support for hardware Tag Matching offload with Dynamically Connected Transport (DCT).

CR-DUMP

Added support for the driver to take an automatic snapshot of the device’s CR-Space in cases of critical failures.

For further information, please refer to “CRDUMP” section in MLNX_EN User Manual.

RoCE Diagnostics and ECN Counters

Added support for additional RoCE diagnostics and ECN congestion counters under /sys/class/infiniband/mlx5_0/ports/1/hw_counters/ directory.

For further information, refer to the Understanding mlx5 Linux Counters and Status Parameters Community post.

rx-fcs Offload (ethtool)

Added support for rx-fcs ethtool offload configuration. Normally, the FCS of the packet will be truncated by the ASIC hardware before sending it to the application socket buffer (skb). Ethtool allows to set the rx-fcs not to be truncated, but to pass it to the application for analysis.

For more information and usage, refer to Understanding ethtool rx-fcs for mlx5 Drivers Community post.

DSCP Trust Mode

Added the option to enable PFC based on the DSCP value. Using this solution, VLAN headers will no longer be mandatory for use.

For further information, refer to the HowTo Configure Trust Mode on Mellanox Adapters Community post.

RoCE ECN Parameters

ECN parameters have been moved to the following directory: /sys/kernel/debug/mlx5/<PCI BUS>/cc_params/

For more information, refer to the HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux) Community post.

Flow Steering Dump Tool

Added support for mlx_fs_dump, which is a python tool that prints the steering rules in a readable manner.

Secure Firmware Updates

Firmware binaries embedded in MLNX_EN package now support Secure Firmware Updates. This feature provides devices with the ability to verify digital signatures of new firmware binaries, in order to ensure that only officially approved versions are installed on the devices.

For further information on this feature, refer to Mellanox Firmware Tools (MFT) User Manual.

PeerDirect

Added the ability to open a device and create a context while giving PCI peer attributes such as name and ID.

For further details, refer to the PeerDirect Programming Community post.

Probed VFs

Added the ability to disable probed VFs on the hypervisor. For further information, see HowTo Configure and Probe VFs on mlx5 Drivers Community post.

Local Loopback

Improved performance by rendering Local loopback (unicast and multicast) disabled by mlx5 driver by default while local loopback is not in use. The mlx5 driver keeps track of the number of transport domains that are opened by user-space applications. If there is more than one user-space transport domain open, local loopback will automatically be enabled.

1PPS Time Synchronization (at alpha level)

Added support for One Pulse Per Second (1PPS), which is a time synchronization feature that allows the adapter to send or receive 1 pulse per second on a dedicated pin on the adapter card.

For further information on this feature, refer to the HowTo Test 1PPS on Mellanox Adapters Community post.

Fast Driver Unload

Added support for fast driver teardown in shutdown and kexec flows.

NVMEoF Target Offload

Added support for NVMe over fabrics (NVMEoF) offload, an implementation of the new NVMEoF standard target (server) side in hardware.

For further information on NVMEoF Target Offload, refer to HowTo Configure NVMEoF Target Offload .

RDMA CM

Changed the default RoCE mode on which RDMA CM runs to RoCEv2 instead of RoCEv1.

RDMA_CM session requires both the client and server sides to support the same RoCE mode. Otherwise, the client will fail to connect to the server.

For further information, refer to RDMA CM and RoCE Version Defaults Community post.

Lustre

Added support for Lustre file system open-source project.

PCIe Error Counting

[ConnectX-4/ConnectX-4 Lx] Added the ability to expose physical layer statistical counters to ethtool.

Standard ethtool

[ConnectX-4/ConnectX-4 Lx] Added support for flow steering and rx-all mode.

SR-IOV Bandwidth Share for Ethernet/RoCE (beta)

[ConnectX-4/ConnectX-4 Lx] Added the ability to guarantee the minimum rate of a certain VF in SR-IOV mode.

Adapter Cards

Added support for ConnectX-5 and ConnectX-5 Ex HCAs.

NFS over RDMA (NFSoRDMA)

Removed support for NFSoRDMA drivers. These drivers are no longer provided along with the MLNX_EN package.