Release Notes Change Log History

Linux Kernel Upstream Release Notes v5.17

This section includes history of changes and new feature of three major (GA) releases back. For older versions' history, please refer to their dedicated release notes.

Supported Cards

Description

All HCAs

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-4 / ConnectX -4 Lx / ConnectX-5 / ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-6 Dx and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-6 and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-5 and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-5 / ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

ConnectX-4 and above

Supported in the following adapter cards unless specifically stated otherwise:

ConnectX-4 / ConnectX -4 Lx / ConnectX-5 / ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx / ConnectX-7 / BlueField-2

Category

Description

4.9-3.1.5.0

mdev, Modules

mdev module is not supported starting MLNX_OFED 4.9-3.1.5.0 .

Category

Description

4.9-0.1.7.0

Devlink Health CR-Space Dump

[ConnectX-5 and above] Added the option to dump configuration space via the devlink tool in order to improve debug capabilities.

Multi-packet TX WQE Support for XDP Transmit Flows

[ConnectX-5 and above] The conventional TX descriptor (WQE or Work Queue Element) describes a single packet for transmission. Added driver support for the HW feature of multi-packet TX WQEs in XDP transmit flows. With this, the HW becomes capable of working with a new and improved WQE layout that describes several packets. In effect, this feature saves PCI bandwidth and transactions, and improves transmit packet rate.

GENEVE Encap/Decap Rules Offload

[ConnectX-5 and above] Added support for GENEVE encapsulation/decapsulation rules offload.

Multi Packet Tx WQE Support for XDP Transmit Flows

[ConnectX-5 and above] Added driver support for the hardware feature of multi-packet Tx to work with a new and improved WQE layout that describes several packets instead of a single packet for XDP transmission flows. This saves PCI bandwidth and transactions, and improves transmit packet rate.

Kernel Software Steering for Connection Tracking (CT)

[ConnectX-5 and above] [Beta] Added support for updating CT rules using the software steering mechanism.

Kernel Software Steering Remote Mirroring

[ConnectX-5 and above] [Beta] Added support for updating remote mirroring rules using the software steering mechanism.

Discard Counters

[ConnectX-4 and above] Exposed rx_prio[p]_discards discard counters per priority that count the number of received packets dropped due to lack of buffers on the physical port.

MPLS Traffic

[ConnectX-4 and above] Added support for reporting TSO and CSUM offload capabilities for MPLS tagged traffic and, allowed the kernel stack to use these offloads.

mlx5e Max Combined Channels

[ConnectX-4 and above] Increased the driver’s maximal combined channels value from 64 to 128 (however, note that OOB value will not cross 64).

128 is the upper bound. Lower maximal value can be seen on the host, depending on the number of cores and MSIX's configured by the firmware.

RoCE Accelerator Counters

[ConnectX-4 and above] Added the following RoCE accelerator counters:

  • roce_adp_retrans - counts the number of adaptive retransmissions for RoCE traffic

  • roce_adp_retrans_to - counts the number of times RoCE traffic reached timeout due to adaptive retransmission

  • roce_slow_restart - counts the number of times RoCE slow restart was used

  • roce_slow_restart_cnps - counts the number of times RoCE slow restart generated CNP packets

  • roce_slow_restart_trans - counts the number of times RoCE slow restart changed state to slow restart

Memory Region

[ConnectX-4 and above] Added support for the user to register memory regions with a relaxed ordering access flag through experimental verbs. This can enhance performance, depending on architecture and scenario.

ibdev2netdev Tool Output

[All HCAs] ibdev2netdev tool output was changed such that the bonding device now points at the bond instead of the slave interface.

Devlink Health Reporters

[All HCAs] Added support for monitoring and recovering from errors that occur on the RX queue, such as cookie errors and timeout.

GSO Optimization

[All HCAs] Improved GSO (Generic Segmentation Offload) workload performance by decreasing doorbells usage to the minimum required.

TX CQE Compression

[All HCAs] Added support for TX CQE (Completion Queue Element) compression. Saves on outgoing PCIe bandwidth by compressing CQEs together. Disabled by default. Configurable via private flags of ethtool.

Firmware Versions Query via Devlink

[All HCAs] Added the option to query for running and stored firmware versions using the devlink tool.

Firmware Flash Update via Devlink

[All HCAs] Added the option to update the firmware image in the flash using the devlink tool.

Usage: devlink dev flash <dev> file <file_name>.mfa2

For further information on how to perform this update, see "Updating Firmware Using ethtool/devlink and .mfa2 File" section in MFT User Manual.

Devlink Health WQE Dump

[All HCAs] Added support for WQE (Work Queue Element) dump, triggered by an error on Rx/Tx reporters. In addition, some dumps (not triggered by an error) can be retrieved by the user via devlink health reporters.

GENEVE Tunnel Stateless Offload

[All HCAs] Added support for GENEVE tunneled hardware offloads of TSO, CSUM and RSS.

TCP Segmentation and Checksum Offload

[All HCAs] Added TCP segmentation and checksum offload support for MPLS-tagged traffic.

NEO-Host SDK

[All HCAs] Added support for NEO-Host SDK installation on MLNX_OFED.

Bug Fixes

See “Bug Fixes" section.

Category

Description

4.7-3.2.9.0

Uplink Representor Modes

[ConnectX-5 and above] Added support for new_netdev and nic_netdev uplink representor modes.

For further information on how to configure these modes, please refer to Configuring Uplink Representor Mode.

mlx5_core

[ConnectX-5 and above] Added new mlx5_core module parameter "num_of_groups", which controls the number of large groups in the FDB flow table.

Note: The default value of num_of_groups may change per MLNX_OFED driver version. The following table lists the values that must be set when upgrading the MLNX_OFED version prior to driver load, in order to achieve the same OOB experience.

MLNX_OFED Version

num_of_groups Default Value

v4.7-3.2.9.0

4

v4.6-3.1.9.0.14

15

v4.6-3.1.9.0.15

15

v4.5-1.0.1.0.19

63

For further information, please refer to Performance Tuning Based on Traffic Patterns section in MLNX_OFED User Manual.

VFs Groups Minimum Bandwidth Rate

[ConnectX-5] Added support for setting a minimum bandwidth rate on a group of VFs (BW guarantee) to ensure this group is able to transmit at least the amount of bandwidth specified on the wire.

Direct Verbs Support for Batch Counters on Root Table

[ConnectX-5] Added support for mlx5dv_dr API to set batch counters for root tables.

Modify Header

[ConnectX-5 and BlueField] Added support for mlx5dv_dr_actions to support up to 32 modify actions.

mlx5dv_dr Memory Consumption

[ConnectX-5 and BlueField] Reduced the mlx5dv_dr API memory consumption by improving the memory allocator.

mlx5dv_dr Memory Allocation

[ConnectX-5 and BlueField] Reduced memory allocation time when using the mlx5dv_dr API. This is particularly significant for the first inserted rules on which memory is allocated.

Mediated Devices

[ConnectX-5 and BlueField] Added support for mediated devices that allows the creation of accelerated devices without SR-IOV on the Bluefield® system.

For further information on mediated devices and how to configure them, please refer to Mediated Devices section in MLNX_EN User Manual.

4.7-1.0.0.1

Counters Monitoring

[ConnectX-4 and above] Added support for monitoring selected counters and generating a notification event (Monitor_Counter_Change event) upon changes made to these counters.

The counters to be monitored are selected using the SET_MONITOR_COUNTER command.

Signature Offload Kernel Verbs Enhancements

[ConnectX-4 and above] Added a new API which enables posting a single WR that completes the Protection Information (PI) operation internally. This reduces CPU utilization for posting and processing multiple WRs and improves performance by choosing the optimal mkey for the hardware according to the buffer memory layout.

EEPROM Device Thresholds via Ethtool

[ConnectX-4 and above] Added support to read additional EEPROM information from high pages of modules such as SFF-8436 and SFF-8636. Such information can be: 1. Application Select table 2. User writable EEPROM 3. Thresholds and alarms - Ethtool dump works on active cables only (e.g. optic), but thresholds and alarms can be read with “offset” and “length” parameters in any cable by running: ethtool -m <DEVNAME> offset X length Y

Performance Improvements

[ConnectX-4 and above]

  • Updated Blueflame capability reporting to prevent redundant use of Blueflame when Write-combining is not supported.

  • Added Blueflame capabilities over VFs.

RDMA_RX RoCE Steering Support

[ConnectX-4 and above] Added the ability to create rules to steer RDMA traffic, with two destinations supported: DevX object and QP. Multiple priorities are also supported.

SRQ and XRC Support on On Demand Paging (ODP) Memory Region (MR)

[ConnectX-4 and above] Added support for using ODP MR with SRQ WQEs and XRC transport.

Indirect Mkey ODP

[ConnectX-4 and above] Added the ability to create indirect Mkeys with ODP support over DevX interface.

DevX Asynchronous Query Commands

[ConnectX-4 and above] Added support for running QUERY commands over the DevX interface in an asynchronous mode. This enables applications to issue many commands in parallel while firmware processes the commands.

Implicit ODP

[ConnectX-4 and above] Added support for reporting implicit ODP support to user applications in order to allow better granularity over ODP creation.

Devlink Health Utility

[ConnectX-4 and above] Added support for real-time alerting of functionality issues that may be found in a system component (reporter). This utility helps detect and recover from a problem with a PCI device. It provides a centralize status of drivers' health activities in the generic Devlink instance and inter alia, supports the following:

  • Storing real-time error dumps

  • Performing automatic (configurable) real-time reporter recovery

  • Performing real-time reporter diagnosis

  • Indicating real-time reporter's health status

  • Providing admins with the ability to dump, diagnose and recover a reporter

  • Providing admins with the ability to configure a reporter

User-Mode Memory Registration (UMR)

[ConnectX-4 and above] Enabled registration of memory patterns that can be used for future RDMA operations.

GENEVE Tunnel Stateless Offload

[ConnectX-4 and above] Added support for Generic Network Virtualization Encapsulation (GENEVE) tunneled hardware offload of TSO, CSUM and RSS.

ODP Pre-fetch

[ConnectX-4 and above] Added support for pre-fetching a range of an on-demand paging (ODP) memory region (MR), this way reducing latency by making pages present with RO/RW permissions before the actual IO is conducted.

Fragmented

QPs Buffer

[ConnectX-4 and above] Added the ability to allocate a fragmented buffer to in-kernel QP creation requests, in cases of large QP size requests that used to fail due to low memory resources on the host.

Flow Counters Batch Query

[ConnectX-4 and above] Allowed flow counters created with the DevX interface to be attached to flows created with the raw flow creation API.

DevX Privilege Enforcement

[ConnectX-4 and above] Enforced DevX privilege by firmware. This enables future device functionality without the need to make driver changes unless a new privilege type is introduced.

DevX Interoperability APIs

[ConnectX-4 and above] Added support for modifying and/or querying for a verb object (including CQ, QP, SRQ, WQ, and IND_TBL APIs) via the DevX interface.

This enables interoperability between verbs and DevX.

Counters Monitoring

[ConnectX-4 and above] Added support for monitoring selected counters and generating a notification event (Monitor_Counter_Change event) upon changes made to these counters.

The counters to be monitored are selected using the SET_MONITOR_COUNTER command.

Rx Hash Fields Configuration

[ConnectX-4 and above] Added the ability to configure Rx hash fields used for traffic spreading into Rx queues using ETHTOOL_SRXFH and ETHTOOL_GRXFH ethtool commands. Built-in Receive Side Scaling (RSS) profiles can now be changed on the following traffic types: UDP4, UDP6, TCP4 and TCP6. This configuration affects both outer and inner headers.

Equal Cost Multi-Path (ECMP)

[ConnectX-4 Lx and above] Added support for offloading ECMP rules by tracking software multipath route and related next-hops, and reflecting this as port affinity to the hardware.

VF LAG

[ConnectX-4 Lx and above] Added support for High Availability and load balancing for Virtual Functions of different physical ports in SwitchDev SR-IOV mode.

Uplink Representors

[ConnectX-4 Lx and above] Exposed PF (uplink) representors in SwitchDev mode, similarly to VF representors, as an infrastructure improvement for SmartNICs.

Userspace Software Steering for eSwitch

[ConnectX-5] Added software steering capabilities to the SR-IOV eSwitch. Software steering enables better rules insertion rate compared to the current firmware-based solution. This is achieved by performing calculations on the main CPU which allows for higher insertion rates.

Userspace Software Steering for NICs

[ConnectX-5] Added software steering capabilities to NIC Rx/Tx. Software steering enables better rules insertion rate compared to the current firmware-based solution. This is achieved by performing calculations on the main CPU which allows for higher insertion rates. This solution was designed to work with Virtio DPDK.

Note: Support will be enabled by default once the support for GID change is added.

ASAP2

[ConnectX-5 and above] Incorporated the documentation of Accelerated Switching And Packet Processing (ASAP2): Hardware Offloading for vSwitches into MLNX_OFED Release Notes and User Manual.

QP Counters and Firmware Errors per PID

[ConnectX-5 and above] QP counters and flow counters are now set per Process ID (PID) to allow better visibility of RDMA error states. Users will be able to manually tune the Q counter to monitor specific QPs, or automatically monitor QPs according to predefined criteria, such as the QP type.

ODP over DC

[ConnectX-5 and above] Added support for On-Demand Paging (ODP) over DC transport.

Address Translation Services

[ConnectX-5 and above] Added support for Address Translation Services (ATS) feature, which improves performance for virtualized PeerDirect applications by caching PA-> MA translations and preventing PCI transactions from going to the root complex.

XDP Inline Transmission of Small Packets

[ConnectX-5 and above] Added support for when forwarding packets with XDP, a packet smaller than 256 bytes would be sent inline within its WQE Tx descriptor for better performance. The number of packets that are transmitted inline depends on CPUs load, where lower load leads to a higher number of inline transmission.

VLAN Rewrite

[ConnectX-5 and above] Added support for offloading VLAN ID modify operation, allowing the user to replace the VLAN tag of the incoming frame with a user-specified VLAN tag value.

CQE Padding

[ConnectX-5 and above] Added support for padding 64B CQEs to 128B cache lines to improve performance on 128B cache line systems, such as PPC.

XDP Multi-Packet Tx Work Queue Element (WQE)

[ConnectX-5 and above] Added support for Multi-Packet Tx WQEs in XDP transmit flows to work with a new and improved WQE layout that describes several packets.This saves PCI bandwidth and transactions, and improves transmit packet rate.

ConnectX Device IDs

[ConnectX-6] Added support for the following new device IDs:

  • ConnectX-6 Dx (PF)

  • ConnectX Family mlx5Gen Virtual Function (VF)

    Note that every new device (adapter) VF will be identified with this device ID. Different VF models will be distinguished by their revision ID.

Ethtool 200Gbps

[ConnectX-6 and above] ConnectX-6 hardware introduces support for 200Gbps and 50Gbps-per-lane link mode. The driver supports full backward compatibility with previous configurations. Note that in order to advertise newly added link-modes, the full bitmap related to the link modes must be advertised from ethtool man page.

NOTE: This feature is firmware-dependent. Currently, ConnectX-6 Ethernet firmware supports up to 100Gbps only. Thus, this capability may not function properly using the current driver and firmware versions.

HDR Link Speed Exposure

[ConnectX-6 and above] Added support for HDR link speed in CapabilityMask2 field in port attributes.

QP Packet Based Credit Mode

[ConnectX-6 and above] Added support for an alternative end-to-end credit mode for QP creation. Credits transported from the responder to the requester are now issued per packet. This is particularly useful for sending large RDMA messages from HCA to switches that are short in memory.

Device Emulation Infrastructure

[BlueField] Added support for Device Emulation in BlueField. This mechanism allows function-A to perform operations on behalf of function-B. The emulation manager creates a channel (named VHCA_TUNNEL general object) that acts as the direct command interface between the emulated function host and the HCA hardware. The emulation software creates this tunnel for every managed function and issues commands via the DevX general command interface.

Verbs Migration to RDMA-Core

[All HCAs] Legacy verbs remain the default userspace installation option in the MLNX_OFED. However, as of MLNX_OFED v4.7, you can opt to install full RDMA-Core based userspace by adding the

--upstream-libs flag to the mlnxofedinstall script.

MLNX_OFED Installation via Repository

[All HCAs] The repository providing legacy verbs has been moved from RPMS or DEBS folders to RPMS/MLNX_LIBS and DEBS/MLNX_LIBS.

In addition, a new repository providing RDMA-Core based userspace has been added to RPMS/UPSTREAM_LIBS and DEBS/UPSTREAM_LIBS.

NFSoRDMA

[All HCAs] Added support for NFS over RDMA (NFSoRDMA) module over the OSs listed in NFSoRDMA Supported OSs section.

[All HCAs] As of MLNX_OFED v4.7, NFSoRDMA driver is no longer installed by default. In order to install it over a supported kernel, add the “--with-nfsrdma” installation option to the “mlnxofedinstall” script.

RDMA-CM QP Timeout Control

[All HCAs] Added a new option to rdma_set_option that allows applications to override the RDMA-CM's QP ACK timeout value.

Object IDs Exportation

[All HCAs] Added a unique ID for each verbs object to allow direct query over rdma-tool and rdma-netlink for enhanced debuggability.

RDMA-CM Application Managed QP

[All HCAs] Added support for the RDMA application to manage its own QPs and use RDMA-CM only for exchanging Address information.

Bug Fixes

See Bug Fixes.

Category

Description

4.6-1.0.1.0

Devlink Configuration Parameters Tool

[ConnectX-3/ConnectX-3 Pro] Added support for a set of configuration parameters that can be changed by the user through the Devlink user interface.

ODP Pre-fetch

[ConnectX-4 and above] Added support for pre-fetching a range of an on-demand paging (ODP) memory region (MR), this way reducing latency by making pages present with RO/RW permissions before the actual IO is conducted.

DevX Privilege Enforcement

[ConnectX-4 and above] Enforced DevX privilege by firmware. This enables future device functionality without the need to make driver changes unless a new privilege type is introduced.

DevX Interoperability APIs

[ConnectX-4 and above] Added support for modifying and/or querying for a verb object (including CQ, QP, SRQ, WQ, and IND_TBL APIs) via the DevX interface.

This enables interoperability between verbs and DevX.

DevX Asynchronous Query Commands

[ConnectX-4 and above] Added support for running QUERY commands over the DevX interface in an asynchronous mode. This enables applications to issue many commands in parallel while firmware processes the commands.

DevX User-space PRM Handles Exposure

[ConnectX-4 and above] Exposed all PRM handles to user-space so DevX user application can mix verbs objects with DevX objects.

For example: Take the cqn from the created ibv_cq and use it on a devx)create(QP).

Indirect Mkey ODP

[ConnectX-4 and above] Added the ability to create indirect Mkeys with ODP support over DevX interface.

XDP Redirect

[ConnectX-4 and above] Added support for XDP_REDIRECT feature for both ingress and egress sides. Using this feature, incoming packets on one interface can be redirected very quickly into the transmission queue of another capable interface. Typically used for load balancing.

RoCE Disablement

[ConnectX-4 and above] Added the option to disable RoCE traffic handling. This enables forwarding of traffic over UDP port 4791 that is handled as RoCE traffic when RoCE is enabled.

When RoCE is disabled, there is no GID table, only Raw Ethernet QP type is supported and RoCE traffic is handled as regular Ethernet traffic.

Forward Error Correction (FEC) Encoding

[ConnectX-4 and above] Added the ability to query and modify Forward Error Correction (FEC) encoding, as well as disabling it via Ethtool.

RAW Per-Lane Counters Exposure

[ConnectX-4 and above] Exposed RAW error counters per cable-module lane via ethtool stats. The counters show the number of errors before FEC correction (if enabled).

For further information, please see phy_raw_errors_lane[i] under Physical Port Counters section in Understanding mlx5 ethtool Counters Community post.

VF LAG

[ConnectX-4 Lx and above] Added support for High Availability and load balancing for Virtual Functions of different physical ports in SwitchDev SR-IOV mode.

ASAP2 Offloading VXLAN Decapsulation with HW LRO

[ConnectX-5 and above] Added support for performing hardware Large Receive Offload (HW LRO) on VFs with HW-decapsulated VXLAN.

For further information on the VXLAN decapsulation feature, please refer to ASAP2 User Manual under nvidia.com/en-us/networking/ → Products → Software → ASAP2.

PCI Atomic Operations

[ConnectX-5 and above] Added the ability to run atomic operations on local memory without involving verbs API or compromising the operation's atomicity.

Equal-Cost Multi-Path (ECMP) Routing Offloading

[ConnectX-5 and above] Enabled Equal-Cost Multi-Path (ECMP) Routing offloading.

Equal-Cost Multi-Path (ECMP) is a forwarding mechanism for routing packets along multiple paths of equal cost with the goal to achieve almost equally distributed link load sharing.

VXLAN over VLAN

[ConnectX-5 and above] VXLAN over VLAN enables the user to use VXLAN offloads' benefit to offload VLAN tagged tunnels thus boost system's performance.

VLAN Rewrite

[ConnectX-5 and above] Rewriting VLAN tags allows the user to replace the VLAN tag of the incoming frame with a user-specified VLAN tag value.

Virtual Ethernet Port Aggregator (VEPA)

[ConnectX-5] Added support for activating/deactivating Virtual Ethernet Port Aggregator (VEPA) mode on a single virtual function (VF). To turn on VEPA on the second VF, run:

bridge link set dev <netdev> hwmode vepa

VFs Rate Limit

[ConnectX-5] Added support for setting a rate limit on groups of Virtual Functions rather on an individual Virtual Function.

ConnectX-6 Support

[ConnectX-6] [Beta] Added support for ConnectX-6 (VPI only) adapter cards.

NOTE: In HDR installations that are built with remotely managed Quantum-based switches, the switch’s firmware must be upgraded to version 27.2000.1142 prior to upgrading the HCA’s (ConnectX-6) firmware to version 20.25.1500. When using ConnectX-6 HCAs with firmware v20.25.1500 and connecting them to Quantum-based switches, make sure the Quantum firmware version is 27.2000.1142 in order to avoid any critical link issues.

Ethtool 200Gbps

[ConnectX-6] ConnectX-6 hardware introduces support for 200Gbps and 50Gbps-per-lane link mode. MLNX_OFED supports full backward compatibility with previous configurations.

Note that in order to advertise newly added link-modes, the full bitmap related to the link modes must be advertised from ethtool man page. For the full bitmap list per link mode, please refer to MLNX_OFED User Manual.

NOTE: This feature is firmware-dependent. Currently, ConnectX-6 Ethernet firmware supports up to 100Gbps only. Thus, this capability may not function properly using the current driver and firmware versions.

PCIe Power State

[ConnectX-6] Added support for the following PCIe power state indications to be printed to dmesg:

  1. Info message #1: PCIe slot power capability was not advertised.

  2. Warning message: Detected insufficient power on the PCIe slot (xxxW).

  3. Info message #2: PCIe slot advertised sufficient power (xxxW).

    When indication #1 or #2 appear in dmesg, user should make sure to use a PCIe slot that is capable of supplying the required power.

Message Signaled

Interrupts-X (MSI-X)

Vectors

[mlx5] Added support for using a single MSI-X vector for all control event queues instead of one MSI-X vector per queue in a virtual function driver. This frees extra MSI-X vectors to be used for completion event queue, allowing for additional traffic channels in the network device.

Send APIs

[mlx5] Introduced a new set of QP Send operations (APIs) which allows extensibility for new Send opcodes.

DC Data-path

[mlx5] Added DC QP data-path support using new Send APIs introduced in Direct Verbs (DV).

BlueField Support

[BlueField] BlueField is now fully supported as part of the NVIDIA OFED mainstream version sharing the same code baseline with all the adapters product line.

Representor Name Change

[BlueField] In SwitchDev mode:

  • Uplink representors are now called p0/p1

  • Host PF representors are now called pf0hpf/pf1hpf

  • VF representors are now called pf0vfN/pf1vfN

ECPF Net Devices

[BlueField] In SwitchDev mode, net devices enp3s0f0 and enp3s0f1 are no longer created.

Setting Host MAC and Tx Rate Limit from ECPF

[BlueField] Expanded to support VFs as well as the host PFs.

RDMA-CM Application Managed QP

[All HCAs] Added support for the RDMA application to manage its own QPs and use RDMA-CM only for exchanging Address information.

RDMA-CM QP Timeout Control

[All HCAs] Added a new option to rdma_set_option that allows applications to override the RDMA-CM's QP ACK timeout value.

MLNX_OFED Verbs API

[All HCAs] As of MLNX_OFED v5.0 release (Q1 2020) onwards, MLNX_OFED Verbs API will be migrated from the legacy version of the user space verbs libraries (libibervs, libmlx5 ..) to the upstream version rdma-core.

More details are available in MLNX_OFED user manual under Installing Upstream rdma-core Libraries.

Bug Fixes

See Bug Fixes.

Category

Description

4.5-1.0.1.0

VFs per PF

[ConnectX-5] Increased the amount of maximum virtual functions (VF) that can be allocated to a physical function (PF) to 127 VF.

SW-Defined UDP Source Port for RoCE v2

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] UDP source port for RoCE v2 packets is now calculated by the driver rather than the firmware, achieving better distribution and less congestion. This mechanism works for RDMA- CM QPs only, and ensures that RDMA connection messages and data messages have the same UDP source port value.

Local Loopback Disable

[mlx5 Driver] Added the ability to manually disable Local Loopback regardless of the number of open user-space transport domains.

Adapter Cards

[ConnectX-6] Added support for ConnectX-6 Ready. For further information, please contact support networking-support@nvidia.com.

NEO-Host

[All HCAs] Integrated NEO-Host for orchestration and management of host networking into MLNX_OFED package.

Bug Fixes

See Bug Fixes.

4.4-2.0.7.0

Bug Fixes

See Bug Fixes.

4.4-1.0.0.0

Adaptive Interrupt Moderation

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for adaptive Tx, which optimizes the moderation values of the Tx CQs on runtime for maximum throughput with minimum CPU overhead.

This mode is enabled by default.

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Updated Adaptive Rx to ignore ACK packets so that queues that only handle ACK packets remain with the default moderation.

Docker Containers [Beta]

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for Docker containers to run over Virtual RoCE and InfiniBand devices using SR-IOV mode.

VF Statistics

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Performed the following virtual function statistics changes:

  • Added tx_broadcast and tx_multicast counters

  • Included RDMA statistics for existing counters

Force TTL

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for setting a global TTL value for all RC QPs and rdma-cm QPs.

Firmware Tracer

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added a new mechanism for the device’s FW/HW to log important events into the event tracing system (/sys/kernel/debug/tracing) without requiring any NVIDIA-specific tool.

Note: This feature is enabled by default.

CR-Dump

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Accelerated the original cr-dump by optimizing the reading process of the device’s CR-Space snapshot.

RoCE ICRC Error Counter

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for a new counter that exposes the amount of corrupted RoCE packets that arrive with bad Invariant Cyclic Redundancy Code (ICRC).

VST Q-in-Q

[ConnectX-4/ConnectX-4 Lx] Added support for C-tag (0x8100) VLAN insertion to tagged packets in VST mode.

Ethernet Tunneling Over IPoIB Driver (eIPoIB)

[ConnectX-4] Re-added support for eth_ipoib driver, which provides a standard Ethernet interface to be used as a Physical Interface (PIF) into the Hypervisor virtual network, and serves one or more Virtual Interfaces (VIF).

OVS Offload using ASAP2

[ConnectX-4 Lx/ConnectX-5] Added support for NVIDIA Accelerated Switching And Packet Processing (ASAP2) technology, which allows OVS offloading by handling OVS data-plane, while maintaining OVS control-plane unmodified. OVS Offload using ASAP2 technology provides significantly higher OVS performance without the associated CPU load.

For further information, refer to ASAP2 Release Notes under nvidia.com/en-us/networking/.com → Products → Software → ASAP2.

Upstream Libraries

[All HCAs] Added a repository repodata to support installing upstream libraries (based on upstream rdma-core), using he Operating System's standard package manager (yum, apt-get, etc.).

For further information, please refer to “Installing Upstream rdma-core Libraries” section in MLNX_OFED User Manual

Note: This is intended only for DPDK users.

Installation

[All HCAs] Added support for new metadata packages that only install userspace packages at a time (without any kernel packages), using the Operating System's standard package manager (yum, apt-get, etc.). These metadata packages will have the suffix “-user-only”. For example: “mlnx-ofed-all-user-only”.

Bug Fixes

See Bug Fixes.

4.3-1.0.1.0

Adaptive Interrupt Moderation

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for adaptive Tx, which optimizes the moderation values of the Tx CQs on runtime for maximum throughput with minimum CPU overhead.

This mode is enabled by default.

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Updated Adaptive Rx to ignore ACK packets so that queues that only handle ACK packets remain with the default moderation.

Docker Containers [Beta]

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for Docker containers to run over Virtual RoCE and InfiniBand devices using SR-IOV mode.

VF Statistics

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Performed the following virtual function statistics changes:

  • Added tx_broadcast and tx_multicast counters

  • Included RDMA statistics for existing counters

Force TTL

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for setting a global TTL value for all RC QPs and rdma-cm QPs.

Firmware Tracer

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added a new mechanism for the device’s FW/HW to log important events into the event tracing system (/sys/kernel/debug/tracing) without requiring any NVIDIA-specific tool.

Note: This feature is enabled by default.

CR-Dump

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Accelerated the original cr-dump by optimizing the reading process of the device’s CR-Space snapshot.

RoCE ICRC Error Counter

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for a new counter that exposes the amount of corrupted RoCE packets that arrive with bad Invariant Cyclic Redundancy Code (ICRC).

VST Q-in-Q

[ConnectX-4 andConnectX-4 Lx] Added support for C-tag (0x8100) VLAN insertion to tagged packets in VST mode.

Ethernet Tunneling Over IPoIB Driver (eIPoIB)

[ConnectX-4] Re-added support for eth_ipoib driver, which provides a standard Ethernet interface to be used as a Physical Interface (PIF) into the Hypervisor virtual network, and serves one or more Virtual Interfaces (VIF).

OVS Offload using ASAP2

[ConnectX-4 Lx/ConnectX-5] Added support for NVIDIA Accelerated Switching And Packet Processing (ASAP2) technology, which allows OVS offloading by handling OVS data-plane, while maintaining OVS control-plane unmodified. OVS Offload using ASAP2 technology provides significantly higher OVS performance without the associated CPU load.

For further information, refer to ASAP2 Release Notes under nvidia.com/en-us/networking/.com → Products → Software → ASAP2.

Upstream Libraries

[All HCAs] Added a repository repodata to support installing upstream libraries (based on upstream rdma-core), using the Operating System's standard package manager (yum, apt-get, etc.).

For further information, please refer to “Installing Upstream rdma-core Libraries” section in MLNX_OFED User Manual

Note: This is intended only for DPDK users.

Installation

[All HCAs] Added support for new metadata packages that only install userspace packages at a time (without any kernel packages), using the Operating System's standard package manager (yum, apt-get, etc.). These metadata packages will have the suffix “-user-only”. For example: “mlnx-ofed-all-user-only”.

Bug Fixes

See Bug Fixes.

4.3-1.0.1.0

Multi-Packet Work Request (WR)

[ConnectX-5] Added support for the following multi-packet WR related verbs for control path:

  • ibv_exp_query_device

  • ibv_exp_create_srq

For further information on the use of these verbs, please refer to the Verbs man page.

Coherent Accelerator Processor Interface (CAPI) [beta]

[ConnectX-5] Added support for CAPI, an interface that enables

ConnectX-5 adapter cards to provide the best performance for Power and OpenPower based platforms.

Tunneled Atomic

[ConnectX-5] Added support for RDMA atomic commands offload so that when an RDMA Write operation is issued, the payload indicates which atomic operation to perform, instead of being written to the Memory Region (MR).

Packet Pacing

[ConnectX-5] Added support for the following advanced burst control parameters:

  • max_burst_sz - for indicating the maximal burst size of packets

  • typical_pkt_sz - for improving the accuracy of the rate limiter

Erasure Coding

Offload verbs

[ConnectX-5] Added support for erasure coding offload software verbs (encode/decode/update API) supporting a number of redundancy blocks (m) greater than 4.

Virtual MAC

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Removed support for Virtual MAC feature.

RoCE LAG

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added out of box RoCE LAG support for RHEL 7.2 and RHEL 6.9.

Dropped Counters

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added a new counter rx_steer_missed_packets which provides the number of packets that were received by the NIC, yet were discarded/dropped since they did not match any flow in the NIC steering flow table.

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added the ability for SR-IOV counter rx_dropped to count the number of packets that were dropped while vport was down.

Relaxed Ordering (RSYNC)

[ConnectX-4/ ConnectX-4 Lx/ConnectX-5] Added support for RSYNC feature to ensure correct ordering of memory operations between the GPU and HCA.

Reset Flow

[mlx5 Driver] Added support for triggering software reset for firmware/driver recovery. When fatal errors occur, firmware can be reset and driver reloaded.

Striding RQ with HW Time-Stamping

[ConnectX-4 Lx/ConnectX-5] Added the option to retrieve the HW timestamp when polling for completions from a completion queue that is attached to a multi-packet RQ (Striding RQ).

4.2-1.2.0.0

DSCP Trust Mode

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for automatically setting the number of TC to 8 when the Trust state is changed to DSCP.

Receive Buffer

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added xon and xoff columns to the Receive Buffer configuration display.

4.2-1.0.0.0

Physical Address Memory Allocation

[mlx5 Driver] Added support to register a specific physical address range.

Innova IPsec Adapter Cards

[Innova IPsec EN] Added support for NVIDIA Innova IPsec EN adapter card, that provides security acceleration for IPsec-enabled networks.

Precision Time Protocol (PTP)

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for PTP feature over PKEY interfaces.

This feature allows for accurate synchronization between the distributed entities over the network. The synchronization is based on symmetric Round Trip Time (RTT) between the master and slave devices, and is enabled by default.

1PPS Time Synchronization

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for One Pulse Per Second (1PPS) over IPoIB interfaces.

Virtual MAC

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for Virtual MAC feature, which allows users to add up to 4 virtual MACs (VMACs) per VF. All traffic that is destined to the VMAC will be forwarded to the relevant VF instead of PF. All traffic going out from the VF with source MAC equal to VMAC will go to the wire also when Spoof Check is enabled.

For further information, please refer to “Virtual MAC” section in MLNX_OFED User Manual.

Receive Buffer

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added the option to change receive buffer size and cable length. Changing cable length will adjust the receive buffer's xon and xoff thresholds.

For further information, please refer to “Receive Buffer” section in MLNX_OFED User Manual.

GRE Tunnel Offloads

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for the following GRE tunnel offloads:

  • TSO over GRE tunnels

  • Checksum offloads over GRE tunnels

  • RSS spread for GRE packets

NVMEoF

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for the host side (RDMA initiator) in RedHat 7.2 and above.

Dropless Receive Queue (RQ)

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for the driver to notify the FW when SW receive queues are overloaded.

PFC Storm Prevention

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for configuring PFC stall prevention in cases where the device unexpectedly becomes unresponsive for a long period of time. PFC stall prevention disables flow control mechanisms when the device is stalled for a period longer than the default pre-configured timeout. Users now have the ability to change the default timeout by moving to auto mode.

For further information, please refer to “PFC Stall Prevention” section in MLNX_OFEDUser Manual.

Force DSCP

[ConnectX-4/ConnectX-4 Lx/ConnectX-5] Added support for this feature that enables setting a global traffic_class value for all RC QPs.

Q-in-Q

[ConnectX-5] Added support for Q-in-Q VST feature in ConnectX-5 adapter cards family.

Device Memory Programming [beta]

[ConnectX-5] Added support for on-chip memory allocation and usage in send/receive and RDMA operations at beta level.

Virtual Guest Tagging (VGT+)

[ConnectX-5] Added support for VGT+ in ConnectX-4/ConnectX-5 HCAs. This feature is s an advanced mode of Virtual Guest Tagging (VGT), in which a VF is allowed to tag its own packets as in VGT, but is still subject to an administrative VLAN trunk policy. The policy determines which VLAN IDs are allowed to be transmitted or received. The policy does not determine the user priority, which is left unchanged.

For further information, please refer to “Virtual Guest Tagging (VGT+)” section in MLNX_OFED User Manual.

Tag Matching Offload

[ConnectX-5] Added support for hardware Tag Matching offload with Dynamically Connected Transport (DCT).

Shared Memory Region (MR)

[ConnectX-3/ConnectX-3 Pro] Removed support for Shared MR feature on ConnectX-3/ConnectX-3 Pro adapter cards. As a result of this change, the following API/flags should not be used:

  • ibv_exp_reg_shared_mr

  • access shared flags for ibv_exp_reg_mr (IBV_EXP_ACCESS_SHARED_MR_XXX)

CR-DUMP

[All HCAs] Added support for the driver to take an automatic snapshot of the device’s CR-Space in cases of critical failures.

For further information, please refer to “CRDUMP” section in MLNX_OFED User Manual.

Upstream Libraries

[All HCAs] Added the option to install upstream libraries (based on upstream rdma-core) for DPDK users only.

For further information, please refer to “Installing Upstream rdma-core Libraries” section in MLNX_OFED User Manual.

DiSNI

[All HCAs] Added the option to install libdisni package as part of MLNX_OFED.

For further information, please refer to section “Installing libdisni Package” in MLNX_OFED User Manual.

Service Scripts

[All HCAs] Added the ability to disable the ‘ stop ’ option in the openibd service script, by setting ALLOW_STOP=no in

/etc/infiniband/openib.conf.

Starting from the next release, ‘ stop ’ option will be disabled by default, and in order to enable it, ALLOW_STOP should be set to ‘yes’ in the conf file, or force-stop should be run.

4.1-1.0.2.0

RoCE Diagnostics and ECN Counters

[mlx5 Driver] Added support for additional RoCE diagnostics and ECN congestion counters under /sys/class/infiniband/mlx5_0/ports/1/hw_counters/ directory.

For further information, refer to the Understanding mlx5 Linux Counters and Status Parameters Community post.

rx-fcs Offload (ethtool)

[mlx5 Driver] Added support for rx-fcs ethtool offload configuration. Normally, the FCS of the packet will be truncated by the ASIC hardware before sending it to the application socket buffer (skb). Ethtool allows to set the rx-fcs not to be truncated, but to pass it to the application for analysis.

For more information and usage, refer to Understanding ethtool rx-fcs for mlx5 Drivers Community post.

DSCP Trust Mode

[mlx5 Driver] Added the option to enable PFC based on the DSCP value. Using this solution, VLAN headers will no longer be mandatory for use.

For further information, refer to the HowTo Configure Trust Mode on NVIDIA Adapters Community post.

RoCE ECN Parameters

[mlx5 Driver] ECN parameters have been moved to the following directory: /sys/kernel/debug/mlx5/<PCI BUS>/cc_params/

For more information, refer to the HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux) Community post.

Flow Steering Dump Tool

[mlx5 Driver] Added support for mlx_fs_dump, which is a python tool that prints the steering rules in a readable manner.

Secure Firmware Updates

[mlx5 Driver] Firmware binaries embedded in MLNX_OFED package now support Secure Firmware Updates. This feature provides devices with the ability to verify digital signatures of new firmware binaries, in order to ensure that only officially approved versions are installed on the devices.

For further information on this feature, refer to NVIDIA Firmware Tools (MFT) User Manual.

Enhanced IPoIB

[mlx5 Driver] Added support for Enhanced IPoIB feature, which enables better utilization of features supported in ConnectX-4 adapter cards, by optimizing IPoIB data path and thus, reaching peak performance in both bandwidth and latency.

Enhanced IPoIB is enabled by default.

PeerDirect

[mlx5 Driver] Added the ability to open a device and create a context while giving PCI peer attributes such as name and ID.

For further details, refer to the PeerDirect Programming Community post.

Probed VFs

[mlx5 Driver] Added the ability to disable probed VFs on the hypervisor. For further information, see HowTo Configure and Probe VFs on mlx5 Drivers Community post.

Local Loopback

[mlx5 Driver] Improved performance by rendering Local loopback (unicast and multicast) disabled by mlx5 driver by default while local loopback is not in use. The mlx5 driver keeps track of the number of transport domains that are opened by user-space applications. If there is more than one user-space transport domain open, local loopback will automatically be enabled.

1PPS Time Synchronization (at alpha level)

[mlx5 Driver] Added support for One Pulse Per Second (1PPS), which is a time synchronization feature that allows the adapter to send or receive 1 pulse per second on a dedicated pin on the adapter card.

For further information on this feature, refer to the HowTo Test 1PPS on NVIDIA Adapters Community post.

Precision Time Protocol (PTP)

[mlx5 Driver] Added support for PTP feature in IPoIB offloaded devices.

This feature allows for accurate synchronization between the distributed entities over the network.

The synchronization is based on symmetric Round Trip Time (RTT) between the master and slave devices.

The feature is enabled by default.

For further information, refer to Running Linux PTP with ConnectX-4 Community post.

Fast Driver Unload

[mlx5 Driver] Added support for fast driver teardown in shutdown and kexec flows.

HCAs: ConnectX-5/ConnectX-5 Ex

NVMEoF Target Offload

[ConnectX-5/ConnectX-5 Ex] Added support for NVMe over fabrics (NVMEoF) offload, an implementation of the new NVMEoF standard target (server) side in hardware.

For further information on NVMEoF Target Offload, refer to HowTo Configure NVMEoF Target Offload.

MPI Tag Matching

[ConnectX-5/ConnectX-5 Ex] Added support for offloading MPI tag matching to HCA.

RDMA CM

[All HCAs] Changed the default RoCE mode on which RDMA CM runs to RoCEv2 instead of RoCEv1.

RDMA_CM session requires both the client and server sides to support the same RoCE mode. Otherwise, the client will fail to connect to the server.

For further information, refer to RDMA CM and RoCE Version Defaults Community post.

Lustre

[All HCAs] Added support for Lustre file system open-source project.

4.0-2.0.2.0

Operating Systems

Added support for Ubuntu v17.04.

4.0-2.0.0.1

PCIe Error Counting

[ConnectX-4/ConnectX-4 Lx] Added the ability to expose physical layer statistical counters to ethtool.

Multiprotocol Label Switching (MPLS) Tagged Packets Classification

[ConnectX-4/ConnectX-4 Lx] Enabled packet flow steering rules with IPv4/IPv6 classification (for raw packet QP (DPDK) only) to work on IPv4/IPv6 over MPLS (Ethertype 0x8847 and 0x8848) encapsulated packets.

RoCE VFs

[ConnectX-4/ConnectX-4 Lx] Added the ability to enable/disable RoCE on VFs.

RoCE LAG

[ConnectX-4/ConnectX-4 Lx] Added support for RoCE over LAG interface.

Standard ethtool

[ConnectX-4/ConnectX-4 Lx] Added support for flow steering and rx-all mode.

SR-IOV Bandwidth Share for Ethernet/RoCE (beta)

[ConnectX-4/ConnectX-4 Lx] Added the ability to guarantee the minimum rate of a certain VF in SR-IOV mode.

Adapter Cards

Added support for ConnectX-5 and ConnectX-5 Ex HCAs.

DSCP ConfigFS Control for RDMA-CM QPs

Added the ability to configure ToS/DSCP for RDMA-CM QPs only.

Soft RoCE (beta)

Add software implementation of RoCE that allows RoCE to run on any Ethernet network adapter whether it offers hardware acceleration or not.

NVMe over Fabrics (NVMEoF)

NVMEoF related module installation has been disabled by default. In order to enable it, add the “ --with-nvmf ” installation option to the “mlnxofedinstall” script.

NFS over RDMA (NFSoRDMA)

Removed support for NFSoRDMA drivers. These drivers are no longer provided along with the MLNX_OFED package.

Customer Affecting Change

Description

Customer Affecting Changes 5.6-1.0.3.3

Interface Renaming, PF/VF, Udev

The OFED driver no longer performs Ethernet NetDev interface renaming for PFs and VFs.

The udev rules file which implemented renaming (82-net-setup-link.rules) and its supporting script vf-net-link-name.sh are no longer installed by default.

Renaming is thus performed by underlying mechanisms -- in udev, in the kernel, and in the BIOS.

Users who wish to continue using the OFED driver renaming mechanism must add option --copy-ifnames-udev to the OFED install command.

To install these files at a later time, copy them from one of the following directories:

  • /usr/share/doc/mlnx-ofa_kernel (RHEL8 and newer)

  • /usr/share/doc/mlnx-ofa_kernel-[1-9]* (RHEL 7.X)

  • /usr/share/doc/packages/mlnx-ofa_kernel (SLES)

  • /usr/share/doc/mlnx-ofed-kernel-utils/examples (Debian-based releases)

Warning
  • File 82-net-setup-link.rules should be copied to directory /etc/udev/rules.d

  • File vf-net-link-name.sh should be copied to directory /etc/infiniband (make sure that it has both read and execute permission)

  • After copying over the files, the driver should be restarted for the copied files to take effect

  • Customers who wish prevent renaming of NetDev names should add "net.ifnames=0 biosdevname=0" to the kernel boot command line, and then reboot the host

Community Operating Systems

Starting OFED 5.6, NVIDIA is introducing a new support model for OFED used on open source community operating systems. The goal of this new support model is to enable customers to use community-maintained variants of the Linux operating system, without being limited to major distributions that NVIDIA provides primary support for. For more information, see "Installation on Community Operating Systems" section in the user manual. For a list of supported Community OSs, please see "Supported Community Operating Systems" section in the release notes.

ar_mgr Subnet Manager Plugin

ar_mgr subnet manager plugin is no longer supported.

For adaptive routing and SHIELD subnet manager configuration, please see the MLNX_OFED user manual.

Fabric Collector in UFM

Starting UFM v6.7, Fabric Collector is no longer supported. For more information, see the UFM release notes.

Warning

Please note that UFM is no longer bundled with OFED.

OVS-DPDK—Partial Offload

Starting OFED 5.6, OVS-DPDK does not support partial offload.

Customer Affecting Change

Description

5.5-1.0.3.2

Disabling RoCE While Using sysfs

When using sysfs to enable/disable roce in kernel 5.5 and up, the "devlink reload" command (using iproute2 with devlink tool) will need to be used to activate the RoCE status change.

Disable RoCE example:

1. echo 0 > /sys/bus/pci/devices/0000:08:00.0/roce_enable

2. devlink dev reload pci/0000:08:00.0

mlnx-ofa_kernel Installation

The source code for mlnx-ofa_kernel is no longer installed by default on RPM-based distributions (e.g., RHEL and SLES).

Notes:

• mlnx-ofa_kernel is included in the <> in the MLNX_OFED distributions under RPMS/ and may be manually installed from there.

• There is no change for deb-based distributions (Debian and Ubuntu). The full source is included, as before, in the package mlnx-ofed-kernel-dkms.

Software Encapsulation Compatibility

There is an encapL2 compatibility issue with accelerated reformat action creation using mlx5dv_dr API.

Using OFED 5.4 with firmware xx.32.1xxx and above or using OFED 5.5 with firmware lower than xx.32.1xxx will not allow accelerated reformat action. (Using OFED 5.4 and 5.5 with bundle firmware works properly.)

xpmem in RHEL8

Added xpmem packages in RHEL8 builds.

Python3

Starting OVS DPDK 2.15, the Python minimum required version is 3 and OVS-DPDK will not be compiled using Python 2.

Customer Affecting Change

Description

Customer Affecting Changes 5.4-3.0.3.0

CUDA, UCX, HCOLL

For UCX-CUDA and hcoll-cuda, CUDA was upgraded from version 10.2 to 11.2.

Customer Affecting Change

Description

Customer Affecting Changes 5.4-1.0.3.0

udev Rules

As of version 5.4, the driver is set so that udev rules will change the names of network interfaces created from NVIDIA adapters.

The udev rules are shipped to "/lib/udev/rules.d" and may be overridden by placing a file with the same name in "/etc/udev/rules.d".

Example: /etc/udev/rules.d/82-net-setup-link.rules

Network Interface Names, udevd

[ConnectX-4 and above] In MLNX_OFED 5.4 GA, ConnectX-4/5/6 Ethernet network interfaces are now provided with permanent names.

Prior to this release, the default interface names were provided by the kernel and udevd (ethX) remained as-is.

From this release onwards, interface names are generated via new udevd rules.

The generated names are now predictable, and the default names are automatically renamed to the predictable names by the udevd daemon, according to udev rules files installed by OFED.

The new interface names look as follows: en[P][p<bus number]sf

For example, a ConnectX device with PCI address: 0005:01:00.1 will be named enP5p1s0f1

The advantage of such a scheme for interface naming is that device whose PCI address is 0005:01:00.1 will always get the same device name since that name now depends on the host geography.

(Previously there were race conditions which sometimes caused the same physical device to get a different interface name upon reboot).

Note: Ethernet interface renaming for ConnectX-4/5/6 is performed only when eswitch is supported.

eswitch is supported on kernels starting from kernel version 4.9; for Linux distro kernels earlier than 4.9.0, eswitch is supported only on RHEL7.x and on XenServer 7.1 CU2.

Deprecated, OvS-DPDK

OvS-DPDK deprecated the command "ovs-appctl dpctl/dump-e2e-stats".

Instead, the command has been integrated into the existing command "ovs-appctl dpctl/offload-stats-show -m" (when e2e-cache is enabled).

OvS-DPDK

OvS-DPDK ct-ct-nat offloads is now disabled by default.

A new knob in OvS was introduced: "ovs-vsctl set open_vswitch . other_config:ct-action-on-nat-conns=" (default value is false).

If disabled, ct-ct-nat configurations will not be fully offloaded, improving connection offloading rate for other cases (ct and ct-nat).

If enabled, ct-ct-nat configurations will be fully offloaded but ct and ct-nat offloading will be created more slowly.

mlnxofedinstall, udev, MLNX_OFED, umad

Before version 5.4, /etc/udev/rules.d/90-ib.rules was potentially automatically edited by installation scripts in case the options --umad-dev-rw or --umad-dev-na were used. From version 5.4 and above, those changes are made in /etc/udev/rules.d/91-ib-permissions.rules which (if exist) only include the settings for those command-line options.

© Copyright 2023, NVIDIA. Last updated on Oct 11, 2023.