[ConnectX-4 and above] Added support for bridge offloads with VLAN support that works on top of mlx5 representors in switchdev mode. 

[ConnectX-6 Dx] Improved OVS failover through support for OVS groups in fast-failover mode + VF_LAG configuration with OVS.

[ConnectX-6 Dx and BlueField-2] Added support for exposing hairpin out of buffer drop counter per device.  This feature shows buffer drops related only to hairpin queues which were opened on the queried device.

To enable this counting mode (this must be done before any hairpin rules are created), use the following: echo "on <peer_devname>" > /sys/class/net/<dev>/hp_oob_cnt_mode where <peer_devname> is the peer device to which traffic coming to the configured device will be forwarded to for transmission.

To read the drop counter, use the following: cat /sys/class/net/<dev>/hp_oob_cnt

[ConnectX-6 Dx and BlueField-2] Added bridge offloads to support bonding (VF LAG), attaching bond device to bridge instead of uplink representors. 

[ConnectX-6 Dx] Added OOB support for VLAN push on Rx (wire to VF) and VLAN pop on Tx (wire to VF) in switchdev mode. 

[ConnectX-4 and above] Expanding the RDMA statistic tool to support setting vendor-specific optional counters dynamically using netlink.
Added to mlx5_ib the following optional counters:
cc_rx_ce_pkts,cc_rx_cnp_pkts,cc_tx_cnp_pkts.

Example:
$ rdma statistic mode supported link rocep8s0f0/1
link rocep8s0f0/1 supported optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts,cc_tx_cnp_pkts

$ sudo rdma statistic set link rocep8s0f0/1 optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts

$ rdma statistic mode link rocep8s0f0/1
link rocep8s0f0/1 optional-counters cc_rx_ce_pkts,cc_rx_cnp_pkts

$ sudo rdma statistic set link rocep8s0f0/1 optional-counters cc_rx_ce_pkts

$ rdma statistic mode link rocep8s0f0/1
link rocep8s0f0/1 optional-counters cc_rx_ce_pkts

$ sudo rdma statistic unset link rocep8s0f0/1 optional-counters

[ConnectX-5 and above] Added ability to use a specific interface for tunnel route lookup if tunnel was created with the current device. 

[ConnectX-5 and above] Added support via mlxdevm to mark a given PCI subfunction (SF) or virtual function (VF) as a trusted function. The device/firmware decides how to define privileges and access to resources. 

[All] Added support for preventing VF memory exhaustion. This feature exposes a sysfs (to the system admin) which can set a limit on each VF memory consumption.

Note: Currently only supported on Ethernet. 

[BlueField-2] Added support for resetting the NIC domain of BlueField-2 while keeping ARM alive. 

[ConnectX-6 Dx and BlueField-2] Added support in multiple flow steering priorities for FDB rules. 

[ConnectX-5 and above] Added support for offloading the HTB qdisc to the NIC, allowing it to scale better by eliminating a single locking point. The configuration is done with the TC commands.

Note: Kernel 5.15 or higher is required. Limited to 256 nodes.

[ConnectX-6 Dx and above] Added support for driver resiliency against high load of RX resync operations. 

Added support for the activation of PTP and CQE compression simultaneously. Since CQE compression might harm the accuracy of the PTP, the feature enables PTP packets to be moved to a dedicated queue where they are not subjected to compression. However, this configuration conflicts with setting aRFS. Turning off CQE compression, causes a hiccup in traffic which may cause a loss of synchronization. To overcome this, restart the synchronization.

Note: This combination is supported only for Ethernet drivers. Other driver profiles, like IPoIB and representors, do not support this combination. 

Added support to expose an option to prefetch ODP MR without faulting. This enables updating the device page table with the presenting CPU pages and reducing page faults in the system. 

[ConnectX-6 and above] Added DV API that allows configuration of MKey with AES-XTS crypto offloads. The MKey can be configured for both crypto and signature offloads. 

[ConnectX-4 and above] Added support to allow DEVX UMEM to be created with larger page sizes than 4K. For some device objects (e.g., RegEx) this is a must. In addition, page size larger than 4K may need less MTTs which may improve performance.

A new API mlx5dv_devx_umem_reg_ex() was added which requests a specific page sizes. It enables better application control on the required UMEM page size. The new API named mlx5dv_devx_umem_reg_ex() will be part of rdma-core V35. 

[ConnectX-4 and above] Added support for cleanup of the synchronize_srcu() from the ODP flow because it was a time-consuming part of dereg_mr.

Note: This only affects the driver and not the firmware. 

[ConnectX-4 and above] Extended support for the "rdma res show" command to SRQ and context resources.

[ConnectX-5 and above] Added support for Raw WQE (mlx5dv_wr_raw_wqe). This feature allows applications to build a new custom work request (WQE) that is not supported by the verbs or driver and post it on normal QP. It is an extension for IBV work request (ibv_wr_*) with mlx5 specific features for sending a work request. 

Added support for mlx5 user space driver over VFIO.

This feature enables an application to take full ownership on the opened device and run any firmware command (e.g., port up/down) without any concern to hurt someone else.

The application look and feel is like regular RDMA application over DEVX. It uses verbs API to open/close a device and then mostly uses DEVX APIs to interact with the device.

New mlx5 DV APIs were added to get ibv_device for a given mlx5 PCI name and to manage device specific events.

For description of the relevant APIs and expected usage of those APIs, look up the following:
mlx5dv_get_vfio_device_list()
mlx5dv_vfio_get_events_fd()
mlx5dv_vfio_process_events() 

[ConnectX-4 and above]  Added support for using system_image_guid to group bonding interfaces.

With some specific NICs, each interface may have different PCIe domain, bus, device, or function IDs. For interfaces with the same system_image_guid, the driver assumes they reside on the same physical device and use a native_port_id to distinguish its index. Fallback is to PCIe BDF, if unsupported. 

[ConnectX-4 and above] mlx_steering_dump  tool is used to parse software steering dump files which includes information about domains, tables, matchers and rules created by software steering (mlx5dv_dr API), it can be used offline by providing a dump file as input, or it can be used to trigger DPDK app (like testpmd) to generate the dump and parse it. 

Allowed installing multiple mlnx-ofa_kernel development headers packages (for different kernel versions of the same mlnx-ofa_kernel package version) side by side on the same system. 

Added signature of kernel modules of EulerOS 2.0 SP8-SP10 (x86_64 and aarch64) builds of MLNX_OFED. 

Enabled SF_CFG (SF config dummy driver, --with-sf-cfg-drv) on EulerOS2.0 SP8 and SP10. 

When using sysfs to enable/disable roce in kernel 5.5 and up,  the "devlink reload" command (using iproute2 with devlink tool) will need to be used to activate the RoCE status change.

Disable RoCE example:
1. echo 0 > /sys/bus/pci/devices/0000:08:00.0/roce_enable
2. devlink dev reload pci/0000:08:00.0 

There is an encapL2 compatibility issue with accelerated reformat action creation using mlx5dv_dr API.
Using OFED 5.4 with firmware xx.32.1xxx and above or using OFED 5.5 with firmware lower than xx.32.1xxx will not allow accelerated reformat action. (Using OFED 5.4 and 5.5 with bundle firmware works properly.)

Starting OVS DPDK 2.15, the Python minimum required version is 3 and OVS-DPDK will not be compiled using Python 2.