Release Notes Change Log History

Linux Kernel Upstream Release Notes v6.5

Feature/Change

Description

Rev 3.0.50000 (DRV 3.0. 25668 )

FwWaWqeTooSmallMode

Added a new mode to FwWaWqeTooSmallMode.
In this mode the firmware will not generate a WQE completion and will discard the arrived packet (Discard Wqe No Cqe) when VF WQE is too small .
By exposing the WqeTooSmall counter in the VM, the new mode enables the counter to count the number of times this action is successfully performed by the firmware.

NVIDIA BlueField-2 NIC Mode

The driver now supports NVIDIA BlueField-2 devices over IPoB running in NIC mode. In this mode, the DPU behaves exactly like an adapter card from the perspective of the external host.

For further information, see section "NVIDIA BlueField-2 DPU NIC Operation Mode" in the NVIDIA BlueField-2 Firmware Release Notes.

Port Traffic Counters

Added new counters to "Mellanox WinOF-2 VF Port Traffic".

For further information see section Adapter Cards Counters.

Diagnostic Counters

Added a new counter "Packets Received dropped due to lack of receive WQEs" to the "Mellanox WinOF-2 VF Diagnostics" counters set.

For further information see section Adapter Cards Counters.

DOCA

As of WinOF-2 v3.0, the DOCA module will be published on a separate package.
DOCA comm channel and socket relay applications were moved to the new DOCA, thus they are removed from the WinOF-2 package.

Note: A DOCA developer should install the DOCA SDK package and not the DevX SDK package.

For more information please see the DOCA documentation.

Bug Fixes

See Bug Fixes.

Rev 2.90.50010 (DRV 2.90.25506)

Adapter Cards

Added support for NVIDIA ConnectX-7 adapter cards.

NVIDIA BlueField-2 NIC Mode

The driver now supports NVIDIA BlueField-2 devices running in NIC mode. In this mode, the DPU behaves exactly like an adapter card from the perspective of the external host.

For further information, see section "NVIDIA BlueField-2 DPU NIC Operation Mode" in the NVIDIA BlueField-2 Firmware Release Notes.

DOCA Socket Relay

Added support for an AF_UNIX connection between applications that run on the Windows host and services that run inside the the DPU.
For further information, see DOCA Socket Relay.

DOCA Communication Channel API

DOCA communication channel API in NVIDIA BlueField-2 SmartNIC adapter cards is now at GA level.

For further information, see DOCA Communication Channel API.

DPKD DevX

Added new interfaces for DevX library to set:

  • the promiscuous mode with the two modes: ALL, MC

  • MTU (limited to Host case, and port MTU >= 1522)

For further information, see Offload Capabilities for Windows DPDK.

NVIDIA BlueField-2 DevX

Added support for flex parser to the DevX steering rule.

Enhanced Connection Establishment

Enhanced Connection Establishment (ECE) is a new negotiation scheme introduced in IBTA v1.4 to exchange extra information about nodes capabilities and later negotiate them at the connection establishment phase. ECE is intended for RDMA connection, i.e., it works in ND and NDK connections.

For further information, see Enhanced Connection Establishment.

CM Packets

This new capability provides the option of ignoring RoCE connections that have differences in the source IP address. Now the user can decide whether or not to allow differences between the IB header source IP and the private data source IP.

To activate this option, the 'EnableCmAntiSpoofing' key must be set to 1 (default value is 0).

For further information, see "RDMA Registry Keys" in section Configuring the Driver Registry Keys.

DriverVersion Utility

Changed the output of mlx5cmd -driverVersion command.
Instead of presenting the OS name, now the tool will present the OS build number + Server\Client information.

Rivermax

Enabled multiple Rivermax applications to listen on the same stream.

Bug Fixes

See Bug Fixes

Rev 2.80.50000 (DRV 2.80.25134)

Operating Systems

Removed support for Windows Server 2012.

DOCA Communication Channel API

[Alpha Level] Added support for DOCA communication channel API in NVIDIA BlueField-2 SmartNIC adapter cards.

DOCA Communication Channel API allows developers to write client applications running on Windows native hosts or Windows Virtual Machines to exchange messages with service applications running on BlueField-2 DPU.

For further information see DOCA Communication Channel API.

GPU Memory Registration

Added support for large GPU memory registration through ibv_reg_mr() and ibv_reg_mr_iova2().

DPU Time Service

Added support for "PTP like" for BlueField-2 devices when using the REAL_TIME timestamping ability.

For further information, see DevX Utility.

Hardware QoS Offload

Added Hardware QoS Offload support to allow egress bandwidth management entirely in the hardware.

For further information, see Hardware QoS Offload.

Reduced RoCE Latency for SMBDirect using Two Queues: One for FRWR and the other for Send

Added a new configuration value ('NdkFmrDedicatedQp') to control whether or not a separated QP is used for NDK fast-register operations.

The dedicated QP will improve latency on systems that phase latency issues.

Warning: The feature is useful only for SMBDirect and can be harmful for other applications using NDK.

Note: This capability is supported in ConnectX-4 and ConnectX-4-Lx adapter cards when in Ethernet (RoCE) mode.

Note: This value is OFF by default.

For further information, see Configuring the Driver Registry Keys.

Asymmetric Number of VFs per PF

This new capability allows the user to set an asymmetric Number of VFs per PF using IOCTL. When using IOCTL to open a second PF on the same port, the new PF will be opened with '0' VFs. The number of VFs can be modified by using the mlxconfig tool, as long as the total number of VFs (on all PFs together) will not exceed the maximum number of VFs allowed.

GPUDirect for Windows

GPUDirect support in Windows is now at GA level.

For further information, see GPUDirect.

Bug Fixes

See Bug Fixes History

Rev 2.70.53000 (DRV 2.70. 24739 )

Bug Fixes

See Bug Fixes History

Rev 2.70.51000 (DRV 2.70.24728)

Bug Fixes

See Bug Fixes History

Rev 2.70.50000 (DRV 2.70. 24708 )

GPUDirect for Windows

[Beta] Added GPUDirect support in Windows to allow the NIC direct access to the GPU memory.

For further information, see GPUDirect.

Operating System

Added support for Windows Server 2022 Operating System.

ND, MlxNdPerf

Added a new performance benchmark tool called MlxNdPerf that replaces all old ND performance tools (e.g. nd_send_bw...).

For further information see MlxNdPerf Utility.

Real Time PTP

Added support for fetching the real_time value from the init_segment.

VXLAN Offloading

VXLAN offloading is now supported on multiple ports. Port configuration can be done through the use of "Mlx5Cmd -Vxlan" command and you can configure up to 4 ports for offloading.

For further information see VXLAN Offloading Configuration Utility.

Vport, Promiscuous mode, DPDK

Added support for promiscuous mode enablement on a Vport when using DPDK.

Note: This capability is support in PF only.

PDDR Cable Information

Added support for cable information in ConnectX-6 / ConnectX-6 Dx / ConnectX-6 Lx and Bluefield-2 adapter cards.

DSFP Connector

Added support for DSFP connector.

Bug Fixes

See Bug Fixes History

Rev 2.60.51000 (DRV 2.60.23957)

Bug Fixes

See Bug Fixes History

Rev 2.60.50000 (DRV 2.60.23957)

Adapter Cards

Added support for NVIDIA® BlueField SmartNIC at GA level.

Hardware vPort Context

Added the option to dump hardware vPort context using mlx5cmd.

Configuration Validator

This tool validates the configuration of registry keys provided in the configuration file.

For further information see Configuration Validator

Link FEC Configuration Utility

The Link FEC Configuration utility provides the ability to query supported link FEC modes by the adapter for the current link speed and for all supported link speeds.

For further information see Link FEC Configuration Utility

Packet Pacing Capabilities

This tools query allocated Packet Pacing objects.

For further information see Packet Pacing Capabilities

DevX Registry Keys

Added new registry keys that configure the DevX feature.

For further information see DevX Registry Keys

NDIS Poll Mode

Windows introduced a new poll mode feature starting NDIS 6.85 onwards. The poll API handles Datapath processing for both TX and/or RX side. When the feature is enabled, the driver registers with NDIS for call backs to poll RX and/or TX data.

For further information see NDIS Poll Mode.

smpquery Utility

smpquery allows querying of various information about the InfiniBand network.

For further information see smpquery Utility.

Counters

Added the following new counters:

  • Packets processed in NDIS poll mode

  • CQ Overrun

For further information see Mellanox WinOF-2 Receive Datapath & Mellanox WinOF-2 Transmit Datapath / Mellanox WinOF-2 PCI Device Diagnostic & Mellanox WinOF-2 Diagnostics Extension 1

Non-encapsulated Packets Steering

Non-encapsulated packet handling enables the user to facilitate the following main flows:

  • Matching by the inner header only (non-encapsulated packets dropped or indicated on default vPort in Promiscuous mode).

  • Matching encapsulated packets by inner header and non-encapsulated packets when registry GreEnableDualTunneling is configured.

  • Matching encapsulated packets by outer header and non-encapsulated packets.

For further information, see Non-encapsulated Packets Steering.

Driver Events

The following event logs severity status was changed from "Error" to "Warning" as they are not fatal errors:

  • MLX_EVENT_LOG_IPOIB_ILLEGAL_Q_KEY (0x000A)

  • MLX_EVENT_LOG_ILLEGAL_MAC_ADDRESS (0x0027)

  • MLX_EVENT_LOG_SM_MTU_MISMATCH (0x0035)

  • MLX_EVENT_ERROR_RESILIENCY_INIT_FAIL (0x0097)

  • MLX_EVENT_ERROR_DUMP_ME_NOW (0x0169)

  • EVENT_NDK_FAILED_TO_BE_ENABLED (0x016f)

  • EVENT_NDK_FAILED_TO_BE_DISABLED (0x0170)

Registry Keys

Added new registry keys to control moving to DPC mode once the maximum RX/TX packet processing limit is reached.

For further information, see Performance Registry Keys.

Counters

Removed "Mellanox WinOF-2 VF Internal Traffic Counters" from Virtual Functions.

Note: Mellanox WinOF-2 VF Internal Traffic Counters are relevant for Physical Functions ONLY.

PCIe Transfer Speed

Added PCIe transfer speed units for event MLX_PCIE_LINK. For further information, see event 0x0191 in Reported Driver Events.

IPoIB Teaming

Added support for IPoIB Teaming in failover mode.

Bug Fixes

See Bug Fixes History

Rev 2.50.50000 (DRV 2.50.23282)

Adapter Cards

Added support for NVIDIA® Mellanox® ConnectX®-6 Lx adapter card.

Adapter Cards

Added support for NVIDIA® Mellanox® BlueField SmartNIC mode.

For further information see NVIDIA BlueField SmartNIC Mode.

Adapter Cards

[Beta] Added support for NVIDIA® Mellanox® BlueField-2 SmartNIC adapter card.

For further information see NVIDIA BlueField SmartNIC Mode.

Bluefield Settings

When using Bluefield in SmartNIC mode, encapsulation offload is done by the SoC, therefore, the encapsulation offload registry keys does not have any impact .
For this reason, the following encapsulation offload registry keys for Bluefield device are not advertised.

  • *EncapsulatedPacketTaskOffloadNvgre

  • EncapsulatedPacketTaskOffloadVxlan

  • *VxlanUDPPortNumber

  • *EncapOverhead

  • *EncapsulatedPacketTaskOffload

INF Modifications

As part of disabling the offloads for Bluefield devices, the following changes we applied to the inf:

  • Added new section:

    • Encap_keys.reg and encap_keys10.0.reg

  • Removed the virt10.0.reg section. The keys that were in this section and moved to encap_keys10.0.reg section are:

    • *EncapsulatedPacketTaskOffloadNvgre

    • EncapsulatedPacketTaskOffloadVxlan

    • *VxlanUDPPortNumber

    • *EncapOverhead

  • *EncapsulatedPacketTaskOffload key was moved from virt.reg section to Encap_keys.reg

Windows Mode Sleep/Hibernation Detection

[Beta] Added support for device removal while the system is in sleep/hibernation mode.

VF’s DHCP Redirections

This feature forces every received\sent DHCP packet to be redirected to PF, including DHCP packets sent or received for VFs. The detection of a packet as a DHCP is done by checking UDP-Ports 67 and 68.

For further information see VF's DHCP Redirections.

Mellanox WinOF-2 PCI Device Diagnostic

The "Available PCI BW" and "Used PCI BW" counters are now deprecated and replaced by "Available PCI BW/Sec" and "Used PCI BW/Sec".

For further information see Mellanox WinOF-2 PCI Device Diagnostic.

DevX SDK

Added a DevX SDK executable file to develop code on top of DevX. The capability requires manual installation.

For further information see DevX Interface.

Features Status Utility

The utility displays the status of driver features.

For further information see Features Status Utility.

Temperature Utility

Mlx5Cmd can now query the external ASIC temperature sensor to get temperature readings using the "Mlx5Cmd -Temperature" command.

For further information see Temperature Utility.

DSCP Based QoS

DSCP Based QoS enables the user to map DSCP to certain priority.

For further information see DSCP Based QoS.

ECN Marking

The driver now updates the IP header with ECN bits based on hints received from the hardware. Additionally, software counters were also added for number of such packets marked and updated by the driver.

Explicit Congestion Notification (ECN) Hint in CQE

In a multi-host system, a single receive buffer is used for all hosts. If one or more host(s) are being congested, the congested host(s) can exhaust the device’s receive buffer and cause service degradation for the other host(s). In order to manage this situation, the device can mark the ECN (Explicit Congestion Notification) bits in the IP header for the congested hosts. When ECN is enabled on the host, the host will sense the ECN marking and will reduce the TCP traffic and by that will throttle the traffic. Additionally, software counters were also added for number of such packets marked and updated by the driver.
The device is capable to either drop or mark the packet (ECN) based on the Aggressive Mode or Dynamic Mode.

For further information see Explicit Congestion Notification (ECN) Hint in CQE.

Additional MAC Addresses in RSS (Native Mode)

This feature allows the user to configure additional MAC addresses for the network adapter without setting the adapter to promiscuous mode. Registering MAC addresses for a network
adapter will allow the adapter to accept packets with the registered MAC address.

For further information see Additional MAC Addresses for the Network Adapter.

Limiting the Event Log from Flooding the Event Viewer

This new capability prevents the event log from flooding the event viewer in case of an unrecoverable error.
This will limit printing similar events up to a defined amount in a defined time range.
The users can enable/disable it per their needs. In addition, users can also configure the time range and threshold events count.

Resource Dump (Debuggability)

Resource Dump is a debuggability utility that extracts and prints data segments generated by the firmware/hardware. For further information see Resource Dump.

vPort RSS Configuration

RSS information is now displayed from the driver. On the Hyper-V it will also display Vport's VMMQ configurations. For further information see Display RSS Information.

DMN Offline Debug Improvements (Debuggability)

Improved the driver's debug information by the following means: Dump-Me-Now collected information, driver traces and mlx5cmd commands.

SR-IOV ATS

Added ATS support on Virtual Functions.

Rshim Drivers

Added support for NVIDIA® Mellanox® RShim drivers. The NVIDIA® Mellanox® BlueField® family of SoC devices combines an array of Arm processors coupled with the Mellanox ConnectX® interconnect. Standard Linux distributions run on the Arm cores allowing common open source development tools to be used. The SoC can be accessed via USB (external cable) or PCIe driven by our RShim drivers. RShim drivers provides functionalities like resetting the Arm cores, pushing a bootstream image, networking functionality and console functionality.

For further information see .RShim Drivers and Usage v2.80.

Dump Me Now

Added a new registry key "DumpMeNowDumpMask" to control the DMN dumps. For further information see Dump Me Now (DMN) Registry Keys.

Bug Fixes

See Bug Fixes.

Rev 2.40.51000 (DRV 2.40.22533)

Link Speed

Added support for InfiniBand link with 2x lanes width.

Software Parsing

Software Parsing is now enabled by default. Meaning, the TCP checksum over IP-in-IP encapsulation of IPv4/6 sent packets calculation is now enabled by default.

Bug Fixes

See Bug Fixes.

Rev 2.40.50000 (DRV 2.40.22511)

Adapter Cards

Added support for ConnectX-6 Dx adapter card.

VM Supported Enhancement

Allows the user to to use a single UAR resource for all Tx priorities queues on the PCI bar, to allow using up to 8 times more RSS queues.

VF Commands Failures Debuggability

The driver generates a detailed (including the command's failure details) Dump-Me-Now file in case it receives error commands from the VF.

Striding RQ

Striding RQ handles the burst of received packets with few PCI access.

For further information, see Striding RQ.

Live Firmware Patch Update (LFWP)

Firmware can be patched with critical bugs fixes live with minimal serviceability impact. The patching can be down only within the same major branch.

For further information, see Live Firmware Patch.

Relaxed Ordering (Read/Write)

Enables the Relaxed Ordering capability for Window hosts and SR-IOV VFs using a new registry key. Additionally, a second registry key allows the host to control the exposure of the relaxed ordering feature (read and write) for VFs.

For further information, see Ethernet Registry Keys.

Nested Virtualization

Added support for Mellanox VF in a VM with Hyper-V enabled

VF BlueFlame

Added support to the ND to execute BlueFlame packet from the VF to achieve low latency.

VF Counters (Anti Spoofing Counters)

Added 3 new counters (see Mellanox WinOF-2 VF Port Traffic) per Virtual Function that count the dropped packets:

  • Counter for dropped packet dues to invalid source MAC Address

  • Counter for dropped packet dues to invalid source VLAN

  • Counter for dropped packet dues to unallowed Ether type

For further information, see .Adapter Cards Counters v2.70.

PCI Device Diagnostic Counters

Added the counter below that report the number of PCI width and lane speed as part of PCI diagnostic counter. Additionally, in case of unexpected speed/width, an event log will generated.

Mellanox WinOF-2 PCI Device Diagnostic:

  • PCI link width the current width of PCIe link

  • PCI link speed the current speed of PCIe link

For further information, see .Adapter Cards Counters v2.70.

Silently Dropped Counters

Added the following counters that will count all dropped packets:

  • Mellanox WinOF-2 Port QoS:

    • Packets received discarded

  • Mellanox WinOF-2 Device Diagnostics:

    • Internal RQ out of buffer

  • Mellanox WinOF-2 VF Diagnostics:

    • Quota exceeded command

    • Send queue priority update flow

    • Mellanox WinOF-2 Port Traffic:

    • Packets Received jabbers Error

    • Packets Received Frame undersize error

    • Packets Received unsupported opcode error

    • Packets Received Frame too long Error

    • Packets Received fragments Error

For further information, see .Adapter Cards Counters v2.70.

Lossy QPs Reporting

Upon creating of a new ND or NDK QP, the driver will send a message to System Event Log if the assigned QP's priority is lossy (=disabled).
The rate of the messages can be limited using the registry parameter RoCEOnLossyPrioEvtRate. For further information, see Lossy QPs Reporting

Bug Fixes

See Bug Fixes History

Rev 2.30.51000 (DRV 2.30.21713)

Windows Client 10

See Bug Fixes History

Rev 2.30.50000 (DRV 2.30.21713)

NIC Teaming

NIC Teaming allows you to group between one and 32 physical Ethernet network adapters into one or more software-based virtual network adapters.

Note: This capability is supported only on Windows 10 Client.

For further information, refer to section NIC Teaming.

Zero Touch RoCE

Zero Touch RoCE is at GA level. For further information see Zero Touch RoCE.

Zero Touch RoCE

Added new Diagnostic Counters. For further information, refer to section Mellanox WinOF-2 Diagnostics Extension 1.

Zero Touch RoCE

Added a new parameter to configure Zero Touch RoCE Facilities. For further information, refer to section Zero Touch RoCE Parameters.

ETL

Changed the maximum size of the ETL files (MaxFileSize) to 500MB.

Counters

Added a counter for packets that their checksum was calculated by the software.

Ethernet Registry Keys

Renamed the RoceMaxFrameSize registry keys to RoceFrameSize and updated its values. For further information, refer to section Ethernet Registry Keys.

Performance Registry Keys

Added a new performance registry key (MaxCallsToNdisIndicate). For further information, refer to section Performance Registry Keys.

Adapter Cards: Link Speed

[Alpha] Added support for ConnectX-6 200GbE link speed only when in Force mode (non-Auto-Negotiation).

Adapter Cards

Updated several ConnectX-6 adapter cards description displayed in the Device Manager window.

Bug Fixes

See Bug Fixes History

Rev 2.20.50000 (DRV2.20.21096)

Zero Touch RoCE

[Beta] Zero Touch RoCE enables RoCE to operate on fabrics where no PFC nor ECN are configured. This makes RoCE configuration a breeze while still maintaining its superior high performance.

Note: This capability is disabled by default.

For further information see Zero Touch RoCE.

Driver Events (PCIe Power)

Added 3 new events that indicate the PCIe power status.

For further information see the events below in section .Reported Driver Events v2.60.

Hardware Timestamping

Hardware Timestamping is used to implement time-stamping functionality directly into the hardware of the Ethernet physical layer (PHY) using Precision Time Protocol (PTP). Time stamping is performed in the PTP stack when receiving packets from the Ethernet buffer queue.

For further information see Hardware Timestamping.

Troubleshooting (GetNetView Tool)

Added support to the mlx5cmd utility for the Microsoft SDN diagnostic script Get-NetView.

mlx5cmd (Registry Keys)

The "mlx5cmd -regkey" command displays now the minimal and maximal optional values for each registry key.

Receive Context Data (RFD)

Increased the minimum size of ReceiveBuffers to 64 to avoid extra barrier at the DataPath.

Event Log

Added new warning event log on the recommended firmware version.

If the firmware version running on the system is less than the defined firmware version it will prompt a warning event stating the minimal recommended firmware version.

Note: The driver supports older firmware version, thus updating the firmware version is not mandatory.

eSwitch Manager (BlueField)

Added support in driver to enable Arm when using BlueField adapter cards to be the eSwitch Manager (eswitch_manager).

Bug Fixes

See Bug Fixes History

© Copyright 2023, NVIDIA. Last updated on Sep 9, 2023.