NVIDIA ConnectX-6 Dx Adapter Cards Firmware Release Notes v22.35.3006 LTS
NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

Changes and New Feature History

Warning

This section includes history of changes and new feature of 3 major releases back. For older releases history, please refer to the relevant firmware versions.

Feature/Change

Description

22.35.2302

Bug Fixes

See Bug Fixes in this Firmware Version section.

Feature/Change

Description

22.35.2000

PCC Algorithm

Enables the users to collect more information from NP to RP for PCC algorithm. To achieve this, the NP ingress bytes information was added to the RTT response packet sent from the NP side.

HPCC: Support per-IP and per-QP Methods

Enables the user to configure the PCC algorithm shaper coalescing mode using nvconfig to select CC algorithm shaper coalescing for IB and ROCE.
The new parameters are IB_CC_SHAPER_COALESCE and ROCE_CC_SHAPER_COALESCE.

Bug Fixes

See Bug Fixes in this Firmware Version section.

Feature/Change

Description

22.35.1012

HPCC, Programmable Congestion Control

HPCC related configurations in is now supported via the mlxconfig utility.

UDP

Added support for copy modify header steering action to/from the UDP field.

Range based Lookup

Added support for range based lookup. This new capability is available using the following new PRM command:

GENERATE WQE which receives GTA WQE, the command supports "match on range" and num_hash_definer=[1,2] and num_match_ste=[1,2].

For further information, refer to section "RTC Object Format" in the PRM.

RoCE based VM Migration

Added support for RoCE based VM migration.

Resource Dump

Added the following resource dump segments:

  • SEG_HW_STE_FULL that includes dump to STE and all its dependencies

  • SEG_FW_STE_FULL that include dump to FW_STE and to HW_STE_FULL in range

Striding WQE - Headroom and Tail-room

As the software requires additional space before and after a packet is scattered for its processing for stridden RQ, the hardware will allocate the required room while scattering packets to spare a copy.

Connections per Second (CPS)

Improved security offload's Connections per Second (CPS) rate using the general object DEK (PSP TLS etc).

VF Migration Flow

Added support for pre-copy commands in VF migration flow in order to reduce the migration downtime.

VF Migration Flow

Optimized performance to support full VF migration flow.

VirtIO vDPA Performance Virtualization

Increased the VirtIO hardware offload message rate to 20/20 MPPS for 256 virtual devices by optimizing the datapath application code.

RoCE: Adaptive Timer

Enabled ADP timer to allow the user to configure RC or DC qp_timeout values lower than 16.

QoS Priority Trust Default State

QoS priority trust default state can now be changed using the new nvconfig below:

  • QOS_TRUST_STATE_P1

  • QOS_TRUST_STATE_P2

The values that can be used to set the default state are:

  • TRUST_PORT

  • TRUST_PCP

  • TRUST_DSCP

  • TRUST_DSCP_PCP

Bug Fixes

See Bug Fixes section.

22.34.4000

Bug Fixes

See Bug Fixes section.

22.34.1002

LLDP Properties Implementation on RDE

Added LLDPEnable, LLDPTransmit and LLDPReceive properties to the RDE Port schema implementation.

PPS Offset

Added a 22 nanosecond of propagation delay to the cable delay of the PPS signal when using PPS out.

Programmable CC, PPCC, MAD, IBCC

Added support for PPCC register with bulk operations, MAD for algorithm configuration and tunable parameters.

Programmable Counters

Added support for programmable counters for PCC via PPCC register and MAD.

RX Rate-limit in Multi-Host

Added support for RX multi-host rate limit using an enabler script.

Queue Counters Allocation

This new capability allows privileged users to allocate queue counters. In this new feature the get_max_qp_cnt_cur_cap() returns a valid value when the UID is with UCTX_CAP_INTERNAL_DEVICE_RESOURCES, otherwise it returns 0.

Bug Fixes

See Bug Fixes section.

22.33.1048

200Gb/s Throughput on Crypto Capable Devices

Enabled 200Gb/s out-of-the-box throughput on crypto capable devices.

Note: If any crypto offloads is in use, 200Gb/s throughput can be achieved only after the next firmware reset

VF Migration

Added support for VF migration. The hypervisor can now suspend its VF, meaning from that point the VF cannot perform action such as send/receive traffic or run any command. In this firmware version only the suspend resume mode is supported (on the same VM).

MADs

Added a new MAD of class SMP that has the attributes hierarchy_Info as defined in the IB Specification and is used to query the hierarchy information stored on the node and the physical port.

NV Configurations via the Relevant Reset Flow

Added pci_rescan_needed field to the MFRL access register to indicate whether a PCI rescan is needed based on the NV configurations issued by the software.

Note: If the Keep Link Up NV configuration is changed, phyless reset will be blocked.

Precision Time Protocol (PTP)

Added Precision Time Protocol (PTP) support.

In this version, the support includes:

  • 16 PTP SQs only

  • only 2 ports

  • only RT clock mode

In this version, the following are not supported:

  • PTP packet drop

  • PTP SQ on VF

Note: All PTP SQs must be closed before operating LFWP (life fw patch).

Resource Dump Support for HW Steering

Added support for HW Steering objects dump via resource dump interface.
This support includes: STC, RTC, STE, modify argument, and modify pattern.

VF Migration

Added support for VF migration.

ICM Pages

Added a new register (vhca_icm_ctrl access_reg) to enable querying and limiting the ICM pages in use.

Steering Definer

Added support for creating a steering definer with a dword selector using create_match_definer_object and the "SELECT" format.

XRQ QP Errors Enhancements

Enhanced the XRQ QP error information provided to the user in case QP goes into an error state. In such case, QUERY_QP will provide information on the syndrome type and which side caused
the error.

HW Steering: WQE Insertion Rules

[Beta] Added HW Steering support for the following:

  • set, add and copy inline STC action

  • set and copy actions for several fields using modify_pattern object and inline stc modify action

  • FDB mode in HW steering using FDB_RX and FDB_TX flow table types

  • ASO flow meter action via STC

  • flow counter query using ASO WQE

  • allocation of large bulks for the objects: STE, ASO flow meter and modify argument

  • jumbo match RTC

  • count action in STC

Holdover Mode

Added support for holdover mode to comply to SyncE specifications (EEC compliance) to limit the maximum phase transient response upon link loss.

SyncE Enhancements

Added support for noise filtering to comply to the SyncE specifications requirements.

vDPA: Performance

Optimized the performance of virtio including: throughput, QoS, and accuracy of min/max bandwidth when virtio works with the QoS settings.

vDPA: virtio-net Full Emulation

This new capability reduces the switchover time of creating a virtq from scratch during live migration, by creating the virtq beforehand on the target server.

When swithover happens, the pre-created virtq will be used and modified with necessary parameters.

ibstat

Updated the ibstat status reported when the phy link is down. Now QUERY_VPORT_STATE.max_tx_speed of UPLINK will not be reported as 0 anymore.

NetworkPort Schema Replacement

Replaced the deprecated NetworkPort schema with Port schema in NIC RDE implementation.

Firmware Steering

Enabled the option to modify the ip_ecn field in the packet header in firmware steering.

ZTRCC

Added support for advanced ZTR_RTTCC algorithm based on the Programmable CC platform to achieve better congestion control without dependency on the switch ECN marking.

Dynamic Completion Event Moderation for vDPA

DIM is used to tune moderation parameter dynamically using an mlxreg command.

To disable this capability, run:

mlxreg -d /dev/mst/mt41686_pciconf0 --reg_id 0xc00d --reg_len 0x8 -s "0x4.1:1=0x0"

SW Steering Cache

Modified the TX or RX cache invalidation behavior. TX or RX cache invalidation now does not occur automatically but only when the software performs the sync operation using the using sync_steering command.

Mega Allocations in Bulk Allocator Mechanism

Modified the maximum bulk size per single allocation from "log_table_size - log_num_unisizes", to allocate any range size, to remove limitations that HWS objects such as counters and modify arguments might encounter.

Dynamic Flex Parser over a VF

Added support for creating a dynamic flex parser on untrusted function, and changed the flex parser cap for untrusted function to the following:

  • maximum flex parser node = 2

  • maximum dw sample = 4

SNAPI: Comm-Channel

Added support for SNAPI (comm-channel) connection while running on raw ETH link.

Changing all the Crypto Features to Wrapped or Cleartext

Crypto features can be in either wrapped or unwrapped mode. Meaning, the key can be wrapped or in plaintext when running the CREATE_DEK PRM command. To comply with the requirements specified in FIPS publication, all the created DEKs must be wrapped.

This feature adds new NV_CONFIG per device to control this mode, and enables the user to change all the crypto features to wrapped or cleartext.

ICM Direct Access by the Software to write/modify the DEK Objects

[Beta] This new capability enables the software to directly access ICM and write/modify the DEK objects. Such change improves the DEK object update rate by re-using DEK object instead of creating a new one.

In addition, added the following:

  • New for DEK object: bulk allocation, modify_dek cmd, and new mode - sw_wrapped.

  • New general object INT_KEK

Bug Fixes

See Bug Fixes section.

22.32.2004

Dynamic Fan Control

Dynamic Fan Control enables cooling down the card and modules by maintaining the minimal possible noise and power consumption.

Bug Fixes

See Bug Fixes section.

22.32.1010

Single PF per NUMA

Added support for BMC with a single PF per NUMA in Socket-Direct adapter cards.

Note: SIngle PF per NUMA should not be enabled on Multi-host.

OpenSNAPI Communication Channel

The communication channel is used to enable communication between processes on different vHCAs regardless of their network connectivity state.

RegC Register

Exposed an additional steering register in the hardware (reg_c_6).

QP Resources

Added a new NvConfig parameter LOG_MAX_QUEUE to set the maximum number of work queue resources (QP, RQ, SQ...) that can be created per function.

The default value is 2^17.

Congestion Control Key

Added a Congestion Control Key to all Congestion Control MADs to authenticate that they are originated from a trusted source.

SMP Firewall

Added an SMP firewall to block the option of sending SMPs (MADS sent on QP0 from the Subnet Manager) from unauthorized hosts to prevent fake SMPs from being recognized as the SM.

Vendor Specific MADs: Class 0x9

Vendor Specific MADs Class 0x9 is no longer supported by the firmware. If case the firmware detects such MAD, the firmware will return a "NOT SUPPORTED" error to the user.

Match Definer Object

Added support for a new steering match definer format (format 33).

Graceful Teardown

The teardown of hotplugged emulated device (a.k.a unplug flow) is in the reverse order of the plug flow. However, certain legacy host software stack does not support surprise removal of the PCIe PF devices.

To support such host software stack, emulation manager software will perform a graceful teardown.

AES-XTS Encryption / Decryption

Enabled disk encryption services using the aes_xts protocol to allow inline data encryption and decryption towards a remote or a local disk/NVDIM.

TLS/XTS/Signature Padding

Blocked the VF's ability to use both padding and signature in order to prevent the NIC from hanging.

Asserts' Severity Level

Added 3 new assert filters (Health buffer, NVlog, FW trace). The assert will be exposed now if its severity level is equal to or above the new filter.

The filters are configurable by the ini file. The "Health buffer" filter is also configurable by new access register.

SyncE

Added support for clock frequency synchronization based on Synchronous Ethernet protocol.

Note: This capability is not supported with link speeds of 50G and higher, and cannot run in parallel with diagnostic counters.

Socket-Direct Adapter Cards

Added support for:

  • Flow Direct

  • LACP

  • GRE Offload

Rate Limit per VM instead of VM-TC

Enabled Rate Limit per VM instead of VM-TC. This capability is implemented by adding support to a new Scheduling element type: rate limit elements that will connect to the rate_limit and will share its rate limit.

Cross GVMI Memory Key

Cross GVMI memory key is used to allow cross GVMI memory access using indirect memory registration which crosses vHCA context.

Steering LAG Mode (Hash LAG)

[Beta] The new LAG mode (PORT_SELECT_FT LAG (hash LAG)) distributes the packets to ports according to the hash on the packet headers, instead of distributing the packets according to the QP (queue affinity – legacy LAG) to avoid cases where the slow/fast path packets are transmitted from different ports.

Identifying the right port is done by using destination type UPLINK with destination_eswitch_owner_vhca_id_valid set and destination_eswitch_owner_vhca_id indicating the PF associated with the port.

The below are the Queue Affinity and Steering LAG (hash) limitations:

  • Queue Affinity (legacy LAG) limitation:

    • LAG cannot be created when other functions on eSwitch are active (VFs, SFs, and x86 PF for SmartNIC).

      Note: This limitation does not exist in older firmware versions (xxx.31.1014 and below) when setting mlxconfig HIDE_PORT2_PF parameter for SmartNICs. This solution is no longer applicable when using firmware version xx.31.2xxx.

  • Steering LAG (hash) limitations:

    • LAG cannot be created when other functions on eSwitch are active (VFs, SFs, and x86 PF for smartnic)

      Note: This limitation can be solved by setting LAG_RESOURCE_ALLOCATION=1

    • Cannot create PORT_SELECT_FT LAG when the SQs on PF (connected to strict TIS) are opened.

      Note: This limitation can be solved by setting LAG_RESOURCE_ALLOCATION=1

      Note: It is recommended to set LAG_RESOURCE_ALLOCATION=1 before configuring the PORT_SELECT_FT lag. LAG_RESOURCE_ALLOCATION will pre-allocate needed firmware and hardware resources in order to CREATE_LAG with less limitations.

    • Can be used only with Software Steering due to a bug with strict SQs in Firmware Steering.

Note: Due to changes in this feature, transmission timestamp in CQE is temporarily unsupported with multi eSwitch.

QSHR Access Register

Added support for QSHR access register to enable Set and Query rate limit per-host per-port.

New Software Steering ICM Resource for VXLAN Encapsulation

The firmware now exposes a new Software Steering ICM resource for VXLAN encap expand in order for the SW Steering to manage this resource directly.

Asymmetrical VFs per PF

Added support for asymmetrical VFs per PF.

To enable it:
PF_NUM_OF_VF_VALID must be true, and PF_NUM_OF_VF to a none zero value.

mlxlink Support to read/write Access Registers by LID

Added 2 new MAD access registers to enable mlxlink to read/write access registers by LID (to the whole subnet).

Virtio-net Full Emulation

Enabled the option to dynamically modify the MSIX and the number of virtio VF device queues.

Note: This modification must be done before loading the driver on the device.

This new capability includes the following limitations:

  • total queue/msix number can not exceed 2k

  • queue/msix per virtio vf device cannot exceed 64

  • the scale of virtio device is limit to below 127 from mlxconfig

Bug Fixes

See Bug Fixes section.

22.31.1014

Using NC-SI Commands for Debugging PCI Link Failures

Implemented a new NC-SI command get_debug_info to get mstdump via the NC-SI protocol to debug a device if the PCI link fails for any given reason.

Eye Margin Measurement

[Beta] Added support for SLRED grade register to provide grade criteria for the link.

Clear Phy Counters x seconds after Link is Up

Phy counters are cleared x seconds after link is up. This feature can be configured and enabled via the ini fields using the NVconfig. The timeout to wait after the link is up before cleaning the counters is configured via NVconfig.

Enable/Disable RDMA via the UEFI HII System Settings

Added support for Enabling/Disabling NIC and RDMA (port/partition) via the UEFI HII system settings.

Note: Values set in this option only take effect when is Ethernet mode.

Single PF per NUMA in Socket-Direct

Added support for a single PF per NUMA for Socket-Direct without a host management.

Dynamic Flex Parser

Extended the Dynamic Flex Parser capabilities in order to support RTP packets parsing.

Tunnel Header Matcher

Added support for full tunnel header matcher to VXLAN header (including VXLAN alert bit).

NC-SI Speed Reporting

Updated the NC-SI speed reporting output to support 200GbE speed. Now when running the NC-SI command, the output presents 200GbE speed as well.

Increased the Maximum Number of MSIX per VF

Increased the maximum number of MSIX per VF to 127.
Note that increasing the number of MSIX per VF (NUM_VF_MSIX) affects the configured number of VFs (NUM_OF_VFS). The firmware may reduce the configured number of MSIX per VF and/or the number of VFs with respect to maximum number of MSIX vectors supported by the device (MAX_TOTAL_MSIX).

Running NVME Traffic on another vHCA than the Emulation Manager vHCA

NVME traffic can run now on another vHCA than the emulation manager vHCA. NVME SQ continues to be created on the emulation manager vHCA and an Alias object is created on the transport vHCA. The NVME SQ BE object is connected then to the Alias object with the QP provided.

Note: With this change, the emulation manager capability for NVME is limited to only PF VHCA.

Extended MSIX

Allows the device to be configured with more than 256 MSIX vectors per physical PCI function.

To use it, please follow the Asymmetrical MSIX configuration (PF_NUM_PF_MSIX_VALID) (see feature Asymmetrical MSIX Configuration below).

Note: When using this feature while Sub-Function configuration (PF_BAR2_ENABLE / PER_PF_NUM_SF) is enabled, it is recommended to configure “above 4G-decoding” in the BIOS due to device BAR space requirements.

Asymmetrical MSIX Configuration

Allows the device to be configured with a different number of MSIX vectors per physical PCI functions.

To use this feature, please follow these steps:

  1. Clear NUM_PF_MSIX_VALID to disable global symmetrical MSIX configuration.

  2. Set PF_NUM_PF_MSIX_VALID to enable asymmetrical per Physical Function MSIX configuration.

  3. Configure PF_NUM_PF_MSIX per physical PCI function.

Notes:

  • When using this configuration, NUM_PF_MSIX (the symmetrical MSIX configuration field) will be ignored by the firmware although it will be present in the mlxconfig query.

  • The asymmetrical MSIX configuration and the legacy symmetrical MSIX configuration are mutually exclusive. When both are enabled, the firmware will work according to the symmetrical MSIX configuration.

  • Step #3 should be done on each of the physical PCI functions, otherwise, the device will use default configurations.

Adaptive Routing (AR): multi_path, data_in_order

Added a new bit ("data_in_order") to query the QP and allow a process/library to detect when the AR is enabled.

flex_parser for GENEVE Hardware Offload and ICMP

Added a new flex parser to support GENEVE hardware offload and ICMP.

Non-Page-Supplier-FLR

When the non-page-supplier-FLR funcion is initiated, the firmware triggers a page event to the page supplier to indicate that all pages should be returned for the FLR function.

Pages are returned by the driver to the kernel without issuing the MANAGE_PAGES commands to the firmare.

PCIe Eye Opening

Enabled measuring PCIe eye dynamic grading over PCIe Gen3 speed.

Real Time Clock through VF

Enabled hardware real time clock for virtualized environment.

Steering

Added support for matching field ipv4_ihl in create_flow_group and set_flow_table_entry commands.

Bug Fixes

See Bug Fixes.

22.30.1004

Precision Time Protocol (PTP)

Enabled Precision Time Protocol (PTP) timestamp (UTC mode) in Multi-Host devices.

PAM4

Added support for PAM4 Auto Negotiation and Link Training in 200GbE link speed.

RoCE, Lossy, slow_restart_idle

Removed triggering unexpected internal CNPs for RoCE Lossy slow_restart_idle feature.

KR-Startup in Auto-Negotiation

Enabled KR-Startup in Auto-Negotiation mode for PAM4.

Performance: Steering

Added support for a new NV config mode “icm_cache_mode_large_scale_steering” that enables less cache misses and improves performance for cases when working with many steering rules.
This capability is enabled using the mlxconfig parameter "ICM_CACHE_MODE".

Active-State Power Management (ASPM)

Added support for power saving in L1 ASPM link state.

Rx Link-Tuning

Added support for Rx Link-Tuning in PAM4 Serdes speeds.

QoS

This new capability allows the user to configure QoS (Rate Limit / BW share) between a network and the storage in a SmartNIC.

Origin bit in the ctrl Segment of the WQE

This new capability adds an origin bit in the ctrl segment of the WQE sent by the Target to the Initiator during frontend controller exceptions in the NVMeoF interface. The WQE sent with the origin bit "set" will have SWQE_OPCODE_SEND_IMM opcode.

Connection Tracking Window Validation

Enabled connection tracking window validation by supporting CONN_TRACK_OFFLOAD general object and WQE with opcode ASO and opmode CONN_TRACK.

Transport Layer Security (TLS) Offload

Enabled TLS offload v1.3 with the key size of 256.

VF/VF-group rate-limiting

This new capability enables VF/VF-group rate-limiting while per-host rate-limiter is also applied.

Sub-Function Scalability

Expanded the Sub-Function capability to support the scale of up to 512 by supporting more MSIX, removing internal obstacles and reducing the ICM memory footprint.

NvConfig: Sub-Functions

This new capability enables asymmetric Sub-Function configuration. It expands the asymmetric Sub-Function configuration to support asymmetric configurations between all PFs, and provides Sub-Function scalability and asymmetric NV configurations.

Match Definer Object

Added support for new steering match definer formats.

  • Match definer format 26

  • Match definer format 27

Steering by Sanity Check Flags

Added support for matching the following flags: l3_ok, l4_ok, ipv4_checksum_ok, l4_checksum_ok bits in create_flow_group and set_flow_table_entry commands, as well as in match definer formats (formats 0,1,2 and 3) to be used in matching actions on the packet headers.

Virtio Full Emulation

Added support for up to 500 virtio-net VFs for full emulation.

Bug Fixes

See Bug Fixes.

22.29.2002

PAM4

PAM4 200GbE linkup time improvement, the l inkup time is now sub 5 seconds.

Remote Loopback in NRZ

[Beta] Enabled remote (PMA) loopback in NRZ, Rx-to-Tx.

Note: To use the PMA loopback, both sides should be in Force Mode (AN Disabled).

Reserved QPN

[Beta] This capability allows the software to reserve a QPN that can be used to establish connection performed over RDMA_CM, and provide the software a unique QP number.

Since RDMA_CM does not support DC, by using CREATE_QPN_RESERVED_OBJECT the software can reserve a QPN value from the firmware's managed QP number namespace range. This allows multiple software processes to hold a unique QPN value instead of using UD-QPs.

Bug Fixes

See Bug Fixes section.

22.29.1016

Eye-Opening

[Beta] Eye-opening is supported only when using NRZ signal.

Multi-Application QoS per QP

Added the option to allow applications to build their own QoS tree over the NIC hierarchy by connecting QPs to responder/requestor Queue Groups.

NRZ Link Performance

Improved NRZ link performance (RX algorithm).

NRZ Link-Up Time

Improved NRZ link-up time (25G\50G\100G speeds).

InfiniBand Support in RDE

Added "InfiniBand" properties set to the Network Device Function Redfish object.

Direct Packet Placement (DPP)

Added support for Direct Packet Placement (DPP). DPP is a receive side transport service in which the Ethernet packets are scattered to the memory according to a packet sequence number (PSN) carried by the packet, and not by their arrival order. To enable DPP offload, the software should create a special RQ by using the CREATE_RQ command, and set DPP relevant attributes.

HW Offloads Enablement on VF

Added trust level for VFs. Once the VF is trusted, it will get a set of trusted capabilities.

vDPA Statistics

Added support for error statistics and reporting for vDPA.

IPv6 TSO Offload

Added support for IPv6 TSO offload.

Enabling Adaptive-Routing (AR) for the Right SL via UCX

UCX can now enable AR by exposing Out-Of-Ordering bitmask per SL with "ooo_per_sl" field in the HCA_VPORT context. It can be also queried by running the QUERY_HCA_VPORT_CONTEXT command.

vDPA Performance

Optimized vDPA performance for PPS in scalability of up to 127 devices per port.

External SmartNIC (ECPF)

When using a Multi-Host device, the firmware can now discover who is the external SmartNIC host (the Embedded CPU PF (ECPF) ) with the highest index that is responsible for performing offload activities to the adapter card.

Steering Dump

Hardware steering dump output used for debugging and troubleshooting.

Please see Known Issue 2213356 for its limitations.

22.28.4000

PAM4 Link Performance

PAM4 link performance improvement.

PAM4 Link Maintenance

Improved PAM4 link maintenance that improved pre-FEC BER (support VL mapping on different physical lanes).

Ethernet wqe_too_small Mode

Added a new counter per vPort that counts the number of packets that reached the Ethernet RQ but cannot fit into the WQE due to their large size. Additionally, we added the option to control if such packet will cause “CQE with Error” or “CQE_MOCK”.

Access Registries

ignore_flow_level is now enabled by the TRUST LEVEL access registry.

Pause Frames from VFs

[Beta] Enabled the capability to allow Virtual Functions to send Pause Frames packets.

Counters

Added support for the cq_overrun counter. The counter represents the number of times CQs enter an error state due to overflow that occur when the device tries to post a CQE into a full CQ buffer.

Bug Fixes

See Bug Fixes.

22.28.2006

Sub Function (SF) BAR Size

Increased the minimum Sub Function (SF) BAR size from 128KB to 256KB. Due to the larger SF BAR size, for the same PF BAR2 size, which can be queried/modified by LOG_PF_BAR2_SIZE NV config, the firmware will support half of the SFs.

To maintain the same amount of supported SFs, software needs to increase the LOG_PF_BAR2_SIZE NV config value by 1.

GPUDirect in Virtualized Environment

Enabled a direct access to ATS from the NIC to GPU buffers using PCIe peer-to-peer transactions. To enable this capability, the “p2p_ordering_mode” parameter was added to the NV_PCI_CONF configuration.

Note: When SECURE_ALL or SECURE_TRUST is configured, ATS and RO must be set identically. When SECURE_NONE is configured, ATS and RO may be set independently as the current firmware behavior allows.

Non-Volatile Configurations

Added a new Non-Volatile Configuration parameter to control VL15 buffer size (VL15_BUFFER_SIZE).

Note: VL15 buffer size enlargement will decrease all other VLs buffers size.

NC-SI

Added a new NC-SI command (get_device_id) to report a unique device identifier.

NC-SI

Added new NC-SI commands (get_lldp_nb, set_lldp_nb) to query the current status of LLDP and to enable/disable it.

ROCE ACCL

Split the SlowRestart ROCE_ACCL into the following:

  • slow-restart – used to reduce rate on retransmission events

  • slow-restart-after-idle – used to reduce rate before first transmission after >1s without transmitting

ROCE ACCL

Enabled TX PSN window size configuration using LOG_TX_PSN_WINDOW NVconfig parameter.

Note: Due to hardware limitations, max log_tx_psn_win value can be set 9.

Bug Fixes

See Bug Fixes.

22.28.1002

GPRS Tunneling Protocol

Added support for GPRS Tunneling Protocol (GTP).

IPSec Full Offload

Enabled with IPsec transport mode (ESP header over IP header). All the bellow settings are done by the hardware (full offload):

  • packet re-formatting, adding/deleting ESP header/trailer, and SN configuration

  • encryption and decryption

Flow Sampling and Mirroring

Added support for flow sampling and mirroring with an associated capable software.

NC-SI 1.2 New Command

Implemented the following new command from NS-SI 1.2 specification:

  • Get PF Assignment

vDPA Bonding

Added support for vDPA bonding on two PFs.

Flow Metering

[Beta] Added support for Flow Metering.

SerDes Lane Receive Eye Diagram (SLRD)

Added a new register to read the SerDes receive eye diagram for each eye.

PTP

Updated the Packet Pacing clock to be in sync with the PTP clock.

PTP

Added support for hardware real time clock by UTC timestamp in PCIE BAR and CQE.

Enhanced Connection Establishment (ECE)

ECE is new negotiation scheme used to exchange extra information about QPs capabilities and later negotiate them at the connection establishment phase. It is used to support various features such as RoCE Selective Repeat and PCC.

RoCE Selective Repeat using Enhanced Connection Establishment (ECE)

RoCE Selective Repeat activation now is available using ECE. This feature is enabled automatically using ECE if NV_config.sr_enable parameter is set 1 otherwise using go back N retransmission.

This new capability comes with the following limitations:

  • Selective Repeat results in packets arriving out of order

  • Signature is not supported with Selective Repeat.

  • Need to create mkey with signature before QPs are opened in gvmi

Non-Volatile Configurations (NVCONFIG)

Added the following new mlxconfig parameters to the Non-Volatile Configurations section.

  • log_max_outstandng_wqe

  • ece_disable_mask

  • NV_config.sr_enable

Hardware Tag Matching

Increased the maximum XRQ number to 512.

Bug Fixes

See Bug Fixes.

22.27.6008

PAM4 Link Speeds when Using 400GbE/200GbE

The following are the minimal software/firmware versions that support PAM4 link speeds when connected using Mellanox NIC to Mellanox Switch and Mellanox NIC to 3rd Party Switches:

  • Mellanox Spectrum-3: 30.2007.1142

  • Mellanox Spectrum-2: 29.2007.1142

  • Switch SDK: 4.4.0920

  • Mellanox Onyx: 3.9.0830-038

  • SONiC/SAI: 201911

  • ConnectX-6 Dx: 22.27.2008*

*Note: NICs with this firmware version support Mellanox-to-Mellanox connectivity with PAM4 link speeds

Programmable Congestion Control (CC)

Programmable Congestion Control (PCC) is at GA level.

TLS

Enabled TLS offload v1.3 with key size 256.

vDPA

Added support for vDPA including LSO and csum offload.

Resourcedump

Added the following segments, as appeared in the PRM, to the Resource Dump:

  • PRM_QUERY_QP

  • PRM_QUERY_CQ

  • PRM_QUERY_MKEY

  • QUERY_VNIC_ENV

Bug Fixes

See Bug Fixes.

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.