NVIDIA DOCA Release Notes

doca-apps

2.7.0

doca-grpc

2.7.0

doca-libs

2.7.0

ucx

1.17.0-1.2404066

gpunetio

2.7.0

ATF

v2.2(release):4.7.0-25-g5569834

Arm-trusted firmware is a reference implementation of secure world software for Arm architectures

UEFI

4.7.0-42-g13081ae

UEFI is a specification that defines the architecture of the platform firmware used for booting and its interface for interaction with the operating system

BlueField-3 NIC firmware

32.41.1000

Firmware is used to run user programs on the BlueField-3 which allow hardware to run

BlueField-2 NIC firmware

24.41.1000

Firmware is used to run user programs on the BlueField-2 which allow hardware to run

BMC firmware

24.04

BlueField BMC firmware

BlueField-3 eROT (Glacier)

cec_ota_BMGP-04.0f

BlueField-3 eROT firmware

BlueField-2 eROT (CEC)

00.02.0182.0000.n02

BlueField-2 eROT firmware

collectx-clxapi

1.17.0-1

A library which exposes the CollectX API, which allows any 3^rd party to easily use CollectX functionality in their own programs

doca-base (MLNX_OFED)

24.04-0.6.6.0

NVIDIA® MLNX_OFED is a single software stack that operates across all NVIDIA network adapter solutions

dpacc

1.7.0-1

DPACC is a high-level compiler for the DPA processor which compiles code targeted for the data-path accelerator (DPA) processor into a device executable and generates a DPA program

dpcp

1.1.48-1.2404066

DPCP provides a unified flexible interface for programming IB devices using DevX

flexio

24.04.2148-0

FlexIO SDK exposes an API for managing the device and executing native code over the DPA processor

ibutils

2.1.1

ibdiagnet scans the fabric using directed route packets and extracts all the available information regarding its connectivity and devices.

libvma

9.8.60-1

The NVIDIA® VMA library accelerates latency-sensitive and throughput-demanding TCP and UDP socket-based applications by offloading traffic from the user-space directly to the NIC, without going through the kernel and the standard IP stack (kernel-bypass)

libxlio

3.30.5-1.2404066

The NVIDIA® XLIO software library boosts the performance of TCP/IP applications based on NGINX (CDN, DoH, etc.) and storage solutions as part of the SPDK

MFT

4.28.0-92

NVIDIA^® MFT is a set of firmware management and debug tools for NVIDIA devices

mlnx-dpdk

22.11.0-2404

Equivalent to DPDK upstream. The versioning of MLNX_DPDK indicates which upstream DPDK it is compatible with it (e.g., 22.11 is compatible with upstream DPDK 2022.11).

mlnx-libsnap

1.6.0-1

Libsnap is a common library designed to assist common tasks for applications wishing to interact with emulated hardware over BlueField and take the most advantage from hardware capabilities

mlnx-snap

3.8.0-3

BlueField SNAP for NVMe and virtio-blk enables hardware-accelerated virtualization of local storage

mlx-regex

1.2-ubuntu1

RegEx is a library that provides RegEx pattern matching to DOCA applications using the regular expression processor (RXP) or software-based engines when required

OpenSM

5.19.0

InfiniBand Subnet Manager and Subnet Administrator based on OpenSM

Rivermax

1.50.7

NVIDIA® Rivermax® is an optimized networking SDK for media and data streaming applications

RShim

2.0.27

The user-space driver to access the BlueField SoC via the RShim interface, providing ways to push boot stream, debug the target, or login via the virtual console or network interface

SHARP

3.7.0

Improves the performance of MPI and Machine Learning collective operation by offloading from CPUs and GPUs to the network and eliminating the need to send data multiple times between endpoints

SPDK

23.01.5-20

SPDK provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications

Virtio-net-controller

1.9.17-1

Virtio-net-controller is a systemd service running on BlueField, with a user interface front-end to communicate with the background service

VMA

9.8.60-1

Accelerates latency-sensitive and throughput-demanding TCP and UDP socket-based applications by offloading traffic from the user-space directly to the network interface card (NIC) or Host Channel Adapter (HCA)

XLIO

3.30.5

Boosts the performance of TCP/IP applications based on NGINX (CDN, DoH, etc.) and storage solutions as part of the SPDK

Host

DOCA Devel

2.7.0

Software development kit package and tools for developing host software

DOCA Runtime

2.7.0

Runtime libraries and tools required to run DOCA-based software applications on host

DOCA Extra

2.7.0

Contains helper scripts (doca-info, doca-kernel-support)

DOCA OFED

2.7.0

Software stack which operates across all NVIDIA network adapter solutions

Arm emulated (QEMU) development container

4.7.0

Linux-based BlueField Arm emulated container for developers

Target BlueField DPU (Arm)

BlueField BSP

4.7.0

BlueField image and firmware

DOCA SDK

2.7.0

Software development kit packages and tools for developing Arm software

DOCA Runtime

2.7.0

Runtime libraries and tools required to run DOCA-based software applications on Arm

1.0.0

1.1.0; 1.1.1

N/A

1.1.0

1.1.1; 1.2.0

1.0.0

1.1.1

1.2.0; 1.3.0

1.1.0; 1.0.0

1.2.0

1.3.0; 1.4.0

1.1.1; 1.1.0

1.3.0

1.4.0; 1.5.0

1.2.0; 1.1.1

1.4.0

1.5.0; 2.0.2

1.3.0; 1.2.0

1.5.0

2.0.2; 2.2.0; 1.5.1; 1.5.2; 1.5.3

1.4.0; 1.3.0

1.5.1

1.5.2

1.5.0

1.5.2

1.5.3

1.5.1; 1.5.0

1.5.3

N/A

1.5.2; 1.5.0

2.0.2

2.2.0; 2.5.0

1.5.0; 1.4.0

2.2.0

2.5.0; 2.6.0

N/A

2.2.1

2.5.0; 2.6.0

N/A

2.5.0

2.5.1; 2.6.0

2.2.1 for BlueField-3; 2.2.0 for BlueField-2

2.5.1

N/A

2.5.0

2.6.0

2.7.0

2.5.0; 2.2.1 for BlueField-3; 2.2.0 for BlueField-2

2.7.0

N/A

2.6.0; 2.5.0; 2.2.1 for BlueField-3; 2.2.0 for BlueField-2

3882794

Description: When working with doca_pcc_np context, the return value from the API doca_pcc_get_max_num_threads() is incorrect. The function has an output parameter that indicates the maximum number of threads allowed for a doca_pcc_np context. The correct value that the library expects is 16 instead of the returned 64.

Workaround: N/A

Keyword: PCC; threads

Reported in version: 2.7.0

3886674

Description: Installing doca-all and other DOCA metapackages does not install the mlnx-nvme driver.

Workaround: mlnx-nvme is only needed for NVMe-over-RDMA remote storage support. If you wish to install it, add the mlnx-nvme package to the install command.

• On RHEL:

  Copy

  Copied!

              
  
              
  apt install doca-all mlnx-nvme-modules
          

• On Ubuntu:

  Copy

  Copied!

              
  
              
  dnf install doca-all-kmod-mlnx-nvme
          

Keyword: NVMe; DOCA profile

Reported in version: 2.7.0

3885930

Description: When installing DOCA-Host on a system using NVMe storage (typically local NVMe disk), and the script doca-kernel-support is used to rebuild and install kernel modules, unloading the mlx5 drivers is only possible after also unmounting the NVMe storage, which would typically necessitate a reboot.

Workaround: N/A

Keyword: NVMe; doca-kernel-support; DOCA for host

Reported in version: 2.7.0

3886315

Description: To reset or shut down the BlueField Arm, it is mandatory to specify the --sync 0 argument with reset level 1 and reset type 3 or 4. For example:

            

            
mlxfwreset -d <device> -l 1 -t 4 --sync 0 r
        

Workaround: N/A

Keyword: Arm; shutdown

Reported in version: 2.7.0

3837255

Description: When running Arm shutdown from the host OS it is expected to get the message -E- Failed to send Register MRSI. This message should be ignored.

Workaround: Wait 2 more minutes before rebooting the host. Before proceeding with host OS reboot, it is recommended to query the operational state of the BlueField Arm cores from the BlueField BMC to verify that shutdown state has been reached. Run the following command:

            

            
ipmitool -C 17 -I lanplus -H <bmc_ip> -U root -P <password> raw 0x32 0xA3
        

Expected output is "06".

Keyword: Host OS; reboot; error

Reported in version: 2.7.0

3881941

Description: When working with RShim 2.0.28, PCIe host crash may rarely occur at the beginning of BFB push after the Arm reset.

Workaround: Downgrade to RShim 2.0.27 or upgrade to RShim 2.0.29.

Keyword: RShim; driver

Reported in version: 2.7.0

3844705

Description: In OpenEuler 20.03, the Linux Kernel version 4.19.90 is affected by an issue that impacts the discard/trim functionality for the BlueField eMMC device which may cause degraded performance of the BlueField eMMC over time.

Workaround: Upgrade to Linux Kernel version 5.10 or later.

Keyword: eMMC discard; trim functionality

Reported in version: 2.7.0

3877725

Description: During BFB installation in NIC mode on BlueField-3, too much information is added into RShim log which fills it, causing the Linux installation progress log to not appear in the RShim log.

            

            
echo "DISPLAY_LEVEL 2" > /dev/rshim0/misc
cat /dev/rshim0/misc
        

Workaround: Monitor the BlueField-3 Arm's UART console to check whether BFB installation has completed or not for NIC mode.

            

            
[13:58:39] INFO: Installation finished
...
[14:01:53] INFO: Rebooting...
        

Keyword: NIC mode; BFB install

Reported in version: 2.7.0

3855702

Description: Trying to jump from a steering level in the hardware to a lower level using software steering is not supported on rdma-core lower than 48.x.

Workaround: N/A

Keyword: RDMA; SWS

Reported in version: 2.7.0

3855485

Description: When enabling the PCI_SWITCH_EMULATION_ENABLE NVconfig, the mlx devices, and potentially the RShim devices disappear. Also, looking at the kernel logs using dmesg shows the following messages:

            

            
pci 0000:29:00.0: BAR 0: no space for [mem size 0x0200 0000 64bit pref]
pci 0000:29:00.0: BAR 2: no space for [mem size 0x0080 0000 64bit pref]
...
        

Workaround: N/A

Keyword: NVconfig; RShim; dmsg

Reported in version: 2.7.0

3831230

Description: In OpenEuler 20.03, the Linux Kernel version 4.19.90 is affected by an issue that impacts the discard/trim functionality for BlueField eMMC device which may cause degraded performance of BlueField eMMC over time.

Workaround: Upgrade to Linux Kernel version 5.10 or later.

Keyword: eMMC discard; trim functionality

Reported in version: 2.7.0

3743879

Description: mlxfwreset could timeout on servers where the RShim driver is running and INTx is not supported. The following error message is printed: BF reset flow encountered a failure due to a reset state error of negotiation timeout.

Workaround: Set PCIE_HAS_VFIO=0 and PCIE_HAS_UIO=0 in /etc/rshim.conf and restart the RShim driver. Then re-run the mlxfwreset command.

If host Linux kernel lockdown is enabled, then manually unbind the RShim driver before mlxfwreset and bind it back after mlxfwreset:

            

            
echo "DROP_MODE 1" > /dev/rshim0/misc
mlxfwreset <arguments>
echo "DROP_MODE 0" > /dev/rshim0/misc
        

Keyword: Timeout; mlxfwreset; INTx

Reported in version: 2.7.0

3869639

Description: Users c annot use --job-output-buffer-size 0 when using remote output memory (--use-remote-output-buffers).

Workaround:

Keyword: DOCA Bench

Reported in version: 2.7.0

3872654

Description: And i ssue occurs when submitting tasks with DOCA SHA with the following error.

            

            
[DOCA][ERR][doca_pe.cpp:177][task_submit] Task 0xaaaaf4865bf0: Failed to submit task: task is already submitted
        

Workaround: Reattempt the submit u sing a different --data-provider-job-count value. This workaround may also fail.

Keyword: DOCA Bench

Reported in version: 2.7.0

3859823

Description: Multi-threaded tests using DOCA Comch may hang or emit an infinite amount of log messages. Single-threaded tests are less likely to cause this issue.

Workaround: N/A

Keyword: DOCA Bench; DOCA Comch

Reported in version: 2.7.0

3857095

Description: Send tasks on DOCA RDMA may fail.

Workaround: N/A

Keyword: DOCA Bench; DOCA RDMA; send

Reported in version: 2.7.0

3857097

Description: DOCA RDMA tests cannot be launched from BlueField side.

Workaround: N/A

Keyword: DOCA Bench; DOCA RDMA

Reported in version: 2.7.0

3849701

Description: DOCA Comch tests can not be launched from BlueField side .

Workaround: N/A

Keyword: DOCA Bench; DOCA Comch

Reported in version: 2.7.0

3840230

Description: Order of cores specified in --core-list is not respected. Cores are picked in ascending order instead.

Workaround: N/A

Keyword: DOCA Bench

Reported in version: 2.7.0

3665070

Description: Virtio-net controller fails to load if DPA_AUTHENTICATION is enabled.

Workaround: N/A

Keyword: Virtio-net; DPA

Reported in version: 2.5.0

3678069

Description: If using BlueField with NVMe and mmcbld and configured to boot from mmcblk, users must create bf.cfg file with device=/dev/mmcblk0, then install the *.bfb as normal.

Workaround: N/A

Keyword: NVMe

Reported in version: 2.5.0

3680538

Description: When using strongSwan or OVS-IPsec as explained in the NVIDIA BlueField DPU BSP, the IPSec Rx data path is not offloaded to hardware and occurs in software running on the Arm cores. As a result, bandwidth performance is substantially low.

Workaround: N/A

Keyword: IPsec

Reported in version: 2.5.0

N/A

Description: Execution unit partitions are still not implemented and would be added in a future release.

Workaround: N/A

Keyword: EU tool

Reported in version: 2.5.0

3666160

Description: Installing BFB using bfb-install when mlxconfig PF_TOTAL_SF>1700, triggers server reboot immediately.

Workaround: Change PF_TOTAL_SF to 0, perform a graceful shutdown, power cycle, then installing BFB.

Keyword: SF; PF_TOTAL_SF; BFB installation

Reported in version: 2.2.1

3594836

Description: When enabling Flex IO SDK tracer at high rates, a slow-down in processing may occur and/or some traces may be lost.

Workaround: Keep tracing limited to ~1M traces per second to avoid a significant processing slow-down. Use tracer for debug purposes and consider disabling it by default.

Keyword: Tracer FlexIO

Reported in version: 2.2.1

3592080

Description: When using UEK8 on the host in DPU mode, creating a VF on the host consumes about 100MB memory on BlueField

Workaround: N/A

Keyword: UEK; VF

Reported in version: 2.2.1

3566042

Description: Virtio hotplug is not supported in GPU-HOST mode on the NVIDIA Converged Accelerator.

Workaround: N/A

Keyword: Virtio; Converged Accelerator

Reported in version: 2.2.0

3546474

Description: PXE boot over ConnectX interface might not work due to an invalid MAC address in the UEFI boot entry.

Workaround: On BlueField, create /etc/bf.cfg file with the relevant PXE boot entries, then run the command bfcfg.

Keyword: PXE; boot; MAC

Reported in version: 2.2.0

3561723

Description: Running mlxfwreset sync 1 on NVIDIA Converged Accelerators may be reported as supported although it is not. Executing the reset will fail.

Workaround: N/A

Keywords: mlxfwreset

Reported in version: 2.2.0

3546202

Description: After rebooting a BlueField-3 DPU running Rocky Linux 8.6 BFB, the kernel log shows the following error:

            

            
[    3.787135] mlxbf_gige MLNXBF17:00: Error getting PHY irq. Use polling instead
        

This message indicates that the Ethernet driver will function normally in all aspects, except that PHY polling is enabled.

Workaround: N/A

Keywords: Linux; PHY; kernel

Reported in version: 2.2.0

3306489

Description: When performing longevity tests (e.g., mlxfwreset, DPU reboot, burning of new BFBs), a host running an Intel CPU may observer errors related to "CPU 0: Machine Check Exception".

Workaround: Add intel_idle.max_cstate=1 entry to the kernel command line.

Keywords: Longevity; mlxfwreset; DPU reboot

Reported in version: 2.2.0

3529297

Description: Enhanced NIC mode is not supported on BlueField-2.

Workaround: N/A

Keywords: Operation; mode

Reported in version: 2.2.0

3538486

Description: When removing LAG configuration from BlueField, a kernel warning for uverbs_destroy_ufile_hw is observed if virtio-net-controller is still running.

Workaround: Stop virtio-net-controller service before cleaning up bond configuration.

Keywords: Virtio-net; LAG

Reported in version: 2.2.0

3534219

Description: On BlueField-3 devices, from DOCA 2.2.0 to 32.37.1306 (or lower), the host crashes when executing partial Arm reset (e.g., Arm reboot; BFB push; mlxfwreset).

Workaround: Before downgrading the firmware:

1. Run:

   Copy

   Copied!

               
   
               
   echo 0 > /sys/bus/platform/drivers/mlxbf-bootctl/large_icm
           

2. Reboot Arm.

Keyword: BlueField-3; downgrade

Reported in version: 2.2.0

3462630

When trying to perform a PXE installation when UEFI Secure Boot is enabled, the following error messages may be observed:

            

            
error: shim_lock protocol not found.
error: you need to load the kernel first.
        

Workaround: Download a Grub EFI binary from the Ubuntu website. For further information on Ubuntu UEFI Secure Boot PXE Boot, please visit Ubuntu's official website.

Keyword: PXE; UEFI Secure Boot

Reported in version: 2.0.2

3448841

Description: While running CentOS 8.2, switchdev Ethernet BlueField runs in "shared" RDMA net namespace mode instead of "exclusive".

Workaround: Use ib_core module parameter netns_mode=0. For example:

            

            
echo "options ib_core netns_mode=0" >> /etc/modprobe.d/mlnx-bf.conf
        

Keyword: RDMA; isolation; Net NS

Reported in version: 2.0.2

2706803

Description: When an NVMe controller, SoC management controller, and DMA controller are configured, the maximum number of VFs is limited to 124.

Workaround: N/A

Keyword: VF; limitation

Reported in version: 2.0.2

3273435

Description: Changing the mode of operation between NIC and DPU modes results in different capabilities for the host driver which might cause unexpected behavior.

Workaround: Reload the host driver or reboot the host.

Keyword: Modes of operation; driver

Reported in version: 2.0.2

3264749

Description: In Rocky and CentOS 8.2 inbox-kernel BFBs, RegEx requires the following extra huge page configuration for it to function properly:

            

            
sudo hugeadm --pool-pages-min DEFAULT:2048M
sudo systemctl start mlx-regex.service
systemctl status mlx-regex.service
        

If these commands have executed successfully you should see active (running) in the last line of the output.

Workaround: N/A

Keyword: RegEx; hugepages

Reported in version: 1.5.1

3240153

Description: DOCA kernel support only works on a non-default kernel.

Workaround: N/A

Keyword: Kernel

Reported in version: 1.5.0

3217627

Description: The doca_devinfo_rep_list_create API returns success on the host instead of Operation not supported.

Workaround: N/A

Keyword: DOCA core; InfiniBand

Reported in version: 1.5.0