DOCA Documentation v2.7.0
1.0

NVIDIA DOCA Release Notes

NVIDIA DOCA SDK release notes containing information on new features, software interoperability, and known issues.

DOCA 2.7.0 introduces NVIDIA® BlueField® networking platform enhancement for high-performance and secure AI bare-metal cloud and DOCA-Host updates for supported BlueField and NVIDIA® ConnectX® devices. With programmable congestion control (PCC) and data-path acceleration (DPA). DOCA SDK provides an extensive framework for developers.

Note

BlueField-3 devices are not supported with MLNX_OFED as the host driver and are required to use DOCA-Host.

  • Spectrum-X 1.0.1 with BlueField-3 SuperNIC

  • DOCA PCC (GA) – Added new telemetry information to the PCC application

  • DOCA Flow enhancements, including DOCA Flow Tune Server and Pipeline Visualization for debugging (alpha support)

  • Added support for DOCA Flow switch unified model

    Note

    DOCA Flow switch has now unified all the representor ports to the switch manager port for traffic management. User should only manage the pipes on the switch manager port.

    Applications must not call DPDK rte_eth_dev_start/configure/stop for VF/SF representors anymore. User should acquire packets (e.g., with DPDK but calling rte_eth_rx/tx_burst()) only with switch manager port.

  • OVS-DOCA – Unified representor for multiple ports for better resource utilization with higher scale; OVS package rename for smoother installation

  • Increased support for virtio-net VF devices on BlueField-3 networking platforms to 2K

  • DOCA HBN Service 2.2.0 enhancements, including GA-level support for Local VRF Route Leaking, EVPN Downstream VNI (DVNI) for symmetric EVPN Route Leaking, Network-to-Network Hairpin routing support on BlueField uplinks

  • SNAP Encryption at Rest with Zero-Copy: Available with BlueField-3 with SNAP 4.4.0

  • DOCA Firefly Service enhancements, including new Telco profile, ptp4l update, new Firefly servo module

  • Traffic Crypto - DOCA IPsec GA and merge into DOCA Flow, New Security protocol - PSP

  • Alpha support for new DOCA Unified Resource and Offload Management (UROM) library

  • Alpha support for DOCA Device Emulation (DevEmu) library – Emulate your own standard/non-standard devices on BlueField

  • DOCA GPUNetIO new API to support RDMA

  • DOCA Comm Channel (Comch) API update, Extend Comch to Arm↔DPA, host↔DPA.

    Note

    Comch API is being updated, the old version is in deprecation process, DOCA 2.8 will be the last version to support it.

  • DOCA Remote Direct Memory Operation (RDMO) reference application

  • Alpha support for DOCA Management Service (DMS) – simplifying BlueField post-boot provisioning and configuration using standard configuration interfaces (API/CLI)

  • DOCA NVQual – H20-BFx support, power stressors improvements

  • DOCA NVCert – BlueField-3 SuperNIC support, SPC-X support using multi-DPU (and ConnectX) and GPU direct, RDMA/TCP-OVS/IPSec and VXLAN workloads

  • Updated the default operation mode of SuperNICs to NIC mode (from DPU mode). This is relevant to the following SKUs:

    • 900-9D3B4-00CC-EA0

    • 900-9D3B4-00SC-EA0

    • 900-9D3B4-00CV-EA0

    • 900-9D3B4-00SV-EA0

    • 900-9D3B4-00EN-EA0

    • 900-9D3B4-00PN-EA0

    • 900-9D3D4-00EN-HA0

    • 900-9D3D4-00NN-HA0

      Note

      When upgrading one of these SuperNICs to 2.7.0, if its mode of operation was changed at any point in the past, then the last configured mode of operation will remain unchanged. Otherwise, the SuperNIC will rise in NIC operation mode.

  • DOCA packaging – new BlueField firmware bundle package (bf-fwbundle-<version>.prod.bfb), a smaller image for Day 2 upgrades, without the OS and DOCA runtime. Includes ATF, UEFI, nic-fw, bmc-fw, and eROT only.

  • BlueField-3 Firmware Components Upgrade – Upgrade all BlueField-3 firmware components in one upgrade flow through either bfb-install from the host (via RShim), or DPU BMC Redfish transfer BFB image

  • Update BlueField NIC-Firmware automatically as part of .bfb image upgrade

  • Improved BlueField BMC robustness –

    • Report LLDP for L2 discovery via Redfish

    • Improved BlueField DPU debuggability

  • Compilation on top of DOCA's SDK

    • DOCA 2.7 – installation now includes additional pkg-config (.pc) definitions per DOCA SDK library on top of the general doca.pc file. This is part of a deprecation process for doca.pc and a focus on modularity of DOCA's SDK.

    • Please refer to DOCA SDK reference samples and applications for an example of using the per library .pc files.

      Note

      Starting with DOCA 2.8, the general doca.pc file will be removed from the release and only files per DOCA SDK library will remain.

  • Added support for new BlueField reset and reboot procedures for loading new firmware and firmware configuration changes which replace previous need for server power cycle

Note

The format of image filenames for the BF-Bundle and DOCA-Host have been updated to the following template:

  • BF-Bundle image file format – bf-bundle-<doca_ver.LTS#>-<build#>[BUILD-LABEL]-<yy.mm>-<OS_distro>-<#os_ver>[OS-LABEL]-<unsigned/dev/prod>.<bfb/iso>

  • DOCA-Host image file format – doca-host-<doca_ver.LTS#>-<build#>[BUILD-LABEL]-<yy.mm>-<OS_distro>-<#os_ver>[OS-LABEL]-<arch>.<rpm/deb/iso>

Where:

  • <doca_ver.LTS#>-<build#> – the DOCA version with the NVIDIA build number in a x.y.z-abcd format (e.g., 2.7.0-1456). If it is an LTS release, it indicates which update number it is.

  • <yy.mm> – the year and month the image was created/released (e.g., 24.04)

  • <OS_distro>-<#os_ver> – the name and version of the operating system (e.g., ubuntu-22.04)

  • <arch>.<rpm/deb/iso> – which processor architecture is supported and how the image is packaged (e.g., x86.rpm or x86.deb)

  • <unsigned/dev/prod>.<bfb/iso> – security signature of the image (no signature, development, production) and how the image is packaged (e.g., prod.rpm or dev.rpm)

Refer to the NVIDIA DOCA Installation Guide for Linux for information on:

  • Setting up DOCA SDK on your BlueField networking platform or SmartNIC

  • Supported BlueField platforms

Note

By default, installing DOCA profiles with standard Linux tools (yum, apt) installs both doca-runtime and doca-devel (previously doca-sdk).

  • doca-runtime includes all the components, libs, drivers, and tools used in the production environment by the DOCA admin

  • doca-devel includes all the components, libs, drivers, and tools used for development, including reference applications, compilers, etc.

Starting with DOCA 2.8, the default installation of BlueField-Bundle and DOCA-Host profiles will only include DOCA runtime. doca-devel can be installed manually as needed.

Embedded DOCA Libraries

Component

Version

doca-apps

2.7.0

doca-grpc

2.7.0

doca-libs

2.7.0

ucx

1.17.0-1.2404066

gpunetio

2.7.0


Embedded DOCA Firmware Components

Component

Version

Description

ATF

v2.2(release):4.7.0-25-g5569834

Arm-trusted firmware is a reference implementation of secure world software for Arm architectures

UEFI

4.7.0-42-g13081ae

UEFI is a specification that defines the architecture of the platform firmware used for booting and its interface for interaction with the operating system

BlueField-3 NIC firmware

32.41.1000

Firmware is used to run user programs on the BlueField-3 which allow hardware to run

BlueField-2 NIC firmware

24.41.1000

Firmware is used to run user programs on the BlueField-2 which allow hardware to run

BMC firmware

24.04

BlueField BMC firmware

BlueField-3 eROT (Glacier)

cec_ota_BMGP-04.0f

BlueField-3 eROT firmware

BlueField-2 eROT (CEC)

00.02.0182.0000.n02

BlueField-2 eROT firmware

Embedded DOCA Drivers

Component

Version

Description

collectx-clxapi

1.17.0-1

A library which exposes the CollectX API, which allows any 3rd party to easily use CollectX functionality in their own programs

doca-base (MLNX_OFED)

24.04-0.6.6.0

NVIDIA® MLNX_OFED is a single software stack that operates across all NVIDIA network adapter solutions

dpacc

1.7.0-1

DPACC is a high-level compiler for the DPA processor which compiles code targeted for the data-path accelerator (DPA) processor into a device executable and generates a DPA program

dpcp

1.1.48-1.2404066

DPCP provides a unified flexible interface for programming IB devices using DevX

flexio

24.04.2148-0

FlexIO SDK exposes an API for managing the device and executing native code over the DPA processor

ibutils

2.1.1

ibdiagnet scans the fabric using directed route packets and extracts all the available information regarding its connectivity and devices.

libvma

9.8.60-1

The NVIDIA® VMA library accelerates latency-sensitive and throughput-demanding TCP and UDP socket-based applications by offloading traffic from the user-space directly to the NIC, without going through the kernel and the standard IP stack (kernel-bypass)

libxlio

3.30.5-1.2404066

The NVIDIA® XLIO software library boosts the performance of TCP/IP applications based on NGINX (CDN, DoH, etc.) and storage solutions as part of the SPDK

MFT

4.28.0-92

NVIDIA® MFT is a set of firmware management and debug tools for NVIDIA devices

mlnx-dpdk

22.11.0-2404

Equivalent to DPDK upstream. The versioning of MLNX_DPDK indicates which upstream DPDK it is compatible with it (e.g., 22.11 is compatible with upstream DPDK 2022.11).

mlnx-libsnap

1.6.0-1

Libsnap is a common library designed to assist common tasks for applications wishing to interact with emulated hardware over BlueField and take the most advantage from hardware capabilities

mlnx-snap

3.8.0-3

BlueField SNAP for NVMe and virtio-blk enables hardware-accelerated virtualization of local storage

mlx-regex

1.2-ubuntu1

RegEx is a library that provides RegEx pattern matching to DOCA applications using the regular expression processor (RXP) or software-based engines when required

OpenSM

5.19.0

InfiniBand Subnet Manager and Subnet Administrator based on OpenSM

Rivermax

1.50.7

NVIDIA® Rivermax® is an optimized networking SDK for media and data streaming applications

RShim

2.0.27

The user-space driver to access the BlueField SoC via the RShim interface, providing ways to push boot stream, debug the target, or login via the virtual console or network interface

SHARP

3.7.0

Improves the performance of MPI and Machine Learning collective operation by offloading from CPUs and GPUs to the network and eliminating the need to send data multiple times between endpoints

SPDK

23.01.5-20

SPDK provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications

Virtio-net-controller

1.9.17-1

Virtio-net-controller is a systemd service running on BlueField, with a user interface front-end to communicate with the background service

VMA

9.8.60-1

Accelerates latency-sensitive and throughput-demanding TCP and UDP socket-based applications by offloading traffic from the user-space directly to the network interface card (NIC) or Host Channel Adapter (HCA)

XLIO

3.30.5

Boosts the performance of TCP/IP applications based on NGINX (CDN, DoH, etc.) and storage solutions as part of the SPDK


DOCA Packages

Device

Component

Version

Description

Host

DOCA Devel

2.7.0

Software development kit package and tools for developing host software

DOCA Runtime

2.7.0

Runtime libraries and tools required to run DOCA-based software applications on host

DOCA Extra

2.7.0

Contains helper scripts (doca-info, doca-kernel-support)

DOCA OFED

2.7.0

Software stack which operates across all NVIDIA network adapter solutions

Arm emulated (QEMU) development container

4.7.0

Linux-based BlueField Arm emulated container for developers

Target BlueField DPU (Arm)

BlueField BSP

4.7.0

BlueField image and firmware

DOCA SDK

2.7.0

Software development kit packages and tools for developing Arm software

DOCA Runtime

2.7.0

Runtime libraries and tools required to run DOCA-based software applications on Arm

Supported Host OS per DOCA-Host Installation Profile

Note

Starting with DOCA version 2.6.0 OSs with kernel versions lower than 4.18 will no longer be supported. DOCA 2.5.0 is the last version to support OS with kernel lower than 4.18.

The default operating system included with the BlueField Bundle (for DPU and SuperNIC) is Ubuntu 22.04.

The supported operating systems on the host machine per DOCA-Host installation profile are the following:

Note

Only the following generic kernel versions are supported for DOCA local repo package for host installation.

OS

OS Version

Default Kernel Version

Arch

DOCA Profile

doca-all

doca-networking

doca-ofed

Alinux

3.2

5.10.134-13.al8.x86_64

x86

Anolis

8.6

5.10.134+

aarch64

x86

BCLinux

21.10SP2

4.19.90-2107.6.0.0098.oe1.bclinux.aarch64

aarch64

4.19.90-2107.6.0.0100.oe1.bclinux.x86_64

x86

CTYunOS

2.0

4.19.90-2102.2.0.0062.ctl2.aarch64

aarch64

4.19.90-2102.2.0.0062.ctl2.x86_64

x86

3.0 (23.01)

5.10.0-136.12.0.86.ctl3.aarch64

aarch64

5.10.0-136.12.0.86.ctl3.x86_64

x86

Debian

10.13

4.19.0-21-arm64

aarch64

4.19.0-21-amd64

x86

10.8

4.19.0-14-arm64

aarch64

4.19.0-14-amd64

x86

10.9

4.19.0-16-amd64

x86

11.3

5.10.0-13-arm64

aarch64

5.10.0-13-amd64

x86

12.1

6.1.0-10-arm64

aarch64

6.1.0-10-amd64

x86

EulerOS

2.0sp11

5.10.0-60.18.0.50.h323.eulerosv2r11.aarch64

aarch64

5.10.0-60.18.0.50.h323.eulerosv2r11.x86_64

x86

2.0sp12

5.10.0-136.12.0.86.h1032.eulerosv2r12.aarch64

aarch64

5.10.0-136.12.0.86.h1032.eulerosv2r12.x86_64

x86

Kylin

10sp2

4.19.90-24.4.v2101.ky10.aarch64

aarch64

4.19.90-24.4.v2101.ky10.x86_64

x86

10sp3

4.19.90-52.22.v2207.ky10.aarch64

aarch64

4.19.90-52.22.v2207.ky10.x86_64

x86

Mariner

2.0

5.15.118.1-1.cm2.x86_64

x86

Oracle Linux

7.9

5.4.17-2011.6.2.el7uek.x86_64

x86

8.4

5.4.17-2102.201.3.el8uek.x86_64

x86

8.6

5.4.17-2136.307.3.1.el8uek.x86_64

x86

8.7

5.15.0-3.60.5.1.el8uek.x86_64

x86

8.8

5.15.0-101.103.2.1.el8uek.x86_64

x86

9.1

5.15.0-3.60.5.1.el9uek.x86_64

x86

9.2

5.15.0-101.103.2.1.el9uek.x86_64

x86

openEuler

20.03sp3

4.19.90-2112.8.0.0131.oe1.aarch64

aarch64

4.19.90-2112.8.0.0131.oe1.x86_64

x86

22.03

5.10.0-60.18.0.50.oe2203.aarch64

aarch64

5.10.0-60.18.0.50.oe2203.x86_64

x86

RHEL/CentOS

8.0

4.18.0-80.el8.aarch64

aarch64

4.18.0-80.el8.x86_64

x86

8.1

4.18.0-147.el8.aarch64

aarch64

4.18.0-147.el8.x86_64

x86

8.2

4.18.0-193.el8.aarch64

aarch64

4.18.0-193.el8.x86_64

x86

8.3

4.18.0-240.el8.aarch64

aarch64

4.18.0-240.el8.x86_64

x86

8.4

4.18.0-305.el8.aarch64

aarch64

4.18.0-305.el8.x86_64

x86

RHEL/Rocky

8.5

4.18.0-348.el8.aarch64

aarch64

4.18.0-348.el8.x86_64

x86

8.6

4.18.0-372.41.1.el8_6.aarch64

aarch64

4.18.0-372.41.1.el8_6.x86_64

x86

8.7

4.18.0-425.14.1.el8_7.aarch64

aarch64

4.18.0-425.14.1.el8_7.x86_64

x86

8.8

4.18.0-477.10.1.el8_8.aarch64

aarch64

4.18.0-477.10.1.el8_8.x86_64

x86

8.9

4.18.0-513.5.1.el8_9.aarch64

aarch64

4.18.0-513.5.1.el8_9.x86_64

x86

9.0

5.14.0-70.46.1.el9_0.aarch64

aarch64

5.14.0-70.46.1.el9_0.x86_64

x86

9.1

5.14.0-162.19.1.el9_1.aarch64

aarch64

5.14.0-162.19.1.el9_1.x86_64

x86

9.2

5.14.0-284.11.1.el9_2.aarch64

aarch64

5.14.0-284.11.1.el9_2.x86_64

x86

9.3

5.14.0-362.8.1.el9_3.aarch64

aarch64

5.14.0-362.8.1.el9_3.x86_64

x86

SLES

15sp2

5.3.18-22-default

aarch64

x86

15sp3

5.3.18-57-default

aarch64

x86

15sp4

5.14.21-150400.22-default

aarch64

x86

15sp5

5.14.21-150500.53-default

aarch64

x86

TKLinux

3.3

5.4.119-19.0009.39

aarch64

5.4.119-19.0009.39

x86

Ubuntu

20.04

5.4.0-26-generic

aarch64

x86

22.04

5.15.0-25-generic

aarch64

x86

24.04

6.8.0-31-generic

aarch64

x86

UOS

20.1060a

5.10.0-46.uelc20.aarch64

aarch64

5.10.0-46.uelc20.x86_64

x86

20.1060e

5.10.0-46.uel20.aarch64

aarch64

5.10.0-46.uel20.x86_64

x86

XenServer

8.2

4.19.0+1

x86

BFB Version Upgrade/Downgrade

The following table provides a matrix for the supported upgrade/downgrade of BFBs across different versions.

Version

Upgrade to

Downgrade to

1.0.0

1.1.0; 1.1.1

N/A

1.1.0

1.1.1; 1.2.0

1.0.0

1.1.1

1.2.0; 1.3.0

1.1.0; 1.0.0

1.2.0

1.3.0; 1.4.0

1.1.1; 1.1.0

1.3.0

1.4.0; 1.5.0

1.2.0; 1.1.1

1.4.0

1.5.0; 2.0.2

1.3.0; 1.2.0

1.5.0

2.0.2; 2.2.0; 1.5.1; 1.5.2; 1.5.3

1.4.0; 1.3.0

1.5.1

1.5.2

1.5.0

1.5.2

1.5.3

1.5.1; 1.5.0

1.5.3

N/A

1.5.2; 1.5.0

2.0.2

2.2.0; 2.5.0

1.5.0; 1.4.0

2.2.0

2.5.0; 2.6.0

N/A

2.2.1

2.5.0; 2.6.0

N/A

2.5.0

2.5.1; 2.6.0

2.2.1 for BlueField-3; 2.2.0 for BlueField-2

2.5.1

N/A

2.5.0

2.6.0

2.7.0

2.5.0; 2.2.1 for BlueField-3; 2.2.0 for BlueField-2

2.7.0

N/A

2.6.0; 2.5.0; 2.2.1 for BlueField-3; 2.2.0 for BlueField-2


Supported DOCA Version Upgrade Using Standard Linux Tools on BlueField

Version

Upgrade to

2.5.0

2.5.1; 2.6.0

2.5.1

N/A

2.6.0

2.7.0


Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods:

Customers who purchased NVIDIA M-1 Global Support Services, please see your contract for details regarding Technical Support.

Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller.

Info

For questions, comments, and feedback, please contact us at DOCA-Feedback@exchange.nvidia.com.

The following table lists the known issues and limitations for this release of DOCA SDK.

Reference

Description

3863011

Description: Restarting openibd results in kernel panic after modifying kernel IPsec configurations.

Workaround: N/A

Keyword: IPsec

Reported in version: 2.7.0

3882794

Description: When working with doca_pcc_np context, the return value from the API doca_pcc_get_max_num_threads() is incorrect. The function has an output parameter that indicates the maximum number of threads allowed for a doca_pcc_np context. The correct value that the library expects is 16 instead of the returned 64.

Workaround: N/A

Keyword: PCC; threads

Reported in version: 2.7.0

3886674

Description: Installing doca-all and other DOCA metapackages does not install the mlnx-nvme driver.

Workaround: mlnx-nvme is only needed for NVMe-over-RDMA remote storage support. If you wish to install it, add the mlnx-nvme package to the install command.

  • On RHEL:

    Copy
    Copied!
                

    apt install doca-all mlnx-nvme-modules

  • On Ubuntu:

    Copy
    Copied!
                

    dnf install doca-all-kmod-mlnx-nvme

Keyword: NVMe; DOCA profile

Reported in version: 2.7.0

3885930

Description: When installing DOCA-Host on a system using NVMe storage (typically local NVMe disk), and the script doca-kernel-support is used to rebuild and install kernel modules, unloading the mlx5 drivers is only possible after also unmounting the NVMe storage, which would typically necessitate a reboot.

Workaround: N/A

Keyword: NVMe; doca-kernel-support; DOCA for host

Reported in version: 2.7.0

3886315

Description: To reset or shut down the BlueField Arm, it is mandatory to specify the --sync 0 argument with reset level 1 and reset type 3 or 4. For example:

Copy
Copied!
            

mlxfwreset -d <device> -l 1 -t 4 --sync 0 r

Workaround: N/A

Keyword: Arm; shutdown

Reported in version: 2.7.0

3837255

Description: When running Arm shutdown from the host OS it is expected to get the message -E- Failed to send Register MRSI. This message should be ignored.

Workaround: Wait 2 more minutes before rebooting the host. Before proceeding with host OS reboot, it is recommended to query the operational state of the BlueField Arm cores from the BlueField BMC to verify that shutdown state has been reached. Run the following command:

Copy
Copied!
            

ipmitool -C 17 -I lanplus -H <bmc_ip> -U root -P <password> raw 0x32 0xA3

Expected output is "06".

Keyword: Host OS; reboot; error

Reported in version: 2.7.0

3881941

Description: When working with RShim 2.0.28, PCIe host crash may rarely occur at the beginning of BFB push after the Arm reset.

Workaround: Downgrade to RShim 2.0.27 or upgrade to RShim 2.0.29.

Keyword: RShim; driver

Reported in version: 2.7.0

3844705

Description: In OpenEuler 20.03, the Linux Kernel version 4.19.90 is affected by an issue that impacts the discard/trim functionality for the BlueField eMMC device which may cause degraded performance of the BlueField eMMC over time.

Workaround: Upgrade to Linux Kernel version 5.10 or later.

Keyword: eMMC discard; trim functionality

Reported in version: 2.7.0

3877725

Description: During BFB installation in NIC mode on BlueField-3, too much information is added into RShim log which fills it, causing the Linux installation progress log to not appear in the RShim log.

Copy
Copied!
            

echo "DISPLAY_LEVEL 2" > /dev/rshim0/misc cat /dev/rshim0/misc

Workaround: Monitor the BlueField-3 Arm's UART console to check whether BFB installation has completed or not for NIC mode.

Copy
Copied!
            

[13:58:39] INFO: Installation finished ... [14:01:53] INFO: Rebooting...

Keyword: NIC mode; BFB install

Reported in version: 2.7.0

3855702

Description: Trying to jump from a steering level in the hardware to a lower level using software steering is not supported on rdma-core lower than 48.x.

Workaround: N/A

Keyword: RDMA; SWS

Reported in version: 2.7.0

3855485

Description: When enabling the PCI_SWITCH_EMULATION_ENABLE NVconfig, the mlx devices, and potentially the RShim devices disappear. Also, looking at the kernel logs using dmesg shows the following messages:

Copy
Copied!
            

pci 0000:29:00.0: BAR 0: no space for [mem size 0x0200 0000 64bit pref] pci 0000:29:00.0: BAR 2: no space for [mem size 0x0080 0000 64bit pref] ...

Workaround: N/A

Keyword: NVconfig; RShim; dmsg

Reported in version: 2.7.0

3831230

Description: In OpenEuler 20.03, the Linux Kernel version 4.19.90 is affected by an issue that impacts the discard/trim functionality for BlueField eMMC device which may cause degraded performance of BlueField eMMC over time.

Workaround: Upgrade to Linux Kernel version 5.10 or later.

Keyword: eMMC discard; trim functionality

Reported in version: 2.7.0

3743879

Description: mlxfwreset could timeout on servers where the RShim driver is running and INTx is not supported. The following error message is printed: BF reset flow encountered a failure due to a reset state error of negotiation timeout.

Workaround: Set PCIE_HAS_VFIO=0 and PCIE_HAS_UIO=0 in /etc/rshim.conf and restart the RShim driver. Then re-run the mlxfwreset command.

If host Linux kernel lockdown is enabled, then manually unbind the RShim driver before mlxfwreset and bind it back after mlxfwreset:

Copy
Copied!
            

echo "DROP_MODE 1" > /dev/rshim0/misc mlxfwreset <arguments> echo "DROP_MODE 0" > /dev/rshim0/misc

Keyword: Timeout; mlxfwreset; INTx

Reported in version: 2.7.0

3869639

Description: Users c annot use --job-output-buffer-size 0 when using remote output memory (--use-remote-output-buffers).

Workaround:

Keyword: DOCA Bench

Reported in version: 2.7.0

3872654

Description: And i ssue occurs when submitting tasks with DOCA SHA with the following error.

Copy
Copied!
            

[DOCA][ERR][doca_pe.cpp:177][task_submit] Task 0xaaaaf4865bf0: Failed to submit task: task is already submitted

Workaround: Reattempt the submit u sing a different --data-provider-job-count value. This workaround may also fail.

Keyword: DOCA Bench

Reported in version: 2.7.0

3859823

Description: Multi-threaded tests using DOCA Comch may hang or emit an infinite amount of log messages. Single-threaded tests are less likely to cause this issue.

Workaround: N/A

Keyword: DOCA Bench; DOCA Comch

Reported in version: 2.7.0

3857095

Description: Send tasks on DOCA RDMA may fail.

Workaround: N/A

Keyword: DOCA Bench; DOCA RDMA; send

Reported in version: 2.7.0

3857097

Description: DOCA RDMA tests cannot be launched from BlueField side.

Workaround: N/A

Keyword: DOCA Bench; DOCA RDMA

Reported in version: 2.7.0

3849701

Description: DOCA Comch tests can not be launched from BlueField side .

Workaround: N/A

Keyword: DOCA Bench; DOCA Comch

Reported in version: 2.7.0

3840230

Description: Order of cores specified in --core-list is not respected. Cores are picked in ascending order instead.

Workaround: N/A

Keyword: DOCA Bench

Reported in version: 2.7.0

3665070

Description: Virtio-net controller fails to load if DPA_AUTHENTICATION is enabled.

Workaround: N/A

Keyword: Virtio-net; DPA

Reported in version: 2.5.0

3678069

Description: If using BlueField with NVMe and mmcbld and configured to boot from mmcblk, users must create bf.cfg file with device=/dev/mmcblk0, then install the *.bfb as normal.

Workaround: N/A

Keyword: NVMe

Reported in version: 2.5.0

3680538

Description: When using strongSwan or OVS-IPsec as explained in the NVIDIA BlueField DPU BSP, the IPSec Rx data path is not offloaded to hardware and occurs in software running on the Arm cores. As a result, bandwidth performance is substantially low.

Workaround: N/A

Keyword: IPsec

Reported in version: 2.5.0

N/A

Description: Execution unit partitions are still not implemented and would be added in a future release.

Workaround: N/A

Keyword: EU tool

Reported in version: 2.5.0

3666160

Description: Installing BFB using bfb-install when mlxconfig PF_TOTAL_SF>1700, triggers server reboot immediately.

Workaround: Change PF_TOTAL_SF to 0, perform a graceful shutdown, power cycle, then installing BFB.

Keyword: SF; PF_TOTAL_SF; BFB installation

Reported in version: 2.2.1

3594836

Description: When enabling Flex IO SDK tracer at high rates, a slow-down in processing may occur and/or some traces may be lost.

Workaround: Keep tracing limited to ~1M traces per second to avoid a significant processing slow-down. Use tracer for debug purposes and consider disabling it by default.

Keyword: Tracer FlexIO

Reported in version: 2.2.1

3592080

Description: When using UEK8 on the host in DPU mode, creating a VF on the host consumes about 100MB memory on BlueField

Workaround: N/A

Keyword: UEK; VF

Reported in version: 2.2.1

3566042

Description: Virtio hotplug is not supported in GPU-HOST mode on the NVIDIA Converged Accelerator.

Workaround: N/A

Keyword: Virtio; Converged Accelerator

Reported in version: 2.2.0

3546474

Description: PXE boot over ConnectX interface might not work due to an invalid MAC address in the UEFI boot entry.

Workaround: On BlueField, create /etc/bf.cfg file with the relevant PXE boot entries, then run the command bfcfg.

Keyword: PXE; boot; MAC

Reported in version: 2.2.0

3561723

Description: Running mlxfwreset sync 1 on NVIDIA Converged Accelerators may be reported as supported although it is not. Executing the reset will fail.

Workaround: N/A

Keywords: mlxfwreset

Reported in version: 2.2.0

3546202

Description: After rebooting a BlueField-3 DPU running Rocky Linux 8.6 BFB, the kernel log shows the following error:

Copy
Copied!
            

[    3.787135] mlxbf_gige MLNXBF17:00: Error getting PHY irq. Use polling instead

This message indicates that the Ethernet driver will function normally in all aspects, except that PHY polling is enabled.

Workaround: N/A

Keywords: Linux; PHY; kernel

Reported in version: 2.2.0

3306489

Description: When performing longevity tests (e.g., mlxfwreset, DPU reboot, burning of new BFBs), a host running an Intel CPU may observer errors related to "CPU 0: Machine Check Exception".

Workaround: Add intel_idle.max_cstate=1 entry to the kernel command line.

Keywords: Longevity; mlxfwreset; DPU reboot

Reported in version: 2.2.0

3529297

Description: Enhanced NIC mode is not supported on BlueField-2.

Workaround: N/A

Keywords: Operation; mode

Reported in version: 2.2.0

3538486

Description: When removing LAG configuration from BlueField, a kernel warning for uverbs_destroy_ufile_hw is observed if virtio-net-controller is still running.

Workaround: Stop virtio-net-controller service before cleaning up bond configuration.

Keywords: Virtio-net; LAG

Reported in version: 2.2.0

3534219

Description: On BlueField-3 devices, from DOCA 2.2.0 to 32.37.1306 (or lower), the host crashes when executing partial Arm reset (e.g., Arm reboot; BFB push; mlxfwreset).

Workaround: Before downgrading the firmware:

  1. Run:

    Copy
    Copied!
                

    echo 0 > /sys/bus/platform/drivers/mlxbf-bootctl/large_icm

  2. Reboot Arm.

Keyword: BlueField-3; downgrade

Reported in version: 2.2.0

3462630

When trying to perform a PXE installation when UEFI Secure Boot is enabled, the following error messages may be observed:

Copy
Copied!
            

error: shim_lock protocol not found. error: you need to load the kernel first.

Workaround: Download a Grub EFI binary from the Ubuntu website. For further information on Ubuntu UEFI Secure Boot PXE Boot, please visit Ubuntu's official website.

Keyword: PXE; UEFI Secure Boot

Reported in version: 2.0.2

3448841

Description: While running CentOS 8.2, switchdev Ethernet BlueField runs in "shared" RDMA net namespace mode instead of "exclusive".

Workaround: Use ib_core module parameter netns_mode=0. For example:

Copy
Copied!
            

echo "options ib_core netns_mode=0" >> /etc/modprobe.d/mlnx-bf.conf

Keyword: RDMA; isolation; Net NS

Reported in version: 2.0.2

2706803

Description: When an NVMe controller, SoC management controller, and DMA controller are configured, the maximum number of VFs is limited to 124.

Workaround: N/A

Keyword: VF; limitation

Reported in version: 2.0.2

3273435

Description: Changing the mode of operation between NIC and DPU modes results in different capabilities for the host driver which might cause unexpected behavior.

Workaround: Reload the host driver or reboot the host.

Keyword: Modes of operation; driver

Reported in version: 2.0.2

3264749

Description: In Rocky and CentOS 8.2 inbox-kernel BFBs, RegEx requires the following extra huge page configuration for it to function properly:

Copy
Copied!
            

sudo hugeadm --pool-pages-min DEFAULT:2048M sudo systemctl start mlx-regex.service systemctl status mlx-regex.service

If these commands have executed successfully you should see active (running) in the last line of the output.

Workaround: N/A

Keyword: RegEx; hugepages

Reported in version: 1.5.1

3240153

Description: DOCA kernel support only works on a non-default kernel.

Workaround: N/A

Keyword: Kernel

Reported in version: 1.5.0

3217627

Description: The doca_devinfo_rep_list_create API returns success on the host instead of Operation not supported.

Workaround: N/A

Keyword: DOCA core; InfiniBand

Reported in version: 1.5.0

© Copyright 2024, NVIDIA. Last updated on Aug 15, 2024.