NVIDIA DOCA Release Notes

NVIDIA DOCA SDK release notes containing information on new features, bug fixes, and known issues.

DOCA 2.6.0 introduces NVIDIA® BlueField® networking platform enhancement for high-performance and secure AI bare-metal cloud. With programmable congestion control (PCC) and data-path acceleration (DPA). DOCA SDK provides an extensive framework for developers.

Warning

Starting from DOCA version 2.6 OSs with kernel versions lower than 4.18 will no longer be supported.

  • Spectrum-X with BlueField SuperNIC Reference Architecture

  • DOCA Flow Enhancements – Improved match and compare capabilities, DOCA Connection Tracking improvements (NAT64, modify multiple fields, performance), Geneve tunneling (support for modifying option fields, exposing HW hash calculation to SW)

  • DOCA DPA – Added beta-level debugger support

  • DOCA GPUNetIO – included in BlueField bundle image (BFB); can be executed on BlueField Arm cores

  • DOCA services enhancements

    • DOCA HBN – L3 EVPN enhancements, VRF route leaking (beta), multi-hop eBGP peering for EVPN, L3 interface binding for stateful ACLs, and additional DPU-facing interfaces

    • DOCA Firefly service – time check for clock jump

  • BlueField platform enhancements

    • Platform security – UEFI password policy

    • Platform management – BMC robustness

  • DOCA NVQual – memory stressor with DMA test between BlueField-3 and the host

  • DOCA NVCert – ConnectX coverage, new basic (FIO) storage workload

Refer to the NVIDIA DOCA Installation Guide for Linux for information on:

  • Setting up DOCA SDK on your BlueField DPU, BlueField SuperNIC, or SmartNIC

  • Supported BlueField platforms

Embedded DOCA Libraries

Component

Version

doca-apps

2.6.0058-1

doca-grpc

2.6.0058-1

doca-libs

2.6.0058-1

ucx

1.15.0-1.2307050

gpunetio

2.6.0058-1


Embedded DOCA Firmware Components

Component

Version

Description

ATF

4.6.0-18-g620c01e

Arm-trusted firmware is a reference implementation of secure world software for Arm architectures

UEFI

4.6.0-17-g45c0149

UEFI is a specification that defines the architecture of the platform firmware used for booting and its interface for interaction with the operating system

BlueField-3 NIC firmware

32.40.1000

Firmware is used to run user programs on the BlueField-3 which allow hardware to run

BlueField-2 NIC firmware

24.40.1000

Firmware is used to run user programs on the BlueField-2 which allow hardware to run

ConnectX-7 firmware

28.40.1000

Firmware is used to run user programs on the ConnectX-7 which allow hardware to run

ConnectX-6 Lx firmware

26.40.1000

Firmware is used to run user programs on the ConnectX-6 Lx which allow hardware to run

ConnectX-6 Dx firmware

22.40.1000

Firmware is used to run user programs on the ConnectX-6 Dx or which allow hardware to run

ConnectX-6 De firmware

22.40.1000

Firmware is used to run user programs on the ConnectX-6 De which allow hardware to run

ConnectX-6 firmware

20.40.1000

Firmware is used to run user programs on the ConnectX-6 which allow hardware to run

BMC firmware

24.01

BlueField BMC firmware

BlueField-3 eROT (Glacier)

00.02.0152.0000

BlueField-3 eROT firmware

BlueField-2 eROT (CEC)

04.0f

BlueField-2 eROT firmware

Embedded DOCA Drivers

Component

Version

Description

collectx-clxapi

1.16.0

A library which exposes the CollectX API, which allows any 3rd party to easily use CollectX functionality in their own programs

doca-base (MLNX_OFED)

24.01-0.3.3.0

NVIDIA® MLNX_OFED is a single software stack that operates across all NVIDIA network adapter solutions

dpacc

1.6.1-35

DPACC is a high-level compiler for the DPA processor which compiles code targeted for the data-path accelerator (DPA) processor into a device executable and generates a DPA program

dpcp

1.1.46-1.2401026

DPCP provides a unified flexible interface for programming IB devices using DevX

flexio

24.01.2011

FlexIO SDK exposes an API for managing the device and executing native code over the DPA processor

ibutils

2.16

ibdiagnet scans the fabric using directed route packets and extracts all the available information regarding its connectivity and devices.

libvma

9.8.51-1

The NVIDIA® VMA library accelerates latency-sensitive and throughput-demanding TCP and UDP socket-based applications by offloading traffic from the user-space directly to the NIC, without going through the kernel and the standard IP stack (kernel-bypass)

libxlio

3.20.8-1.2310120

The NVIDIA® XLIO software library boosts the performance of TCP/IP applications based on NGINX (CDN, DoH, etc.) and storage solutions as part of the SPDK

MFT

4.27.0-83

NVIDIA® MFT is a set of firmware management and debug tools for NVIDIA devices

mlnx-dpdk

22.11.0-2401

Equivalent to DPDK upstream. The versioning of MLNX_DPDK indicates which upstream DPDK it is compatible with it (e.g., 22.11 is compatible with upstream DPDK 2022.11).

mlnx-libsnap

1.6.0-1

Libsnap is a common library designed to assist common tasks for applications wishing to interact with emulated hardware over BlueField DPUs and take the most advantage from hardware capabilities

mlnx-snap

3.8.0-3

BlueField SNAP for NVMe and virtio-blk enables hardware-accelerated virtualization of local storage

mlx-regex

1.2-ubuntu1

RegEx is a library that provides RegEx pattern matching to DOCA applications using the regular expression processor (RXP) or software-based engines when required

OpenSM

5.18.0

InfiniBand Subnet Manager and Subnet Administrator based on OpenSM

Rivermax

1.41.9

NVIDIA® Rivermax® is an optimized networking SDK for media and data streaming applications

RShim

2.0.17-0.g0caa378

The user-space driver to access the BlueField SoC via the RShim interface, providing ways to push boot stream, debug the target, or login via the virtual console or network interface

SHARP

3.6.0

Improves the performance of MPI and Machine Learning collective operation by offloading from CPUs and GPUs to the network and eliminating the need to send data multiple times between endpoints

SPDK

23.01.5-20

SPDK provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications

Virtio-net-controller

1.8.8-1

Virtio-net-controller is a systemd service running on the DPU, with a user interface front-end to communicate with the background service

VMA

9.8.51

Accelerates latency-sensitive and throughput-demanding TCP and UDP socket-based applications by offloading traffic from the user-space directly to the network interface card (NIC) or Host Channel Adapter (HCA)

XLIO

3.21.2

Boosts the performance of TCP/IP applications based on NGINX (CDN, DoH, etc.) and storage solutions as part of the SPDK


DOCA Packages

Device

Component

Version

Description

Host

DOCA SDK

2.6.0

Software development kit package for developing host software

DOCA Runtime

2.6.0

Runtime libraries required to run DOCA-based software applications on host

DOCA Tools

2.6.0

Tools for developers and administrators on host

DOCA Extra

2.6.0

Contains helper scripts (doca-info, doca-kernel-support)

DOCA OFED

2.6.0

Software stack which operates across all NVIDIA network adapter solutions

Arm emulated (QEMU) development container

4.6.0

Linux-based BlueField Arm emulated container for developers

Target BlueField DPU (Arm)

BlueField BSP

4.6.0

BlueField image and firmware

DOCA SDK

2.6.0

Software development kit packages for developing Arm software

DOCA Runtime

2.6.0

Runtime libraries required to run DOCA-based software applications on Arm

DOCA Tools

2.6.0

Tools for developers and administrators for Arm target

Supported Operating System Distributions

Warning

Starting from DOCA version 2.6 OSs with kernel versions lower than 4.18 will no longer be supported. DOCA 2.5 is the last version to support OS with kernel lower than 4.18.

The default operating system of the BlueField DPU (Arm) is Ubuntu 22.04.

The supported operating systems on the host machine per DOCA profile are the following:

Warning

Only the following generic kernel versions are supported for DOCA local repo package for host installation.

DOCA for Host

Kernel

Arch

doca-all

doca-cx

doca-ofed

CTYunOS3 23.01

5.10

aarch64

RHEL/CentOS 8.2

4.18

x86

RHEL/Rocky 8.6

4.18

x86

Ubuntu 20.04

5.4

x86

Ubuntu 22.04

5.15

x86

5.15

aarch64

Debian 10.8

4.19

x86

Debian 10.13

5.10.135 / 5.4.210

x86

Allinux 3.2

5.10

X86

Oracle Linux 8.7

5.15

x86

RHEL/Rocky Linux 9.1

5.14

x86

BCLinux 21.10 SP2

4.19.90

x86 / aarch64

CTYunOS2.0

4.19.90

x86 / aarch64

Debian10.9

4.19.0-16

x86

Debian11.3

5.10.0-13

x86 / aarch64

Debian12.1

6.1.0-10

x86 / aarch64

Kylin 10 SP2

4.19.90

x86 / aarch64

Oracle Linux 8.6

5.4

x86

openEuler 20.03 SP3

4.19.90

x86 / aarch64

openEuler 22.03

5.10.0

x86 / aarch64

RHEL/CentOS 8.0

4.18.0-80.el8

x86

RHEL/CentOS 8.2

4.18.0-193.el8

aarch64

RHEL/CentOS 8.4

4.18.0-305.el8

x86 / aarch64

RHEL/Rocky 8.6

4.18.0-372.41.1.el8

aarch64

RHEL/Rocky 8.8

4.18.0-477.10.1.el8_8

x86 / aarch64

RHEL/Rocky 8.9

4.18.0-513.5.1.el8_9

x86 / aarch64

RHEL/Rocky 9.0

5.14.0-70.46.1.el9_0

x86 / aarch64

RHEL/Rocky 9.1

5.14.0-162.19.1.el9_1

x86 / aarch64

RHEL/Rocky 9.2

5.14.0-284.11.1.el9_2

x86 / aarch64

RHEL/Rocky 9.3

5.14.0-362.8.1.el9_3

x86 / aarch64

sles15sp3

5.3.18-57

x86 / aarch64

sles15sp4

5.14.21-150400.22

x86 / aarch64

sles15sp5

5.14.21-150500.53

x86 / aarch64

Version Upgrade/Downgrade

Version

Upgrade to

Downgrade to

1.0.0

1.1.0; 1.1.1

N/A

1.1.0

1.1.1; 1.2.0

1.0.0

1.1.1

1.2.0; 1.3.0

1.1.0; 1.0.0

1.2.0

1.3.0; 1.4.0

1.1.1; 1.1.0

1.3.0

1.4.0; 1.5.0

1.2.0; 1.1.1

1.4.0

1.5.0; 2.0.2

1.3.0; 1.2.0

1.5.0

2.0.2; 2.2.0; 1.5.1; 1.5.2; 1.5.3

1.4.0; 1.3.0

1.5.1

1.5.2

1.5.0

1.5.2

1.5.3

1.5.1; 1.5.0

1.5.3

N/A

1.5.2; 1.5.0

2.0.2

2.2.0; 2.5.0

1.5.0; 1.4.0

2.2.0

2.5.0; 2.6.0

N/A

2.2.1

2.5.0; 2.6.0

N/A

2.5.0

2.6.0

2.2.1 for BlueField-3; 2.2.0 for BlueField-2

2.6.0

N/A

2.5.0; 2.2.1 for BlueField-3; 2.2.0 for BlueField-2


Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods:

Customers who purchased NVIDIA M-1 Global Support Services, please see your contract for details regarding Technical Support.

Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller.

Note

For questions, comments, and feedback, please contact us at DOCA-Feedback@exchange.nvidia.com.

The following table lists the known issues and limitations for this release of DOCA SDK.

ReferenceDescription
3814526Description: The location of the official Kubernetes repository changed, causing apt/yum failures.
Workaround:
  • For Ubuntu/Debian:
    Copy
    Copied!
                

    rm -f /etc/apt/sources.list.d/kubernetes.list apt update

  • On RPM based OSs:
    Copy
    Copied!
                

    rm -f /etc/yum.repos.d/kubernetes.repo

Keywords: Kubernetes
Reported in version: 2.6.0
3683801Description: Starting from kernel 5.14, the virtio-net TX path has a logic which may trigger infinite loop when vq is broken (e.g., device is removed) under heavy traffic.
Workaround: N/A
Keyword: Virtio-net
Reported in version: 2.6.0
3714522Description: When creating/destroying VFs back to back, make sure the virtio-net controller side does not see any alive VF before recreating them from the guest OS (i.e., virtnet list).
Workaround: N/A
Keyword: Virtio-net; VFs
Reported in version: 2.6.0
3694402 Description: When restarting the virtio-net-controller from the DPU while the guest OS is booting, the guest OS may see kernel call trace while the controller is preparing the device. It recovers once the controller starts.
Workaround: N/A
Keyword: Virtio-net; hotplug; restart
Reported in version: 2.6.0
3709873Description: In some setups, GGA is limited to 512 bytes even though it exposes a capability of 2M.
Workaround: N/A
Keyword: GGA
Reported in version: 2.6.0
3665070Description: Virtio-net controller fails to load if DPA_AUTHENTICATION is enabled.
Workaround: N/A
Keyword: Virtio-net; DPA
Reported in version: 2.5.0
3631257Description: When installing DOCA for host on a SLES15.5 system that uses NVMe storage (e.g., NVMe disk) and a non-default kernel that requires rebuilding the kernel, the script doca-kernel-support builds and installs the mlnx-nvme drivers that override standard NVMe kernel drivers, which renders the system unbootable.

Workaround: After installing doca-extra, edit the doca-kernel-extra script before running it to avoid it building mlnx-nvme modules. The option --without-mlnx-nvme must be added to the invocation of install.pl:

Copy
Copied!
            

sed -i -e '/install\.pl/a\\t--without-mlnx-nvme \\' /opt/mellanox/doca/tools/doca-kernel-support

Keyword: NVMe; SLES
Reported in version: 2.5.0
3678069Description: If using DPUs with NVMe and mmcbld and configured to boot from mmcblk, users must create bf.cfg file with device=/dev/mmcblk0, then install the *.bfb as normal.
Workaround: N/A
Keyword: NVMe
Reported in version: 2.5.0
3669736Description: DOCA can run only one instance of a DOCA library per process on setups with a page size different than 4K (e.g., Rockey). Combination of different libraries cannot be run on the same process.
Workaround: N/A
Keyword: OpenEuler; Rocky; DOCA libs
Reported in version: 2.5.0
3469692Description: The overall number of IPsec sessions is limited to 64K per port.
Workaround: N/A
Keyword: IPsec
Reported in version: 2.5.0
3680538Description: When using strongSwan or OVS-IPsec as explained in the NVIDIA BlueField DPU BSP, the IPSec Rx data path is not offloaded to hardware and occurs in software running on the Arm cores. As a result, bandwidth performance is substantially low.
Workaround: N/A
Keyword: IPsec
Reported in version: 2.5.0

3674295
Description: The script for the East-West Overlay Encryption application is not supported in DOCA 2.5.0. Users who wish to configure OVS IPsec with hardware offload must should follow the steps explained in the "Configuration Flow" section of the guide instead of running the application.
Workaround: N/A
Keyword: IPsec
Reported in version: 2.5.0
N/ADescription: Execution unit partitions are still not implemented and would be added in future releases.
Workaround: N/A
Keyword: EU tool
Reported in version: 2.5.0
3618936Description: When moving to DPU mode from NIC mode, it is necessary to reinstall the BFB and perform a graceful reboot to the DPU by shutting down the Arm cores before rebooting the host system.
Workaround: N/A
Keyword: NIC mode
Reported in version: 2.5.0
3666160Description: Installing BFB using bfb-install when mlxconfig PF_TOTAL_SF>1700, triggers server reboot immediately.
Workaround: Change PF_TOTAL_SF to 0, perform a graceful shutdown, power cycle, then installing BFB.
Keyword: SF; PF_TOTAL_SF; BFB installation
Reported in version: 2.2.1
3603146Description: Running mlxfwreset on BlueField-3 may cause the external host to crash when the RShim driver is running on that host.
Workaround: Stop the RShim driver on the external host using systemctl stop rshim before performing mlxfwreset.
Keyword: RShim; mlxfwreset
Reported in version: 2.2.1
3594836Description: When enabling Flex IO SDK tracer at high rates, a slow-down in processing may occur and/or some traces may be lost.
Workaround: Keep tracing limited to ~1M traces per second to avoid a significant processing slow-down. Use tracer for debug purposes and consider disabling it by default.
Keyword: Tracer FlexIO
Reported in version: 2.2.1
3592080Description: When using UEK8 on the host in DPU mode, creating a VF on the host consumes about 100MB memory on the DPU.
Workaround: N/A
Keyword: UEK; VF
Reported in version: 2.2.1
3568924Description: doca_rdma does not support cross-subnet routing.
Workaround: N/A
Keyword: RDMA
Reported in version: 2.2.1
3566042Description: Virtio hotplug is not supported in GPU-HOST mode on the NVIDIA Converged Accelerator.
Workaround: N/A
Keyword: Virtio; Converged Accelerator
Reported in version: 2.2.0
3546474Description: PXE boot over ConnectX interface might not work due to an invalid MAC address in the UEFI boot entry.
Workaround: On the DPU, create /etc/bf.cfg file with the relevant PXE boot entries, then run the command bfcfg.
Keyword: PXE; boot; MAC
Reported in version: 2.2.0
3561723Description: Running mlxfwreset sync 1 on NVIDIA Converged Accelerators may be reported as supported although it is not. Executing the reset will fail.
Workaround: N/A
Keywords: mlxfwreset
Reported in version: 2.2.0
3546202

Description: After rebooting a BlueField-3 DPU running Rocky Linux 8.6 BFB, the kernel log shows the following error:

Copy
Copied!
            

[    3.787135] mlxbf_gige MLNXBF17:00: Error getting PHY irq. Use polling instead

This message indicates that the Ethernet driver will function normally in all aspects, except that PHY polling is enabled.

Workaround: N/A
Keywords: Linux; PHY; kernel
Reported in version: 2.2.0
3306489Description: When performing longevity tests (e.g., mlxfwreset, DPU reboot, burning of new BFBs), a host running an Intel CPU may observer errors related to "CPU 0: Machine Check Exception".
Workaround: Add intel_idle.max_cstate=1 entry to the kernel command line.
Keywords: Longevity; mlxfwreset; DPU reboot
Reported in version: 2.2.0
3529297Description: Enhanced NIC mode is not supported on BlueField-2 DPUs.
Workaround: N/A
Keywords: Operation; mode
Reported in version: 2.2.0
3538486Description: When removing LAG configuration from the DPU, a kernel warning for uverbs_destroy_ufile_hw is observed if virtio-net-controller is still running.
Workaround: Stop virtio-net-controller service before cleaning up bond configuration.
Keywords: Virtio-net; LAG
Reported in version: 2.2.0
3534219Description: On BlueField-3 devices, from DOCA 2.2.0 to 32.37.1306 (or lower), the host crashes when executing partial Arm reset (e.g., Arm reboot; BFB push; mlxfwreset).

Workaround: Before downgrading the firmware:

  1. Run:

    Copy
    Copied!
                

    echo 0 > /sys/bus/platform/drivers/mlxbf-bootctl/large_icm

  2. Reboot Arm.
Keyword: BlueField-3; downgrade
Reported in version: 2.2.0
3516279Description: The NVIDIA DOCA East-West Overlay Encryption Reference Application (and the underlying DPU OS Kernel driver IPsec functionality) is not supported. User space DOCA IPsec is not impacted.
Workaround: N/A
Keyword: IPsec
Reported in version: 2.2.0
3444073Description: mlxfwreset is not supported in this release.
Workaround: Perform a graceful shutdown, then power cycle the host.
Keyword: mlxfwreset; support
Reported in version: 2.0.2
3462630

When trying to perform a PXE installation when UEFI Secure Boot is enabled, the following error messages may be observed:

Copy
Copied!
            

error: shim_lock protocol not found. error: you need to load the kernel first.

Workaround: Download a Grub EFI binary from the Ubuntu website. For further information on Ubuntu UEFI Secure Boot PXE Boot, please visit Ubuntu's official website.
Keyword: PXE; UEFI Secure Boot
Reported in version: 2.0.2
3448841Description: While running CentOS 8.2, switchdev Ethernet DPU runs in "shared" RDMA net namespace mode instead of "exclusive".

Workaround: Use ib_core module parameter netns_mode=0. For example:

Copy
Copied!
            

echo "options ib_core netns_mode=0" >> /etc/modprobe.d/mlnx-bf.conf

Keyword: RDMA; isolation; Net NS
Reported in version: 2.0.2
3232444Description: After live migration of virtio-net devices using the VFE driver, the max_queues_size output from the virtnet list may be wrong. This does not affect the actual value.
Workaround: N/A
Keyword: Virtio-net; live migration
Reported in version: 2.0.2
3441287Description: Failure occurs when attempting to raise static LAG with ifenslave_2.10ubuntu3 package.
Workaround: Use ifenslave_2.9ubuntu1.
Keywords: ifenslave; bonding
Reported in version: 4.0.2
2706803Description: When an NVMe controller, SoC management controller, and DMA controller are configured, the maximum number of VFs is limited to 124.
Workaround: N/A
Keyword: VF; limitation
Reported in version: 2.0.2
3273435Description: Changing the mode of operation between NIC and DPU modes results in different capabilities for the host driver which might cause unexpected behavior.
Workaround: Reload the host driver or reboot the host.
Keyword: Modes of operation; driver
Reported in version: 2.0.2
3264749

Description: In Rocky and CentOS 8.2 inbox-kernel BFBs, RegEx requires the following extra huge page configuration for it to function properly:

Copy
Copied!
            

sudo hugeadm --pool-pages-min DEFAULT:2048M sudo systemctl start mlx-regex.service systemctl status mlx-regex.service

If these commands have executed successfully you should see active (running) in the last line of the output.

Workaround: N/A
Keyword: RegEx; hugepages
Reported in version: 1.5.1
3240153Description: DOCA kernel support only works on a non-default kernel.
Workaround: N/A
Keyword: Kernel
Reported in version: 1.5.0
3217627Description: The doca_devinfo_rep_list_create API returns success on the host instead of Operation not supported.
Workaround: N/A
Keyword: DOCA core; InfiniBand
Reported in version: 1.5.0

© Copyright 2023, NVIDIA. Last updated on Feb 9, 2024.