NVIDIA DOCA Installation Guide for Linux

1.0

This guide details the necessary steps to set up NVIDIA DOCA in your Linux environment.

Installation of the NVIDIA® BlueField® networking platform (DPU or SuperNIC) software requires following the following step-by-step procedure.

Supported Platforms

Supported BlueField Platforms

The following NVIDIA® BlueField® Platforms are supported with DOCA:

NVIDIA SKU

Legacy OPN

PSID

Description

900-9D3B6-00CV-AA0

N/A

MT_0000000884

BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

900-9D3B6-00SV-AA0

N/A

MT_0000000965

BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Disabled

900-9D3B6-00CC-AA0

N/A

MT_0000001024

BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

900-9D3B6-00SC-AA0

N/A

MT_0000001025

BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Disabled

900-9D219-0086-ST1

MBF2M516A-CECOT

MT_0000000375

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0086-ST0

MBF2M516A-EECOT

MT_0000000376

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-ST1

MBF2M516A-EENOT

MT_0000000377

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D206-0053-SQ0

MBF2H332A-AENOT

MT_0000000539

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0063-ST2

MBF2H332A-AEEOT

MT_0000000540

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0083-ST3

MBF2H332A-AECOT

MT_0000000541

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0083-ST1

MBF2H322A-AECOT

MT_0000000542

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 8GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0063-ST1

MBF2H322A-AEEOT

MT_0000000543

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 8GB on-board DDR; 1GbE OOB management; HHHL

900-9D219-0066-ST0

MBF2M516A-EEEOT

MT_0000000559

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-SN1

MBF2M516A-CENOT

MT_0000000560

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0066-ST2

MBF2M516A-CEEOT

MT_0000000561

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0006-ST0

MBF2H516A-CEEOT

MT_0000000702

BlueField-2 DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-ST2

MBF2H516A-CENOT

MT_0000000703

BlueField-2 DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0066-ST3

MBF2H516A-EEEOT

MT_0000000704

BlueField-2 DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-SQ0

MBF2H516A-EENOT

MT_0000000705

BlueField-2 DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D250-0038-ST1

MBF2M345A-HESOT

MT_0000000715

BlueField-2 E-Series DPU; 200GbE/HDR single-port QSFP56; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D250-0048-ST1

MBF2M345A-HECOT

MT_0000000716

BlueField-2 E-Series DPU; 200GbE/HDR single-port QSFP56; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D218-0073-ST1

MBF2H512C-AESOT

MT_0000000723

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0083-ST2

MBF2H512C-AECOT

MT_0000000724

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0086-ST4

MBF2M516C-EECOT

MT_0000000728

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0086-SQ0

MBF2H516C-CECOT

MT_0000000729

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST5

MBF2M516C-CESOT

MT_0000000731

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST6

MBF2M516C-EESOT

MT_0000000732

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0086-ST3

MBF2M516C-CECOT

MT_0000000733

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST2

MBF2H516C-EESOT

MT_0000000737

BlueField-2 P-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST1

MBF2H516C-CESOT

MT_0000000738

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D218-0083-ST4

MBF2H532C-AECOT

MT_0000000765

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0073-ST0

MBF2H532C-AESOT

MT_0000000766

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Disabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0076-ST3

MBF2H536C-CESOT

MT_0000000767

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0086-ST2

MBF2H536C-CECOT

MT_0000000768

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0073-ST4

MBF2H512C-AEUOT

MT_0000000972

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled with UEFI disabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management

900-9D208-0076-STA

MBF2H516C-CEUOT

MT_0000000973

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled with UEFI disabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management

900-9D208-0076-STB

MBF2H536C-CEUOT

MT_0000001008

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56, integrated BMC, PCIe Gen4 x16, Secure Boot Enabled with UEFI Disabled, Crypto Disabled, 32GB on-board DDR, 1GbE OOB management, Tall Bracket, FHHL

P1004/699210040230

N/A

NVD0000000015

BlueField-2 A30X, P1004 SKU 205, Generic, GA100, 24GB HBM2e, PCIe passive Dual Slot 230W GEN4, DPU Crypto ON W/ Bkt, 1 Dongle, Black, HF, VCPD

P4028/699140280000

N/A

NVD0000000020

ZAM / NAS

Supported ConnectX NICs

The NVIDIA® ConnectX® NICs supported with DOCA-Host can be found in: NVIDIA DOCA Profiles

Hardware Prerequisites

For BlueField Platform users, this guide assumes that a BlueField device has been installed in a server according to the instructions detailed in your DPU's hardware user guide .

DOCA Packages

Device

Component

Version

Description

Host

DOCA Devel

2.7.0

Software development kit package and tools for developing host software

DOCA Runtime

2.7.0

Runtime libraries and tools required to run DOCA-based software applications on host

DOCA Extra

2.7.0

Contains helper scripts (doca-info, doca-kernel-support)

DOCA OFED

2.7.0

Software stack which operates across all NVIDIA network adapter solutions

Arm emulated (QEMU) development container

4.7.0

Linux-based BlueField Arm emulated container for developers

Target BlueField DPU (Arm)

BlueField BSP

4.7.0

BlueField image and firmware

DOCA SDK

2.7.0

Software development kit packages and tools for developing Arm software

DOCA Runtime

2.7.0

Runtime libraries and tools required to run DOCA-based software applications on Arm

Supported Host OS per DOCA-Host Installation Profile

The default operating system included with the BlueField Bundle (for DPU and SuperNIC) is Ubuntu 22.04.

The supported operating systems on the host machine per DOCA-Host installation profile are the following:

Note

Only the following generic kernel versions are supported for DOCA local repo package for host installation.

OS
OS Version
Default Kernel Version
Arch
doca-all
doca-networking
doca-ofed
Alinux
3.2
5.10.134-13.al8.x86_64
x86



Anolis
8.6
5.10.134+
aarch64



x86



BCLinux
21.10SP2
4.19.90-2107.6.0.0098.oe1.bclinux.aarch64
aarch64



4.19.90-2107.6.0.0100.oe1.bclinux.x86_64
x86



CTYunOS
2.0
4.19.90-2102.2.0.0062.ctl2.aarch64
aarch64



4.19.90-2102.2.0.0062.ctl2.x86_64
x86



3.0 (23.01)
5.10.0-136.12.0.86.ctl3.aarch64
aarch64



5.10.0-136.12.0.86.ctl3.x86_64
x86



Debian
10.13
4.19.0-21-arm64
aarch64



4.19.0-21-amd64
x86



10.8
4.19.0-14-arm64
aarch64



4.19.0-14-amd64
x86



10.9
4.19.0-16-amd64
x86



11.3
5.10.0-13-arm64
aarch64



5.10.0-13-amd64
x86



12.1
6.1.0-10-arm64
aarch64



6.1.0-10-amd64
x86



EulerOS
2.0sp11
5.10.0-60.18.0.50.h323.eulerosv2r11.aarch64
aarch64



5.10.0-60.18.0.50.h323.eulerosv2r11.x86_64
x86



2.0sp12
5.10.0-136.12.0.86.h1032.eulerosv2r12.aarch64
aarch64



5.10.0-136.12.0.86.h1032.eulerosv2r12.x86_64
x86



Kylin
10sp2
4.19.90-24.4.v2101.ky10.aarch64
aarch64



4.19.90-24.4.v2101.ky10.x86_64
x86



10sp3
4.19.90-52.22.v2207.ky10.aarch64
aarch64



4.19.90-52.22.v2207.ky10.x86_64
x86



Mariner
2.0
5.15.118.1-1.cm2.x86_64
x86



Oracle Linux
7.9
5.4.17-2011.6.2.el7uek.x86_64
x86



8.4
5.4.17-2102.201.3.el8uek.x86_64
x86



8.6
5.4.17-2136.307.3.1.el8uek.x86_64
x86



8.7
5.15.0-3.60.5.1.el8uek.x86_64
x86



8.8
5.15.0-101.103.2.1.el8uek.x86_64
x86



9.1
5.15.0-3.60.5.1.el9uek.x86_64
x86



9.2
5.15.0-101.103.2.1.el9uek.x86_64
x86



openEuler
20.03sp3
4.19.90-2112.8.0.0131.oe1.aarch64
aarch64



4.19.90-2112.8.0.0131.oe1.x86_64
x86



22.03
5.10.0-60.18.0.50.oe2203.aarch64
aarch64



5.10.0-60.18.0.50.oe2203.x86_64
x86



RHEL/CentOS
8.0
4.18.0-80.el8.aarch64
aarch64



4.18.0-80.el8.x86_64
x86



8.1
4.18.0-147.el8.aarch64
aarch64



4.18.0-147.el8.x86_64
x86



8.2
4.18.0-193.el8.aarch64
aarch64



4.18.0-193.el8.x86_64
x86



8.3
4.18.0-240.el8.aarch64
aarch64



4.18.0-240.el8.x86_64
x86



8.4
4.18.0-305.el8.aarch64
aarch64



4.18.0-305.el8.x86_64
x86



RHEL/Rocky
8.5
4.18.0-348.el8.aarch64
aarch64



4.18.0-348.el8.x86_64
x86



8.6
4.18.0-372.41.1.el8_6.aarch64
aarch64



4.18.0-372.41.1.el8_6.x86_64
x86



8.7
4.18.0-425.14.1.el8_7.aarch64
aarch64



4.18.0-425.14.1.el8_7.x86_64
x86



8.8
4.18.0-477.10.1.el8_8.aarch64
aarch64



4.18.0-477.10.1.el8_8.x86_64
x86



8.9
4.18.0-513.5.1.el8_9.aarch64
aarch64



4.18.0-513.5.1.el8_9.x86_64
x86



9.0
5.14.0-70.46.1.el9_0.aarch64
aarch64



5.14.0-70.46.1.el9_0.x86_64
x86



9.1
5.14.0-162.19.1.el9_1.aarch64
aarch64



5.14.0-162.19.1.el9_1.x86_64
x86



9.2
5.14.0-284.11.1.el9_2.aarch64
aarch64



5.14.0-284.11.1.el9_2.x86_64
x86



9.3
5.14.0-362.8.1.el9_3.aarch64
aarch64



5.14.0-362.8.1.el9_3.x86_64
x86



RHEL/Rocky
9.4
5.14.0-427.13.1.el9_4.aarch64
aarch64



5.14.0-427.13.1.el9_4.x86_64
x86



SLES
15sp2
5.3.18-22-default
aarch64



x86



15sp3
5.3.18-57-default
aarch64



x86



15sp4
5.14.21-150400.22-default
aarch64



x86



15sp5
5.14.21-150500.53-default
aarch64



x86



TKLinux
3.3
5.4.119-19.0009.39
aarch64



5.4.119-19.0009.39
x86



Ubuntu
20.04
5.4.0-26-generic
aarch64



x86



22.04
5.15.0-25-generic
aarch64



x86



24.04
6.8.0-31-generic
aarch64



x86



UOS
20.1060a
5.10.0-46.uelc20.aarch64
aarch64



5.10.0-46.uelc20.x86_64
x86



20.1060e
5.10.0-46.uel20.aarch64
aarch64



5.10.0-46.uel20.x86_64
x86



This guide provides the minimal instructions for setting up DOCA on a standard system.

Note

Make sure to follow the instructions in this section sequentially. Make sure to update DOCA on the host side first before installing the BFB Bundle on the BlueField.

Installation Files

DeviceComponentOSArchLink
Host

These files contain the following components suitable for their respective OS version.

  • DOCA Devel v2.7.0
  • DOCA Runtime v2.7.0
  • DOCA Extra v2.7.0
  • DOCA OFED v2.7.0
Alinux 3.2x86doca-host-2.7.0-209000_24.04_alinux32.x86_64.rpm
Anolisaarch64doca-host-2.7.0-209000_24.04_anolis86.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_anolis86.x86_64.rpm
BCLinux 21.10 SP2aarch64doca-host-2.7.0-209000_24.04_bclinux2110sp2.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_bclinux2110sp2.x86_64.rpm
CTyunOS 2.0aarch64doca-host-2.7.0-209000_24.04_ctyunos20.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_ctyunos20.x86_64.rpm
CTyunOS 23.01aarch64doca-host-2.7.0-209000_24.04_ctyunos2301.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_ctyunos2301.x86_64.rpm
Debian 10.13aarch64doca-host_2.7.0-209000-24.04-debian1013_arm64.deb
x86doca-host_2.7.0-209000-24.04-debian1013_amd64.deb
Debian 10.8aarch64doca-host_2.7.0-209000-24.04-debian108_arm64.deb
x86doca-host_2.7.0-209000-24.04-debian108_amd64.deb
Debian 10.9x86doca-host_2.7.0-209000-24.04-debian109_amd64.deb
Debian 11.3aarch64doca-host_2.7.0-209000-24.04-debian113_arm64.deb
x86doca-host_2.7.0-209000-24.04-debian113_amd64.deb
Debian 12.1aarch64doca-host_2.7.0-209000-24.04-debian121_arm64.deb
x86doca-host_2.7.0-209000-24.04-debian121_amd64.deb
EulerOS 20 SP11aarch64doca-host-2.7.0-209000_24.04_euleros20sp11.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_euleros20sp11.x86_64.rpm
EulerOS 20 SP12aarch64doca-host-2.7.0-209000_24.04_euleros20sp12.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_euleros20sp12.x86_64.rpm
Kylin 1.0 SP2aarch64doca-host-2.7.0-209000_24.04_kylin10sp2.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_kylin10sp2.x86_64.rpm
Kylin 1.0 SP3aarch64doca-host-2.7.0-209000_24.04_kylin10sp3.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_kylin10sp3.x86_64.rpm
Mariner 2.0x86doca-host-2.7.0-209000_24.04_mariner20.x86_64.rpm
Oracle Linux 7.9x86doca-host-2.7.0-209000_24.04_ol79.x86_64.rpm
Oracle Linux 8.4x86doca-host-2.7.0-209000_24.04_ol84.x86_64.rpm
Oracle Linux 8.6x86doca-host-2.7.0-209000_24.04_ol86.x86_64.rpm
Oracle Linux 8.7x86doca-host-2.7.0-209000_24.04_ol87.x86_64.rpm
Oracle Linux 8.8x86doca-host-2.7.0-209000_24.04_ol88.x86_64.rpm
Oracle Linux 9.0x86doca-host-2.7.0-209000_24.04_ol90.x86_64.rpm
Oracle Linux 9.1x86doca-host-2.7.0-209000_24.04_ol91.x86_64.rpm
Oracle Linux 9.2x86doca-host-2.7.0-209000_24.04_ol92.x86_64.rpm
openEuler 20.03 SP3aarch64doca-host-2.7.0-209000_24.04_openeuler2003sp3.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_openeuler2003sp3.x86_64.rpm
openEuler 22.03aarch64doca-host-2.7.0-209000_24.04_openeuler2203.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_openeuler2203.x86_64.rpm
RHEL/CentOS 8.0aarch64doca-host-2.7.0-209000_24.04_rhel80.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel80.x86_64.rpm
RHEL/CentOS 8.1aarch64doca-host-2.7.0-209000_24.04_rhel81.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel81.x86_64.rpm
RHEL/CentOS 8.2aarch64doca-host-2.7.0-209000_24.04_rhel82.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel82.x86_64.rpm
RHEL/CentOS 8.3aarch64doca-host-2.7.0-209000_24.04_rhel83.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel83.x86_64.rpm
RHEL/CentOS 8.4aarch64doca-host-2.7.0-209000_24.04_rhel84.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel84.x86_64.rpm
RHEL/CentOS 8.5aarch64doca-host-2.7.0-209000_24.04_rhel85.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel85.x86_64.rpm
RHEL/Rocky 8.6aarch64doca-host-2.7.0-209000_24.04_rhel86.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel86.x86_64.rpm
RHEL/Rocky 8.7aarch64doca-host-2.7.0-209000_24.04_rhel87.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel87.x86_64.rpm
RHEL/Rocky 8.8aarch64doca-host-2.7.0-209000_24.04_rhel88.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel88.x86_64.rpm
RHEL/Rocky 8.9aarch64doca-host-2.7.0-209000_24.04_rhel89.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel89.x86_64.rpm
RHEL/Rocky 9.0aarch64doca-host-2.7.0-209000_24.04_rhel90.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel90.x86_64.rpm
RHEL/Rocky 9.1aarch64doca-host-2.7.0-209000_24.04_rhel91.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel91.x86_64.rpm
RHEL/Rocky 9.2aarch64doca-host-2.7.0-209000_24.04_rhel92.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel92.x86_64.rpm
RHEL/Rocky 9.3aarch64doca-host-2.7.0-209000_24.04_rhel93.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel93.x86_64.rpm
RHEL/Rocky 9.4aarch64doca-host-2.7.0-209000_24.04_rhel94.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_rhel94.x86_64.rpm
SLES 15 SP2aarch64doca-host-2.7.0-209000_24.04_sles15sp2.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_sles15sp2.x86_64.rpm
SLES 15 SP3aarch64doca-host-2.7.0-209000_24.04_sles15sp3.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_sles15sp3.x86_64.rpm
SLES 15 SP4aarch64doca-host-2.7.0-209000_24.04_sles15sp4.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_sles15sp4.x86_64.rpm
SLES 15 SP5aarch64doca-host-2.7.0-209000_24.04_sles15sp5.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_sles15sp5.x86_64.rpm
TencentOS 3.3aarch64doca-host-2.7.0-209000_24.04_tencentos33.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_tencentos33.x86_64.rpm
Ubuntu 20.04aarch64doca-host_2.7.0-209000-24.04-ubuntu2004_arm64.deb
x86doca-host_2.7.0-209000-24.04-ubuntu2004_amd64.deb
Ubuntu 22.04aarch64doca-host_2.7.0-209000-24.04-ubuntu2204_arm64.deb
x86doca-host_2.7.0-209000-24.04-ubuntu2204_amd64.deb
Ubuntu 24.04aarch64doca-host_2.7.0-209000-24.04-ubuntu2404_arm64.deb
x86doca-host_2.7.0-209000-24.04-ubuntu2404_amd64.deb
UOS20.1060aarch64doca-host-2.7.0-209000_24.04_uos201060.aarch64.rpm
x86doca-host-2.7.0-209000_24.04_uos201060.x86_64.rpm
UOS20.1060aarch64
&
x86
doca-host-2.7.0-209000_24.04_uos201060a.aarch64.rpm
&
doca-host-2.7.0-209000_24.04_uos201060a.x86_64.rpm
Target BlueField Platform (Arm)BlueField Software v4.7.0 Ubuntu 22.04aarch64bf-bundle-2.7.0-33_24.04_ubuntu-22.04_prod.bfb
DOCA SDK v2.7.0Ubuntu 22.04aarch64doca-dpu-repo-ubuntu2204-local_2.7.0085-1.24.04.0.6.6.0.bf.4.7.0.13127_arm64.deb
DOCA Runtime v2.7.0

Uninstalling Software from Host

If an older DOCA (or MLNX_OFED) software version is installed on your host, make sure to uninstall it before proceeding with the installation of the new version:

Deb-based

Copy
Copied!
            

$ for f in $( dpkg --list | grep doca | awk '{print $2}' ); do echo $f ; apt remove --purge $f -y ; done $ /usr/sbin/ofed_uninstall.sh --force $ sudo apt-get autoremove

RPM-based

Copy
Copied!
            

host# for f in $(rpm -qa | grep -i doca ) ; do yum -y remove $f; done host# /usr/sbin/ofed_uninstall.sh --force host# yum autoremove host# yum makecache

Then perform the following steps:

  1. Download NVIDIA's RPM-GPG-KEY-Mellanox-SHA256 key:

    Copy
    Copied!
                

    # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox-SHA256 --2018-01-25 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox-SHA256 Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox-SHA256?   100%[=================================================>] 1,354 --.-K/s in 0s   2018-01-25 13:52:30 (247 MB/s) - ?RPM-GPG-KEY-Mellanox-SHA256? saved [1354/1354]

  2. Install the key:

    Copy
    Copied!
                

    # sudo rpm --import RPM-GPG-KEY-Mellanox-SHA256 warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED//RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) " From : /repos/MLNX_OFED//RPM-GPG-KEY-Mellanox-SHA256 Is this ok [y/N]:

  3. Verify that the key was successfully imported:

    Copy
    Copied!
                

    # rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n' | grep Mellanox gpg-pubkey-a9e4b643-520791ba gpg(Mellanox Technologies )

Installing Prerequisites on Host for Target BlueField

Install RShim to manage and flash the BlueField Platform.

OS

Procedure

Deb-based

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the deb repo. Run:

    Copy
    Copied!
                

    host# sudo dpkg -i <repo_file>

  3. Perform apt update. Run:

    Copy
    Copied!
                

    host# sudo apt-get update

  4. Run apt install for RShim:

    Copy
    Copied!
                

    host# sudo apt install rshim

RPM-based

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the RPM repo. Run:

    Copy
    Copied!
                

    host# sudo rpm -Uvh <repo_file>

  3. Enable new dnf repos. Run:

    Copy
    Copied!
                

    host# sudo dnf makecache

  4. Run dnf install to install RShim:

    Copy
    Copied!
                

    host# sudo dnf install rshim

Note

Skip section "Installing Software on Host" to proceed without the DOCA local repo package for host.

Determining BlueField Device ID

It is important to learn your BlueField's device-id to perform some of the software installations or upgrades in this guide.

To determine the device ID of the BlueField Platform on your setup, run:

Copy
Copied!
            

host# mst start host# mst status -v

Example output:

Copy
Copied!
            

MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 3b:00.1 mlx5_1 net-ens1f1 0   BlueField2(rev:1) /dev/mst/mt41686_pciconf0 3b:00.0 mlx5_0 net-ens1f0 0   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   e2:00.1   mlx5_1          net-ens7f1np1             4   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     e2:00.0   mlx5_0          net-ens7f0np0             4

Info

The device IDs for the BlueField-2 and BlueField-3 networking platforms in this example are /dev/mst/mt41686_pciconf0 and /dev/mst/mt41692_pciconf0 respectively.

Installing Software on Host

Note

Skip this section if you intend to update only the BlueField software (*.bfb).

Note

Make sure to have followed the instructions under "Installing Prerequisites on Host for Target DPU".

  1. Install DOCA local repo package for host:

    Info

    The following table provides instructions for installing the DOCA host repo on your device depending on your OS and desired profile.

    OS

    Profile

    Instructions

    Deb-based

    doca-all

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Ensure that the kernel headers installed match the version of the currently running kernel.

      Info

      If the build directory exists in under /lib/modules/$(uname -r)/build, then the kernel headers are installed.

    6. Run apt install for DOCA SDK and DOCA runtime:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-all mlnx-fw-updater

    doca-networking

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Ensure that the kernel headers installed match the version of the currently running kernel.

      Info

      If the build directory exists in under /lib/modules/$(uname -r)/build, then the kernel headers are installed.

    6. Run apt install for DOCA SDK and DOCA runtime:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-networking mlnx-fw-updater

    doca-ofed

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# sudo dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# sudo apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Ensure that the kernel headers installed match the version of the currently running kernel.

      Info

      If the build directory exists in under /lib/modules/$(uname -r)/build, then the kernel headers are installed.

    6. Install doca-ofed. Run:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-ofed mlnx-fw-updater

    RPM-based

    doca-all

    1. Download the DOCA host repo from section "Installation Files" for the host .

    2. Unpack the rpm repo. Run:

      Copy
      Copied!
                  

      host# rpm -Uvh <repo_file>.rpm

    3. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Run yum install for DOCA SDK and DOCA runtime:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-all mlnx-fw-updater

    doca-networking

    1. Download the DOCA host repo from section "Installation Files" for the host .

    2. Unpack the rpm repo. Run:

      Copy
      Copied!
                  

      host# rpm -Uvh <repo_file>.rpm

    3. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Run yum install for DOCA SDK and DOCA runtime:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-networking mlnx-fw-updater

    doca-ofed

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the RPM repo. Run:

      Copy
      Copied!
                  

      host# sudo rpm -Uvh <repo_file>.rpm

    3. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Install doca-ofed. Run:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-ofed mlnx-fw-updater

  2. Load the drivers:

    Copy
    Copied!
                

    host# sudo /etc/init.d/openibd restart

  3. Initialize MST. Run:

    Copy
    Copied!
                

    host# sudo mst restart

  4. Skip this step if your BlueField Platform is Ethernet only. Please refer to Supported Platforms to learn your Bluefield type.

    If you have a VPI-capable BlueField, the default link type of the ports will be configured to IB. To verify your link type, run:

    Copy
    Copied!
                

    host# sudo mst start host# sudo mlxconfig -d <device-id> -e q | grep -i link_type Configurations: Default Current Next Boot * LINK_TYPE_P1 IB(1) ETH(2) IB(1) * LINK_TYPE_P2 IB(1) ETH(2) IB(1)

    Note

    If your BlueField is Ethernet capable only, then the sudo mlxconfig -d <device> command will not provide an output.

    If the current link type is set to IB, run the following command to change it to Ethernet:

    Copy
    Copied!
                

    host# sudo mlxconfig -d <device-id> s LINK_TYPE_P1=2 LINK_TYPE_P2=2

  5. Verify that RShim is active.

    Copy
    Copied!
                

    host# sudo systemctl status rshim

    This command is expected to display active (running). If RShim service does not launch automatically, run:

    Copy
    Copied!
                

    host# sudo systemctl enable rshim host# sudo systemctl start rshim

  6. Assign a dynamic IP to tmfifo_net0 interface (RShim host interface).

    Note

    Skip this step if you are installing the DOCA image on multiple DPUs.

    Copy
    Copied!
                

    host# ifconfig tmfifo_net0 192.168.100.1 netmask 255.255.255.252 up

DOCA Extra Package

If the kernel version on on your host is not supported (not shown under "Supported Operating System Distributions"), two options are available:

  • Switch to a compatible kernel.
  • Install doca-extra package:

    1. Run:

      Copy
      Copied!
                  

      host# sudo apt/yum install -y doca-extra

    2. Execute the doca-kernel-support script which rebuilds and installs the DOCA-Host kernel modules with the running kernel:

      Copy
      Copied!
                  

      host# sudo /opt/mellanox/doca/tools/doca-kernel-support

    3. Install user-space packages:

      Copy
      Copied!
                  

      host# sudo apt/yum install -y doca-ofed-userspace

      Note

      doca-kernel-support does not support customized or unofficial kernels.

Installing Software on DPU

Users have two options for installing DOCA on BlueField DPU or SuperNIC:

  • Upgrading the full DOCA image on BlueField (recommended) – this option overwrites the entire boot partition with an Ubuntu 22.04 installation and updates BlueField and NIC firmware.
  • Upgrading DOCA local repo package on BlueField – this option upgrades DOCA components without overwriting the boot partition. Use this option to preserve configurations or files on BlueField itself.

Installing Full DOCA Image on DPU via Host

Warning

This step overwrites the entire boot partition.

Note

This installation sets up the OVS bridge.

Note

If you are installing DOCA on multiple BlueField platforms, skip to section Installing Full DOCA Image on Multiple BlueField Platforms.

Option 1 – No Pre-defined Password

Note

To change the default Ubuntu password during the BFB bundle installation, proceed to Option 2.

BFB installation is executed as follows:

Copy
Copied!
            

host# sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb>

Where rshim<N> is rshim0 if you only have one Bluefield. You may run the following command to verify:

Copy
Copied!
            

host# ls -la /dev/ | grep rshim

Option 2 – Set Pre-defined Password

Ubuntu users can provide a unique password that will be applied at the end of the BlueField BFB bundle installation. This password needs to be defined in a bf.cfg configuration file.

To set the password for the "ubuntu" user:

  1. Create password hash. Run:

    Copy
    Copied!
                

    host# openssl passwd -1 Password: Verifying - Password: $1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1

  2. Add the password hash in quotes to the bf.cfg file:

    Copy
    Copied!
                

    host# sudo vim bf.cfg ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'

    When running the installation command, use the --config flag to provide the file containing the password:

    Copy
    Copied!
                

    host# sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb> --config bf.cfg

    Note

    Optionally, to upgrade the BlueField integrated BMC firmware using BFB bundle, please provide the current BMC root credentials in a bf.cfg file, as shown in the following:

    Copy
    Copied!
                

    BMC_PASSWORD="<root password>" BMC_USER="root" BMC_REBOOT="yes"

    Unless previously changed, the default BMC root password is 0penBmc.

    Note

    If --config is not used, then upon first login to the BlueField device, users will be prompted to update the default 'ubuntu' password.

    The following is an example of Ubuntu-22.04 BFB bundle installation (Release version may vary in the future).

    Copy
    Copied!
                

    host# sudo bfb-install --rshim rshim0 --bfb bf-bundle-2.7.0_24.04_ubuntu-22.04_prod.bfb --config bf.cfg Pushing bfb 1.41GiB 0:02:02 [11.7MiB/s] [ <=> ] Collecting BlueField booting status. Press Ctrl+C to stop INFO[PSC]: PSC BL1 START INFO[BL2]: start INFO[BL2]: boot mode (rshim) INFO[BL2]: VDDQ: 1120 mV INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: lifecycle GA Secured INFO[BL31]: VDD: 850 mV INFO[BL31]: runtime INFO[BL31]: MB ping success INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: UPVS valid INFO[UEFI]: PMI: updates started INFO[UEFI]: PMI: total updates: 1 INFO[UEFI]: PMI: updates completed, status 0 INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[UEFI]: UEFI Secure Boot INFO[UEFI]: PK configured INFO[UEFI]: Redfish enabled INFO[UEFI]: exit Boot Service INFO[MISC]: Found bf.cfg INFO[MISC]: Ubuntu installation started INFO[MISC]: Installing OS image INFO[MISC]: Changing the default password for user ubuntu INFO[MISC]: Ubuntu installation completed INFO[MISC]: Updating NIC firmware... INFO[MISC]: NIC firmware update done INFO[MISC]: Installation finished

    To verify the BlueField has completed booting up, allow additional 90 seconds then perform the following:

    Copy
    Copied!
                

    host# sudo cat /dev/rshim<N>/misc ...  INFO[MISC]: Linux up  INFO[MISC]: DPU is ready  

Installing Full DOCA Image on Multiple BlueField Platforms

On a host with multiple BlueField devices, the BFB image can be installed on all of them using the multi-bfb-install script.

Copy
Copied!
            

host# ./multi-bfb-install --bfb <image_path.bfb> --password <password>

This script detects the number of RShim devices and configures them statically.

  • For Ubuntu – the script creates a configuration file /etc/netplan/20-tmfifo.yaml
  • For CentOS/RH 8.0 and 8.2 – the script installs the bridge-utils package to use the brctl command, creates the tm-br bridge, and connects all RShim interfaces to it

After the installation is complete, the configuration of the bridge and each RShim interface can be observed using ifconfig. The expected result is to see the IP on the tm-br bridge configured to 192.168.100.1 with subnet 255.255.255.0.

Note

To log into BlueField with rshim0, run:

Copy
Copied!
            

ssh ubuntu@192.168.100.2

For each RShim after that, add 1 to the fourth octet of the IP address (e.g., ubuntu@192.168.100.3 for rshim1, ubuntu@192.168.100.4 for rshim2, etc).

The script burns a new MAC address to each BlueField and configures a new IP, 192.168.100.x, as described earlier.

Installing DOCA Local Repo Package on BlueField

Note

If you have already installed BlueField image, be aware that the DOCA SDK, Runtime, and Tools are already contained in the BFB, and this installation is not mandatory. If you have not installed the BlueField image and wish to update DOCA Local Repo package, proceed with the following procedure.

Note

Before installing DOCA on the target BlueField, make sure the out-of-band interface (mgmt) is connected to the internet.

  1. Download the DOCA SDK and DOCA Runtime package from section Installation Files.
  2. Copy deb repo package into BlueField. Run:

    Copy
    Copied!
                

    host# sudo scp -r doca-repo-aarch64-ubuntu2204-local_<version>_arm64.deb ubuntu@192.168.100.2:/tmp/

  3. Unpack the deb repo. Run:

    Copy
    Copied!
                

    dpu# sudo dpkg -i doca-dpu-repo-ubuntu2204-local_<version>_arm64.deb

  4. Run apt update.

    Copy
    Copied!
                

    dpu# sudo apt-get update

  5. Run apt install for DOCA Runtime and DOCA SDK:

    Copy
    Copied!
                

    dpu# sudo apt install doca-runtime doca-sdk

Upgrading Firmware

Note

This operation is only required if the user skipped NIC firmware update during BFB bundle installation using the parameter WITH_NIC_FW_UPDATE=no in the bf.cfg file.

This section explains how to update the NIC firmware on a DOCA installed BlueField OS.

Note

If multiple BlueFields are installed, the following steps must be performed on all of them after BFB installation.

An up-to-date NIC firmware image is provided in BlueField BFB bundle and copied to the BlueField filesystem during BFB installation.

To upgrade firmware in the BlueField Arm OS:

  1. SSH to your BlueField Arm OS by any means available.

    The following instructions enable to login to the BlueField Arm OS from the host OS over the RShim virtual interface, tmfifo_net<N> and do not require LAN connectivity with the BlueField OOB network port.

    Note

    This operation can be performed over the host's tmfifo_net0 IPv4, 192.168.100.1 (preconfigured) with BlueField Arm OS at 192.168.100.2 (default).

    If multiple BlueField DPUs were updated using the multi-bfb-install script, as explained above, then each target BlueField OS IPv4 address changes in its last octate according to the underlaying RShim interface number: 192.168.100.3 for rshim1, 192.168.100.4 for rshim2, etc.

    The default credentials for Ubuntu are as follows:

    Username

    Password

    ubuntu

    ubuntu

    For example, to log into BlueField Arm OS over IPv6:

    Copy
    Copied!
                

    host]# systemctl restart rshim // Wait 10 seconds   host]# ssh -6 fe80::21a:caff:feff:ff01%tmfifo_net<N> Password: <configured-password>

  2. Upgrade firmware in BlueField. Run:

    Copy
    Copied!
                

    dpu# sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

    Example output:

    Copy
    Copied!
                

    Device #1: ----------   Device Type: BlueField-2 [...] Versions: Current Available FW <Old_FW> <New_FW>

  3. For the firmware upgrade to take effect perform a BlueField system reboot.

Post-installation Procedure

  1. Restart the driver. Run:

    Copy
    Copied!
                

    host# sudo /etc/init.d/openibd restart Unloading HCA driver: [ OK ] Loading HCA driver and Access Layer: [ OK ]

  2. Configure the physical function (PF) interfaces.

    Copy
    Copied!
                

    host# sudo ifconfig <interface-1> <network-1/mask> up host# sudo ifconfig <interface-2> <network-2/mask> up

    For example:

    Copy
    Copied!
                

    host# sudo ifconfig p2p1 192.168.200.32/24 up host# sudo ifconfig p2p2 192.168.201.32/24 up

    Pings between the source and destination should now be operational.

This dpu-upgrade procedure enables upgrading DOCA components using standard Linux tools (e.g., apt update and yum update). This process utilizes native package manager repositories to upgrade DPUs without the need for a full installation, and has the following benefits :

  • Only updates components that include modifications

    • Configurable – user can select specific components (e.g., UEFI-ATF, NIC-FW)

  • Includes upgrade of:

    • DOCA drivers and libraries

    • DOCA reference applications

    • BSP (UEFI/ATF) upgrade while maintaining the configuration

    • NIC firmware upgrade while maintaining the configuration

  • Does not:

    • Impact user binaries

    • Upgrade non-Ubuntu OS kernels

    • Upgrade DPU BMC firmware

  • After completion of DPU upgrade:

    • If NIC firmware was not updated, perform DPU Arm reset (software reset / reboot DPU)

    • If NIC firmware was updated, perform firmware reset (mlxfwreset) or perform a graceful shutdown and power cycle

OS

Action

Instructions

Ubuntu/

Debian

Remove mlxbf-bootimages package

Copy
Copied!
            

<dpu> $ apt remove --purge mlxbf-bootimages* -y

Install the the GPG key

Copy
Copied!
            

<dpu> $ apt update <dpu> $ apt install gnupg2

Export the desired distribution

Export DOCA_REPO with the relevant URL. The following is an example for Ubuntu 22.04:

Copy
Copied!
            

<dpu> $ export DOCA_REPO="https://linux.mellanox.com/public/repo/doca/2.7.0/ubuntu22.04/dpu-arm64"

Add GPG key to APT trusted keyring

Copy
Copied!
            

<dpu> $ curl $DOCA_REPO/GPG-KEY-Mellanox.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub

Add DOCA online repository

Copy
Copied!
            

<dpu> $ echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_REPO ./" > /etc/apt/sources.list.d/doca.list

Update index

Copy
Copied!
            

<dpu> $ apt update

Upgrade UEFI/ATF firmware

Run:

Copy
Copied!
            

<dpu> $ apt install mlxbf-bootimages-signed

Then i nitiate upgrade for UEFI/ATF firmware:

Copy
Copied!
            

<dpu> $ apt install mlxbf-scripts <dpu> $ bfrec

Upgrade BlueField DPU NIC firmware

Run:

Copy
Copied!
            

<dpu> $ apt install mlnx-fw-updater-signed.aarch64

Note

This immediately starts NIC firmware upgrade.

To prevent automatic upgrade, run:

Copy
Copied!
            

<dpu> $ export RUN_FW_UPDATER=no

Remove old metapackages

Copy
Copied!
            

<dpu> $ apt-get remove doca-tools doca-sdk doca-runtime -y

Install new metapackages

Copy
Copied!
            

<dpu> $ apt-get install doca-runtime doca-devel -y

Upgrade system

Copy
Copied!
            

<dpu> $ apt upgrade

Apply the new changes,

NIC firmware, and UEFI/ATF

For the upgrade to take effect, perform BlueField system reboot as explained in the "NVIDIA BlueField Reset and Reboot Procedures" troubleshooting page.

Note

This step triggers immediate reboot of the BlueField Arm cores.

CentOS/RHEL/

Anolis/Rocky

Remove mlxbf-bootimages package

Copy
Copied!
            

<dpu> $ yum -y remove mlxbf-bootimages* <dpu> $ yum makecache

Export the desired distribution

Export DOCA_REPO with the relevant URL. The following is an example for Rocky Linux 8.6:

Copy
Copied!
            

<dpu> $ export DOCA_REPO="https://linux.mellanox.com/public/repo/doca/2.7.0/rhel8.6/dpu-arm64/"

Add DOCA online repository

Copy
Copied!
            

echo "[doca] name=DOCA Online Repo baseurl=$DOCA_REPO enabled=1 gpgcheck=0 priority=10 cost=10" > /etc/yum.repos.d/doca.repo

A file is created under /etc/yum.repos.d/doca.repo .

Update index

Copy
Copied!
            

<dpu> $ yum makecache

Upgrade UEFI/ATF firmware

Run:

Copy
Copied!
            

<dpu> $ yum install mlxbf-bootimages-signed.aarch64 mlxbf-bfscripts

Then i nitiate the upgrade for UEFI/ATF firmware:

Copy
Copied!
            

<dpu> $ bfrec

Upgrade BlueField DPU NIC firmware

The following command updates the firmware package and automatically attempts to flash the firmware to the NIC:

Copy
Copied!
            

<dpu> $ yum install mlnx-fw-updater-signed.aarch64

Info

This step can be used as a standalone firmware update. In any case, it is performed as part of the upgrade flow.

Note

To prevent automatic flashing of the firmware to the NIC, run the following first:

Copy
Copied!
            

<dpu> $ export RUN_FW_UPDATER=no

Info

Flashing the firmware to the NIC can be performed manually by running the following command, after the firmware package had been updated:

Copy
Copied!
            

sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

Remove old metapackages

Copy
Copied!
            

<dpu> $ yum -y remove doca-tools doca-sdk doca-runtime

Install new metapackages

Copy
Copied!
            

<dpu> $ yum -y install doca-runtime doca-devel

Upgrade system

Copy
Copied!
            

<dpu> $ yum upgrade --nobest

Apply the new changes,

NIC firmware, and UEFI/ATF

For the upgrade to take effect, perform BlueField system reboot as explained in the "NVIDIA BlueField Reset and Reboot Procedures" troubleshooting page.

Note

This step triggers immediate reboot of the BlueField Arm cores.

Users wishing to build their own customized BlueField OS image can use the BFB build environment. Please refer to the bfb-build project in this GitHub webpage for more information.

Note

For a customized BlueField OS image to boot on the UEFI secure-boot-enabled BlueField (default BlueField secure boot setting), the OS must be either signed with an existing key in the UEFI DB (e.g., the Microsoft key), or UEFI secure boot must be disabled. Please refer to the "Secure Boot" page under NVIDIA BlueField DPU Platform Operating System Documentation for more details.

For full instructions about setting up a development environment, refer to the NVIDIA DOCA Developer Guide.

Installing CUDA on NVIDIA Converged Accelerator

NVIDIA® CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing GPUs.

This section details the necessary steps to set up CUDA on your environment. This section assumes that a BFB image has already been installed on your environment.
To install CUDA on your converged accelerator:

  1. Download and install the latest NVIDIA Data Center GPU driver.

  2. Download and install CUDA

    Note

    The CUDA version tested to work with DOCA SDK is 11.8.0.

Note

Downloading CUDA includes the latest NVIDIA Data Center GPU driver and CUDA toolkit. For more information about CUDA and driver compatibility, refer to the NVIDIA CUDA Toolkit Release Notes.

Configuring Operation Mode

There are two modes that the NVIDIA Converged Accelerator may operate in:

  • Standard mode (default) – the BlueField and the GPU operate separately

  • BlueField-X mode – the GPU is exposed to BlueField and is no longer visible on the host

To verify which mode the system is operating in, run:

Copy
Copied!
            

host# sudo mst start host# sudo mlxconfig -d <device-id> q PCI_DOWNSTREAM_PORT_OWNER[4]

Note

To learn your BlueField Platform's device ID, refer to section "Determining BlueField Device ID".

  • Standard mode output:

    Copy
    Copied!
                

    Device #1: […] Configurations: Next Boot PCI_DOWNSTREAM_PORT_OWNER[4] DEVICE_DEFAULT(0)

  • BlueField-X mode output:

    Copy
    Copied!
                

    Device #1: […] Configurations: Next Boot PCI_DOWNSTREAM_PORT_OWNER[4] EMBEDDED_CPU(15)

To configure BlueField-X mode, run:

Copy
Copied!
            

host# mlxconfig -d <device-id> s PCI_DOWNSTREAM_PORT_OWNER[4]=0xF

To configure standard mode, run:

Copy
Copied!
            

host# mlxconfig -d <device-id> s PCI_DOWNSTREAM_PORT_OWNER[4]=0x0

Note

To learn your BlueField Platform's device ID, refer to section "Determining BlueField Device ID".

Power cycle is required for configuration to take effect. For power cycle the host run:

Copy
Copied!
            

host# ipmitool power cycle


Downloading and Installing CUDA Toolkit and Driver

This section details the necessary steps to set up CUDA on your environment. It assumes that a BFB image has already been installed on your environment.

  1. Install CUDA by visiting the CUDA Toolkit Downloads webpage.

    Note

    Select the Linux distribution and version relevant for your environment.

    Note

    This section shows the native compilation option either on x86 or aarch64 hosts.

  2. Test that the driver installation completed successfully. Run:

    Copy
    Copied!
                

    dpu# nvidia-smi   Tue Apr 5 13:37:59 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA BF A10 Off | 00000000:06:00.0 Off | 0 | | 0% 43C P0 N/A / 225W | 0MiB / 23028MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

  3. Verify that the installation completed successfully.

    1. Download CUDA samples repo. Run:

      Copy
      Copied!
                  

      dpu# git clone https://github.com/NVIDIA/cuda-samples.git

    2. Build and run vectorAdd CUDA sample. Run:

      Copy
      Copied!
                  

      dpu# cd cuda-samples/Samples/0_Introduction/vectorAdd dpu# make dpu# ./vectorAdd

    Note

    If the vectorAdd sample works as expected, it should output "Test Passed".

    Note

    If it seems that the GPU is slow or stuck, stop execution and run:

    Copy
    Copied!
                

    dpu# sudo setpci -v -d ::0302 800.L=201 # CPL_VC0 = 32

GPUDirect RDMA

For information on GPUDirect RMDA and more, refer to DOCA GPUNetIO documentation.

Installing Rivermax on BlueField

NVIDIA Rivermax offers a unique IP-based solution for any media and data streaming use case.

This section provides the steps to install Rivermax assuming that a BFB image has already been installed on your environment.

Downloading Rivermax Driver

  1. Navigate to the NVIDIA Rivermax SDK product page.

  2. Register to be able to download the driver package using the JOIN button at the top of the page.

  3. D ownload the appropriate driver package according to your BFB under the "Linux" subsection. For example, for Ubuntu 22.04 BFB, download rivermax_ubuntu2204_<version>.tar.gz.

Installing Rivermax Driver

  1. Copy the .tgz file to BlueField:

    Copy
    Copied!
                

    host# sudo scp -r rivermax_ubuntu2204_<version>.tar.gz ubuntu@192.168.100.2:/tmp/

  2. Extract the Rivermax file:

    Copy
    Copied!
                

    dpu# sudo tar xzf rivermax_ubuntu2204_<version>.tar.gz

  3. Install the Rivermax driver package:

    Copy
    Copied!
                

    dpu# cd <rivermax-version>/Ubuntu.22.04/deb-dist/aarch64/ dpu# sudo dpkg -i rivermax_<version>.deb

Installing Rivermax Libraries from DOCA

Rivermax libraries are compatibles with DOCA components and can be found inside the doca-dpu-repo.

  1. Unpack the doca-dpu-repo:

    Copy
    Copied!
                

    dpu# sudo dpkg -i doca-dpu-repo-ubuntu2204-local_<version>_arm64.deb

  2. Run apt update:

    Copy
    Copied!
                

    dpu# sudo apt-get update

  3. Install the Rivermax libraries:

    Copy
    Copied!
                

    dpu# sudo apt install doca-rmax-libs dpu# sudo apt install libdoca-rmax-libs-dev

For additional details and guidelines, please visit the NVIDIA Rivermax SDK product page.

Info

For questions, comments, and feedback, please contact us at DOCA-Feedback@exchange.nvidia.com.

© Copyright 2024, NVIDIA. Last updated on Feb 9, 2024.