DOCA Documentation v2.6.0
DOCA SDK 2.6.0

NVIDIA DOCA Installation Guide for Linux

This guide details the necessary steps to set up NVIDIA DOCA in your Linux environment.

Installation of the NVIDIA® BlueField® networking platform software requires following the following step-by-step procedure.

Supported Platforms

Supported BlueField DPUs and SuperNICs

The following NVIDIA® BlueField® DPUs and SuperNICs are supported with DOCA:

NVIDIA SKU

Legacy OPN

PSID

Description

900-9D3B6-00CV-AA0

N/A

MT_0000000884

BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

900-9D3B6-00SV-AA0

N/A

MT_0000000965

BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Disabled

900-9D3B6-00CC-AA0

N/A

MT_0000001024

BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

900-9D3B6-00SC-AA0

N/A

MT_0000001025

BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Disabled

900-9D219-0086-ST1

MBF2M516A-CECOT

MT_0000000375

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0086-ST0

MBF2M516A-EECOT

MT_0000000376

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-ST1

MBF2M516A-EENOT

MT_0000000377

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D206-0053-SQ0

MBF2H332A-AENOT

MT_0000000539

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0063-ST2

MBF2H332A-AEEOT

MT_0000000540

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0083-ST3

MBF2H332A-AECOT

MT_0000000541

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0083-ST1

MBF2H322A-AECOT

MT_0000000542

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 8GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0063-ST1

MBF2H322A-AEEOT

MT_0000000543

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 8GB on-board DDR; 1GbE OOB management; HHHL

900-9D219-0066-ST0

MBF2M516A-EEEOT

MT_0000000559

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-SN1

MBF2M516A-CENOT

MT_0000000560

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0066-ST2

MBF2M516A-CEEOT

MT_0000000561

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0006-ST0

MBF2H516A-CEEOT

MT_0000000702

BlueField-2 DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-ST2

MBF2H516A-CENOT

MT_0000000703

BlueField-2 DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0066-ST3

MBF2H516A-EEEOT

MT_0000000704

BlueField-2 DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-SQ0

MBF2H516A-EENOT

MT_0000000705

BlueField-2 DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D250-0038-ST1

MBF2M345A-HESOT

MT_0000000715

BlueField-2 E-Series DPU; 200GbE/HDR single-port QSFP56; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D250-0048-ST1

MBF2M345A-HECOT

MT_0000000716

BlueField-2 E-Series DPU; 200GbE/HDR single-port QSFP56; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D218-0073-ST1

MBF2H512C-AESOT

MT_0000000723

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0083-ST2

MBF2H512C-AECOT

MT_0000000724

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0086-ST4

MBF2M516C-EECOT

MT_0000000728

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0086-SQ0

MBF2H516C-CECOT

MT_0000000729

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST5

MBF2M516C-CESOT

MT_0000000731

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST6

MBF2M516C-EESOT

MT_0000000732

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0086-ST3

MBF2M516C-CECOT

MT_0000000733

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST2

MBF2H516C-EESOT

MT_0000000737

BlueField-2 P-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST1

MBF2H516C-CESOT

MT_0000000738

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D218-0083-ST4

MBF2H532C-AECOT

MT_0000000765

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0073-ST0

MBF2H532C-AESOT

MT_0000000766

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Disabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0076-ST3

MBF2H536C-CESOT

MT_0000000767

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0086-ST2

MBF2H536C-CECOT

MT_0000000768

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0073-ST4

MBF2H512C-AEUOT

MT_0000000972

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled with UEFI disabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management

900-9D208-0076-STA

MBF2H516C-CEUOT

MT_0000000973

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled with UEFI disabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management

900-9D208-0076-STB

MBF2H536C-CEUOT

MT_0000001008

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56, integrated BMC, PCIe Gen4 x16, Secure Boot Enabled with UEFI Disabled, Crypto Disabled, 32GB on-board DDR, 1GbE OOB management, Tall Bracket, FHHL

P1004/699210040230

N/A

NVD0000000015

BlueField-2 A30X, P1004 SKU 205, Generic, GA100, 24GB HBM2e, PCIe passive Dual Slot 230W GEN4, DPU Crypto ON W/ Bkt, 1 Dongle, Black, HF, VCPD

P4028/699140280000

N/A

NVD0000000020

ZAM / NAS


Supported ConnectX NICs

The following NVIDIA® ConnectX® NICs are supported with DOCA on the host:

Hardware Prerequisites

This quick start guide assumes that an NVIDIA® BlueField® networking platform has been installed in a server according to the instructions detailed in your DPU's hardware user guide .

DOCA Packages

Device

Component

Version

Description

Host

DOCA SDK

2.6.0

Software development kit package for developing host software

DOCA Runtime

2.6.0

Runtime libraries required to run DOCA-based software applications on host

DOCA Tools

2.6.0

Tools for developers and administrators on host

DOCA Extra

2.6.0

Contains helper scripts (doca-info, doca-kernel-support)

DOCA OFED

2.6.0

Software stack which operates across all NVIDIA network adapter solutions

Arm emulated (QEMU) development container

4.6.0

Linux-based BlueField Arm emulated container for developers

Target BlueField DPU (Arm)

BlueField BSP

4.6.0

BlueField image and firmware

DOCA SDK

2.6.0

Software development kit packages for developing Arm software

DOCA Runtime

2.6.0

Runtime libraries required to run DOCA-based software applications on Arm

DOCA Tools

2.6.0

Tools for developers and administrators for Arm target

Supported Operating System Distributions

The default operating system of the BlueField DPU (Arm) is Ubuntu 22.04.

The supported operating systems on the host machine per DOCA profile are the following:

Note

Only the following generic kernel versions are supported for DOCA local repo package for host installation.

DOCA for Host

Kernel

Arch

doca-all

doca-cx

doca-ofed

CTYunOS3 23.01

5.10

aarch64

RHEL/CentOS 8.2

4.18

x86

RHEL/Rocky 8.6

4.18

x86

Ubuntu 20.04

5.4

x86

Ubuntu 22.04

5.15

x86

5.15

aarch64

Debian 10.8

4.19

x86

Debian 10.13

5.10.135 / 5.4.210

x86

Allinux 3.2

5.10

X86

Oracle Linux 8.7

5.15

x86

RHEL/Rocky Linux 9.1

5.14

x86

BCLinux 21.10 SP2

4.19.90

x86 / aarch64

CTYunOS2.0

4.19.90

x86 / aarch64

Debian10.9

4.19.0-16

x86

Debian11.3

5.10.0-13

x86 / aarch64

Debian12.1

6.1.0-10

x86 / aarch64

Kylin 10 SP2

4.19.90

x86 / aarch64

Oracle Linux 8.6

5.4

x86

openEuler 20.03 SP3

4.19.90

x86 / aarch64

openEuler 22.03

5.10.0

x86 / aarch64

RHEL/CentOS 8.0

4.18.0-80.el8

x86

RHEL/CentOS 8.2

4.18.0-193.el8

aarch64

RHEL/CentOS 8.4

4.18.0-305.el8

x86 / aarch64

RHEL/Rocky 8.6

4.18.0-372.41.1.el8

aarch64

RHEL/Rocky 8.8

4.18.0-477.10.1.el8_8

x86 / aarch64

RHEL/Rocky 8.9

4.18.0-513.5.1.el8_9

x86 / aarch64

RHEL/Rocky 9.0

5.14.0-70.46.1.el9_0

x86 / aarch64

RHEL/Rocky 9.1

5.14.0-162.19.1.el9_1

x86 / aarch64

RHEL/Rocky 9.2

5.14.0-284.11.1.el9_2

x86 / aarch64

RHEL/Rocky 9.3

5.14.0-362.8.1.el9_3

x86 / aarch64

sles15sp3

5.3.18-57

x86 / aarch64

sles15sp4

5.14.21-150400.22

x86 / aarch64

sles15sp5

5.14.21-150500.53

x86 / aarch64

This guide provides the minimal instructions for setting up DOCA on a standard system.

Installation Files

Device

Component

OS

Arch

Link

Host

These files contain the following components suitable for their respective OS version.

  • DOCA SDK v2.6.0

  • DOCA Runtime v2.6.0

  • DOCA Tools v2.6.0

  • DOCA Extra v2.6.0

  • DOCA OFED v2.6.0

Alinux 3.2

x86

doca-host-repo-alinux32-2.6.0-0.0.1.2.6.0058.1.al8.24.01.0.3.3.1.x86_64.rpm

BCLinux 21.10 SP2

aarch64

doca-host-repo-bclinux2110sp2-2.6.0-0.0.1.24.01.0.3.3.1.oe1.bclinux.aarch64.rpm

x86

doca-host-repo-bclinux2110sp2-2.6.0-0.0.1.24.01.0.3.3.1.oe1.bclinux.x86_64.rpm

CTyunOS 2.0

aarch64

doca-host-repo-ctyunos20-2.6.0-0.0.1.24.01.0.3.3.1.ctl2.aarch64.rpm

x86

doca-host-repo-ctyunos20-2.6.0-0.0.1.24.01.0.3.3.1.ctl2.x86_64.rpm

CTyunOS 23.01

aarch64

doca-host-repo-ctyunos2301-2.6.0-0.0.1.2.6.0058.1.ctl3.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-ctyunos2301-2.6.0-0.0.1.2.6.0058.1.ctl3.24.01.0.3.3.1.x86_64.rpm

Debian 10.13

x86

doca-host-repo-debian1013_2.6.0-0.0.1.2.6.0058.1.24.01.0.3.3.1_amd64.deb

Debian 10.8

x86

doca-host-repo-debian108_2.6.0-0.0.1.2.6.0058.1.24.01.0.3.3.1_amd64.deb

Debian 10.9

x86

doca-host-repo-debian109_2.6.0-0.0.1.24.01.0.3.3.1_amd64.deb

Debian 11.3

aarch64

doca-host-repo-debian113_2.6.0-0.0.1.24.01.0.3.3.1_arm64.deb

x86

doca-host-repo-debian113_2.6.0-0.0.1.24.01.0.3.3.1_amd64.deb

Debian 12.1

aarch64

doca-host-repo-debian121_2.6.0-0.0.1.24.01.0.3.3.1_arm64.deb

x86

doca-host-repo-debian121_2.6.0-0.0.1.24.01.0.3.3.1_amd64.deb

Kylin 1.0 SP2

aarch64

doca-host-repo-kylin10sp2-2.6.0-0.0.1.24.01.0.3.3.1.ky10.aarch64.rpm

x86

doca-host-repo-kylin10sp2-2.6.0-0.0.1.24.01.0.3.3.1.ky10.x86_64.rpm

Oracle Linux 7.9

x86

doca-host-repo-ol79-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

Oracle Linux 8.4

x86

doca-host-repo-ol84-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

Oracle Linux 8.6

x86

doca-host-repo-ol86-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

Oracle Linux 8.7

x86

doca-host-repo-ol87-2.6.0-0.0.1.2.6.0058.1.el8.24.01.0.3.3.1.x86_64.rpm

openEuler 20.03 SP3

aarch64

doca-host-repo-openeuler2003sp3-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-openeuler2003sp3-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

openEuler 22.03

aarch64

doca-host-repo-openeuler2203-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-openeuler2203-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/CentOS 8.0

x86

doca-host-repo-rhel80-2.6.0-0.0.1.2.6.0058.1.el8.24.01.0.3.3.1.x86_64.rpm

RHEL/CentOS 8.1

aarch64

doca-host-repo-rhel81-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel81-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/CentOS 8.2

aarch64

doca-host-repo-rhel82-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel82-2.6.0-0.0.1.2.6.0058.1.el8.24.01.0.3.3.1.x86_64.rpm

RHEL/CentOS 8.3

aarch64

doca-host-repo-rhel83-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel83-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/CentOS 8.4

aarch64

doca-host-repo-rhel84-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel84-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/CentOS 8.5

aarch64

doca-host-repo-rhel85-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel85-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 8.6

aarch64

doca-host-repo-rhel86-2.6.0-0.0.1.2.6.0058.1.el8.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel86-2.6.0-0.0.1.2.6.0058.1.el8.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 8.8

aarch64

doca-host-repo-rhel88-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel88-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 8.9

aarch64

doca-host-repo-rhel89-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel89-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 9.0

aarch64

doca-host-repo-rhel90-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel90-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 9.1

aarch64

doca-host-repo-rhel91-2.6.0-0.0.1.2.6.0058.1.el9.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel91-2.6.0-0.0.1.2.6.0058.1.el9.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 9.2

aarch64

doca-host-repo-rhel92-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel92-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

RHEL/Rocky 9.3

aarch64

doca-host-repo-rhel93-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-rhel93-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

SLES 12 SP4

aarch64

doca-host-repo-sles12sp4-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-sles12sp4-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

SLES 12 SP5

aarch64

doca-host-repo-sles12sp5-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-sles12sp5-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

SLES 15 SP2

aarch64

doca-host-repo-sles15sp2-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-sles15sp2-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

SLES 15 SP3

aarch64

doca-host-repo-sles15sp3-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-sles15sp3-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

SLES 15 SP4

aarch64

doca-host-repo-sles15sp4-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-sles15sp4-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

SLES 15 SP5

aarch64

doca-host-repo-sles15sp5-2.6.0-0.0.1.24.01.0.3.3.1.aarch64.rpm

x86

doca-host-repo-sles15sp5-2.6.0-0.0.1.24.01.0.3.3.1.x86_64.rpm

Ubuntu 18.04

x86

doca-host-repo-ubuntu1804_2.6.0-0.0.1.2.6.0058.1.24.01.0.3.3.1_amd64.deb

Ubuntu 20.04

aarch64

doca-host-repo-ubuntu2004_2.6.0-0.0.1.24.01.0.3.3.1_arm64.deb

x86

doca-host-repo-ubuntu2004_2.6.0-0.0.1.2.6.0058.1.24.01.0.3.3.1_amd64.deb

Ubuntu 22.04

aarch64

doca-host-repo-ubuntu2204_2.6.0-0.0.1.2.6.0058.1.24.01.0.3.3.1_arm64.deb

x86

doca-host-repo-ubuntu2204_2.6.0-0.0.1.2.6.0058.1.24.01.0.3.3.1_amd64.deb

UOS 20 1040d

aarch64

doca-host-repo-uos201040_2.6.0-0.0.1.24.01.0.3.3.1_arm64.deb

x86

doca-host-repo-uos201040_2.6.0-0.0.1.24.01.0.3.3.1_amd64.deb

Target BlueField DPU or SuperNIC (Arm)

BlueField Software v 4.6.0

Ubuntu 22.04

aarch64

doca_2.6.0_bsp_4.6.0_ubuntu_22.04-5.24-01.prod.bfb

DOCA SDK v2.6.0

Ubuntu 22.04

aarch64

doca-dpu-repo-ubuntu2204-local_2.6.0058-1.24.01.0.3.3.0.bf.4.6.0.13035_arm64.deb

DOCA Runtime v2.6.0

DOCA Tools v2.6.0


Uninstalling Software from Host

If an older DOCA software version is installed on your host, make sure to uninstall it before proceeding with the installation of the new version:

Ubuntu/Debian

Copy
Copied!
            

$ for f in $( dpkg --list | grep doca | awk '{print $2}' ); do echo $f ; apt remove --purge $f -y ; done $ ofed_uninstall.sh --force $ sudo apt-get autoremove

CentOS/RHEL/Rocky

Copy
Copied!
            

host# for f in $(rpm -qa | grep -i doca ) ; do yum -y remove $f; done host# ofed_uninstall.sh --force host# yum autoremove host# yum makecache

Then perform the following steps:

  1. Download NVIDIA's RPM-GPG-KEY-Mellanox-SHA256 key:

    Copy
    Copied!
                

    # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox-SHA256 --2018-01-25 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox-SHA256 Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox-SHA256?   100%[=================================================>] 1,354 --.-K/s in 0s   2018-01-25 13:52:30 (247 MB/s) - ?RPM-GPG-KEY-Mellanox-SHA256? saved [1354/1354]

  2. Install the key:

    Copy
    Copied!
                

    # sudo rpm --import RPM-GPG-KEY-Mellanox-SHA256 warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED//RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) " From : /repos/MLNX_OFED//RPM-GPG-KEY-Mellanox-SHA256 Is this ok [y/N]:

  3. Verify that the key was successfully imported:

    Copy
    Copied!
                

    # rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n' | grep Mellanox gpg-pubkey-a9e4b643-520791ba gpg(Mellanox Technologies )

Installing Prerequisites on Host for Target BlueField

Install doca-tools to manage and flash the BlueField DPU or SuperNIC.

OS

Procedure

Ubuntu/Debian

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the deb repo. Run:

    Copy
    Copied!
                

    host# sudo dpkg -i doca-host-repo-ubuntu<version>_amd64.deb

  3. Perform apt update. Run:

    Copy
    Copied!
                

    host# sudo apt-get update

  4. Run apt install for DOCA Tools:

    Copy
    Copied!
                

    host# sudo apt install doca-tools

CentOS/RHEL 7.x

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the RPM repo. Run:

    Copy
    Copied!
                

    host# sudo rpm -Uvh doca-host-repo-rhel<version>.x86_64.rpm

  3. Enable new yum repos. Run:

    Copy
    Copied!
                

    host# sudo yum makecache

  4. Run yum install to install DOCA Tools:

    Copy
    Copied!
                

    host# sudo yum install doca-tools

CentOS/RHEL 8.x or Rocky 8.6

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the RPM repo. Run:

    Copy
    Copied!
                

    host# sudo rpm -Uvh doca-host-repo-rhel<version>.x86_64.rpm

  3. Enable new dnf repos. Run:

    Copy
    Copied!
                

    host# sudo dnf makecache

  4. Run dnf install to install DOCA Tools:

    Copy
    Copied!
                

    host# sudo dnf install doca-tools

Note

Skip section "Installing Software on Host" to proceed without the DOCA local repo package for host.


Determining BlueField Device ID

It is important to learn your BlueField's device-id for performing some of the software installations or upgrades in this guide.

To determine the device ID of the DPUs and SuperNICs on your setup, run:

Copy
Copied!
            

host# mst start host# mst status -v

Example output:

Copy
Copied!
            

MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 3b:00.1 mlx5_1 net-ens1f1 0   BlueField2(rev:1) /dev/mst/mt41686_pciconf0 3b:00.0 mlx5_0 net-ens1f0 0   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   e2:00.1   mlx5_1          net-ens7f1np1             4   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     e2:00.0   mlx5_0          net-ens7f0np0             4

The device IDs for the BlueField-2 and BlueField-3 networking platforms in this example are /dev/mst/mt41686_pciconf0 and /dev/mst/mt41692_pciconf0 respectively.

Installing Software on Host

Note

Skip this section if you intend to update only the BlueField software (*.bfb).

Note

Make sure to have followed the instructions under "Installing Prerequisites on Host for Target DPU".

  1. Install DOCA local repo package for host:

    Info

    The following table provides instructions for installing the DOCA host repo on your device depending on your OS and desired profile.

    OS

    Profile

    Instructions

    Deb-based

    doca-all

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Ensure that the kernel headers installed match the version of the currently running kernel.

      Info

      If the build directory exists in under /lib/modules/$(uname -r)/build, then the kernel headers are installed.

    6. Run apt install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-all

    doca-networking

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Ensure that the kernel headers installed match the version of the currently running kernel.

      Info

      If the build directory exists in under /lib/modules/$(uname -r)/build, then the kernel headers are installed.

    6. Run apt install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-networking

    doca-ofed

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# sudo dpkg -i doca-host-repo-ubuntu<version>_amd64.deb

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# sudo apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Ensure that the kernel headers installed match the version of the currently running kernel.

      Info

      If the build directory exists in under /lib/modules/$(uname -r)/build, then the kernel headers are installed.

    6. Install doca-ofed. Run:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-ofed mlnx-fw-updater

    RPM-based

    doca-all

    Note

    RHEL users need a valid subscription to install packages. See "HowTo Open RedHat Account".

    1. Install the following software dependencies. Run:

      • For RH 8.x, run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Alinux 3.x (Alibaba Cloud Linux), run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Rocky 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf makecache host# sudo dnf install epel-release host# sudo dnf install -y yum-utils host# sudo dnf config-manager --enable PowerTools host# sudo dnf clean dbcache

      • For CentOS 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf install epel-release host# sudo dnf config-manager --set-enabled PowerTools

    2. Download the DOCA host repo from section "Installation Files" for the host .

    3. Unpack the rpm repo. Run:

      Copy
      Copied!
                  

      host# rpm -Uvh <repo_file>.rpm

    4. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    5. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    6. Run yum install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-all

    doca-networking

    1. Install the following software dependencies. Run:

      • For RH 8.x, run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Alinux 3.x (Alibaba Cloud Linux), run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Rocky 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf makecache host# sudo dnf install epel-release host# sudo dnf install -y yum-utils host# sudo dnf config-manager --enable PowerTools host# sudo dnf clean dbcache

      • For CentOS 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf install epel-release host# sudo dnf config-manager --set-enabled PowerTools

      • For Oracle Linux 8.7, run:

        Copy
        Copied!
                    

        host# yum config-manager --set-enabled ol8_codeready_builder

    2. Download the DOCA host repo from section "Installation Files" for the host .

    3. Unpack the rpm repo. Run:

      Copy
      Copied!
                  

      host# rpm -Uvh <repo_file>.rpm

    4. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    5. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    6. Run yum install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-networking

    doca-ofed

    1. Download the DOCA host repo from section "Installation Files" for the host.

    2. Unpack the RPM repo. Run:

      Copy
      Copied!
                  

      host# sudo rpm -Uvh doca-host-repo-rhel<version>.x86_64.rpm

    3. Install doca-extra. Run:

      Copy
      Copied!
                  

      host# sudo yum/dnf install -y doca-extra

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Install doca-ofed. Run:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-ofed mlnx-fw-updater

  2. Load the drivers:

    Copy
    Copied!
                

    host# sudo /etc/init.d/openibd restart

  3. Initialize MST. Run:

    Copy
    Copied!
                

    host# sudo mst start

  4. Reset the nvconfig parameters to their default values:

    Copy
    Copied!
                

    host# sudo mlxconfig -d <device-id> -y reset   Reset configuration for device <device-id>? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

    Note

    To learn the BlueField's device ID, refer to section "Determining Bluefield Device ID".

  5. Skip this step if your BlueField DPU or SuperNIC is Ethernet only. Please refer to Supported Platforms to learn your Bluefield type.

    If you have a VPI networking platform, the default link type of the ports will be configured to IB. To verify your link type, run:

    Copy
    Copied!
                

    host# sudo mst start host# sudo mlxconfig -d <device-id> -e q | grep -i link_type Configurations: Default Current Next Boot * LINK_TYPE_P1 IB(1) ETH(2) IB(1) * LINK_TYPE_P2 IB(1) ETH(2) IB(1)

    Note

    If your BlueField is Ethernet capable only, then the sudo mlxconfig -d <device> command will not provide an output.

    If the current link type is set to IB, run the following command to change it to Ethernet:

    Copy
    Copied!
                

    host# sudo mlxconfig -d <device-id> s LINK_TYPE_P1=2 LINK_TYPE_P2=2

  6. Verify that RShim is active.

    Copy
    Copied!
                

    host# sudo systemctl status rshim

    This command is expected to display active (running). If RShim service does not launch automatically, run:

    Copy
    Copied!
                

    host# sudo systemctl enable rshim host# sudo systemctl start rshim

  7. Assign a dynamic IP to tmfifo_net0 interface (RShim host interface).

    Note

    Skip this step if you are installing the DOCA image on multiple DPUs.

    Copy
    Copied!
                

    host# ifconfig tmfifo_net0 192.168.100.1 netmask 255.255.255.252 up

DOCA Extra Package

If the kernel version on on your host is not supported (not shown under "Supported Operating System Distributions"), two options are available:

  • Switch to a compatible kernel.

  • Install doca-extra package:

    1. Run:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-tools doca-extra

    2. Execute the doca-kernel-support script which downloads the appropriate missing packages from your kernel to support DOCA:

      Copy
      Copied!
                  

      host# sudo /opt/mellanox/doca/tools/doca-kernel-support

      Note

      doca-kernel-support does not support customized or unofficial kernels.

Installing Software on DPU

Users have two options for installing DOCA on the DPU:

  • Upgrading the full DOCA image on the DPU (recommended) – this option overwrites the entire boot partition

  • Upgrading DOCA local repo package on the DPU – this option upgrades DOCA components without overwriting the boot partition. Use this option to preserve configurations or files on the DPU itself.

Installing Full DOCA Image on DPU via Host

Warning

This step overwrites the entire boot partition.

Note

This installation sets up the OVS bridge.

Note

If you are installing DOCA on multiple networking platforms, skip to section Installing Full DOCA Image on Multiple Networking Platforms.

Option 1 – No Pre-defined Password

Note

To set the password in advance, proceed to Option 2.

BFB installation is executed as follows:

Copy
Copied!
            

host# sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb>

Where rshimN is rshim0 if you only have one Bluefield. You may run the following command to verify:

Copy
Copied!
            

host# ls -la /dev/ | grep rshim


Option 2 – Set Pre-defined Password

Ubuntu users can provide a unique password that will be applied at the end of the BlueField software image installation. This password needs to be defined in a bf.cfg configuration file.

To set the password for the "ubuntu" user:

  1. Create password hash. Run:

    Copy
    Copied!
                

    host# openssl passwd -1 Password: Verifying - Password: $1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1

  2. Add the password hash in quotes to the bf.cfg file:

    Copy
    Copied!
                

    host# sudo vim bf.cfg ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'

    When running the installation command, use the --config flag to provide the file containing the password:

    Copy
    Copied!
                

    host# sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb> --config bf.cfg

    Note

    If --config is not used, then upon first login to the BlueField device, users will be asked to update their password.

    The following is an example of Ubuntu installation assuming the pv Linux tool has been installed (to view the installation progress).

    Copy
    Copied!
                

    host# sudo bfb-install --rshim rshim0 --bfb DOCA_<version>-aarch64.bfb --config bf.cfg Pushing bfb 1.08GiB 0:00:57 [19.5MiB/s] [ <=> ] Collecting BlueField booting status. Press Ctrl+C to stop… INFO[BL2]: start INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: runtime INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[MISC]: Ubuntu installation started INFO[MISC]: Installation finished INFO[MISC]: Rebooting...

Installing Full DOCA Image on Multiple Networking Platforms

On a host with multiple networking platforms, the BFB image can be installed on all of them using the multi-bfb-install script.

Copy
Copied!
            

host# ./multi-bfb-install --bfb <bfb-file> --password <password>

This script detects the number of RShim devices and configures them statically.

  • For Ubuntu – the script creates a configuration file /etc/netplan/20-tmfifo.yaml

  • For CentOS/RH 8.0 and 8.2 – the script installs the bridge-utils package to use the brctl command, creates the tm-br bridge, and connects all RShim interfaces to it

After the installation is complete, the configuration of the bridge and each RShim interface can be observed using ifconfig. The expected result is to see the IP on the tm-br bridge configured to 192.168.100.1 with subnet 255.255.255.0.

Note

To log into BlueField with rshim0, run:

Copy
Copied!
            

ssh ubuntu@192.168.100.2

For each RShim after that, add 1 to the fourth octet of the IP address (e.g., ubuntu@192.168.100.3 for rshim1, ubuntu@192.168.100.4 for rshim2, etc).

The script burns a new MAC address to each BlueField and configures a new IP, 192.168.100.x, as described earlier.

Installing DOCA Local Repo Package on BlueField

Note

If you have already installed BlueField image, be aware that the DOCA SDK, Runtime, and Tools are already contained in the BFB, and this installation is not mandatory. If you have not installed the BlueField image and wish to update DOCA Local Repo package, proceed with the following procedure.

Note

Before installing DOCA on the target BlueField networking platform, make sure the out-of-band interface (mgmt) is connected to the internet.

  1. Download the DOCA SDK, DOCA Runtime, and DOCA Tools package from section Installation Files.

  2. Copy deb repo package into BlueField. Run:

    Copy
    Copied!
                

    host# sudo scp -r doca-repo-aarch64-ubuntu2204-local_<version>_arm64.deb ubuntu@192.168.100.2:/tmp/

  3. Unpack the deb repo. Run:

    Copy
    Copied!
                

    dpu# sudo dpkg -i doca-dpu-repo-ubuntu2204-local_<version>_arm64.deb

  4. Run apt update.

    Copy
    Copied!
                

    dpu# sudo apt-get update

  5. Run apt install for DOCA Runtime, DOCA Tools, and DOCA SDK:

    Copy
    Copied!
                

    dpu# sudo apt install doca-runtime dpu# sudo apt install doca-tools dpu# sudo apt install doca-sdk

Upgrading Firmware

Note

If multiple Bluefield's are installed, the following steps must be performed on all of them after BFB installation.

To upgrade firmware:

  1. SSH to your BlueField device via 192.168.100.2 (preconfigured).

    Note

    If multiple BlueField's are installed, the tmfifo IP interface does not have to be 192.168.100.2. The last octate changes and depends on the RShim number.

    The default credentials for Ubuntu are as follows:

    Username

    Password

    ubuntu

    ubuntu or a unique password that you set in bf.cfg

    For example:

    Copy
    Copied!
                

    host# ssh ubuntu@192.168.100.2 Password: <configured-password>

  2. Upgrade firmware in BlueField networking platform. Run:

    Copy
    Copied!
                

    dpu# sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

    Example output:

    Copy
    Copied!
                

    Device #1: ----------   Device Type: BlueField-2 [...] Versions: Current Available FW <Old_FW> <New_FW>

  3. For the firmware upgrade to take effect:

    1. Run the following command on BlueField and host:

      Copy
      Copied!
                  

      dpu# sudo mst start

    2. Otherwise, trigger reset by running the following:

      Copy
      Copied!
                  

      dpu# sudo mlxfwreset -d /dev/mst/mt41686_pciconf0 --sync 1 -y reset

      If the mlxfwreset command fails, perform host power cycle.

      Note

      The entire BlueField networking platform will experience reset.

Post-installation Procedure

  1. Restart the driver. Run:

    Copy
    Copied!
                

    host# sudo /etc/init.d/openibd restart Unloading HCA driver: [ OK ] Loading HCA driver and Access Layer: [ OK ]

  2. Configure the physical function (PF) interfaces.

    Copy
    Copied!
                

    host# sudo ifconfig <interface-1> <network-1/mask> up host# sudo ifconfig <interface-2> <network-2/mask> up

    For example:

    Copy
    Copied!
                

    host# sudo ifconfig p2p1 192.168.200.32/24 up host# sudo ifconfig p2p2 192.168.201.32/24 up

    Pings between the source and destination should now be operational.

This dpu-upgrade procedure enables upgrading DOCA components using standard Linux tools (e.g., apt update and yum update). This process utilizes native package manager repositories to upgrade DPUs without the need for a full installation, and has the following benefits :

  • Only updates components that include modifications

    • Configurable – user can select specific components (e.g., UEFI-ATF, NIC-FW)

  • Includes upgrade of:

    • DOCA drivers and libraries

    • DOCA reference applications

    • BSP (UEFI/ATF) upgrade while maintaining the configuration

    • NIC firmware upgrade while maintaining the configuration

  • Does not:

    • Impact user binaries

    • Upgrade non-Ubuntu OS kernels

    • Upgrade DPU BMC firmware

  • After completion of DPU upgrade:

    • If NIC firmware was not updated, perform DPU Arm reset (software reset / reboot DPU)

    • If NIC firmware was updated, perform firmware reset (mlxfwreset) or perform a graceful shutdown and power cycle

OS

Action

Instructions

Ubuntu/

Debian

Remove mlxbf-bootimages package

Copy
Copied!
            

<dpu> $ apt remove --purge mlxbf-bootimages* -y

Install the the GPG key

Copy
Copied!
            

<dpu> $ apt update <dpu> $ apt install gnupg2

Export the desired distribution

Export DOCA_REPO with the relevant URL. The following is an example for Ubuntu 22.04:

Copy
Copied!
            

<dpu> $ export DOCA_REPO="https://linux.mellanox.com/public/repo/doca/2.6.0/ubuntu22.04/dpu-arm64"

Add GPG key to APT trusted keyring

Copy
Copied!
            

<dpu> $ curl $DOCA_REPO/GPG-KEY-Mellanox.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub

Add DOCA online repository

Copy
Copied!
            

<dpu> $ echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_REPO ./" > /etc/apt/sources.list.d/doca.list

Update index

Copy
Copied!
            

<dpu> $ apt update

Upgrade UEFI/ATF firmware

Run:

Copy
Copied!
            

<dpu> $ apt install mlxbf-bootimages-signed

Then i nitiate upgrade for UEFI/ATF firmware:

Copy
Copied!
            

<dpu> $ bfrec

Upgrade BlueField DPU NIC firmware

Run:

Copy
Copied!
            

<dpu> $ apt install mlnx-fw-updater-signed.aarch64

Note

This immediately starts NIC firmware upgrade.

To prevent automatic upgrade, run:

Copy
Copied!
            

<dpu> $ export RUN_FW_UPDATER=no

Upgrade system

Copy
Copied!
            

<dpu> $ apt upgrade

Apply the new changes,

NIC firmware, and UEFI/ATF

Copy
Copied!
            

<dpu> $ mlxfwreset -d /dev/mst/mt*_pciconf0 -y -l 3 --sync 1 r

Note

If mlxfwreset is not supported, graceful shutdown and host power cycle are required for the NIC firmware upgrade to take effect.

CentOS/RHEL/

Anolis/Rocky

Remove mlxbf-bootimages package

Copy
Copied!
            

<dpu> $ yum -y remove mlxbf-bootimages* <dpu> $ yum makecache

Export the desired distribution

Export DOCA_REPO with the relevant URL. The following is an example for Rocky Linux 8.6:

Copy
Copied!
            

<dpu> $ export DOCA_REPO="https://linux.mellanox.com/public/repo/doca/2.6.0/rhel8.6/dpu-arm64/"

Add DOCA online repository

Copy
Copied!
            

echo "[doca] name=DOCA Online Repo baseurl=$DOCA_REPO enabled=1 gpgcheck=0 priority=10 cost=10" > /etc/yum.repos.d/doca.repo

A file is created under /etc/yum.repos.d/doca.repo .

Update index

Copy
Copied!
            

<dpu> $ yum makecache

Upgrade UEFI/ATF firmware

Run:

Copy
Copied!
            

<dpu> $ yum install mlxbf-bootimages-signed.aarch64 mlxbf-bfscripts

Then i nitiate the upgrade for UEFI/ATF firmware:

Copy
Copied!
            

<dpu> $ bfrec

Upgrade BlueField DPU NIC firmware

The following command updates the firmware package and automatically attempts to flash the firmware to the NIC:

Copy
Copied!
            

<dpu> $ yum install mlnx-fw-updater-signed.aarch64

Note

To prevent automatic flashing of the firmware to the NIC, run the following first:

Copy
Copied!
            

<dpu> $ export RUN_FW_UPDATER=no

Info

This step can be used as a standalone firmware update. In any case, it is performed as part of the upgrade flow.

Info

Flashing the firmware to the NIC can be performed manually by running the following command, after the firmware package had been updated:

Copy
Copied!
            

sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

Upgrade system

Copy
Copied!
            

<dpu> $ yum upgrade --nobest

Apply the new changes,

NIC firmware, and UEFI/ATF

Copy
Copied!
            

<dpu> $ mlxfwreset -d /dev/mst/mt*_pciconf0 -y -l 3 --sync 1 r

Note

If mlxfwreset is not supported, a graceful shutdown and host power cycle are required for the NIC firmware upgrade to take effect.

Users wishing to build their own customized BlueField OS image can use the BFB build environment. Please refer to the bfb-build project in this GitHub webpage for more information.

Note

For a customized BlueField OS image to boot on the UEFI secure-boot-enabled BlueField (default BlueField secure boot setting), the OS must be either signed with an existing key in the UEFI DB (e.g., the Microsoft key), or UEFI secure boot must be disabled. Please refer to the "Secure Boot" page under NVIDIA BlueField DPU Platform Operating System Documentation for more details.

For full instructions about setting up a development environment, refer to the NVIDIA DOCA Developer Guide.

Installing CUDA on NVIDIA Converged Accelerator

NVIDIA® CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing GPUs.

This section details the necessary steps to set up CUDA on your environment. This section assumes that a BFB image has already been installed on your environment.
To install CUDA on your converged accelerator:

  1. Download and install the latest NVIDIA Data Center GPU driver.

  2. Download and install CUDA

    Note

    The CUDA version tested to work with DOCA SDK is 11.8.0.

Note

Downloading CUDA includes the latest NVIDIA Data Center GPU driver and CUDA toolkit. For more information about CUDA and driver compatibility, refer to the NVIDIA CUDA Toolkit Release Notes.

Configuring Operation Mode

There are two modes that the NVIDIA Converged Accelerator may operate in:

  • Standard mode (default) – the BlueField networking platform and the GPU operate separately

  • BlueField-X mode – the GPU is exposed to BlueField and is no longer visible on the host

To verify which mode the system is operating in, run:

Copy
Copied!
            

host# sudo mst start host# sudo mlxconfig -d <device-id> q PCI_DOWNSTREAM_PORT_OWNER[4]

Note

To learn the BlueField's device ID, refer to section "Determining BlueField Device ID".

  • Standard mode output:

    Copy
    Copied!
                

    Device #1: […] Configurations: Next Boot PCI_DOWNSTREAM_PORT_OWNER[4] DEVICE_DEFAULT(0)

  • BlueField-X mode output:

    Copy
    Copied!
                

    Device #1: […] Configurations: Next Boot PCI_DOWNSTREAM_PORT_OWNER[4] EMBEDDED_CPU(15)

To configure BlueField-X mode, run:

Copy
Copied!
            

host# mlxconfig -d <device-id> s PCI_DOWNSTREAM_PORT_OWNER[4]=0xF

To configure standard mode, run:

Copy
Copied!
            

host# mlxconfig -d <device-id> s PCI_DOWNSTREAM_PORT_OWNER[4]=0x0

Note

To learn the BlueField's device ID, refer to section "Determining BlueField Device ID".

Power cycle is required for configuration to take effect. For power cycle the host run:

Copy
Copied!
            

host# ipmitool power cycle


Downloading and Installing CUDA Toolkit and Driver

This section details the necessary steps to set up CUDA on your environment. It assumes that a BFB image has already been installed on your environment.

  1. Install CUDA by visiting the CUDA Toolkit Downloads webpage.

    Note

    Select the Linux distribution and version relevant for your environment.

    Note

    This section shows the native compilation option either on x86 or aarch64 hosts.

  2. Test that the driver installation completed successfully. Run:

    Copy
    Copied!
                

    dpu# nvidia-smi   Tue Apr 5 13:37:59 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA BF A10 Off | 00000000:06:00.0 Off | 0 | | 0% 43C P0 N/A / 225W | 0MiB / 23028MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

  3. Verify that the installation completed successfully.

    1. Download CUDA samples repo. Run:

      Copy
      Copied!
                  

      dpu# git clone https://github.com/NVIDIA/cuda-samples.git

    2. Build and run vectorAdd CUDA sample. Run:

      Copy
      Copied!
                  

      dpu# cd cuda-samples/Samples/0_Introduction/vectorAdd dpu# make dpu# ./vectorAdd

    Note

    If the vectorAdd sample works as expected, it should output "Test Passed".

    Note

    If it seems that the GPU is slow or stuck, stop execution and run:

    Copy
    Copied!
                

    dpu# sudo setpci -v -d ::0302 800.L=201 # CPL_VC0 = 32

GPUDirect RDMA

For information on GPUDirect RMDA and more, refer to DOCA GPUNetIO documentation.

Installing Rivermax on BlueField Networking Platform

NVIDIA Rivermax offers a unique IP-based solution for any media and data streaming use case.

This section provides the steps to install Rivermax assuming that a BFB image has already been installed on your environment.

Downloading Rivermax Driver

  1. Navigate to the NVIDIA Rivermax SDK product page.

  2. Register to be able to download the driver package using the JOIN button at the top of the page.

  3. D ownload the appropriate driver package according to your BFB under the "Linux" subsection. For example, for Ubuntu 22.04 BFB, download rivermax_ubuntu2204_<version>.tar.gz.

Installing Rivermax Driver

  1. Copy the .tgz file to BlueField:

    Copy
    Copied!
                

    host# sudo scp -r rivermax_ubuntu2204_<version>.tar.gz ubuntu@192.168.100.2:/tmp/

  2. Extract the Rivermax file:

    Copy
    Copied!
                

    dpu# sudo tar xzf rivermax_ubuntu2204_<version>.tar.gz

  3. Install the Rivermax driver package:

    Copy
    Copied!
                

    dpu# cd <rivermax-version>/Ubuntu.22.04/deb-dist/aarch64/ dpu# sudo dpkg -i rivermax_<version>.deb

Installing Rivermax Libraries from DOCA

Rivermax libraries are compatibles with DOCA components and can be found inside the doca-dpu-repo.

  1. Unpack the doca-dpu-repo:

    Copy
    Copied!
                

    dpu# sudo dpkg -i doca-dpu-repo-ubuntu2204-local_<version>_arm64.deb

  2. Run apt update:

    Copy
    Copied!
                

    dpu# sudo apt-get update

  3. Install the Rivermax libraries:

    Copy
    Copied!
                

    dpu# sudo apt install doca-rmax-libs dpu# sudo apt install libdoca-rmax-libs-dev

For additional details and guidelines, please visit the NVIDIA Rivermax SDK product page.

Info

For questions, comments, and feedback, please contact us at DOCA-Feedback@exchange.nvidia.com.

© Copyright 2024, NVIDIA. Last updated on May 7, 2024.