DOCA Documentation v2.5.1 LTS
1.0

NVIDIA DOCA Installation Guide for Linux

This guide details the necessary steps to set up NVIDIA DOCA in your Linux environment.

Installation of the NVIDIA® BlueField® DPU software requires following the following step-by-step procedure.

Supported Platforms

Supported BlueField DPUs

The following NVIDIA® BlueField® DPUs are supported with DOCA:

NVIDIA SKU

Legacy OPN

PSID

Description

900-9D3B6-00CV-AA0

N/A

MT_0000000884

BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

900-9D3B6-00SV-AA0

N/A

MT_0000000965

BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Disabled

900-9D3B6-00CC-AA0

N/A

MT_0000001024

BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

900-9D3B6-00SC-AA0

N/A

MT_0000001025

BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Disabled

900-9D219-0086-ST1

MBF2M516A-CECOT

MT_0000000375

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0086-ST0

MBF2M516A-EECOT

MT_0000000376

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-ST1

MBF2M516A-EENOT

MT_0000000377

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D206-0053-SQ0

MBF2H332A-AENOT

MT_0000000539

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0063-ST2

MBF2H332A-AEEOT

MT_0000000540

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0083-ST3

MBF2H332A-AECOT

MT_0000000541

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0083-ST1

MBF2H322A-AECOT

MT_0000000542

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto and Secure Boot Enabled; 8GB on-board DDR; 1GbE OOB management; HHHL

900-9D206-0063-ST1

MBF2H322A-AEEOT

MT_0000000543

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 8GB on-board DDR; 1GbE OOB management; HHHL

900-9D219-0066-ST0

MBF2M516A-EEEOT

MT_0000000559

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-SN1

MBF2M516A-CENOT

MT_0000000560

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0066-ST2

MBF2M516A-CEEOT

MT_0000000561

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0006-ST0

MBF2H516A-CEEOT

MT_0000000702

BlueField-2 DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-ST2

MBF2H516A-CENOT

MT_0000000703

BlueField-2 DPU 100GbE Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0066-ST3

MBF2H516A-EEEOT

MT_0000000704

BlueField-2 DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D219-0056-SQ0

MBF2H516A-EENOT

MT_0000000705

BlueField-2 DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; PCIe Gen4 x16; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D250-0038-ST1

MBF2M345A-HESOT

MT_0000000715

BlueField-2 E-Series DPU; 200GbE/HDR single-port QSFP56; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D250-0048-ST1

MBF2M345A-HECOT

MT_0000000716

BlueField-2 E-Series DPU; 200GbE/HDR single-port QSFP56; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

900-9D218-0073-ST1

MBF2H512C-AESOT

MT_0000000723

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0083-ST2

MBF2H512C-AECOT

MT_0000000724

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0086-ST4

MBF2M516C-EECOT

MT_0000000728

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0086-SQ0

MBF2H516C-CECOT

MT_0000000729

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST5

MBF2M516C-CESOT

MT_0000000731

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST6

MBF2M516C-EESOT

MT_0000000732

BlueField-2 E-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0086-ST3

MBF2M516C-CECOT

MT_0000000733

BlueField-2 E-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST2

MBF2H516C-EESOT

MT_0000000737

BlueField-2 P-Series DPU 100GbE/EDR/HDR100 VPI Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D208-0076-ST1

MBF2H516C-CESOT

MT_0000000738

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management; Tall Bracket; FHHL

900-9D218-0083-ST4

MBF2H532C-AECOT

MT_0000000765

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0073-ST0

MBF2H532C-AESOT

MT_0000000766

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Disabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0076-ST3

MBF2H536C-CESOT

MT_0000000767

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Disabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D208-0086-ST2

MBF2H536C-CECOT

MT_0000000768

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

900-9D218-0073-ST4

MBF2H512C-AEUOT

MT_0000000972

BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled with UEFI disabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management

900-9D208-0076-STA

MBF2H516C-CEUOT

MT_0000000973

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled with UEFI disabled; Crypto Disabled; 16GB on-board DDR; 1GbE OOB management

900-9D208-0076-STB

MBF2H536C-CEUOT

MT_0000001008

BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56, integrated BMC, PCIe Gen4 x16, Secure Boot Enabled with UEFI Disabled, Crypto Disabled, 32GB on-board DDR, 1GbE OOB management, Tall Bracket, FHHL

P1004/699210040230

N/A

NVD0000000015

BlueField-2 A30X, P1004 SKU 205, Generic, GA100, 24GB HBM2e, PCIe passive Dual Slot 230W GEN4, DPU Crypto ON W/ Bkt, 1 Dongle, Black, HF, VCPD

P4028/699140280000

N/A

NVD0000000020

ZAM / NAS


Supported ConnectX NICs

The following NVIDIA® ConnectX® NICs are supported with DOCA on the host:

Hardware Prerequisites

This quick start guide assumes that an NVIDIA® BlueField® DPU has been installed in a server according to the instructions detailed in your DPU's hardware user guide .

DOCA Packages

Device

Component

Version

Description

Host

DOCA SDK

2.5.1

Software development kit package for developing host software

DOCA Runtime

2.5.1

Runtime libraries required to run DOCA-based software applications on host

DOCA Tools

2.5.1

Tools for developers and administrators on host

DOCA Extra

2.5.1

Contains helper scripts (doca-info, doca-kernel-support)

DOCA OFED

2.5.1

Software stack which operates across all NVIDIA network adapter solutions

Arm emulated (QEMU) development container

4.5.1

Linux-based BlueField Arm emulated container for developers

Target BlueField DPU (Arm)

BlueField BSP

4.5.1

BlueField image and firmware

DOCA SDK

2.5.1

Software development kit packages for developing Arm software

DOCA Runtime

2.5.1

Runtime libraries required to run DOCA-based software applications on Arm

DOCA Tools

2.5.1

Tools for developers and administrators for Arm target

Supported Operating System Distributions

This guide provides the minimal instructions for setting up DOCA on a standard system.

Installation Files

DeviceComponentOSArchLink
Host


These files contain the following components suitable for their respective OS version.

  • DOCA SDK v2.5.1
  • DOCA Runtime v2.5.1
  • DOCA Tools v2.5.1
  • DOCA Extra v2.5.1
  • DOCA OFED v2.5.1
Alinux 3.2x86doca-host-repo-alinux32-2.5.1-0.0.4.2.5.1007.1.al8.23.10.2.1.3.1.x86_64.rpm
BCLinux 21.10 SP2aarch64doca-host-repo-bclinux2110sp2-2.5.1-0.0.4.23.10.2.1.3.1.oe1.bclinux.aarch64.rpm
x86doca-host-repo-bclinux2110sp2-2.5.1-0.0.4.23.10.2.1.3.1.oe1.bclinux.x86_64.rpm
CTyunOS 2.0aarch64doca-host-repo-ctyunos20-2.5.1-0.0.4.23.10.2.1.3.1.ctl2.aarch64.rpm
x86doca-host-repo-ctyunos20-2.5.1-0.0.4.23.10.2.1.3.1.ctl2.x86_64.rpm
CTyunOS 23.01aarch64doca-host-repo-ctyunos2301-2.5.1-0.0.4.2.5.1007.1.ctl3.23.10.2.1.3.1.x86_64.rpm
x86doca-host-repo-ctyunos2301-2.5.1-0.0.4.23.10.2.1.3.1.ctl3.aarch64.rpm
Debian 10.13x86doca-host-repo-debian1013_2.5.1-0.0.4.2.5.1007.1.23.10.2.1.3.1_amd64.deb
Debian 10.8x86doca-host-repo-debian108_2.5.1-0.0.4.2.5.1007.1.23.10.2.1.3.1_amd64.deb
Debian 10.9x86doca-host-repo-debian109_2.5.1-0.0.4.23.10.2.1.3.1_amd64.deb
Kylin 1.0aarch64doca-host-repo-kylin10sp2-2.5.1-0.0.4.23.10.2.1.3.1.ky10.aarch64.rpm
x86doca-host-repo-kylin10sp2-2.5.1-0.0.4.23.10.2.1.3.1.ky10.x86_64.rpm
Oracle Linux 7.9x86doca-host-repo-ol79-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
Oracle Linux 8.4x86doca-host-repo-ol84-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
Oracle Linux 8.6x86doca-host-repo-ol86-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
Oracle Linux 8.7x86doca-host-repo-ol87-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.x86_64.rpm
Oracle Linux 9.0x86doca-host-repo-ol90-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
openEuler 20.03 SP3aarch64doca-host-repo-openeuler2003sp3-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-openeuler2003sp3-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
openEuler 22.03aarch64doca-host-repo-openeuler2203-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-openeuler2203-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 7.2x86doca-host-repo-rhel72-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 7.4x86doca-host-repo-rhel74-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 7.6aarch64doca-host-repo-rhel76-2.5.1-0.0.4.2.5.1007.1.el7a.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel76-2.5.1-0.0.4.2.5.1007.1.el7.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 7.7x86doca-host-repo-rhel77-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 7.8x86doca-host-repo-rhel78-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 7.9x86doca-host-repo-rhel79-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 8.0x86doca-host-repo-rhel80-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 8.1aarch64doca-host-repo-rhel81-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel81-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 8.2x86doca-host-repo-rhel82-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 8.3aarch64doca-host-repo-rhel83-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel83-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 8.4aarch64doca-host-repo-rhel84-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel84-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/CentOS 8.5aarch64doca-host-repo-rhel85-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel85-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 8.6aarch64doca-host-repo-rhel86-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel86-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 8.8aarch64doca-host-repo-rhel88-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel88-2.5.1-0.0.4.2.5.1007.1.el8.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 8.9aarch64doca-host-repo-rhel89-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel89-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 9.0aarch64doca-host-repo-rhel90-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel90-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 9.1aarch64doca-host-repo-rhel91-2.5.1-0.0.4.2.5.1007.1.el9.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel91-2.5.1-0.0.4.2.5.1007.1.el9.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 9.2aarch64doca-host-repo-rhel92-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel92-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
RHEL/Rocky 9.3aarch64doca-host-repo-rhel93-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-rhel93-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
SLES 12 SP4aarch64doca-host-repo-sles12sp4-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-sles12sp4-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
SLES 12 SP5aarch64doca-host-repo-sles12sp5-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-sles12sp5-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
SLES 15 SP2aarch64doca-host-repo-sles15sp2-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-sles15sp2-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
SLES 15 SP3aarch64doca-host-repo-sles15sp3-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-sles15sp3-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
SLES 15 SP4aarch64doca-host-repo-sles15sp4-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-sles15sp4-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
SLES 15 SP5aarch64doca-host-repo-sles15sp5-2.5.1-0.0.4.23.10.2.1.3.1.aarch64.rpm
x86doca-host-repo-sles15sp5-2.5.1-0.0.4.23.10.2.1.3.1.x86_64.rpm
Ubuntu 18.04x86doca-host-repo-ubuntu1804_2.5.1-0.0.4.2.5.1007.1.23.10.2.1.3.1_amd64.deb
Ubuntu 20.04x86doca-host-repo-ubuntu2004_2.5.1-0.0.4.2.5.1007.1.23.10.2.1.3.1_amd64.deb
Ubuntu 22.04aarch64doca-host-repo-ubuntu2204_2.5.1-0.0.4.2.5.1007.1.23.10.2.1.3.1_arm64.deb
x86doca-host-repo-ubuntu2204_2.5.1-0.0.4.2.5.1007.1.23.10.2.1.3.1_amd64.deb
UOS 20 1040daarch64doca-host-repo-uos201040_2.5.1-0.0.4.23.10.2.1.3.1_arm64.deb
x86doca-host-repo-uos201040_2.5.1-0.0.4.23.10.2.1.3.1_amd64.deb
Target BlueField DPU (Arm)BlueField Software v4.5.1 Ubuntu 22.04aarch64doca_2.5.1_bsp_4.5.1_ubuntu_22.04-6.24-02-lts.prod.bfb
DOCA SDK v2.5.1Ubuntu 22.04aarch64
doca-dpu-repo-ubuntu2204-local_2.5.1007-1.23.10.2.1.3.0.bf.4.5.1.13053_arm64.deb
DOCA Runtime v2.5.1
DOCA Tools v2.5.1

Uninstalling Software from Host

If an older DOCA software version is installed on your host, make sure to uninstall it before proceeding with the installation of the new version:

Ubuntu/Debian

Copy
Copied!
            

$ for f in $( dpkg --list | grep doca | awk '{print $2}' ); do echo $f ; apt remove --purge $f -y ; done $ ofed_uninstall.sh --force $ sudo apt-get autoremove

CentOS/RHEL/Rocky

Copy
Copied!
            

host# for f in $(rpm -qa | grep -i doca ) ; do yum -y remove $f; done host# ofed_uninstall.sh --force host# yum autoremove host# yum makecache

Then perform the following steps:

  1. Download NVIDIA's RPM-GPG-KEY-Mellanox-SHA256 key:

    Copy
    Copied!
                

    # wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox-SHA256 --2018-01-25 13:52:30-- http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox-SHA256 Resolving www.mellanox.com... 72.3.194.0 Connecting to www.mellanox.com|72.3.194.0|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1354 (1.3K) [text/plain] Saving to: ?RPM-GPG-KEY-Mellanox-SHA256?   100%[=================================================>] 1,354 --.-K/s in 0s   2018-01-25 13:52:30 (247 MB/s) - ?RPM-GPG-KEY-Mellanox-SHA256? saved [1354/1354]

  2. Install the key:

    Copy
    Copied!
                

    # sudo rpm --import RPM-GPG-KEY-Mellanox-SHA256 warning: rpmts_HdrFromFdno: Header V3 DSA/SHA1 Signature, key ID 6224c050: NOKEY Retrieving key from file:///repos/MLNX_OFED//RPM-GPG-KEY-Mellanox Importing GPG key 0x6224C050: Userid: "Mellanox Technologies (Mellanox Technologies - Signing Key v2) " From : /repos/MLNX_OFED//RPM-GPG-KEY-Mellanox-SHA256 Is this ok [y/N]:

  3. Verify that the key was successfully imported:

    Copy
    Copied!
                

    # rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n' | grep Mellanox gpg-pubkey-a9e4b643-520791ba gpg(Mellanox Technologies )

Installing Prerequisites on Host for Target DPU

Install doca-tools to manage and flash the BlueField DPU.

OS

Procedure

Ubuntu/Debian

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the deb repo. Run:

    Copy
    Copied!
                

    host# sudo dpkg -i doca-host-repo-ubuntu<version>_amd64.deb

  3. Perform apt update. Run:

    Copy
    Copied!
                

    host# sudo apt-get update

  4. Run apt install for DOCA Tools:

    Copy
    Copied!
                

    host# sudo apt install doca-tools

CentOS/RHEL 7.x

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the RPM repo. Run:

    Copy
    Copied!
                

    host# sudo rpm -Uvh doca-host-repo-rhel<version>.x86_64.rpm

  3. Enable new yum repos. Run:

    Copy
    Copied!
                

    host# sudo yum makecache

  4. Run yum install to install DOCA Tools:

    Copy
    Copied!
                

    host# sudo yum install doca-tools

CentOS/RHEL 8.x or Rocky 8.6

  1. Download the DOCA host repo package from the "Installation Files" section.

  2. Unpack the RPM repo. Run:

    Copy
    Copied!
                

    host# sudo rpm -Uvh doca-host-repo-rhel<version>.x86_64.rpm

  3. Enable new dnf repos. Run:

    Copy
    Copied!
                

    host# sudo dnf makecache

  4. Run dnf install to install DOCA Tools:

    Copy
    Copied!
                

    host# sudo dnf install doca-tools

Warning

Skip section "Installing Software on Host" to proceed without the DOCA local repo package for host.

Determining DPU Device ID

It is is important to learn your DPU's device-id for performing some of the software installations or upgrades in this guide.

To determine the device ID of the DPUs on your setup, run:

Copy
Copied!
            

host# mst start host# mst status -v

Example output:

Copy
Copied!
            

MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 3b:00.1 mlx5_1 net-ens1f1 0   BlueField2(rev:1) /dev/mst/mt41686_pciconf0 3b:00.0 mlx5_0 net-ens1f0 0   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   e2:00.1   mlx5_1          net-ens7f1np1             4   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     e2:00.0   mlx5_0          net-ens7f0np0             4

The device IDs for the BlueField-2 and BlueField-3 DPUs in this example are /dev/mst/mt41686_pciconf0 and /dev/mst/mt41692_pciconf0 respectively.

Installing Software on Host

Warning

Skip this section if you intend to update only the BlueField software (*.bfb).

  1. Follow the instructions under "Installing Prerequisites on Host for Target DPU".
  2. Install DOCA local repo package for host:

    Note

    The following table provides instructions for installing the DOCA host repo on your device depending on your OS and desired profile.

    OS

    Profile

    Device

    Instructions

    Ubuntu/

    Debian

    doca-all

    BlueField

    1. Download the DOCA SDK and DOCA Runtime packages from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Run apt install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-runtime doca-sdk doca-tools

    doca-cx

    ConnectX

    1. Download the DOCA SDK and DOCA Runtime packages from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# dpkg -i <repo_file>

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Run apt install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-cx-runtime doca-cx-sdk doca-cx-tools

    doca-ofed

    All

    1. Download the DOCA SDK and DOCA Runtime packages from section "Installation Files" for the host.

    2. Unpack the deb repo. Run:

      Copy
      Copied!
                  

      host# sudo dpkg -i doca-host-repo-ubuntu<version>_amd64.deb

    3. Perform apt update. Run:

      Copy
      Copied!
                  

      host# sudo apt-get update

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Install doca-ofed. Run:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-ofed

    CentOS/RHEL/

    Alinux/Rocky/

    Oracle Linux

    doca-all

    BlueField

    Warning

    RHEL users need a valid subscription to install packages. See "HowTo Open RedHat Account".

    1. Install the following software dependencies. Run:

      • For RH 8.x, run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Alinux 3.x (Alibaba Cloud Linux), run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Rocky 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf makecache host# sudo dnf install epel-release host# sudo dnf install -y yum-utils host# sudo dnf config-manager --enable PowerTools host# sudo dnf clean dbcache

      • For CentOS 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf install epel-release host# sudo dnf config-manager --set-enabled PowerTools

      • For CentOS 7.x, run:

        Copy
        Copied!
                    

        host# sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm host# sudo yum-config-manager --add-repo http://mirror.centos.org/centos/7/os/x86_64 host# sudo rpm --import http://mirror.centos.org/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-7 host# sudo yum-config-manager --add-repo http://mirror.centos.org/centos/7/extras/x86_64 host# sudo yum-config-manager --save --setopt=mirror.centos.org_centos_7_os_x86_64.exclude='pciutils* libnl3*' host# yum makecache

    2. Download the DOCA SDK and DOCA Runtime packages from section "Installation Files" for the host .

    3. Unpack the rpm repo. Run:

      Copy
      Copied!
                  

      host# rpm -Uvh <repo_file>.rpm

    4. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    5. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    6. Run yum install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-runtime doca-sdk doca-tools

    doca-cx

    ConnectX

    1. Install the following software dependencies. Run:

      • For RH 8.x, run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Alinux 3.x (Alibaba Cloud Linux), run:

        Copy
        Copied!
                    

        host# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

      • For Rocky 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf makecache host# sudo dnf install epel-release host# sudo dnf install -y yum-utils host# sudo dnf config-manager --enable PowerTools host# sudo dnf clean dbcache

      • For CentOS 8.x, run:

        Copy
        Copied!
                    

        host# sudo dnf install epel-release host# sudo dnf config-manager --set-enabled PowerTools

      • For CentOS 7.x, run:

        Copy
        Copied!
                    

        host# sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm host# sudo yum-config-manager --add-repo http://mirror.centos.org/centos/7/os/x86_64 host# sudo rpm --import http://mirror.centos.org/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-7 host# sudo yum-config-manager --add-repo http://mirror.centos.org/centos/7/extras/x86_64 host# sudo yum-config-manager --save --setopt=mirror.centos.org_centos_7_os_x86_64.exclude='pciutils* libnl3*' host# yum makecache

      • For Oracle Linux 8.7, run:

        Copy
        Copied!
                    

        host# yum config-manager --set-enabled ol8_codeready_builder

    2. Download the DOCA SDK and DOCA Runtime packages from section "Installation Files" for the host .

    3. Unpack the rpm repo. Run:

      Copy
      Copied!
                  

      host# rpm -Uvh <repo_file>.rpm

    4. Perform yum update. Run:

      Copy
      Copied!
                  

      host# sudo yum makecache

    5. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    6. Run yum install for DOCA SDK, DOCA runtime, DOCA tools:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-cx-runtime doca-cx-sdk doca-cx-tools

    doca-ofed

    All

    1. Download the DOCA SDK and DOCA Runtime packages from section "Installation Files" for the host.

    2. Unpack the RPM repo. Run:

      Copy
      Copied!
                  

      host# sudo rpm -Uvh doca-host-repo-rhel<version>.x86_64.rpm

    3. Install doca-extra. Run:

      Copy
      Copied!
                  

      host# sudo yum/dnf install -y doca-extra

    4. If the kernel version on your host is not supported (not shown under "Supported Operating System Distributions"), refer to section "DOCA Extra Package".

    5. Install doca-ofed. Run:

      Copy
      Copied!
                  

      host# sudo yum install -y doca-ofed

  3. Initialize MST. Run:

    Copy
    Copied!
                

    host# sudo mst start

  4. Reset the nvconfig parameters to their default values:

    Copy
    Copied!
                

    host# sudo mlxconfig -d <device-id> -y reset   Reset configuration for device <device-id>? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

    Warning

    To learn the DPU's device ID, refer to section "Determining DPU Device ID".

  5. Skip this step if your BlueField DPU is Ethernet only. Please refer to Supported Platforms to learn your DPU type.

    If you have a VPI DPU, the default link type of the ports will be configured to IB. To verify your link type, run:

    Copy
    Copied!
                

    host# sudo mst start host# sudo mlxconfig -d <device-id> -e q | grep -i link_type Configurations: Default Current Next Boot * LINK_TYPE_P1 IB(1) ETH(2) IB(1) * LINK_TYPE_P2 IB(1) ETH(2) IB(1)

    Warning

    If your DPU is Ethernet capable only, then the sudo mlxconfig -d <device> command will not provide an output.

    If the current link type is set to IB, run the following command to change it to Ethernet:

    Copy
    Copied!
                

    host# sudo mlxconfig -d <device-id> s LINK_TYPE_P1=2 LINK_TYPE_P2=2

  6. Verify that RShim is active.

    Copy
    Copied!
                

    host# sudo systemctl status rshim

    This command is expected to display active (running). If RShim service does not launch automatically, run:

    Copy
    Copied!
                

    host# sudo systemctl enable rshim host# sudo systemctl start rshim

  7. Assign a dynamic IP to tmfifo_net0 interface (RShim host interface).

    Warning

    Skip this step if you are installing the DOCA image on multiple DPUs.

    Copy
    Copied!
                

    host# ifconfig tmfifo_net0 192.168.100.1 netmask 255.255.255.252 up

DOCA Extra Package

If the kernel version on on your host is not supported (not shown under "Supported Operating System Distributions"), two options are available:

  • Switch to a compatible kernel.
  • Install doca-extra package:

    1. Run:

      Copy
      Copied!
                  

      host# sudo apt install -y doca-tools doca-extra

    2. Execute the doca-kernel-support script which downloads the appropriate missing packages from your kernel to support DOCA:

      Copy
      Copied!
                  

      host# sudo /opt/mellanox/doca/tools/doca-kernel-support

      Warning

      doca-kernel-support does not support customized or unofficial kernels.

Installing Software on DPU

Users have two options for installing DOCA on the DPU:

  • Upgrading the full DOCA image on the DPU (recommended) – this option overwrites the entire boot partition
  • Upgrading DOCA local repo package on the DPU – this option upgrades DOCA components without overwriting the boot partition. Use this option to preserve configurations or files on the DPU itself.

Installing Full DOCA Image on DPU via Host

Important

This step overwrites the entire boot partition.

Warning

This installation sets up the OVS bridge.

Warning

If you are installing DOCA on multiple DPUs, skip to section Installing Full DOCA Image on Multiple DPUs.

Option 1 – No Pre-defined Password

Warning

To set the password in advance, proceed to Option 2.

BFB installation is executed as follows:

Copy
Copied!
            

host# sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb>

Where rshimN is rshim0 if you only have one DPU. You may run the following command to verify:

Copy
Copied!
            

host# ls -la /dev/ | grep rshim

Option 2 – Set Pre-defined Password

Ubuntu users can provide a unique password that will be applied at the end of the BlueField software image installation. This password needs to be defined in a bf.cfg configuration file.

To set the password for the "ubuntu" user:

  1. Create password hash. Run:

    Copy
    Copied!
                

    host# openssl passwd -1 Password: Verifying - Password: $1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1

  2. Add the password hash in quotes to the bf.cfg file:

    Copy
    Copied!
                

    host# sudo vim bf.cfg ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'

    When running the installation command, use the --config flag to provide the file containing the password:

    Copy
    Copied!
                

    host# sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb> --config bf.cfg

    Warning

    If --config is not used, then upon first login to the BlueField device, users will be asked to update their password.

    The following is an example of Ubuntu installation assuming the pv Linux tool has been installed (to view the installation progress).

    Copy
    Copied!
                

    host# sudo bfb-install --rshim rshim0 --bfb DOCA_<version>-aarch64.bfb --config bf.cfg Pushing bfb 1.08GiB 0:00:57 [19.5MiB/s] [ <=> ] Collecting BlueField booting status. Press Ctrl+C to stop… INFO[BL2]: start INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: runtime INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[MISC]: Ubuntu installation started INFO[MISC]: Installation finished INFO[MISC]: Rebooting...

Installing Full DOCA Image on Multiple DPUs

On a host with multiple DPUs, the BFB image can be installed on all of them using the multi-bfb-install script.

Copy
Copied!
            

host# ./multi-bfb-install --bfb <bfb-file> --password <password>

This script detects the number of RShim devices and configures them statically.

  • For Ubuntu – the script creates a configuration file /etc/netplan/20-tmfifo.yaml
  • For CentOS/RH 7.6 – the script creates a configuration file /etc/sysconfig/network-scripts/ifcfg-br_tmfifo
  • For CentOS/RH 8.0 and 8.2 – the script installs the bridge-utils package to use the brctl command, creates the tm-br bridge, and connects all RShim interfaces to it

After the installation is complete, the configuration of the bridge and each RShim interface can be observed using ifconfig. The expected result is to see the IP on the tm-br bridge configured to 192.168.100.1 with subnet 255.255.255.0.

Warning

To log into BlueField with rshim0, run:

Copy
Copied!
            

ssh ubuntu@192.168.100.2

For each RShim after that, add 1 to the fourth octet of the IP address (e.g., ubuntu@192.168.100.3 for rshim1, ubuntu@192.168.100.4 for rshim2, etc).

The script burns a new MAC address to each DPU and configures a new IP, 192.168.100.x, as described earlier.

Installing DOCA Local Repo Package on DPU

Warning

If you have already installed BlueField image, be aware that the DOCA SDK, Runtime, and Tools are already contained in the BFB, and this installation is not mandatory. If you have not installed the BlueField image and wish to update DOCA Local Repo package, proceed with the following procedure.

Warning

Before installing DOCA on the target DPU, make sure the out-of-band interface (mgmt) is connected to the internet.

  1. Download the DOCA SDK, DOCA Runtime, and DOCA Tools package from section Installation Files.
  2. Copy deb repo package into BlueField. Run:

    Copy
    Copied!
                

    host# sudo scp -r doca-repo-aarch64-ubuntu2204-local_<version>_arm64.deb ubuntu@192.168.100.2:/tmp/

  3. Unpack the deb repo. Run:

    Copy
    Copied!
                

    dpu# sudo dpkg -i doca-dpu-repo-ubuntu2204-local_<version>_arm64.deb

  4. Run apt update.

    Copy
    Copied!
                

    dpu# sudo apt-get update

  5. Run apt install for DOCA Runtime, DOCA Tools, and DOCA SDK:

    Copy
    Copied!
                

    dpu# sudo apt install doca-runtime dpu# sudo apt install doca-tools dpu# sudo apt install doca-sdk

Upgrading Firmware

Warning

If multiple DPUs are installed, the following steps must be performed on all of them after BFB installation.

To upgrade firmware:

  1. SSH to your BlueField device via 192.168.100.2 (preconfigured).

    Warning

    If multiple DPUs are installed, the tmfifo IP interface does not have to be 192.168.100.2. The last octate changes and depends on the RShim number.

    The default credentials for Ubuntu are as follows:

    Username

    Password

    ubuntu

    ubuntu or a unique password that you set in bf.cfg

    For example:

    Copy
    Copied!
                

    host# ssh ubuntu@192.168.100.2 Password: <configured-password>

  2. Upgrade firmware in BlueField DPU. Run:

    Copy
    Copied!
                

    dpu# sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

    Example output:

    Copy
    Copied!
                

    Device #1: ----------   Device Type: BlueField-2 [...] Versions: Current Available FW <Old_FW> <New_FW>

  3. For the firmware upgrade to take effect:

    1. Run the following command on the BlueField DPU and host:

      Copy
      Copied!
                  

      dpu# sudo mst start

    2. Otherwise, trigger reset by running the following:

      Copy
      Copied!
                  

      dpu# sudo mlxfwreset -d /dev/mst/mt41686_pciconf0 --sync 1 -y reset

      If the mlxfwreset command fails, perform a graceful reboot and host power cycle.

      Warning

      The entire DPU will experience reset.

Post-installation Procedure

  1. Restart the driver. Run:

    Copy
    Copied!
                

    host# sudo /etc/init.d/openibd restart Unloading HCA driver: [ OK ] Loading HCA driver and Access Layer: [ OK ]

  2. Configure the physical function (PF) interfaces.

    Copy
    Copied!
                

    host# sudo ifconfig <interface-1> <network-1/mask> up host# sudo ifconfig <interface-2> <network-2/mask> up

    For example:

    Copy
    Copied!
                

    host# sudo ifconfig p2p1 192.168.200.32/24 up host# sudo ifconfig p2p2 192.168.201.32/24 up

    Pings between the source and destination should now be operational.

This dpu-upgrade procedure enables upgrading DOCA components using standard Linux tools (e.g., apt update and yum update). This process utilizes native package manager repositories to upgrade DPUs without the need for a full installation, and has the following benefits :

  • Only updates components that include modifications

    • Configurable – user can select specific components (e.g., UEFI-ATF, NIC-FW)

  • Includes upgrade of:

    • DOCA drivers and libraries

    • DOCA reference applications

    • BSP (UEFI/ATF) upgrade while maintaining the configuration

    • NIC firmware upgrade while maintaining the configuration

  • Does not:

    • Impact user binaries

    • Upgrade non-Ubuntu OS kernels

    • Upgrade DPU BMC firmware

  • After completion of DPU upgrade:

    • If NIC firmware was not updated, perform DPU Arm reset (software reset / reboot DPU)

    • If NIC firmware was updated, perform firmware reset (mlxfwreset) or perform a graceful shutdown and power cycle

OS

Action

Instructions

Ubuntu/

Debian

Remove mlxbf-bootimages package

Copy
Copied!
            

<dpu> $ apt remove --purge mlxbf-bootimages* -y

Install the the GPG key

Copy
Copied!
            

<dpu> $ apt update <dpu> $ apt install gnupg2

Export the desired distribution

Export DOCA_REPO with the relevant URL. The following is an example for Ubuntu 22.04:

Copy
Copied!
            

<dpu> $ export DOCA_REPO="https://linux.mellanox.com/public/repo/doca/2.5.1/ubuntu22.04/dpu-arm64"

Add GPG key to APT trusted keyring

Copy
Copied!
            

<dpu> $ curl $DOCA_REPO/GPG-KEY-Mellanox.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub

Add DOCA online repository

Copy
Copied!
            

<dpu> $ echo "deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_REPO ./" > /etc/apt/sources.list.d/doca.list

Update index

Copy
Copied!
            

<dpu> $ apt update

Upgrade UEFI/ATF firmware

Run:

Copy
Copied!
            

<dpu> $ apt install mlxbf-bootimages-signed mlxbf-scripts

Then i nitiate upgrade for UEFI/ATF firmware:

Copy
Copied!
            

<dpu> $ bfrec

Upgrade BlueField DPU NIC firmware

Run:

Copy
Copied!
            

<dpu> $ apt install mlnx-fw-updater-signed

Warning

This immediately starts NIC firmware upgrade.

To prevent automatic upgrade, run:

Copy
Copied!
            

<dpu> $ export RUN_FW_UPDATER=no

Upgrade system

Copy
Copied!
            

<dpu> $ apt upgrade

Apply the new changes, NIC firmware, and UEFI/ATF

Copy
Copied!
            

<dpu> $ mlxfwreset -d /dev/mst/mt*_pciconf0 -y -l 3 --sync 1 r

Warning

If mlxfwreset is not supported, graceful shutdown and host power cycle are required for the NIC firmware upgrade to take effect.

CentOS/RHEL/

Anolis/Rocky

Remove mlxbf-bootimages, librerswan, and openvswitch-ipsec packages

Copy
Copied!
            

<dpu> $ yum -y remove mlxbf-bootimages* <dpu> $ yum remove libreswan openvswitch-ipsec <dpu> $ yum makecache

Export the desired distribution

Export DOCA_REPO with the relevant URL. The following is an example for Rocky Linux 8.6:

Copy
Copied!
            

<dpu> $ export DOCA_REPO="https://linux.mellanox.com/public/repo/doca/2.5.1/rhel8.6/dpu-arm64/"

Add DOCA online repository

Copy
Copied!
            

echo "[doca] name=DOCA Online Repo baseurl=$DOCA_REPO enabled=1 gpgcheck=0 priority=10 cost=10" > /etc/yum.repos.d/doca.repo

A file is created under /etc/yum.repos.d/doca.repo .

Update index

Copy
Copied!
            

<dpu> $ yum makecache

Upgrade UEFI/ATF firmware

Run:

Copy
Copied!
            

<dpu> $ yum install mlxbf-bootimages-signed.aarch64 mlxbf-bfscripts

Then i nitiate the upgrade for UEFI/ATF firmware:

Copy
Copied!
            

<dpu> $ bfrec

Upgrade BlueField DPU NIC firmware

The following command updates the firmware package and automatically attempts to flash the firmware to the NIC:

Copy
Copied!
            

<dpu> $ yum install mlnx-fw-updater-signed.aarch64

Warning

To prevent automatic flashing of the firmware to the NIC, run the following first:

Copy
Copied!
            

<dpu> $ export RUN_FW_UPDATER=no

Note

This step can be used as a standalone firmware update. In any case, it is performed as part of the upgrade flow.

Note

Flashing the firmware to the NIC can be performed manually by running the following command, after the firmware package had been updated:

Copy
Copied!
            

sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

Upgrade system

Copy
Copied!
            

<dpu> $ yum upgrade --nobest

Apply the new changes, NIC firmware, and UEFI/ATF

Copy
Copied!
            

<dpu> $ mlxfwreset -d /dev/mst/mt*_pciconf0 -y -l 3 --sync 1 r

Warning

If mlxfwreset is not supported, a graceful shutdown and host power cycle are required for the NIC firmware upgrade to take effect.

Users wishing to build their own customized BlueField OS image can use the BFB build environment. Please refer to the bfb-build project in this GitHub webpage for more information.

Warning

For a customized BlueField OS image to boot on the UEFI secure-boot-enabled DPU (default DPU secure boot setting), the OS must be either signed with an existing key in the UEFI DB (e.g., the Microsoft key), or UEFI secure boot must be disabled. Please refer to the "Secure Boot" page under NVIDIA BlueField DPU Platform Operating System Documentation for more details.

For full instructions about setting up a development environment, refer to the NVIDIA DOCA Developer Guide.

Installing CUDA on NVIDIA Converged Accelerator

NVIDIA® CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing GPUs.

This section details the necessary steps to set up CUDA on your environment. This section assumes that a BFB image has already been installed on your environment.
To install CUDA on your converged accelerator:

  1. Download and install the latest NVIDIA Data Center GPU driver.

  2. Download and install CUDA

    Warning

    The CUDA version tested to work with DOCA SDK is 11.8.0.

Warning

Downloading CUDA includes the latest NVIDIA Data Center GPU driver and CUDA toolkit. For more information about CUDA and driver compatibility, refer to the NVIDIA CUDA Toolkit Release Notes.

Configuring Operation Mode

There are two modes that the NVIDIA Converged Accelerator may operate in:

  • Standard mode (default) – the BlueField DPU and the GPU operate separately

  • BlueField-X mode – the GPU is exposed to the DPU and is no longer visible on the host

To verify which mode the system is operating in, run:

Copy
Copied!
            

host# sudo mst start host# sudo mlxconfig -d <device-id> q PCI_DOWNSTREAM_PORT_OWNER[4]

Warning

To learn the DPU's device ID, refer to section "Determining DPU Device ID".

  • Standard mode output:

    Copy
    Copied!
                

    Device #1: […] Configurations: Next Boot PCI_DOWNSTREAM_PORT_OWNER[4] DEVICE_DEFAULT(0)

  • BlueField-X mode output:

    Copy
    Copied!
                

    Device #1: […] Configurations: Next Boot PCI_DOWNSTREAM_PORT_OWNER[4] EMBEDDED_CPU(15)

To configure BlueField-X mode, run:

Copy
Copied!
            

host# mlxconfig -d <device-id> s PCI_DOWNSTREAM_PORT_OWNER[4]=0xF

To configure standard mode, run:

Copy
Copied!
            

host# mlxconfig -d <device-id> s PCI_DOWNSTREAM_PORT_OWNER[4]=0x0

Warning

To learn the DPU's device ID, refer to section "Determining DPU Device ID".

Graceful reboot and power cycle are required for configuration to take effect. To power cycle the host run:

Copy
Copied!
            

host# ipmitool power cycle


Downloading and Installing CUDA Toolkit and Driver

This section details the necessary steps to set up CUDA on your environment. It assumes that a BFB image has already been installed on your environment.

  1. Install CUDA by visiting the CUDA Toolkit Downloads webpage.

    Warning

    Select the Linux distribution and version relevant for your environment.

    Warning

    This section shows the native compilation option either on x86 or aarch64 hosts.

  2. Test that the driver installation completed successfully. Run:

    Copy
    Copied!
                

    dpu# nvidia-smi   Tue Apr 5 13:37:59 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA BF A10 Off | 00000000:06:00.0 Off | 0 | | 0% 43C P0 N/A / 225W | 0MiB / 23028MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

  3. Verify that the installation completed successfully.

    1. Download CUDA samples repo. Run:

      Copy
      Copied!
                  

      dpu# git clone https://github.com/NVIDIA/cuda-samples.git

    2. Build and run vectorAdd CUDA sample. Run:

      Copy
      Copied!
                  

      dpu# cd cuda-samples/Samples/0_Introduction/vectorAdd dpu# make dpu# ./vectorAdd

    Warning

    If the vectorAdd sample works as expected, it should output "Test Passed".

    Warning

    If it seems that the GPU is slow or stuck, stop execution and run:

    Copy
    Copied!
                

    dpu# sudo setpci -v -d ::0302 800.L=201 # CPL_VC0 = 32

GPUDirect RDMA

For information on GPUDirect RMDA and more, refer to DOCA GPUNetIO documentation.

Installing Rivermax on DPU

NVIDIA Rivermax offers a unique IP-based solution for any media and data streaming use case.

This section provides the steps to install Rivermax assuming that a BFB image has already been installed on your environment.

Downloading Rivermax Driver

  1. Navigate to the NVIDIA Rivermax SDK product page.

  2. Register to be able to download the driver package using the JOIN button at the top of the page.

  3. D ownload the appropriate driver package according to your BFB under the "Linux" subsection. For example, for Ubuntu 22.04 BFB, download rivermax_ubuntu2204_<version>.tar.gz.

Installing Rivermax Driver

  1. Copy the .tgz file to the DPU:

    Copy
    Copied!
                

    host# sudo scp -r rivermax_ubuntu2204_<version>.tar.gz ubuntu@192.168.100.2:/tmp/

  2. Extract the Rivermax file:

    Copy
    Copied!
                

    dpu# sudo tar xzf rivermax_ubuntu2204_<version>.tar.gz

  3. Install the Rivermax driver package:

    Copy
    Copied!
                

    dpu# cd <rivermax-version>/Ubuntu.22.04/deb-dist/aarch64/ dpu# sudo dpkg -i rivermax_<version>.deb

Installing Rivermax Libraries from DOCA

Rivermax libraries are compatibles with DOCA components and can be found inside the doca-dpu-repo.

  1. Unpack the doca-dpu-repo:

    Copy
    Copied!
                

    dpu# sudo dpkg -i doca-dpu-repo-ubuntu2204-local_<version>_arm64.deb

  2. Run apt update:

    Copy
    Copied!
                

    dpu# sudo apt-get update

  3. Install the Rivermax libraries:

    Copy
    Copied!
                

    dpu# sudo apt install doca-rmax-libs dpu# sudo apt install libdoca-rmax-libs-dev

For additional details and guidelines, please visit the NVIDIA Rivermax SDK product page.

Note

For questions, comments, and feedback, please contact us at DOCA-Feedback@exchange.nvidia.com.

© Copyright 2023, NVIDIA. Last updated on Mar 5, 2024.