NVIDIA BlueField DPU BSP v3.9.0

Deploying DPU OS Using BFB from Host

Warning

This procedure assumes that a BlueField DPU has already been installed in a server according to the instructions detailed in the DPU's hardware user guide.

The following table lists an overview of the steps required to install Ubuntu BFB on your DPU:

Step

Procedure

Link to Section

1

Install RShim on the host

Install RShim on Host

2

Verify that RShim is running on the host

Ensure RShim Running on Host

3

Change the default credentials using bf.cfg file (optional)

Changing Default Credentials Using bf.cfg

4

Install the Ubuntu BFB image

BFB Installation

5

Verify installation completed successfully

Verify BFB is Installed

6

Upgrade the firmware on your DPU

Firmware Upgrade

Warning

It is is important to learn your DPU's device-id for performing some of the software installations or upgrades in this guide.

To determine the device ID of the DPUs on your setup, run:

Copy
Copied!
            

mst start mst status -v

Example output:

Copy
Copied!
            

MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA BlueField2(rev:1) /dev/mst/mt41686_pciconf0.1 3b:00.1 mlx5_1 net-ens1f1 0   BlueField2(rev:1) /dev/mst/mt41686_pciconf0 3b:00.0 mlx5_0 net-ens1f0 0   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   e2:00.1   mlx5_1          net-ens7f1np1             4   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     e2:00.0   mlx5_0          net-ens7f0np0             4

The device IDs for the BlueField-2 and BlueField-3 DPUs in this example are /dev/mst/mt41686_pciconf0 and /dev/mst/mt41692_pciconf0 respectively.

Warning

If you installed the latest DOCA local repo for host, then RShim will be included as part of it. You may skip this section.

Before installing the RShim driver, verify that the RShim devices, which will be probed by the driver, are listed under lsusb/lspci.

Copy
Copied!
            

lspci | grep -i nox

Output example:

Copy
Copied!
            

27:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller 27:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller 27:00.2 Non-Volatile memory controller: Mellanox Technologies NVMe SNAP Controller 27:00.3 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface

Warning

For RShim over PCIe in multi-host configuration, host[0] which owns PCIe lane 0, will have RShim access by default. Other hosts will not see the RShim function.

RShim Installation for DEB-based Distributions

Retrieve the .deb package from the URL provided in the BlueField Drivers for Host downloader under the "Download/Documentation" column on this page. Run:

Copy
Copied!
            

# dpkg -i rshim_2.0.6-1.ga97dc5d_amd64.deb

rshim.service is enabled and started automatically.

Once RShim is started, it probes all the BlueField DPUs and creates the network interface tmfifo_net<N> and directory /dev/rshim<N> on each DPU.

RShim Installation for RPM-based Distributions

Retrieve the .rpm package from the URL provided in the BlueField Drivers for Host downloader under the "Download/Documentation" column on this page. Run:

Copy
Copied!
            

# rpm -Uvh rshim-2.0.6-1.ga97dc5d.el7.centos.x86_64.rpm

rshim.service is enabled and started automatically.

Once RShim is started, it probes all the BlueField DPUs and creates the network interface tmfifo_net<N> and directory /dev/rshim<N> on each DPU.

  1. Verify RShim status. Run:

    Copy
    Copied!
                

    sudo systemctl status rshim

    Expected output:

    Copy
    Copied!
                

    active (running) ... Probing pcie-0000:<DPU’s PCIe Bus address on host> create rshim pcie-0000:<DPU’s PCIe Bus address on host> rshim<N> attached

    Where <N> denotes RShim enumeration starting with 0 (then 1, 2, etc.) for every additional DPU installed on the server.

    If the text "another backend already attached" is displayed, users will not be able to use RShim on the host. Please refer to "RShim Troubleshooting and How-Tos" to troubleshoot RShim issues.

    1. If the previous command displays inactive, restart RShim service. Run:

      Copy
      Copied!
                  

      sudo systemctl restart rshim

    2. Verify RShim status again. Run:

      Copy
      Copied!
                  

      sudo systemctl status rshim

      If this command does not display "active (running)", then refer to "RShim Troubleshooting and How-Tos".

  2. Display the current setting. Run:

    Copy
    Copied!
                

    # cat /dev/rshim<N>/misc | grep DEV_NAME DEV_NAME pcie-04:00.2 (ro)

    This output indicates that the RShim service is ready to use.

Changing Default Credentials Using bf.cfg

Ubuntu users are prompted to change the default password (ubuntu) for the default user (ubuntu) upon first login. Logging in will not be possible even if the login prompt appears until all services are up ("DPU is ready" message appears in /dev/rshim0/misc).

Warning

Attempting to log in before all services are up prints the following message: "Permission denied, please try again."

Alternatively, Ubuntu users can provide a unique password that will be applied at the end of the BFB installation. This password would need to be defined in a bf.cfg configuration file. To set the password for the ubuntu user:

  1. Create password hash. Run:

    Copy
    Copied!
                

    # openssl passwd -1 Password: Verifying - Password: $1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1

  2. Add the password hash in quotes to the bf.cfg file:

    Copy
    Copied!
                

    # vim bf.cfg ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'

    The bf.cfg file will be used with the bfb-install script in the following step.

    Warning

    Password policy:

    • Minimum password length – 8

    • At least one upper-case letter

    • At least one lower-case letter

    • At least one numerical character

GRUB Password Protection

GRUB menu entries are protected by a username and password to prevent unwanted changes to the default boot options or parameters.

The default credentials are as follows:

Username

admin

Password

BlueField

The password can be changed during BFB installation by providing a new grub_admin_PASSWORD parameter in bf.cfg:

Copy
Copied!
            

# vim bf.cfg grub_admin_PASSWORD=’ grub.pbkdf2.sha512.10000.5EB1FF92FDD89BDAF3395174282C77430656A6DBEC1F9289D5F5DAD17811AD0E2196D0E49B49EF31C21972669D180713E265BB2D1D4452B2EA9C7413C3471C53.F533423479EE7465785CC2C79B637BDF77004B5CC16C1DDE806BCEA50BF411DE04DFCCE42279E2E1F605459F1ABA3A0928CE9271F2C84E7FE7BF575DC22935B1’

To get a new encrypted password value use the command grub-mkpasswd-pbkdf2.

After the installation, the password can be updated by editing the file /etc/grub.d/40_custom and then running the command update-grub which updates the file /boot/grub/grub.cfg.

BFB Installation

Warning

Installing the BFB does not update the firmware. To do that, refer to section Firmware Upgrade.

A pre-built BFB of Ubuntu 20.04 with DOCA Runtime and DOCA packages installed is available on the NVIDIA DOCA SDK developer zone page.

Warning

All new BlueField-2 devices are secure boot enabled, hence all SW images must be signed by NVIDIA in order to boot. All formally published SW images are signed.

To install Ubuntu BFB, run on the host side:

Copy
Copied!
            

# bfb-install -h syntax: bfb-install --bfb|-b <BFBFILE> [--config|-c <bf.cfg>] \ [--rootfs|-f <rootfs.tar.xz>] --rshim|-r <rshimN> [--help|-h]

The bfb-install utility is installed by the RShim package.

This utility script pushes the BFB image and optional configuration to the BlueField side and checks and prints the BFB installation progress. To see the BFB installation progress, please install the pv Linux tool.

Important

The first boot after BFB installation includes OS configuration. If BlueField is restarted before the configuration is complete, it will not operate as expected. For example, it may not be accessible using SSH.

The following is an output example of Ubuntu 20.04 installation with the bfb-install script assuming pv has been installed.

Copy
Copied!
            

# bfb-install --bfb <BlueField-OS>.bfb --config bf.cfg --rshim rshim0 Pushing bfb + cfg 1.21GiB 0:01:14 [16.5MiB/s] [ <=> ] Collecting BlueField booting status. Press Ctrl+C to stop… INFO[BL2]: start INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: runtime INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: PMI: updates started INFO[UEFI]: PMI: boot image update INFO[UEFI]: PMI: updates completed, status 0 INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[MISC]: Found bf.cfg INFO[MISC]: Ubuntu installation started INFO[MISC]: Installation finished INFO[MISC]: Rebooting...

Important

Interrupting the update process during its " Pushing bfb " stage may cause issues that render the DPU inoperable. To recover the DPU, re-initiate the update process with bfb-install .


Verify BFB is Installed

After installation of the Ubuntu OS is complete, the following note appears in /dev/rshim0/misc on first boot:

Copy
Copied!
            

... INFO[MISC]: Linux up INFO[MISC]: DPU is ready

"DPU is ready" indicates that all the relevant services are up and users can login the system.

After the installation of the Ubuntu 20.04 BFB, the configuration detailed in the following sections is generated.

Warning

Make sure all the services (including cloud-init) are started on BlueField before power cycling the host server.

BlueField OS image version is stored under /etc/mlnx-release in the DPU.

Copy
Copied!
            

# cat /etc/mlnx-release DOCA_v1.1_BlueField_OS_Ubuntu_20.04-<version>


Firmware Upgrade

Warning

mlxfwreset is not supported in this release. Please power cycle the host where mlxfwreset is requested.

To upgrade firmware:

  1. Set a temporary static IP on the host. Run:

    Copy
    Copied!
                

    sudo ip addr add 192.168.100.1/24 dev tmfifo_net0

  2. SSH to your DPU via 192.168.100.2 (preconfigured). The default credentials for Ubuntu are as follows.

    Username

    Password

    ubuntu

    Set during installation

    For example:

    Copy
    Copied!
                

    ssh ubuntu@192.168.100.2 Password: <unique-password>

  3. Upgrade the firmware on the DPU. Run:

    Copy
    Copied!
                

    sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

    Example output:

    Copy
    Copied!
                

    Device #1: ----------   Device Type: BlueField-2 [...] Versions: Current Available FW <Old_FW> <New_FW>

    Warning

    Important! To apply NVConfig changes, stop here and follow the steps in section "Updating NVConfig Params".

  4. Power cycle the host for the changes to take effect.

Customizations During BFB Installation

Additional customizations can be done to the Ubuntu OS and BlueField device during BFB installation using bf.cfg file.

Add any of the following functions to the bf.cfg file for them to be called by the install.sh script embedded in the BFB:

  • bfb_modify_os – called after file the system is extracted on the target partitions. It can be used to modify files or create new files on the target file system mounted under /mnt. So the file path should look as follows: /mnt/<expected_path_on_target_OS>. This can be used to run a specific tool from the target OS (remember to add /mnt to the path for the tool).

  • bfb_pre_install – called before EMMC partitions format and OS filesystem is extracted

  • bfb_post_install – called as a last step before reboot. All EMMC partitions are unmounted at this stage.

For example, using the bf.cfg script below configures the BlueField DPU with Separated Host mode and disables the port owner from the Arm side:

Copy
Copied!
            

# cat /root/bf.cfg   bfb_modify_os() { log ===================== bfb_modify_os ===================== log "Disable OVS bridges creation upon boot" sed -i -r -e 's/(CREATE_OVS_BRIDGES=).*/\1"no"/' /mnt/etc/mellanox/mlnx-ovs.conf }   bfb_pre_install() { log ===================== bfb_pre_install ===================== }   bfb_post_install() { log ===================== bfb_post_install ===================== mst start mst_device=$(/bin/ls /dev/mst/mt*pciconf0 2> /dev/null)   log "Setting Separated Host mode for $mst_device" mlxconfig -y -d $mst_device s INTERNAL_CPU_MODEL=0   for mst_device in /dev/mst/mt*pciconf* do log "Disable port owner from ARM side for $mst_device" mlxconfig -y -d $mst_device s PORT_OWNER=0 done}   # bfb-install -b /tmp/<BlueField-OS>.bfb -c /root/bf.cfg -r rshim0 Pushing bfb + cfg 1.18GiB 0:01:11 [17.0MiB/s] [ <=> ] Collecting BlueField booting status. Press Ctrl+C to stop… INFO[BL2]: start INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: runtime INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: PMI: updates started INFO[UEFI]: PMI: boot image update INFO[UEFI]: PMI: updates completed, status 0 INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[MISC]: Found bf.cfg INFO[MISC]: Ubuntu installation started INFO[MISC]: Installing OS image INFO[MISC]: Disable OVS bridges creation upon boot INFO[MISC]: Setting Separated Host mode for /dev/mst/mt41686_pciconf0 INFO[MISC]: Disable port owner from ARM side for /dev/mst/mt41686_pciconf0 INFO[MISC]: Disable port owner from ARM side for /dev/mst/mt41686_pciconf0.1 INFO[MISC]: Installation finished INFO[MISC]: Rebooting...

Warning

After modifying files on the BlueField DPU, run the command sync to flush file system buffers to EMMC flash memory to avoid data loss during reboot or power cycle.


Default Ports and OVS Configuration

The /sbin/mlnx_bf_configure script runs automatically with mlx5_ib kernel module loaded (see /etc/modprobe.d/mlnx-bf.conf) and performs the following configurations:

  1. Ports are configured with switchdev mode and software steering.

  2. Two scalable function (SF) interfaces are created (one per port) if BlueField is configured with Embedded CPU mode (default):

    Copy
    Copied!
                

    # mlnx-sf -a show   SF Index: pci/0000:03:00.0/229408 Parent PCI dev: 0000:03:00.0 Representor netdev: en3f0pf0sf0 Function HWADDR: 02:61:f6:21:32:8c Auxiliary device: mlx5_core.sf.2 netdev: enp3s0f0s0 RDMA dev: mlx5_2   SF Index: pci/0000:03:00.1/294944 Parent PCI dev: 0000:03:00.1 Representor netdev: en3f1pf1sf0 Function HWADDR: 02:30:13:6a:2d:2c Auxiliary device: mlx5_core.sf.3 netdev: enp3s0f1s0 RDMA dev: mlx5_3

    The parameters for these SFs are defined in configuration file /etc/mellanox/mlnx-sf.conf.

    Copy
    Copied!
                

    /sbin/mlnx-sf --action create --device 0000:03:00.0 --sfnum 0 --hwaddr 02:61:f6:21:32:8c /sbin/mlnx-sf --action create --device 0000:03:00.1 --sfnum 0 --hwaddr 02:30:13:6a:2d:2c

    Warning

    To avoid repeating a MAC address in the your network, the SF MAC address is set randomly upon BFB installation. You may choose to configure a different MAC address that better suit your network needs.

  3. Two OVS bridges are created:

    Copy
    Copied!
                

    # ovs-vsctl show f08652a8-92bf-4000-ba0b-7996c772aff6 Bridge ovsbr2 Port ovsbr2 Interface ovsbr2 type: internal Port p1 Interface p1 Port en3f1pf1sf0 Interface en3f1pf1sf0 Port pf1hpf Interface pf1hpf Bridge ovsbr1 Port p0 Interface p0 Port pf0hpf Interface pf0hpf Port ovsbr1 Interface ovsbr1 type: internal Port en3f0pf0sf0 Interface en3f0pf0sf0 ovs_version: "2.14.1"

    The parameters for these bridges are defined in configuration file /etc/mellanox/mlnx-ovs.conf:

    Copy
    Copied!
                

    CREATE_OVS_BRIDGES="yes" OVS_BRIDGE1="ovsbr1" OVS_BRIDGE1_PORTS="p0 pf0hpf en3f0pf0sf0" OVS_BRIDGE2="ovsbr2" OVS_BRIDGE2_PORTS="p1 pf1hpf en3f1pf1sf0" OVS_HW_OFFLOAD="yes" OVS_START_TIMEOUT=30

    Warning

    If failures occur in /sbin/mlnx_bf_configure or configuration changes happen (e.g. switching to separated host mode) OVS bridges are not created even if CREATE_OVS_BRIDGES="yes".

  4. OVS HW offload is configured.

Default Network Interface Configuration

Network interfaces are configured using the netplan utility:

Copy
Copied!
            

# cat /etc/netplan/50-cloud-init.yaml # This file is generated from information provided by the datasource. Changes # to it will not persist across an instance reboot. To disable cloud-init's # network configuration capabilities, write a file # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: ethernets: tmfifo_net0: addresses: - 192.168.100.2/30 dhcp4: false nameservers: addresses: - 192.168.100.1 routes: - metric: 1025 to: 0.0.0.0/0 via: 192.168.100.1 oob_net0: dhcp4: true renderer: NetworkManager version: 2   # cat /etc/netplan/60-mlnx.yaml network: ethernets: enp3s0f0s0: dhcp4: 'true' enp3s0f1s0: dhcp4: 'true' renderer: networkd version: 2


To improve the boot time, the following optimizations were made to Ubuntu OS image:

Copy
Copied!
            

# cat /etc/systemd/system/systemd-networkd-wait-online.service.d/override.conf [Service] ExecStart= ExecStart=/usr/bin/nm-online -s -q --timeout=5   # cat /etc/systemd/system/NetworkManager-wait-online.service.d/override.conf [Service] ExecStart= ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --timeout=5   # cat /etc/systemd/system/networking.service.d/override.conf [Service] TimeoutStartSec=5 ExecStop= ExecStop=/sbin/ifdown -a --read-environment --exclude=lo --force --ignore-errors

This configuration may affect network interface configuration if DHCP is used. If a network device fails to get configuration from the DHCP server, then the timeout value in the two files above must be increased.

Grub Configuration:

Added quiet to GRUB_CMDLINE_LINUX under /etc/default/grub which disables kernel output on the screen.

Copy
Copied!
            

GRUB_CMDLINE_LINUX="console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc quiet"

Setting the Grub timeout at 2 seconds with GRUB_TIMEOUT=2 under /etc/default/grub. In conjunction with the GRUB_TIMEOUT_STYLE=countdown parameter, Grub will show the countdown of 2 seconds in the console before booting Ubuntu. Please note that, with this short timeout, the standard Grub method for entering the Grub menu (i.e., SHIFT or Esc) does not work. Function key F4 can be used to enter the Grub menu.

System Services:

The following services were disabled in the default Ubuntu OS image as they dramatically affect the boot time:

  • docker.service

  • containerd.service

The kexec utility can be used to reduce the reboot time. Script /usr/sbin/kexec_reboot is included in the default Ubuntu 20.04 OS image to run corresponding kexec commands.

Copy
Copied!
            

# kexec_reboot

BlueField DPU may be installed with support for dual boot. That is, two identical images of the BlueField OS may be installed using BFB.

Proposed EMMC partitioning layout for 64GB EMMC is:

Copy
Copied!
            

Device Start End Sectors Size Type /dev/mmcblk0p1 2048 104447 102400 50M EFI System /dev/mmcblk0p2 104448 50660334 50555887 24.1G Linux filesystem /dev/mmcblk0p3 50660335 50762734 102400 50M EFI System /dev/mmcblk0p4 50762735 101318621 50555887 24.1G Linux filesystem /dev/mmcblk0p5 101318622 122290141 20971520 10G Linux filesystem

Where:

  • /dev/mmcblk0p1 - boot EFI partition for the first OS image

  • /dev/mmcblk0p2 - root FS partition for the first OS image

  • /dev/mmcblk0p3 - boot EFI partition for the second OS image

  • /dev/mmcblk0p4 - root FS partition for the second OS image

  • /dev/mmcblk0p5 - common partition for both OS images

    Warning

    The common partition can be used to store BFB files that will be used for OS image update on the non-active OS partition.

Installing Ubuntu OS Image Using Dual Boot

Warning

For software upgrade procedure, please refer to section "Upgrading Ubuntu OS Image Using Dual Boot".

Add the values below to the bf.cfg configuration file (see section "Ubuntu 20.04 with DOCA Runtime and DOCA Installation" for more information).

Copy
Copied!
            

DUAL_BOOT=yes

If EMMC size is ≤16GB, dual boot support is disabled by default, but it can be forced by setting the following parameter in bf.cfg:

Copy
Copied!
            

FORCE_DUAL_BOOT=yes

To modify the default size of the /common partition, add the following parameter:

Copy
Copied!
            

COMMON_SIZE_SECTORS=<number-of-sectors>

The number of sectors is the size in bytes divided by the block size (512). For example, for 10GB, the COMMON_SIZE_SECTORS=$((10*2**30/512)).

After assigning size for the /common partition, what remains is divided equally between the two OS images.

Copy
Copied!
            

# bfb-install --bfb <BFB> --config bf.cfg --rshim rshim0

This will result in the Ubuntu OS image to be installed twice on the BlueField DPU.

Upgrading Ubuntu OS Image Using Dual Boot

  1. Download the new BFB to the BlueField DPU into the /common partition. Use bfb_tool.py script to install the new BFB on the inactive BlueField DPU partition:

    Copy
    Copied!
                

    /opt/mellanox/mlnx_snap/exec_files/bfb_tool.py --op fw_activate_bfb --bfb <BFB>

  2. Reset BlueField DPU to load the new OS image:

    Copy
    Copied!
                

    /sbin/shutdown -r 0

    BlueField DPU will now boot into the new OS image.

Use efibootmgr utility to manage the boot order if necessary.

  • Change the boot order with:

    Copy
    Copied!
                

    # efibootmgr -o

  • Remove stale boot entries with:

    Copy
    Copied!
                

    # efibootmgr -b <E> -B

    Where <E> is the last character of the boot entry (i.e., Boot000<E>). You can find that by running:

    Copy
    Copied!
                

    # efibootmgr BootCurrent: 0040 Timeout: 3 seconds BootOrder: 0040,0000,0001,0002,0003 Boot0000* NET-NIC_P0-IPV4 Boot0001* NET-NIC_P0-IPV6 Boot0002* NET-NIC_P1-IPV4 Boot0003* NET-NIC_P1-IPV6 Boot0040* focal0 ....

Warning

Modifying the boot order with efibootmgr -o does not remove unused boot options. For example, changing a boot order from 0001,0002, 0003 to just 0001 does not actually remove 0002 and 0003. 0002 and 0003 need to be explicitly removed using efibootmgr -B .


© Copyright 2023, NVIDIA. Last updated on Sep 9, 2023.