The following table lists an overview of the steps required to install Ubuntu BFB on your DPU:

mst start
mst status -v

MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
BlueField2(rev:1)       /dev/mst/mt41686_pciconf0.1   3b:00.1   mlx5_1          net-ens1f1                0
 
BlueField2(rev:1)       /dev/mst/mt41686_pciconf0     3b:00.0   mlx5_0          net-ens1f0                0

BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   e2:00.1   mlx5_1          net-ens7f1np1             4

BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     e2:00.0   mlx5_0          net-ens7f0np0             4

Install RShim on Host

Before installing the RShim driver, verify that the RShim devices, which will be probed by the driver, are listed under lsusb/lspci.

lspci | grep -i nox

Output example:

27:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller
27:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller
27:00.2 Non-Volatile memory controller: Mellanox Technologies NVMe SNAP Controller
27:00.3 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface

RShim Installation for DEB-based Distributions

Retrieve the .deb package from the URL provided in the BlueField Drivers for Host downloader under the "Download/Documentation" column on this page. Run:

# dpkg -i rshim_2.0.6-1.ga97dc5d_amd64.deb

rshim.service is enabled and started automatically.

Once RShim is started, it probes all the BlueField DPUs and creates the network interface tmfifo_net<N> and directory /dev/rshim<N> on each DPU.

RShim Installation for RPM-based Distributions

Retrieve the .rpm package from the URL provided in the BlueField Drivers for Host downloader under the "Download/Documentation" column on this page. Run:

# rpm -Uvh rshim-2.0.6-1.ga97dc5d.el7.centos.x86_64.rpm

rshim.service is enabled and started automatically.

Once RShim is started, it probes all the BlueField DPUs and creates the network interface tmfifo_net<N> and directory /dev/rshim<N> on each DPU.

Ensure RShim Running on Host

1. Verify RShim status. Run:

   sudo systemctl status rshim

   This command is expected to display "active (running)".

   1. If the previous command displays inactive, restart RShim service. Run:

      sudo systemctl restart rshim

   2. Verify RShim status again. Run:

      sudo systemctl status rshim

      If this command does not display "active (running)", then refer to "RShim Troubleshooting and How-Tos".

2. Display the current setting. Run: 

   # cat /dev/rshim<N>/misc | grep DEV_NAME
   DEV_NAME        pcie-04:00.2 (ro)

   This output indicates that the RShim service is ready to use.

Installing Ubuntu on BlueField

Changing Default Credentials Using bf.cfg

BFB Installation

A pre-built BFB of Ubuntu 20.04 with DOCA Runtime and DOCA packages installed is available on the NVIDIA DOCA SDK developer zone page.

To install Ubuntu BFB, run on the host side: 

# bfb-install -h
syntax: bfb-install --bfb|-b <BFBFILE> [--config|-c <bf.cfg>] \
  [--rootfs|-f <rootfs.tar.xz>] --rshim|-r <rshimN> [--help|-h]

This utility script pushes the BFB image and optional configuration to the BlueField side and checks and prints the BFB installation progress. To see the BFB installation progress, please install the pv Linux tool.

The following is an output example of Ubuntu 20.04 installation with the bfb-install script assuming pv has been installed.

# bfb-install --bfb <BlueField-OS>.bfb --config bf.cfg --rshim rshim0
Pushing bfb + cfg
1.21GiB 0:01:14 [16.5MiB/s] [ <=> ]
Collecting BlueField booting status. Press Ctrl+C to stop…
 INFO[BL2]: start
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: runtime
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: PMI: updates started
 INFO[UEFI]: PMI: boot image update
 INFO[UEFI]: PMI: updates completed, status 0
 INFO[UEFI]: PCIe enum start
 INFO[UEFI]: PCIe enum end
 INFO[MISC]: Found bf.cfg
 INFO[MISC]: Ubuntu installation started
 INFO[MISC]: Installation finished
 INFO[MISC]: Rebooting...

Verify BFB is Installed

...
INFO[MISC]: Linux up
INFO[MISC]: DPU is ready

# cat /etc/mlnx-release
DOCA_v1.1_BlueField_OS_Ubuntu_20.04-<version>

Firmware Upgrade

Customizations During BFB Installation

Additional customizations can be done to the Ubuntu OS and BlueField device during BFB installation using bf.cfg file.

Add any of the following functions to the bf.cfg file for them to be called by the install.sh script embedded in the BFB:

• bfb_modify_os – called after file the system is extracted on the target partitions. It can be used to modify files or create new files on the target file system mounted under /mnt. So the file path should look as follows: /mnt/<expected_path_on_target_OS>. This can be used to run a specific tool from the target OS (remember to add /mnt to the path for the tool).
• bfb_pre_install – called before EMMC partitions format and OS filesystem is extracted
• bfb_post_install – called as a last step before reboot. All EMMC partitions are unmounted at this stage.

For example, using the BlueField.cfg script below configures the BlueField DPU with Separated Host mode and disables the port owner from the Arm side:

# cat /root/bf.cfg

bfb_modify_os()
{
        log ===================== bfb_modify_os =====================
        log "Disable OVS bridges creation upon boot"
        sed -i -r -e 's/(CREATE_OVS_BRIDGES=).*/\1"no"/' /mnt/etc/mellanox/mlnx-ovs.conf
}

bfb_pre_install()
{
        log ===================== bfb_pre_install =====================
}

bfb_post_install()
{
        log ===================== bfb_post_install =====================
        mst start
        mst_device=$(/bin/ls /dev/mst/mt*pciconf0 2> /dev/null)

        log "Setting Separated Host mode for $mst_device"
        mlxconfig -y -d $mst_device s INTERNAL_CPU_MODEL=0

        for mst_device in /dev/mst/mt*pciconf*
        do
                log "Disable port owner from ARM side for $mst_device"
                mlxconfig -y -d $mst_device s PORT_OWNER=0
        done}

# bfb-install -b /tmp/<BlueField-OS>.bfb -c /root/bf.cfg -r rshim0
Pushing bfb + cfg
1.18GiB 0:01:11 [17.0MiB/s] [      <=>       ]
Collecting BlueField booting status. Press Ctrl+C to stop…
 INFO[BL2]: start
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: runtime
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: PMI: updates started
 INFO[UEFI]: PMI: boot image update
 INFO[UEFI]: PMI: updates completed, status 0
 INFO[UEFI]: PCIe enum start
 INFO[UEFI]: PCIe enum end
 INFO[MISC]: Found bf.cfg
 INFO[MISC]: Ubuntu installation started
 INFO[MISC]: Installing OS image
 INFO[MISC]: Disable OVS bridges creation upon boot
 INFO[MISC]: Setting Separated Host mode for /dev/mst/mt41686_pciconf0
 INFO[MISC]: Disable port owner from ARM side for /dev/mst/mt41686_pciconf0
 INFO[MISC]: Disable port owner from ARM side for /dev/mst/mt41686_pciconf0.1
 INFO[MISC]: Installation finished
 INFO[MISC]: Rebooting...

Default Ports and OVS Configuration

The /sbin/mlnx_bf_configure script runs automatically with mlx5_ib kernel module loaded (see /etc/modprobe.d/mlnx-bf.conf) and performs the following configurations:

1. Ports are configured with switchdev mode and software steering.

2. Two scalable function (SF) interfaces are created (one per port) if BlueField is configured with Embedded CPU mode (default):

   # mlnx-sf -a show
    
   SF Index: pci/0000:03:00.0/229408
     Parent PCI dev: 0000:03:00.0
     Representor netdev: en3f0pf0sf0
     Function HWADDR: 02:61:f6:21:32:8c
     Auxiliary device: mlx5_core.sf.2
       netdev: enp3s0f0s0
       RDMA dev: mlx5_2
    
   SF Index: pci/0000:03:00.1/294944
     Parent PCI dev: 0000:03:00.1
     Representor netdev: en3f1pf1sf0
     Function HWADDR: 02:30:13:6a:2d:2c
     Auxiliary device: mlx5_core.sf.3
       netdev: enp3s0f1s0
       RDMA dev: mlx5_3

   The parameters for these SFs are defined in configuration file /etc/mellanox/mlnx-sf.conf.

   /sbin/mlnx-sf --action create --device 0000:03:00.0 --sfnum 0 --hwaddr 02:61:f6:21:32:8c
   /sbin/mlnx-sf --action create --device 0000:03:00.1 --sfnum 0 --hwaddr 02:30:13:6a:2d:2c

3. Two OVS bridges are created:

   # ovs-vsctl show
   f08652a8-92bf-4000-ba0b-7996c772aff6
       Bridge ovsbr2
           Port ovsbr2
               Interface ovsbr2
                   type: internal
           Port p1
               Interface p1
           Port en3f1pf1sf0
               Interface en3f1pf1sf0
           Port pf1hpf
               Interface pf1hpf
       Bridge ovsbr1
           Port p0
               Interface p0
           Port pf0hpf
               Interface pf0hpf
           Port ovsbr1
               Interface ovsbr1
                   type: internal
           Port en3f0pf0sf0
               Interface en3f0pf0sf0
       ovs_version: "2.14.1"

   The parameters for these SFs are defined in configuration file /etc/mellanox/mlnx-ovs.conf:

   CREATE_OVS_BRIDGES="yes"
   OVS_BRIDGE1="ovsbr1"
   OVS_BRIDGE1_PORTS="p0 pf0hpf en3f0pf0sf0"
   OVS_BRIDGE2="ovsbr2"
   OVS_BRIDGE2_PORTS="p1 pf1hpf en3f1pf1sf0"
   OVS_HW_OFFLOAD="yes"
   OVS_START_TIMEOUT=30

   If failures occur in /sbin/mlnx_bf_configure or configuration changes happen (e.g. switching to separated host mode) OVS bridges are not created even if CREATE_OVS_BRIDGES="yes".

4. OVS HW offload is configured.

Default Network Interface Configuration

Network interfaces are configured using the netplan utility:

# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        tmfifo_net0:
            addresses:
            - 192.168.100.2/30
            dhcp4: false
            nameservers:
                addresses:
                - 192.168.100.1
            routes:
            -   metric: 1025
                to: 0.0.0.0/0
                via: 192.168.100.1
        oob_net0:
            dhcp4: true
    renderer: NetworkManager
    version: 2
 
 
# cat /etc/netplan/60-mlnx.yaml
network:
  ethernets:
    enp3s0f0s0:
      dhcp4: 'true'
    enp3s0f1s0:
      dhcp4: 'true'
  renderer: networkd
  version: 2

Ubuntu Boot Time Optimizations

To improve the boot time, the following optimizations were made to Ubuntu OS image: 

# cat /etc/systemd/system/systemd-networkd-wait-online.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --timeout=5

# cat /etc/systemd/system/NetworkManager-wait-online.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online --timeout=5
 
# cat /etc/systemd/system/networking.service.d/override.conf
[Service]
TimeoutStartSec=5
ExecStop=
ExecStop=/sbin/ifdown -a --read-environment --exclude=lo --force --ignore-errors

This configuration may affect network interface configuration if DHCP is used. If a network device fails to get configuration from the DHCP server, then the timeout value in the two files above must be increased.

Grub Configuration:

Added quiet to GRUB_CMDLINE_LINUX under /etc/default/grub which disables kernel output on the screen.

GRUB_CMDLINE_LINUX="console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc quiet"

Setting the Grub timeout at 2 seconds with GRUB_TIMEOUT=2 under /etc/default/grub. In conjunction with the GRUB_TIMEOUT_STYLE=countdown parameter, Grub will show the countdown of 2 seconds in the console before booting Ubuntu. Please note that, with this short timeout, the standard Grub method for entering the Grub menu (i.e., SHIFT or Esc) does not work. Function key F4 can be used to enter the Grub menu.

System Services:

The following services were disabled in the default Ubuntu OS image as they dramatically affect the boot time:

• docker.service
• containerd.service

The kexec utility can be used to reduce the reboot time. Script /usr/sbin/kexec_reboot is included in the default Ubuntu 20.04 OS image to run corresponding kexec commands.

# kexec_reboot

Ubuntu Dual Boot Support

BlueField DPU may be installed with support for dual boot. That is, two identical images of the BlueField OS may be installed using BFB.

Proposed EMMC partitioning layout for 64GB EMMC is:

Device             Start       End  Sectors  Size Type
/dev/mmcblk0p1      2048    104447   102400   50M EFI System
/dev/mmcblk0p2    104448  50660334 50555887 24.1G Linux filesystem
/dev/mmcblk0p3  50660335  50762734   102400   50M EFI System
/dev/mmcblk0p4  50762735 101318621 50555887 24.1G Linux filesystem
/dev/mmcblk0p5 101318622 122290141 20971520   10G Linux filesystem

Where:

• /dev/mmcblk0p1 - boot EFI partition for the first OS image

• /dev/mmcblk0p2 - root FS partition for the first OS image

• /dev/mmcblk0p3 - boot EFI partition for the second OS image

• /dev/mmcblk0p4 - root FS partition for the second OS image

• /dev/mmcblk0p5 - common partition for both OS images 

  The common partition can be used to store BFB files that will be used for OS image update on the non-active OS partition.

Installing Ubuntu OS Image Using Dual Boot

Add the values below to the bf.cfg configuration file (see section "Ubuntu 20.04 with DOCA Runtime and DOCA Installation" for more information).

DUAL_BOOT=yes

If EMMC size is ≤16GB, dual boot support is disabled by default, but it can be forced by setting the following parameter in bf.cfg:

FORCE_DUAL_BOOT=yes

To modify the default size of the /common partition, add the following parameter:

COMMON_SIZE_SECTORS=<number-of-sectors>

The number of sectors is the size in bytes divided by the block size (512). For example, for 10GB, the COMMON_SIZE_SECTORS=$((10*2**30/512)).

After assigning size for the /common partition, what remains is divided equally between the two OS images.

# bfb-install --bfb <BFB> --config bf.cfg --rshim rshim0

This will result in the Ubuntu OS image to be installed twice on the BlueField DPU.

Upgrading Ubuntu OS Image Using Dual Boot

1. Download the new BFB to the BlueField DPU into the /common partition. Use bfb_tool.py script to install the new BFB on the inactive BlueField DPU partition:

   /opt/mellanox/mlnx_snap/exec_files/bfb_tool.py --op fw_activate_bfb --bfb <BFB>

2. Reset BlueField DPU to load the new OS image:

   /sbin/shutdown -r 0

   BlueField DPU will now boot into the new OS image.

Use efibootmgr utility to manage the boot order if necessary.

• Change the boot order with:

  # efibootmgr -o

• Remove stale boot entries with:

  # efibootmgr -b <E> -B

  Where <E> is the last character of the boot entry (i.e., Boot000<E>). You can find that by running: 

  # efibootmgr
  BootCurrent: 0000
  Timeout: 3 seconds
  BootOrder: 0000,0001,0002
  Boot0000* CentOS
  Boot0001* Linux from mmc0
  Boot0002* EFI Internal Shell
  ....