Building Your Own BFB Installation Image

Users wishing to customize their own kernel or root file system may do so using docker images supplied by Mellanox. BlueField docker images can be obtained at: https://hub.docker.com/repository/docker/mellanox/bluefield.

Docker images give users the ability to customize their own Linux distribution and create a BFB. That BFB file can then be installed on any BlueField device. Instructions are provided on the BlueField docker page as well as in the docker image itself.

Users can build all BlueField software, the Linux kernel, and OFED using the docker image and "native" or "cross" compile:

- Build using QEMU emulation on an x86 server
- Build on an Arm system (such as a BF1200 or an AWS Arm EC2 instance)
- The same docker image is used regardless of if you build native on an Arm server or use emulation on an x86 server. The only difference is how long the build takes.

Installing Linux Distribution With BFB Installation Image

Select an image according to the usage model and push it from the host machine via the RShim interface (USB or PCIe).

$ cat <Distro.bfb> > /dev/rshim0/boot

Wait for about 10-15 minutes. At the end of the installation BlueField will be rebooted and ready to use.

Pre-built BFBs of CentOS 7.6, Ubuntu 20.04, and Debian 10 with MLNX_OFED_LINUX installed are available. Please contact Mellanox Support to obtain these BFBs.

CentOS Partitioning on BlueField Device

The CentOS BFBs released by Mellanox support two partitioning schemes:

SCHEME_A (default partitioning for CentOS):

/dev/mmcblk0p1 200M /boot/efi
/dev/mmcblk0p2 1014M /boot
/dev/mmcblk0p3 13G /

SCHEME_B:

/dev/mmcblk0p1 200M /boot/efi
/dev/mmcblk0p2 4.0G /
/dev/mmcblk0p3 8.0G /home
/dev/mmcblk0p4 1.6G /var

To choose one of these schemes create the file "bf.cfg" on the host with the following line:

PART_SCHEME=<name>

Where "name" is either SCHEME_A or SCHEME_B.

Then run:

cat <CentOS.bfb> bf.cfg > /dev/rshim0/boot

Installing Official CentOS 7.x Distribution

This section provides instructions on how to set up a local environment to install the official Arm-based CentOS 7.x distribution on a BlueField system.

Requirements

Host machine running CentOS 7.

CentOS 6.2+ would need slight modification in the “setup.sh” script to set up the tftp/dhcpd services.
BlueField prebuilt packages installed under the directory BF_INST_DIR. If they are not installed yet, get the tarball file BlueField-1.2.0.xxxxxx.yyyyy.tar.xz and run:
```
$ tar Jxvf BlueField-1.2.0.xxxxxx.yyyyy.tar.xz -C <PATH>.
```

The “<BF_INST_DIR>” could then be found under the directory “<PATH>/BlueField-1.2.0.xxxxxx.yyyyy”.

Host Machine Setup

Download the CentOS installation ISO file from the following URL: http://archive.kernel.org/centos-vault/altarch/7.4.1708/isos/aarch64/CentOS-7-aarch64-Everything.iso.
If ConnectX interfaces are expected during the installation (rather than installing OFED later), download the “mlnx-ofed” file from the Mellanox Flexboot webpage by selecting “DUD & kIOS Download” → Version (e.g. 4.2-1.4.10.0) → RHEL/CentOS 7.4 → aarch64 → dd-rhel7.4-mlnx-ofed-4.2-1.4.10.0-aarch64.iso.gz. Download the file and decompress it. (This step is needed if you are performing PXE boot over the ConnectX interface.)
Navigate to "pxeboot" directory:
```
$ cd <BF_INST_DIR>/distro/rhel/pxeboot
```
Run the setup script:
```
$ ./setup.sh -d <BF_INST_DIR> -i <centos-installation.iso> [-c <ttyAMA0 | ttyAMA1 | rshim>] [-o ofed-dud.iso] [-k]
```
The option “-k” enables automatic installation according to the kickstart file ks.cfg. This option is recommended. When specified, the sample centos.ks is installed under /var/pxe/ks/ which is used for CentOS installation.

UART1 (ttyAMA1) is used by default. To specify a different console use “-c xxx”. SmartNIC uses UART0, so it takes “-c ttyAMA0”. For SmartNICs without UART connections, please use “-c rshim”.

A nonpxe.bfb is also generated which can be used to boot the device via the RShim interface. It starts CentOS installation by skipping the UEFI PXE process which should be faster. The nonpxe.bfb is generated on the same directory where the ./setup.sh is run. So you must have write permissions on this directory.

SELinux is used on CentOS/RedHat to orchestrate what programs are allowed to do, especially with regards to network access. The "setup.sh" script relies on setting up the server host as a network server and providing DHCP/PXE/NAT functionalities to the BlueField. SELinux may not allow this behavior. For example, if pinging the server host machine from the BlueField does not work, it could be that SELinux is disallowing the host. Perform "setenforce 0" on the server host in order to temporarily disable SELinux to verify whether this is the case.

Basic Yocto Installation

Connect the UART console.
Find the device file and connect to it using minicom or screen. For example:
```
$ screen /dev/ttyUSB0 115200
```
Use “yum install screen” or “yum install minicom” to install minicom/screen if not found.
For minicom, set:
• Bps/Par/Bits – 115200 8N1
• Hardware Flow Control – No
• Software Flow Control – No
Power cycle the board.
Select an image according to the board type and push it from the host machine via the RShim interface (USB or PCIe).
```
$ cat <BF_INST_DIR>/sample/install.bfb > /dev/rshim0/boot
```
Log into Linux from the UART console (root with no password). Run the following script to flash the default image and wait until it is done.The board will boot into Linux.
```
$ /opt/mlnx/scripts/bfinst --minifs
```
This step is needed to update the boot partition images.

RShim PXE Boot

Reboot the board. Once the “UEFI firmware ...” message appears on the UART console, press the “Esc” key several times to enter the UEFI boot menu.

Restart the dhcpd/tftp-server services. Run:

$ systemctl restart dhcpd; systemctl restart xinetd

Select the “Boot Manager” in the UART console and press Enter. Then select “EFI Network” in the Boot Manager and press Enter to start the PXE boot.
Check the Rx/Tx statistics on host side. Run:
```
$ ifconfig tmfifo_net0
```
After some time, a list of OS appears. Select “Install centos/7.4 AArch64 – BlueField” and press Enter to start the CentOS installation.

It takes time to fetch the Linux kernel image and initrd. So please be patient and check the Rx/Tx packet counters. The installation starts when the counters reach ~50K.

OOB PXE Boot

The OOB interface is available only on BlueField-2 based devices.

Set up the PXE boot environment by running the setup.sh script on the server host:
```
/root/BlueField-3.0.alphaX.XXXXX/distro/rhel/pxeboot/setup.sh \
-d /root/BlueField-3.0.alphaX.XXXXX/ \
-i /root/CentOS-7-aarch64-Everything.iso \
-o  /root/dd-rhel7.4-mlnx-ofed-4.2-1.4.10.0-aarch64.iso \
-c ttyAMA0 \ 
-p yocto-full-image-path \  
-k \
-s <oob-interface> \  
-m <oob-mac>
```
Where:
- -d points to the location from which the tar file has been extracted. The script uses this directory to find all the source code it needs.
- -i points to the OS installation disk. This is the image that is accessed via PXE boot to install the OS on BlueField-2.
- -o points to the Mellanox OFED driver disk for Arm. Download and extract it from the Mellanox FlexBoot page.
- -c specifies the default UART port for the OS to use since the BlueField-2 DPU has two Arm UARTs. Use "ttyAMA0" for BlueField-2, which is UART0.
- -p (optional) points to the Yocto full image path.
- -k (optional) kickstarts auto-installation based on a default kickstart file which is installed as /var/pxe/ks/oob_centos.ks or /var/pxe/ks/oob_yocto.ks.
- -s points to the name of the OOB network interface on the server host. It can be retrieved from the host via "ifconfig -a".
  
  This parameter needs to be passed along with -m <oob-mac>.
- -m points to the BlueField-2 OOB MAC address, which can be retrieved from the BlueField prompt via "ifconfig oob_net0". The format should be xx:xx:xx:xx:xx:xx.
  
  This parameter needs to be passed along with -s <oob-interface>.
Reboot the board. Once the “UEFI firmware ...” message appears on the UART console, press the “Esc” key several times to enter the UEFI boot menu.
Select the “Boot Manager” in the UART console and press Enter.
Then from the Boot Manager, select “EFI Network” which corresponds to the IPv4 OOB network interface, and press Enter to start the PXE boot.
Check the Rx/Tx statistics on host side. Run:
```
$ ifconfig <oob-interface>
```
After some time, a list of OS appears. Select “Install centos/7 AArch64 via OOB - BlueField” to boot via OOB. Then press Enter to start the CentOS installation.

CentOS Installation

Follow the installation wizard.

Post-installation

Enable “yum install” or external network access after CentOS installation.

On the host side, run:

$ systemctl restart dhcpd
$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ iptables -t nat -A POSTROUTING -o <out_intf> -j MASQUERADE
'<out_intf>' is the outgoing network interface to the network.

Check the IP address for tmfifo_net0 on the BlueField device side. Run:

$ ifconfig tmfifo_net0
tmfifo_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.2  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::21a:caff:feff:ff01  prefixlen 64  scopeid 0x20<link>
        ether 00:1a:ca:ff:ff:01  txqueuelen 1000  (Ethernet)
        RX packets 95  bytes 14106 (13.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 104  bytes 13219 (12.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

If an IP address is not assigned, run:

$ ifdown tmfifo_net0; ifup tmfifo_net0

Install driver RPMs from source.

This step is not needed if OFED is installed.
- If “rpmbuild” is not available, run:
```
$ yum install rpm-build
```
- If development tools are not available, run:
```
$ yum group install "Development Tools"
```
- If kernel-devel is not installed, run:
```
$ yum install kernel-devel-`uname -r`
```
All driver source RPMs are located at <BF_INST_DIR>/distro/SRPMS. Upload them to the target (e.g. under /opt).
The following is an example which exhibits how to install i2c-mlnx-1.0-0.g6af3317.src.rpm.
```
$ cd /opt
$ rpmbuild --rebuild i2c-mlnx-1.0-0.g6af3317.src.rpm
$ cd ~/rpmbuild/RPMS/aarch64/
$ rpm -ivh i2c-mlnx-1.0-0.g6af3317_4.11.0_22.el7a.aarch64.rpm
```
The tmfifo driver is also included in the initramfs. Remove it from initramfs as instructed (3.a) below when upgrading the tmfifo driver.
Install OFED (optional) as instructed in section "Installing MLNX_OFED on the SmartNIC".

If MLNX_OFED_LINUX is installed with “--add-kernel-support”, run “dracut -f” to update drivers in initramfs after MLNX_OFED_LINUX installation.

Building a New bluefield_dd ISO Image

To build an updated driver disk for a different version of RHEL, you may use the sources located in the “bluefield_dd” directory. Run the build-dd.sh script there and provide it as an argument the path to the kernel-devel RPM, and it then builds a matching driver disk.

Typically, you would do this as a cross-build when initially booting up a BlueField system, so the cross-build environment must be configured prior to running the script.

Note that for the Yocto SDK, you must “unset LDFLAGS” before running the script, since the kernel build uses raw LDFLAGS via “ld” rather than via “gcc” as the Yocto SDK assumes.

So, for example:

source /path/to/SDK/environment-setup-aarch64-poky-linux
unset LDFLAGS
./build-dd.sh /path/to/kernel-devel-4.11.0-22.el7a.aarch64.rpm

This generates a suitable .iso file in the current directory.

PXE Boot Flow

UEFI allows booting over PXE in the same way that is familiar with other operating system installations.

If the BlueField eMMC is already provisioned with a bootstream containing ATF and UEFI, it should power on and boot up with that bootstream. If the eMMC also contains a bootable kernel, you would need to interrupt the boot by hitting “Esc” quickly once UEFI starts up. This takes you to the UEFI main menu on the serial console. If the eMMC is provisioned with a bootstream but does not contain a bootable kernel, you would enter the UEFI main menu on the serial console automatically at power on.

If the eMMC is not yet provisioned with a bootstream, you would need to boot it externally using USB or PCIe. In that case, providing a bootstream containing only ATF and UEFI would boot the chip and you automatically enter the UEFI main menu on the serial console.

From this point, you may navigate to the network boot option and select the primary ConnectX network Interface; or the OOB network interface; or select the RShim network interface, which would bridge to the external host over USB or PCIe, and use its network interface instead.

At this stage, you should be able to boot the CentOS installation media from a PXE server in the normal manner, loading Grub and then selecting your kernel of choice from the media.

RShim PXE Boot Over VLAN

This function is available only on BlueField-2 DPU.

This section describes how to perform Yocto/CentOS PXE boot and installation over VLAN interface. For simplicity, this procedure sets up the PXE server on the server host machine which has the RShim (USB or PCIe) connection to the BlueField board, and assumes that the BlueField release tarball has been installed under directory <BF_INSTALL>.

Set up the PXE server on the server host as root.
```
$ cd <BF_INSTALL>/distro/rhel/pxeboot/
$ ./setup.sh -i <centos-iso> -d <BF_INSTALL> -c <ttyAMA0 | ttyAMA1 | rshim> -k -p <yocto-full-image-path>
```
The parameter “-p <yocto-full-image-path>” is not needed for CentOS. Once specified, however, the script sets up PXE server environment for Yocto as well. Type “./setup.sh” to see more help information.
The default grub configuration file is /var/lib/tftpboot/grub.cfg. It can be customized it as needed, such as adding VLAN support (see step 4 below).
The default kickstart file is /var/pxe/ks/yocto.ks (for Yocto) or /var/pxe/ks/centos.ks (for CentOS). They can be customized as well.
Note that this script also copies the patched grubaa64.efi file from the directory <BF_INST_DIR>/distro/rhel/pxeboot/patches/ to /var/lib/tftpboot/. This file is needed since the default grubaa64.efi from the CentOS ISO does not support PXE over VLAN. See section “Patch to Support PXE Boot Over VLAN” for more details about this patch.
Once done, reset BlueField and enter the UEFI Boot Manager to pick the network interface for PXE boot.

Configure VLAN in UEFI.
In order to PXE boot over VLAN, VLAN needs to be created in UEFI and on the server host.VLANs can be created inside the UEFI boot menu. Once created, they are stored in the UEFI persistent variables. It is possible to write the UEFI variables directly if needed without having to enter the UEFI boot menu.Press “ESC” key to enter UEFI boot menu, then follow the steps below.

2.1) Select "Device Manager".
==============================================================================
   Continue                                             This selection will
   Select Language            <Standard English>        take you to the
   Boot Manager                                         Device Manager
>  Device Manager
   Boot Maintenance Manager
 
 
2.2) Select "Network Device List".
==============================================================================
   Devices List                                         Select the network
   Secure Boot Configuration                            device according the
   iSCSI Configuration                                  MAC address
>  Network Device List
   Driver Health
     The platform is healthy
 
 
2.3) Select "MAC:00:1A:CA:FF:FF:01".
==============================================================================
   Network Device List                                  Network Device
>  MAC:00:1A:CA:FF:FF:01
   MAC:50:6B:4B:08:CE:EB
   MAC:50:6B:4B:08:CE:EC
   Driver Health
     The platform is healthy
 
2.4) Select "VLAN configuration".
==============================================================================
   Network Device                                       VLAN Configuration
>  VLAN Configuration                                   (MAC:001ACAFFFF01)
   Driver Health
     The platform is healthy
 
 
2.5) Select "Enter Configuration Menu".
==============================================================================
>  Enter Configuration Menu                             Press ENTER to enter
                                                        configuration menu
                                                        for VLAN configuration.
 
2.6) Fill in the VLAN ID value, then move down to "Add VLAN" and press enter.
==============================================================================
   Create new VLAN                                      VLAN ID of new VLAN
     VLAN ID                  [100  ]                   or existing VLAN,
     Priority                 [0]                       valid value is 0~4094
   Add VLAN
 
   Configured VLAN List
   Remove VLAN
 
2.7) Repeat step 2.6 to create all the VLANs. They will have an output like the
     one presented below. Select the VLAN ID and run the action "Remove VLAN" to
     delete the selected VLAN.
==============================================================================
   Create new VLAN                                      Create a new VLAN or
     VLAN ID                  [0]                       update existing VLAN
     Priority                 [0]
   Add VLAN
 
   Configured VLAN List
     VLAN ID: 100, Priority:0 [ ]
     VLAN ID: 200, Priority:0 [ ]
   Remove VLAN
 

2.8) Reboot or Press 'ESC' keys to go back to the "Boot Manager".
     VLAN interfaces will be show up in the EFI Network list. Select it and press
     enter to start PXE boot over this VLAN.
==============================================================================
                                                        Device Path :
   Boot Option Menu                                     MAC(001ACAFFFF01,0x1)/
                                                        Vlan(100)/IPv4(0.0.0.0)
   EFI Network                                         
>  EFI Network 2
   EFI Network 3
   CentOS 7.4
   EFI Network 6
   EFI Network 7
   EFI Internal Shell
   Linux from mmc0
   EFI Misc Device
   ...

If needed, the boot order can be modified in the UEFI shell to boot from VLAN interface by default as follows:

Shell> bcfg boot mv 4 0

This moves boot entry 4 to 0.

Create VLAN interface on the server host.
The VLAN interface needs to be configured on the server host side, such as modifying /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0 by changing “DEVICE=tmfifo_net0” to “DEVICE=tmfifo_net0.100” and add line “VLAN=yes”.
Then restart the RShim service (systemctl restart rshim) to make it take effect.

Configure VLAN in /var/lib/tftpboot/grub.cfg on the server host. The examples below enable VLAN 100 and 200 on the RShim network interface tmfifo_net0.

For Yocto:

menuentry 'Install Yocto AArch64 - BlueField' --class red --class gnu-linux --class gnu --class os {
 linux /yocto/vmlinuz ro ip=dhcp console=ttyAMA0 bfks=http://192.168.100.1/ks/yocto.ks bfnet="tmfifo_net0.100:dhcp" bfnet="tmfifo_net0.200:dhcp" initrd /yocto/initrd.img 
}

For CentOS:

menuentry 'Install centos/7 AArch64 - BlueField' --class red --class gnu-linux --class gnu --class os {
 linux (tftp)/centos/7/vmlinuz ro ip=dhcp repo=http://192.168.100.1/centos7 inst.dd=/bluefield_dd.iso vlan=tmfifo_net0.100:tmfifo_net0 vlan=tmfifo_net0.200:tmfifo_net0 console=ttyAMA1 initrd (tftp)/centos/7/initrd.img
}

In order to perform PXE over VLAN for CentOS, /var/pxe/ks/centos.ks needs to be updated as well by adding “--vlanid=100 --vlanid=200” to the line of “network --bootproto...” to create VLAN 100 and 200.

Patch to Support PXE Boot Over VLAN

The directory “<BF_INST_DIR>/distro/rhel/pxeboot/patches/” contains patches to the default grub from CentOS to support PXE boot over VLAN, and a prebuilt grubaa64.efi binary in order to simplify matters. The patch is based on the https://git.centos.org/r/rpms/grub2.git branch remotes/origin/c7-alt. Below is the top commit of this branch after applying all the existing patches according to the RPM SPEC file.

commit 66d2044d7fb6c3bd0b170a9b0828d9a0b5dce582
Author: CentOS Sources <bugs@centos.org>
Date:   Tue Oct 30 01:04:32 2018 -0400
 
    debrand grub2-2.02-0.76.el7

Copy the patches to the SOURCE/ directory and add them into SOURCE/grub.patches as given below. Then rebuild the RPM SPEC file to generate the grubaa64.efi.

diff --git a/SOURCES/grub.patches b/SOURCES/grub.patches
index 26b949d..38f7b03 100644
--- a/SOURCES/grub.patches
+++ b/SOURCES/grub.patches
@@ -286,3 +286,6 @@ Patch0285: 0285-editenv-handle-relative-symlinks.patch
 Patch0286: 0286-efinet-also-use-the-firmware-acceleration-for-http.patch
 Patch0287: 0287-Make-root_url-reflect-the-protocol-hostname-of-our-b.patch
 Patch0289: 0288-efi-uga-Fix-PCIe-LER-when-GRUB2-accesses-non-enabled.patch
+Patch0290: 0290-efinet-Add-EFI-boot-path-support-for-VLAN.patch
+Patch0291: 0291-Fix-potential-IP-fragment-issue.patch
+Patch0292: 0292-Increase-tftp-blksize-to-boost-performance.patch

This VLAN issue and patches have been submitted to CentOS Bug Tracker https://bugs.centos.org/view.php?id=15307.

OOB PXE Boot Over VLAN

This section describes how to perform Yocto/CentOS PXE boot and installation over VLAN interface. For simplicity, this procedure sets up the PXE server on the server host machine which has the OOB connection to the BlueField board, and assumes that the BlueField release tarball has been installed under directory <BF_INSTALL>.

Please follow the steps in "34262210"

Configure VLAN in UEFI.
In order to PXE boot over VLAN, VLAN needs to be created in UEFI and on the server host.VLANs can be created inside the UEFI boot menu. Once created, they are stored in the UEFI persistent variables. It is possible to write the UEFI variables directly if needed without having to enter the UEFI boot menu.Press “ESC” key to enter UEFI boot menu, then follow the steps below.

2.1) Select "Device Manager".
==============================================================================
   Continue                                             This selection will
   Select Language            <Standard English>        take you to the
   Boot Manager                                         Device Manager
>  Device Manager
   Boot Maintenance Manager




  2.2) Select "Network Device List".
==============================================================================
   Devices List                                         Select the network
   Secure Boot Configuration                            device according the
   iSCSI Configuration                                  MAC address
>  Network Device List
   Driver Health
     The platform is healthy


2.3) Select the OOB MAC Address
==============================================================================
   Network Device List                                  Network Device
>  MAC:00:1A:CA:FF:FF:01
   MAC:50:6B:4B:08:CE:EB
   MAC:50:6B:4B:08:CE:EC
   Driver Health
     The platform is healthy 




2.4) Select "VLAN configuration".
==============================================================================
   Network Device                                       VLAN Configuration
>  VLAN Configuration                                   (MAC:************)
   Driver Health
     The platform is healthy




2.5) Select "Enter Configuration Menu".
==============================================================================
>  Enter Configuration Menu                             Press ENTER to enter
                                                        configuration menu
                                                        for VLAN configuration. 




2.6) Fill in the VLAN ID value, then move down to "Add VLAN" and press enter.
==============================================================================
   Create new VLAN                                      VLAN ID of new VLAN
     VLAN ID                  [100  ]                   or existing VLAN,
     Priority                 [0]                       valid value is 0~4094
   Add VLAN


   Configured VLAN List
   Remove VLAN 




2.7) Repeat step 2.6 to create all the VLANs. They will have an output like the
     one presented below. Select the VLAN ID and run the action "Remove VLAN" to
     delete the selected VLAN.
===============================================================================
  Create new VLAN                                      Create a new VLAN or
     VLAN ID                  [0]                      update existing VLAN
     Priority                 [0]
   Add VLAN


   Configured VLAN List
     VLAN ID: 100, Priority:0 [ ]
     VLAN ID: 200, Priority:0 [ ]
   Remove VLAN 




2.8) Reboot or Press 'ESC' keys to go back to the "Boot Manager".
     VLAN interfaces will be show up in the EFI Network list. Select it and press
     enter to start PXE boot over this VLAN.
=================================================================================
                                                        Device Path :
   Boot Option Menu                                     MAC(************,0x1)/
                                                        Vlan(100)/IPv4(0.0.0.0)
   EFI Network
>  EFI Network 2
   EFI Network 3
   CentOS 7.4
   EFI Network 6
   EFI Network 7
   EFI Internal Shell
   Linux from mmc0
   EFI Misc Device
   ...

If needed, the boot order can be modified in the UEFI shell to boot from VLAN interface by default as follows:

Shell> bcfg boot mv 4 0

This moves boot entry 4 to 0.

Create OOB VLAN interface on the server host.
The VLAN interface needs to be configured on the server host side, by:
```
cp /etc/sysconfig/network-scripts/ifcfg-<oob-interface> /etc/sysconfig/network-scripts/ifcfg-<oob-interface>.100
```
Change “DEVICE=<oob-interface>” to “DEVICE=<oob-interface>.100” and add line “VLAN=yes”.
```
vi /etc/sysconfig/network-scripts/ifcfg-em2
#comment out this line "IPADDR=192.168.101.1"
```
Restart the networking service in order for the changes to take effect. As root issue the following command:
```
systemctl restart network
```

Configure VLAN in /var/lib/tftpboot/grub.cfg on the server host. The examples below enable VLAN 100 and 200 on the OOB network interface.

For Yocto:

menuentry 'Install Yocto AArch64 - BlueField' --class red --class gnu-linux --class gnu --class os {
 linux /yocto/vmlinuz ro ip=dhcp console=ttyAMA0 bfks=http://192.168.101.1/ks/yocto.ks bfnet="<oob-interface>.100:dhcp" bfnet="<oob-interface>.200:dhcp" initrd /yocto/initrd.img 
}

For CentOS:

menuentry 'Install centos/7 AArch64 - BlueField' --class red --class gnu-linux --class gnu --class os {
 linux (tftp)/centos/7/vmlinuz ro ip=dhcp repo=http://192.168.101.1/centos7 inst.dd=/bluefield_dd.iso vlan=<oob-interface>.100:<oob-interface> vlan=<oob-interface>.200:<oob-interface> console=ttyAMA1 initrd (tftp)/centos/7/initrd.img
}

In order to perform PXE over VLAN for CentOS, /var/pxe/ks/oob_centos.ks needs to be updated as well by adding “--vlanid=100 --vlanid=200” to the line of “network --bootproto...” to create VLAN 100 and 200.

Non-PXE Boot Flow

It is possible to explicitly boot the PXE boot components of CentOS directly over USB or PCIe to avoid the requirement of a PXE server being available on the network. This is somewhat equivalent to booting a local bootable CD and then using another media source to find all the packages to install.

The root of the CentOS install image, for example $ROOT, corresponds to the root of the ISO image file; that is, the directory which contains EFI, EULA, GPL, LiveOS, Packages, etc. You should create a bootstream which includes the pxeboot kernel and initramfs and use that to boot the image. This is assuming the initrd.img file in this directory has already been updated to include bluefield_dd.iso, as described in the previous section.

You must also determine from where to load the ISO image for the installation. In this example, that destination is http://1.2.3.4/rhel.

After the kernel image is uncompressed, it must be placed, along with the initrd.img, in a BlueField bootstream. To do so, “cd” to the “samples” directory of the BlueField Runtime Distribution, make sure that the “bin” directory is on your $PATH, and run:

gunzip < $ROOT/images/pxeboot/vmlinuz > /tmp/Image
build-bfb \
  --bfb ../boot/default.bfb \
  --kernel Image \
  --initramfs $ROOT/images/pxeboot/initrd.img \
  --bootarg "inst.dd=/dw_mmc.iso inst.repo=http://1.2.3.4/rhel" \
  --no-gpt -i rshim pxeboot.bfb

This bootstream can then be used to boot the BlueField in the normal way. It comes up in text mode on the console; you may also select to run the installer from a VNC client. When running the setup.sh script, a nonpxe.bfb is generated in the same way.

Installation Troubleshooting and FAQ

How to reset the board or NIC via the RShim interface from host side

Run the following:

$ echo "SW_RESET 1" > /dev/rshim<N>/misc

Make sure to replace “rshim<N>” with the actual name (e.g. rshim0).

How to upgrade existing CentOS to some BlueField release without reinstallation

Follow Step 3 of section “‎Basic Yocto Installation” to push the installation image via RShim (USB or PCIe).

Do not run the bfinst script, or else a new installation will start.
Run “/opt/mlnx/scripts/bfrec” to upgrade the boot partitions. This step upgrades the ATF & UEFI images to this release.
Reboot into CentOS. Follow Step 2 of section “‎Post-installation” to install/upgrade the tmfifo driver and other drivers as needed from the source RPM.

To re-run the “setup.sh” script after host reboot before installing CentOS

Either re-run the script, or mount /var/pxe/centos7 to the CentOS ISO file.

How to change the MAC address of the tmfifo network interface (Arm side)

See section “‎‎Permanently Changing the MAC Address of the Arm Side”.

Why CentOS (Arm side) did not get DHCP address on tmfifo interface (eth0) after re-boot

The host-side DHCP daemon must be restarted after board reboot in order to provide DHCP service. A configuration file could accomplish this automatically.

$ Create /sbin/ifup-local or add to it.
INTF=$1
if [ "$INTF" = "tmfifo_net0" ]; then
    systemctl restart dhcpd
fi

How to kickstart auto-installation

Run setup.sh with the “-k” option. The default kickstart file is installed as /var/pxe/ks/ks.cfg. It should have all packages needed for OFED. Add more packages if needed.

Running RedHat on BlueField

In general, running RedHat Enterprise Linux or CentOS on BlueField is similar to setting it up on any other ARM64 server.

A driver disk is required to support the eMMC hardware typically used to install the media onto. The driver disk also supports the tmfifo networking interface that allows creating a network interface over the USB or PCIe connection to an external host. For newer RedHat releases, or if the specific storage or networking drivers mentioned are not needed, you can skip the driver disk.

The way to manage bootflow components with BlueField is through grub boot manager. The installation should create a /boot/efi VFAT partition that holds the binaries visible to UEFI for bootup. The standard grub tools then manage the contents of that partition, and the UEFI EEPROM persistent variables, to control the boot.

It is also possible to use the BlueField runtime distribution tools to directly configure UEFI to load the kernel and initramfs from the UEFI VFAT boot partition if desired, but typically using grub is preferred. In particular, you would need to explicitly copy the kernel image to the VFAT partition whenever it is upgraded so that UEFI could access it; normally it is kept on an XFS partition.

Provisioning ConnectX Firmware

Prior to installing RedHat, you should ensure that the ConnectX SPI ROM firmware has been provisioned. If the BlueField is connected to an external host via PCIe, and is not running in Secure Boot mode, this is typically done by using the Mellanox MFT tools on the external host to provision the BlueField. If the BlueField is connected via USB or is configured in Secure Boot mode, you must provision the SPI ROM by booting a dedicated bootstream that allows the SPI ROM to be configured by the MFT running on the BlueField ARM cores.

There are multiple ways to access the RedHat installation media from a BlueField device for installation.

You may use the primary ConnectX interfaces on the BlueField to reach the media over the network.
You may configure a USB or PCIe connection to the BlueField as a network bridge to reach the media over the network.

Requires installing and running the RShim drivers on the host side of the USB or PCIe connection.
You may connect other network or storage devices to the BlueField via PCIe and use them to connect to or host the RedHat install media.

This method has not been tested.

Note that, in principle, it is possible to perform the installation according to the second method above without first provisioning the ConnectX SPI ROM, but since you need to do that provisioning anyway, it is recommended to perform it first. In particular, the PCIe network interface available via the external host’s RShim driver is likely too slow prior to provisioning to be usable for a distribution installation.

Managing Driver Disk

As discussed previously, you likely need a driver disk for RedHat installations. Mellanox provides a number of pre-built driver disks, as well as a documented flow for building one for any particular RedHat version. See section “Building a New bluefield_dd ISO Image” for details on how to do that.

Normally a driver disk can be placed on removable media (like a CDROM or USB stick) and is auto-detected by the RedHat installer. However, since BlueField typically has no removable media slots, you must provide it over the network. Although, if you are installing over the network connection via the PCIe/USB link to an external host, you will not have a network connection either. As a result, the procedure documented is for modifying the default RedHat images/pxeboot/initrd.img file to include the driver disk itself.

To create the updated initrd.img, you should locate the “image/pxeboot” directory in the RedHat installation media. This will have a kernel image file (vmlinuz) and initrd.img (initial RAM disk). The “bluefield_dd/update-initrd.sh” script takes the path to the initrd.img as an argument and adds the appropriate BlueField driver disk ISO file to the initrd.img.

When booting the installation media, make sure to include “inst.dd=/bluefield_dd.iso” on the kernel command line, which will instruct Anaconda to use that driver disk, enabling the use of the IP over USB/PCIe link (tmfifo) and the DesignWare eMMC (dw_mmc).

Installing Reference Yocto Distribution

The BlueField processor should be attached to an external host via a USB connection. The external host is required to be running Linux. This process has been tested with an external host running CentOS 7.4.

You will need to install the provided “rshim”, “rshim_net”, and “rshim_usb” drivers on the external host. These drivers are required to communicate with the BlueField device over the USB interface.

The initramfs contains ConnectX^® firmware in the /lib/firmware/mellanox directory. MFT is also installed on the initramfs if you wish to update the ConnectX firmware.

On the external host.
To use the default console, run:

$ cat <BF_INST_DIR>/sample/install.bfb > /dev/rshim0/boot

To use Serial-Over-LAN for BlueField console on the reference platform:

$ echo "console=ttyAMA0 earlycon=pl011,0x01000000 initrd=initramfs" > bootarg 
<BF_INST_DIR>/bin/mlx-mkbfb --boot-args bootarg <bluefield_local_path>/install/sample/install.bfb install.bfb

This creates a new install.bfb in your current directory which uses Serial-Over-LAN. Use this image in place of the install.bfb:

$ cat install.bfb > /dev/rshim0/boot

Run this on the external host:
```
$ ifconfig eth<x> 192.168.100.1/24 up
```
Determine what root file system you wish to install and download it from the BlueField Software page on Mellanox's website. This example uses the core-image-full-dev-bluefield.tar.xz image file.

On the BlueField console port, run:

$ /opt/mlnx/scripts/bfinst --fullfs /tmp/core-image-full-bluefield.tar.xz
$ shutdown -r now

Maintaining User Data After Software Upgrade

By default, when installing Yocto for the first time, a special partition is created and mounted to /data. This is the persistent partition.

Any files written to this partition are treated specially by the installer, and are not overwritten on subsequent installs of Yocto.

To forcefully format the /data partition on install, run "bfinst" with the "--fmtpersist" option during the install process.

This section describes the persistence features of the BlueField Yocto distribution.

Persistent Configuration

Yocto offers a persistent configuration feature that uses the persistent partition.

To enable it, create an empty file called "persistconfig" at the root of the data partition, and a folder called "etc", then reboot.

$ mkdir /data/etc
$ touch /data/persistconfig
$ reboot

Upon startup, /etc is mounted as a special overlay filesystem. Any changes made to /etc in this state persist after software upgrades.

This feature depends on a file called /etc.img; do not delete it.

Do not modify /data/etc while persistconfig is enabled. To disable persistconfig, just delete the file in /data:

$ rm /data/persistconfig
$ reboot

Upon reboot, /etc is mounted from the root filesystem, and any changed files are reverted to what they were before persistconfig was enabled. Any changes made during persistconfig mode will be remembered in /data/etc.

Any /etc files updated during a software upgrade are masked if any user-made changes are in persistent /etc mode. For example, if you modified /etc/foo, and a software upgrade updates /etc/foo, the vendor-made changes will not be visible, and manual intervention will be required.

Understanding Persistent Data Partition Feature Implementation

The persistent partition "/data" is created at install time with a known GUID. Upon subsequent installs, the bfinst script will check for a partition with this GUID, and if it exists, will leave it intact and only format the other partitions.

To implement persistent configuration, an overlayfs has been used (see Documentation/filesystems/overlayfs.txt in the Linux source tree for more information). When persistence is turned on, after the data partition is mounted, the script checks that /data/persistconfig exists. If it does, /etc is mounted as an overlayfs. The lower filesystem is /etc.img, a read-only etc image created at install time, and the upper filesystem is /data/etc, residing on the persistent partition.

When the persistent config is first turned on, all read accesses to /etc will go to the lower filesystem. But, when something is written to /etc, the file being written is copied from the lower to the upper filesystem, and then opened in write mode on the upper filesystem, redirecting all writes. Any further reads and writes on that file will now go to the copy residing on the upper filesystem. The overlayfs effectively "merges" the two folders and presents a new filesystem composed of both on /etc.

Since all changes to /etc are automatically written to /data/etc, all configuration changes survive software upgrades.

The /data and /etc mounts are the first initialization tasks that happen on the system, preempting even systemd. This is accomplished with the script /sbin/preinit, which performs the mounts and then execs into /sbin/init. A kernel argument is passed by grub to force the kernel to use /sbin/preinit as its initialization process.

The reason this is done is to get around an issue involving systemd, and to ensure any initialization processes open files in the /etc overlayfs, not the rootfs. The systemd software reads its configuration from two locations: /etc/systemd and /lib/systemd. It is also normally responsible for all mounts in the system. When the user adds their own services to /etc/systemd, it is possible to get systemd to pick up the new changes when /etc is mounted. Unfortunately, it is not possible to _execute_ newly acquired services during the boot process. To fix this natively with systemd is very complex and unreliable, so we just go under systemd entirely to simplify the solution.

Installing Ubuntu on BlueField

Ubuntu Hardware and Software Requirements

Host system with RShim driver installed
BlueField NIC inserted in a host PCIe slot
USB cable connecting the NIC card and the host

Installation Procedure for Canonical's Ubuntu BFB

If Ubuntu 18.04 is installed on the host, make sure to have the HWE kernel also. Run:
```
$ sudo apt install linux-signed-generic-hwe-18.04
$ sudo reboot
```
Install the image by pushing the downloaded image to the NIC. Run:
```
$ sudo sh -c 'xzcat server-2004-ubuntu-20.04-alpha2-20200725-66.bfb.xz > /dev/rshim0/boot'
```
This will start an Ubuntu kernel that automatically installs an image contained in the initramfs included inside the bfb. After flashing this image, the installation script reboots, and eventually a login prompt appears in the serial console.

Ubuntu With MLNX_OFED Installation

A pre-built BFB of Ubuntu 18.04 with MLNX_OFED_LINUX installed is available. Please contact Mellanox Support to get this BFB.

To install Ubuntu BFB, run on the host side:

$ cat <ubuntu.bfb> > /dev/rshim0/boot

Username: ubuntu
Password: ubuntu

Installing Debian on BlueField

Debian Hardware and Software Requirements

Host system with RShim driver installed
BlueField NIC inserted in a host PCIe slot

Debian with MLNX_OFED Installation

A pre-built BFB of Debian10 with MLNX_OFED_LINUX installed is available. Please contact Mellanox Support to get this BFB.

To install Debian BFB, run the following on the host side:

$ cat <debian.bfb> > /dev/rshim0/boot
 
Username: root
Password: debian

Content

Space Tools

Installing Popular Linux Distributions on BlueField

On This Page