Users wishing to customize their own kernel or root file system may do so using docker images supplied by NVIDIA. BlueField docker images can be obtained at: https://hub.docker.com/repository/docker/mellanox/bluefield.
Docker images give users the ability to customize their own Linux distribution and create a BFB. That BFB file can then be installed on any BlueField device. Instructions are provided on the BlueField docker page as well as in the docker image itself.
Users can build all BlueField software, the Linux kernel, and OFED using the docker image and "native" or "cross" compile:
Build using QEMU emulation on an x86 server
Build on an Arm system (such as a BF1200 or an AWS Arm EC2 instance)
The same docker image is used regardless of if you build native on an Arm server or use emulation on an x86 server. The only difference is how long the build takes.
In general, running RedHat Enterprise Linux or CentOS on BlueField is similar to setting it up on any other ARM64 server.
A driver disk is required to support the eMMC hardware typically used to install the media onto. The driver disk also supports the tmfifo networking interface that allows creating a network interface over the USB or PCIe connection to an external host. For newer RedHat releases, or if the specific storage or networking drivers mentioned are not needed, you can skip the driver disk.
The way to manage boot flow components with BlueField is through grub boot manager. The installation should create a /boot/efi VFAT partition that holds the binaries visible to UEFI for bootup. The standard grub tools then manage the contents of that partition, and the UEFI EEPROM persistent variables, to control the boot.
It is also possible to use the BlueField runtime distribution tools to directly configure UEFI to load the kernel and initramfs from the UEFI VFAT boot partition if desired, but typically using grub is preferred. In particular, you would need to explicitly copy the kernel image to the VFAT partition whenever it is upgraded so that UEFI could access it; normally it is kept on an XFS partition.
Provisioning ConnectX Firmware
Prior to installing RedHat, you should ensure that the ConnectX SPI ROM firmware has been provisioned. If the BlueField is connected to an external host via PCIe, and is not running in Secure Boot mode, this is typically done by using MFT on the external host to provision the BlueField. If the BlueField is connected via USB or is configured in Secure Boot mode, you must provision the SPI ROM by booting a dedicated bootstream that allows the SPI ROM to be configured by the MFT running on the BlueField Arm cores.
There are multiple ways to access the RedHat installation media from a BlueField device for installation.
You may use the primary ConnectX interfaces on the BlueField to reach the media over the network.
You may configure a USB or PCIe connection to the BlueField as a network bridge to reach the media over the network.
WarningRequires installing and running the RShim drivers on the host side of the USB or PCIe connection.
You may connect other network or storage devices to the BlueField via PCIe and use them to connect to or host the RedHat install media.
ImportantThis method has not been tested.
Note that, in principle, it is possible to perform the installation according to the second method above without first provisioning the ConnectX SPI ROM, but since you need to do that provisioning anyway, it is recommended to perform it first. In particular, the PCIe network interface available via the external host’s RShim driver is likely too slow prior to provisioning to be usable for a distribution installation.
Managing Driver Disk
NVIDIA provides a number of pre-built driver disks, as well as a documented flow for building one for any particular RedHat version.
Normally a driver disk can be placed on removable media (like a CDROM or USB stick) and is auto-detected by the RedHat installer. However, since BlueField has no removable media slots, you must provide it over the network. Although, if you are installing over the network connection via the PCIe/USB link to an external host, you will not have a network connection either. As a result, the procedure documented is for modifying the default RedHat images/pxeboot/initrd.img file to include the driver disk itself.
To create the updated initrd.img, you should locate the image/pxeboot directory in the RedHat installation media. This will have a kernel image file (vmlinuz) and initrd.img (initial RAM disk). The bluefield_dd/update-initrd.sh script takes the path to the initrd.img as an argument and adds the appropriate BlueField driver disk ISO file to the initrd.img.
When booting the installation media, make sure to include inst.dd=/bluefield_dd.iso on the kernel command line, which will instruct Anaconda to use that driver disk, enabling the use of the IP over USB/PCIe link (tmfifo) and the DesignWare eMMC (dw_mmc).
For information on the installation of CentOS distributions, please contact NVIDIA Support.