This section describes how to use the BlueField alternate boot partition support feature to safely upgrade the boot software. We give the requirements that motivate the feature and explain the software interfaces that are used to configure it.
BFB File Overview
The default BlueField bootstream (BFB) shown above (located at
/lib/firmware/mellanox/boot/default.bfb) is assumed to be loaded from the eMMC. In it, there is a hard-coded boot path pointing to a GUID partition table (GPT) on the eMMC device. Once loaded, as a side effect, this path would be also stored in the UPVS (UEFI Persistent Variable Store) EEPROM. That is, if you use the
bfrec tools provided in the
mlxbf-bfscripts package to write this BFB to the eMMC boot partition (see bfrec man for more information), then during boot, the DPU would load this from the boot FIFO, and the UEFI would assume to boot off the eMMC.
BFB files can be useful for many things such as installing new software on a BlueField DPU. For example, the installation BFB for BlueField platforms normally contains an initramfs file in the BFB chain. Using the initramfs (and Linux kernel Image also found in the BFB) you can do things like set the boot partition on the eMMC using
mlx-bootctl or flash new HCA firmware using MFT utilities. You can also install a full root file system on the eMMC while running out of the initramfs.
The following table presents the types of files possible in a BFB.
|Bl2r-cert||Secure Firmware BL2R (RIoT Core) certificate||33||BL1|
|Bl2r||Secure Firmware BL2R (RIoT Core)||28||BL1|
|bl2-cert||Trusted Boot Firmware BL2 certificate||6||BL1/BL2R*|
|bl2||Trusted Boot Firmware BL2||1||BL1/BL2R*|
|trusted-key-cert||Trusted key certificate||7||BL2|
|bl31-key-cert||EL3 Runtime Firmware BL3-1 key certificate||9||BL2|
|bl31-cert||EL3 Runtime Firmware BL3-1 certificate||13||BL2|
|bl31||EL3 Runtime Firmware BL3-1||3||BL2|
|Secure Payload BL3-2 (Trusted OS) key certificate||10||BL2|
Secure Payload BL3-2 (Trusted OS) certificate
Secure Payload BL3-2 (Trusted OS)
Non-Trusted Firmware BL3-3 key certificate
Non-Trusted Firmware BL3-3 certificate
Non-Trusted Firmware BL3-3
Name of the ACPI table
Name of the DTB file
Default boot menu item description
Boot image path
Arguments for boot image
Boot menu timeout
* When BL2R is booted in BlueField-2 devices, both the BL2 image and the BL2 certificate are read by BL2R. Thus, the BL2 image and certificate are read by BL1. BL2R is not booted in BlueField-1 devices.
Before explaining the implementation of the solution, the BlueField boot process needs to be expanded upon.
BlueField Boot Process
The BlueField boot flow is comprised of 4 main phases:
- Hardware loads Arm Trusted Firmware (ATF)
- ATF loads UEFI—together ATF and UEFI make up the booter software
- UEFI loads the operating system, such as the Linux kernel
- The operating system loads applications and user data
When booting from eMMC, these stages make use of two different types of storage within the eMMC part:
- ATF and UEFI are loaded from a special area known as the eMMC boot partition. Data from a boot partition is automatically streamed from the eMMC device to the eMMC controller under hardware control during the initial boot-up. Each eMMC device has two boot partitions, and the partition which is used to stream the boot data is chosen by a non-volatile configuration register in the eMMC.
- The operating system, applications, and user data come from the remainder of the chip, known as the user area. This area is accessed via block-size reads and writes, done by a device driver or similar software routine.
In most deployments, the Arm cores of BlueField are expected to obtain their bootloader from an on-board eMMC device. Even in environments where the final OS kernel is not kept on eMMC—for instance, systems which boot over a network—the initial booter code still comes from the eMMC.
Most software stacks need to be modified or upgraded in their lifetime. Ideally, the user can to install the new software version on their BlueField system, test it, and then fall back to an older version if the new one does not work. In some environments, it is important that this fallback operation happen automatically since there may be no physical access to the system. In others, there may be an external agent, such as a service processor, which could manage the process.
In order to satisfy the requests listed above, the following must be performed:
- Provision two software partitions on the eMMC, 0 and 1. At any given time, one area must be designated the primary partition, and the other the backup partition. The primary partition is the one booted on the next reboot or reset.
- Allow software running on the Arm cores to declare that the primary partition is now the backup partition, and vice versa. (For the remainder of this section, this operation is referred to as "swapping the partitions" even though only the pointer is modified, and the data on the partitions does not move.)
- Allow an external agent, such as a service processor, to swap the primary and backup partitions.
- Allow software running on the Arm cores to reboot the system, while activating an upgrade watchdog timer. If the upgrade watchdog expires (due to the new image being broken, invalid, or corrupt), the system automatically reboots after swapping the primary and backup partitions.
Updating Boot Partition
The Bluefield software distribution provides a boot file that can be used to update the eMMC boot partitions. The BlueField boot file (BFB) is located in the boot directory <BF_INST_DIR>/boot/ and contains all the necessary boot loader images (i.e. ATF binary file images and UEFI binary image).
The table below presents the pre-built boot images included within the BlueField software release:
The trusted firmware bootloader stage 1 (BL1) image, already stored into the on-chip boot ROM. It is executed when the device is reset.
|bl2r.bin||The secure firmware (RIoT core) image. This image provides support for crypto operation and calculating measurements for security attestation and is relevant to BlueField-2 devices only.|
The trusted firmware bootloader stage 2 (BL2) image
|bl31.bin||The trusted firmware bootloader stage 3-1 (BL31) image|
|The UEFI firmware image. It is also referred to as the non-trusted firmware bootloader stage 3-3 (BL33) image.|
The BlueField boot file (BFB) which encapsulates all bootloader components such as bl2r.bin, bl2.bin, bl31.bin, and BLUEFIELD_EFI.fd. This file may be used to boot the BlueField devices from the RShim interface. It also could be installed into the eMMC boot partition.
It is also possible to build bootloader images from sources and create the BlueField boot file (BFB). Please refer to the sections below for more details.
The software image includes various tools and utilities to update the eMMC boot partitions. It also embeds a boot file in "/lib/firmware/mellanox/boot/default.bfb". To update the eMMC boot partitions using the embedded boot file, execute the following command from the BlueField console:
bfrec is also available under "
The boot partitions update is initiated by the
bfrec tool at runtime. With no options specified, the "bfrec" uses the default boot file "/lib/firmware/mellanox/boot/default.bfb" to update the boot partitions of device /dev/mmcblk0. This might be done directly in an OS using the "mlxbf-bootctl" utility, or at a later stage after reset using the capsule interface.
The syntax of
bfrec is as follows:
bfrec is called with the option
--bootctl, the tool uses the boot file FILE, if given, rather than the default
/lib/firmware/mellanox/boot/default.bfb in order to update the boot partitions. The command line usage is as follows:
Where FILE represents the BlueField boot file encapsulating the new bootloader images to be written to the eMMC boot partitions.
For example, if the new bootstream file which we would like to install and validate is called
newdefault.bfb, download the file to the BlueField and update the eMMC boot partitions by executing the following commands from the BlueField console:
--capsule option updates the boot partition via the capsule interface. The capsule update image is reported in UEFI, so that at a later point the bootloader consumes the capsule file and performs the boot partition update. This option might be executed with or without additional arguments. The command line usage is as follows:
Where FILE represents the capsule update image file encapsulating the new boot image to be written to the eMMC boot partitions.
For example, if the new bootstream file which we want to install and validate is called "
newdefault.bfb", download the file to the BlueField and update the eMMC boot partitions by executing the following commands from the BlueField console:
For more information about the capsule updates, please refer to
After reset, the BlueField platform boots from the newly updated boot partition. To verify the version of ATF and UEFI, execute the following command:
It is also possible to update the eMMC boot partitions directly with the
mlxbf-bootctl tool. The tool is shipped as part of the software image (under
/sbin) and the sources are shipped in the
src directory in the BlueField Runtime Distribution. A simple
make command builds the utility.
The syntax of
mlxbf-bootctl is as follows:
--device– use a device other than the default
--bootstream– write the specified bootstream to the alternate partition of the device. This queries the base device (e.g.
/dev/mmcblk0) for the alternate partition, and uses that information to open the appropriate boot partition device (e.g.
--overwrite-current(used with "
--bootstream") – overwrite the current boot partition instead of the alternate one
Not recommended as there is no easy way to recover if the new bootloader code does not bring the system up. Use
--output(used with "
--bootstream") – specify a file to which to write the boot partition data (creating it if necessary), rather than using an existing master device and deriving the boot partition device
--watchdog-swap– arrange to start the Arm watchdog timer with a countdown of the specified number of seconds until it triggers; also, set the boot software so that it swaps the primary and alternate partitions at the next reset
--nowatchdog-swap– ensure that after the next reset, no watchdog is started, and no swapping of boot partitions occurs
To update the boot partitions, execute the following command:
This writes the new bootstream to the alternate boot partition, swaps alternate and primary so that the new bootstream is used on the next reboot.
It is recommended to enable the watchdog when calling
mlxbf-bootcl in order to ensure that the Arm bootloader can perform alternate boot in case of a nonfunctional bootloader code within the primary boot partition. If something goes wrong on the next reboot and the system does not come up properly, it will reboot and return to the original configuration. To do so, the user may run:
This reboots the system, and if it hangs for 60 seconds or more, the watchdog fires and resets the chip, the bootloader swaps the partitions back again to the way they were before, and the system reboots back with the original boot partition data. Similarly, if the system comes up but panics and resets, the bootloader will again swap the boot partition back to the way it was before.
The user must ensure that Linux after the reboot is configured to boot up with the
sbsa_gwdt driver enabled. This is the Server Base System Architecture (SBSA) Generic WatchDog Timer. As soon as the driver is loaded, it begins refreshing the watchdog and preventing it from firing, which allows the system to finish booting up safely. In the example above, 60 seconds are allowed from system reset until the Linux watchdog kernel driver is loaded. At that point, the user’s application may open /dev/watchdog explicitly, and the application would then become responsible for refreshing the watchdog frequently enough to keep the system from rebooting.
For documentation on the Linux watchdog subsystem, see Linux watchdog documentation.
To disable the watchdog completely, run:
The user may select to incorporate other features of the Arm generic watchdog into their application code using the programming API as well.
Once the system has booted up, in addition to disabling or reconfiguring the watchdog itself if the user desires, they must also clear the "swap on next reset" functionality from the bootloader by running:
Otherwise, next time the system is reset (via reboot, external reset, etc.) it assumes a failure or watchdog reset occurred and swaps the eMMC boot partition automatically.
LVFS and fwupd
Officially released bootloaders (ATF and UEFI) may be alternatively installed from the LVFS (Linux Vendor Firmware Service). LVFS is a free service operated by the Linux Foundation, which allows vendors to host stable firmware images for easy download and installation.
The DPU must have a functioning connection to the internet.
Interaction with LVFS is carried out through a standard open-source tool called fwupd. fwupd is an updater daemon that runs in the background, waiting for commands from a management application. fwupd and the command line manager, fwupdmgr, comes pre-installed on the BlueField Ubuntu image.
To verify bootloader support for a fwupd update, run the following command:
If "UEFI Device Firmware" device appears, then your currently installed bootloader supports the update process. Other devices may appear depending on your distribution of choice. Version numbers similar to 0.0.0.1 may appear if you are using an older version of the bootloader.
Before updating, a fresh list of release metadata must be obtained. Run:
Optionally, to confirm if a new release is available, run:
Update your system bootloader, run "upgrade" with the GUID of the UEFI device. Run:
This will upgrade the ATF and UEFI to the latest available stable version of the bootloader through a UEFI capsule update, without upgrading the root file system. If your system is already at the latest available version, this upgrade command will do nothing.
- Reboot the DPU to complete the upgrade.
Installing boot firmware directly through mlxbf-bootctl may cause fwupdmgr to detect an incorrect version string. If your workflow depends on fwupd, try to update the bootloader through capsule update (i.e. bfrec --capsule) or fwupdmgr only.
For more information about LVFS and fwupd, please refer to the official website of LVFS.
Updating Boot Partitions with BMC
The Arm cores notify the BMC prior to the reboot that an upgrade is about to happen. Software running on the BMC can then be implemented to watch the Arm cores after reboot. If after some time the BMC does not detect the Arm cores come up properly, it can use its USB debug connection to the Arm cores to properly reset the Arm cores. It first sets a suitable mode bit that the Arm bootloader responds to by switching the primary and alternating boot partitions as part of resetting into its original state.
Creating BlueField Boot File
The BlueField software distribution provides tools to format and to package the bootloader images into a single bootable file.
To create the BlueField boot file, use the
mlx-mkbfb tool with the appropriate images. The bootloader images are embedded within the BSD under <BF_INST_DIR>/boot/. It is also possible to build the binary images from sources. Please refer to the following sections for further details.
First, set the PATH variable:
Then, generate the boot file by using the
This command creates the
BLUEFILED_EFI.fd. The generated file might be used to update the eMMC boot partitions.
To verify the content of the boot file, run:
To extract the bootloader images from the boot file, run:
To obtain further details about the tool options, run the tool with
UEFI Boot Management
The UEFI firmware provides boot management function that can be configured by modifying architecturally defined global variables which are stored in the UPVS EEPROM. The boot manager will attempt to load and boot the OS in an order defined by the persistent variables.
The UEFI boot manager can be configured; boot entries may be added or removed from the boot menu. The UEFI firmware can also effectively generate entries in this boot menu, according to the available network interfaces and possibly the disks attached to the system.
The boot option is a unique identifier for a UEFI boot entry. This identifier is assigned when the boot entry is created, and it does not change. It also represents the boot option in several lists, including the BootOrder array, and it is the name of the directory on disk in which the system stores data related to the boot entry, including backup copies of the boot entry. A UEFI boot entry ID has the format "
xxxx is a hexadecimal number that reflects the order in which the boot entries are created.
Besides the boot entry ID, the UEFI boot entry has the following fields:
- Description (e.g. Yocto, CentOS, Linux from RShim)
- Device Path (e.g. VenHw(F019E406-8C9C-11E5-8797-001ACA00BFC4)/Image)
- Boot arguments (e.g. console=ttyAMA0 earlycon=pl011,0x01000000 initrd=initramfs)
List UEFI Boot Options
To display the boot option already installed in the NVIDIA® BlueField® system, reboot and go to the UEFI menu screen. To get to the UEFI menu, hit Esc when prompted (in the RShim or UART console) before the countdown timer runs out.
Boot options are listed as soon as you select the "Boot Manager" entry.
It is also possible to retrieve more details about the boot entries. To do so, select "EFI Internal Shell" entry from the Boot Manager screen.
From the UEFI shell, you may run the following command to display the option list:
-v displays the option list with extra info including boot parameters. The following is an output example:
Boot arguments are printed in Hex mode, but you may recognize the boot parameters printed on the side in ASCII format.
UEFI System Configuration
UEFI System Configuration menu can be accessed under UEFI menu → Device Manager → System Configuration.
The following options are supported:
- Set Password – set a password for UEFI. Default: No password.
- Select SPCR UART – choose UART for Port Console Redirection. Default: Disabled.
- Enable SMMU – enable SMMU in ACPI. Default: Disabled.
- Disable SPMI – disable/enable ACPI SPMI Table. Default: Enabled.
- Enable 2nd eMMC – this option is relevant only for some BlueField Reference Platform boards. Default: Disabled.
- Boot Partition Protection – enable eMMC boot partition so it can be updated by the UEFI capsule only
- Disable PCIe – disable PCIe in ACPI. Default: Enabled.
- Disable ForcePXERetry – if ForcePXE is enabled from the BMC, the boot process keeps retrying PXE boot if it fails unless this option is enabled. If ForcePXERetry is disabled, the boot process only attempts PXE boot once, then it retries the normal boot flow if all PXE boot entries fail.
- Reset EFI Variables – clears all EFI variables to factory default state and disables SMMU and wipes the BOOT option variables and secure boot keys
- Reset MFG Info – clears the manufacturing information
All the above options, except for password and the two reset options, are also programmatically configurable via the BlueField Linux
/etc/bf.cfg. Refer to section "bf.cfg Parameters" for further information.