BlueField target is stuck inside UEFI menu
Upgrade to the latest stable boot partition images, see "How to upgrade the boot partition (ATF & UEFI) without re-installation".
BFB does not recognize the BlueField board type
If the .bfb file cannot recognize the BlueField board type, it reverts to low core operation. The following message will be printed on your screen:
***System type can't be determined*** ***Booting as a minimal system***
Please contact NVIDIA Support if this occurs.
CentOS fails into "dracut" mode during installation
This is most likely configuration related.
- If installing through the RShim interface, check whether /var/pxe/centos7 is mounted or not. If not, either manually mount it or re-run the setup.sh script.
- Check the Linux boot message to see whether eMMC is found or not. If not, the BlueField driver patch is missing. For local installation via RShim, run the setup.sh script with the absolute path and check if there are any errors. For a corporate PXE server, make sure the BlueField and ConnectX driver disk are patched into the initrd image.
How to find the software versions of the running system
Run the following:
/opt/mellanox/scripts/bfvcheck: root@bluefield:/opt/mellanox/scripts# ./bfvcheck Beginning version check... -RECOMMENDED VERSIONS- ATF: v1.5(release):BL2.0-1-gf9f7cdd UEFI: 2.0-6004a6b FW: 18.25.1010 -INSTALLED VERSIONS- ATF: v1.5(release):BL2.0-1-gf9f7cdd UEFI: 2.0-6004a6b FW: 18.25.1010 Version checked
Also, the version information is printed to the console.
For ATF, a version string is printed as the system boots.
"NOTICE: BL2: v1.3(release):v1.3-554-ga622cde"
For UEFI, a version string is printed as the system boots.
"UEFI firmware (version 0.99-18d57e3 built at 00:55:30 on Apr 13 2018)"
For Yocto, run:
$ cat /etc/bluefield_version 184.108.40.20617
How to upgrade the host RShim driver
See the readme at
How to upgrade the boot partition (ATF & UEFI) without re-installation
Boot the target through the RShim interface from a host machine:
$ cat <BF_INST_DIR>/sample/install.bfb > /dev/rshim<N>/boot
Log into the BlueField target:
How to upgrade ConnectX firmware from Arm side
flint tools can be used to update firmware.
For Ubuntu, CentOS and Debian, run the following command from the Arm side:
The file mlxfwmanager_sriov_dis_aarch64_41686 is intended for BlueField-2.
The file mlxfwmanager_sriov_dis_aarch64_41682 is intended for BlueField.
How to configure ConnectX firmware
Configuring ConnectX firmware can be done using the mlxconfig tool.
It is possible to configure privileges of both the internal (Arm) and the external host (for DPUs) from a privileged host. According to the configured privilege, a host may or may not perform certain operations related to the NIC (e.g. determine if a certain host is allowed to read port counters).
For more information and examples please refer to the MFT User Manual which can be found at the following link.
How to use the UEFI boot menu
Press the "Esc" key when prompted after booting (before the countdown timer runs out) to enter the UEFI boot menu and use the arrows to select the menu option.
It could take 1-2 minutes to enter the Boot Manager depending on how many devices are installed or whether the EXPROM is programmed or not.
Once in the boot manager:
- "EFI Network xxx" entries with device path "PciRoot..." are ConnectX interface
- "EFI Network xxx" entries with device path "MAC(..." are for the RShim interface and the BlueField-2 OOB Ethernet interface
Select the interface and press ENTER will start PXE boot.
The following are several useful commands under UEFI shell:
Shell> ls FS0: # display file Shell> ls FS0:\EFI # display file Shell> cls # clear screen Shell> ifconfig -l # show interfaces Shell> ifconfig -s eth0 dhcp # request DHCP Shell> ifconfig -l eth0 # show one interface Shell> tftp 192.168.100.1 grub.cfg FS0:\grub.cfg # tftp download a file Shell> bcfg boot dump # dump boot variables Shell> bcfg boot add 0 FS0:\EFI\centos\shim.efi "CentOS" # create an entry
How to get installation images from the BlueField release tarball
The BlueField software tarball is released as BlueField-<ver>.<num>.tar.xz (e.g. BlueField-220.127.116.1139.tar.xz). In this FAQ section, it is assumed that the tarball is uncompressed under directory "<BF_INST_DIR>".
How to install Yocto
See the readme file at
How to install CentOS
See the readme file at
<BF_INST_DIR>/distro/rhel/pxeboot/README. For more on the installation of CentOS distributions, please contact NVIDIA Support.
How to Use the Kernel Debugger (KGDB)
The default Yocto kernel has CONFIG_KGDB and CONFIG_KGDB_SERIAL_CONSOLE enabled. This allows the Linux kernel on BlueField to be debugged over the serial port. A single serial port cannot be used both as a console and by KGDB at the same time. It is recommended to use the RShim for console access (/dev/rshim0/console) and the UART port (/dev/ttyAMA0 or /dev/ttyAMA1) for KGDB. Kernel GDB over console (KGDBOC) does not work over the RShim console. If the RShim console is not available, there are open source packages such as KGDB demux and agent-proxy which allow a single serial port to be shared.
There are two ways to configure KGDBOC. If the OS is already booted, then write the name of the serial device to the KGDBOC module parameter. For example:
$ echo ttyAMA1 > /sys/module/kgdboc/parameters/kgdboc
In order to attach GDB to the kernel, it must be stopped first. One way to do that is to send a "g" to /proc/sysrq-trigger.
$ echo g > /proc/sysrq-trigger
If you want to debug incidents that occur at boot time, that has to be configured through the kernel boot parameters. Add "kgdboc=ttyAMA1,115200 kgdwait" to the boot arguments to use UART1 for debugging and force it to wait for GDB to attach before booting.
Once the KGDBOC module is configured and the kernel stopped, run the Arm64 GDB on the host machine connected to the serial port, then set the remote target to the serial device on the host side.
<BF_INST_DIR>/sdk/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux/aarch64-poky-linux-gdb <BF_INST_DIR>/sample/vmlinux (gdb) target remote /dev/ttyUSB3 Remote debugging using /dev/ttyUSB3 arch_kgdb_breakpoint () at /labhome/dwoods/src/bf/linux/arch/arm64/include/asm/kgdb.h:32 32 asm ("brk %0" : : "I" (KGDB_COMPILED_DBG_BRK_IMM)); (gdb)
<BF_INST_DIR> is the directory where the BlueField software is installed. It is assumed that the SDK has been unpacked in the same directory.
How to enable/disable SMMU
SMMU could affect performance for certain applications. It is disabled by default and can be modified in different ways.
- Enable/disable SMMU in the UEFI System Configuration
- Set it in
bf.cfgand push it together with the
install.bfb(see section "Installing Popular Linux Distributions on BlueField")
- In BlueField Linux, create a file with one line with
SYS_ENABLE_SMMU=TRUE, then run
The configuration change will take effect after reboot. The configuration value is stored in a persistent UEFI variable. It is not modified by OS installation.
UEFI System Configuration menu can be accessed under UEFI menu → Device Manager → System Configuration. The following actions are supported from it:
- Set Password: Set a password for UEFI. Default: No password.
- Select SPCR UART: Choose UART for Port Console Redirection. Default: Disabled.
- Enable SMMU: Enable SMMU in ACPI. Default: Disabled.
- Disable SPMI: Disable/enable ACPI SPMI Table. Default: Enabled.
- Disable PCIe: Disable PCIe in ACPI. Default: Enabled.
- Reset EFI Variables: Clears all EFI variables to factory default state.
- Reset MFG Info: Clears the manufacturing information.
How to change the default console of the install image
$ echo "console=ttyAMA0 earlycon=pl011,0x01000000 initrd=initramfs" > bootarg $ <BF_INST_DIR>/bin/mlx-mkbfb --boot-args bootarg \ <BF_INST_DIR>/sample/ install.bfb
$ echo "console=ttyAMA1 earlycon=pl011,0x01000000 initrd=initramfs" > bootarg $ <BF_INST_DIR>/bin/mlx-mkbfb --boot-args bootarg \ <BF_INST_DIR>/sample/install.bfb
$ echo "console=hvc0 initrd=initramfs" > bootarg $ <BF_INST_DIR>/bin/mlx-mkbfb --boot-args bootarg \ <BF_INST_DIR>/sample/install.bfb