image image image image image

On This Page

It is recommended to upgrade your BlueField product to the latest software and firmware versions in order to enjoy the latest features and bug fixes.

BlueField Software

NVIDIA provides software that enables users to fully utilize the BlueField® DPU and enjoy the rich feature-set it provides. Using BlueField software packages, users are able to:

  • Quickly and easily boot an initial Linux image on your development board
  • Port existing applications to and develop new applications for BlueField
  • Patch, configure, rebuild, update or otherwise customize your image
    • Debug, profile, and tune their development system using open source development tools taking advantage of the diverse and vibrant Arm ecosystem.

The BlueField family of DPU devices combines an array of 64-bit Arm v8 A72 cores coupled with the ConnectX® interconnect. Standard Linux distributions run on the Arm cores allowing common open source development tools to be used. Developers should find the programming environment familiar and intuitive which in turn allows them to quickly and efficiently design, implement and verify their control-plane and data-plane applications.

BlueField SW ships with the NVIDIA BlueField Controller Cards. BlueField SW is a reference Linux distribution based on the Yocto Poky distribution and extended to include the NVIDIA OFED stack for Arm and a Linux kernel which supports NVMe-oF. This SW distribution is capable of running all customer-based Linux applications seamlessly. Yocto also provides an SDK that contains an extremely flexible cross-build environment allowing software targeted for the BlueField DPU to build on virtually any x86 server running any Linux distribution.

The following are other software elements delivered with BlueField DPU:

  • Arm Trusted Firmware (ATF) for BlueField
  • UEFI for BlueField
  • Hardware Diagnostics
  • NVIDIA OFED stack
  • NVIDIA MFT

Software On eMMC

The BlueField Controller Card boots off eMMC upon power-up. The image flashed on the eMMC from the factory is the Yocto Linux.

Run the following command to discover the BlueField Software version:

cat /etc/bluefield_version

Yocto Distribution Installation

The BlueField tarball comes with pre-built Yocto images that can be installed.  The core-image-full image is a full root filesystem image that is appropriate for imaging on the rootfs partition of the eMMC. 

  1. Prepare the host environment. For more information, refer to Preparing the Host-Side Environment.
  2. Boot the BlueField Controller Card over USB using the samples/install.bfb image.
  3. Refer to samples/README.install for instructions.

  4. To boot the BlueField Controller Card over USB from the server host, run:

    cat install.bfb > /dev/rshim0/boot

  5. To prepare the eMMC for Yocto installation from the Arm, run:

    /opt/mlnx/scripts/bfinst --fullfs /tmp/core-image-full-bluefield.tar.xz

  6. After the installation is done, execute lowercase reboot on the Arm.

    shutdown -r now

  7. Verify the version via:

    cat /etc/bluefield_version

PXE Server Configuration on Host Side

Before installing CentOS 7 on the BlueField Controller Card, you need to configure the PXE server on the host side (x86) to allow the deployment of the CentOS image over the BlueField Controller Card.

Download the CentOS installation iso file from the following link:

# Download the centos installation iso file from http://mirror.centos.org/altarch/7/isos/aarch64/CentOS-7-aarch64-Everything.iso
# cd <BF_INST_DIR>/distro/rhel/pxeboot
# ./setup.sh -d <BF_INST_DIR> -i <centos-installation.iso> [-c <ttyAMA0 >]

UART0 (ttyAMA0) is used by default, or you can use "-c ttyAMA0" to manually specify UART0.

Installing Linux on BlueField Controller Card

This section demonstrates CentOS 7.4 installation on the BlueField Controller Card. Other OSs work similarly with the PXE boot installation process.

Before installing the preferred OS on the BlueField Controller Card, make sure you install the BlueField Controller Card in a JBOF System. Installing it in a host system may damage the card.

Software Requirements

  • CentOS 7.4 Linux OS. To get CentOS 7.4 image, run:

    wget http://archive.kernel.org/centos-vault/altarch/7.4.1708/isos/aarch64/CentOS-7- aarch64-Everything.iso

    Some required drivers do not compile and load if running CentOS 5.x or earlier.

    Please note that CentOS 7.5 is not supported.

  • Access to the latest BlueField Controller Card SW bundle:
    NVIDIA uses box.com to distribute BlueField software. Contact your sales/support representative for a custom link to download BlueField software releases.

  • In this document, we assume the tarball BlueField-1.0.alphaX.XXXXX.tar.gz is extracted at / root, to do this, run the following command:

      tar -xvf BlueField-1.0.alphaX.XXXXX.tar.xz -C /root

Preparing Host-Side Environment

Before installing the preferred OS on the BlueField Controller Card, the host must be set up for it to be capable of provisioning the BlueField Controller Card. The RShim USB driver is installed on the host to communicate with the RShim device on the BlueField DPU. The RShim USB driver must be installed so that it can push the initial bootloader and supply the OS image for PXE boot through the USB connection.

This process only needs to be done on the host machine which is provisioning the BlueField Controller Card, it is not required on the end machine.

Setup Procedure With Installation Script

If the host is running CentOS 7 (or equivalent) on the host, you may run a script to complete all the steps detailed in Preparing the Host-Side Environment.

/root/BlueField-1.0.alphaX.XXXXX/distro/rhel/pxeboot/setup.sh    \
-d /root/BlueField-1.0.alphaX.XXXXX/ \
-i /root/CentOS-7-aarch64-Everything.iso \
-o /root/dd-rhel7.4-mlnx-ofed-4.2-1.4.10.0-aarch64.iso \
-c ttyAMA0\
-k

Note that there should be no firewall blocking the IP communication between the BlueField Controller Card and the server host machine. If a firewall exists, disable it with the following commands:

iptables -F 
iptables -t 
nat -F
  • The “-d” flag points to where the tar file has been extracted from, the script uses this directory to find all the source code it needs.
  • The “-i” flag points to the OS installation disk. This is the image that is accessed via PXE boot to install the OS on the BlueField Controller Card.
  • The “-o” flag points to the NVIDIA OFED driver disk for Arm. Download and extract it from  http://www.mellanox.com/page/products_dyn?product_family=34.
  • The “-c” flag specifies the default UART port for the OS to use since the BlueField DPU has two Arm UARTs. For the BlueField Controller Card, “ttyAMA0” is used, which is UART0.
  • The “-t” flag is optional and needed for nonpxe boot. When specified and given the argument of what Controller card is set (BlueField Controller Card in this case), it generates a “nonpxe.bfb” file which contains the install kernel and rootfs. If this file is pushed to the RShim boot device, it automatically runs the installation process and skips the initial UEFI PXE boot operations. (the -t flag). Please refer to distro/rhel/pxe/README.
  • The optional “-k” flag kickstarts auto-installation based on a default kickstart file which is installed as /var/pxe/ks/ks.cfg (optional).

Setup Procedure Without Installation Script

If the host is running CentOS 7 or equivalent, please refer to Preparing the Host-Side Environment for a simpler way to perform the installation using an installation script.

The following sections demonstrate CentOS 7 installation, however, installation in other environments should be relatively similar.

Step 1: Set up RShim Interface

The RShim driver communicates with the RShim device on the BlueField DPU. The RShim is in charge of many miscellaneous functions of the DPU, including resetting the Arm cores, providing the initial bootstream, and using the TMFIFO and the RShim network, to exchange network and console data with the host.

The RShim device can be reached by the host via the USB connector.

Step 2: Install RShim Drivers

To install the kernel modules, please follow the instruction in section RShim Host Driver.

Step 3: Configure TFTP Server

The host should be configured to act as a TFTP server to the BlueField Controller Card via the USB RShim network. This server provides the required files by the BlueField Controller Card to perform the PXE boot for installing the preferred OS.

Configuring the TFTP server requires a TFTP package. If it is not installed, install it via “yum install tftp” or “apt-get tftp”, depending on your Linux distribution.

Note: On some versions, the TFTP package cannot be found. In such cases, install “xinetd”.

  1. Extract the OS image and copy the required PXE boot components:

    mount -t iso9660 -o loop CentOS-7-aarch64-Everything.iso /mnt
mkdir -p /var/lib/tftpboot/centos/7.4
cp /mnt/EFI/BOOT/BOOTAA64.EFI /var/lib/tftpboot/ cp /mnt/EFI/BOOT/grubaa64.efi /var/lib/tftpboot/
cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/centos/7.4
cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/centos/7.4/initrd-orig.img

  2. Patch the initrd with the eMMC driver and TMFIFO (RShim network) driver:

    mkdir -p /tmp/.bfcentos mkdir -p $/tmp/.bfinstdd cd /tmp/.bfcentos
xzcat /var/lib/tftpboot/centos/7.4/initrd-orig.img | cpio -idm mount     
/root/BlueField-1.0.alpha3.10409/distro/rhel/bluefield_dd/bluefield_dd-4.11.0- 22.el7a.aarch64.iso 
/tmp/.bfinstddmkdir -p usr/lib/modules/4.11.0-22.el7a.aarch64/updates/cp 
/tmp/.bfinstdd/lib/modules/4.11.0-22.el7a.aarch64/updates/dw_mmc*.ko usr/lib/mod- ules/4.11.0-22.el7a.aarch64/updates/cp 
/tmp/.bfinstdd/lib/modules/4.11.0-22.el7a.aarch64/updates/tmfifo.ko usr/lib/mod- ules/4.11.0-22.el7a.aarch64/updates/cp     
/root/BlueField-1.0.alpha3.10409/distro/rhel/bluefield_dd/bluefield_dd-4.11.0- 22.el7a.aarch64.iso ./bluefield_dd.iso
umount /tmp/.bfinstdd; rmdir /tmp/.bfinstdd chown root:root
* -R
depmod -b 
/tmp/.bfcentos 4.11.0-22.el7a.aarch64
find . | cpio
-oc | xz --check=crc32 --lzma2=dict=32MiB >
/var/lib/tftpboot/centos/7.4/ initrd.img

    These commands assume that you are using kernel version “4.11.0-22.el7a.aarch64”. If you are using a different version, utilize the corresponding bluefield_dd.iso. If none is found, compile one by running the following: source /path/to/SDK/environment-setup-aarch64-poky-linux; unset LDFLAGS; ./build- dd.sh    /path/to/kernel-devel-4.11.0-44.el7a.aarch64.rpm

  3. Change the grub configuration to PXE boot over the right location:

    cat >/var/lib/tftpboot/grub.cfg <<EOF
menuentry 'Install centos/7.4 AArch64 - BlueField'
--class red --class
gnu-linux
--class gnu --class os {
linux (tftp)/centos/7.4/vmlinuz ro ip=dhcp method=http://192.168.100.1/centos7 inst.dd=/bluefield_dd.iso console=ttyAMA0
initrd (tftp)/centos/7.4/initrd.img
} EOF

  4. Start the TFTP server:

    systemctl restart tftp

    Based on the system, the user may need to use “system TFTP restart” instead. Also, if required, the user might need to switch use “xinetd” instead of “TFTP”.


Step 4: Set Up the DHCP Server

DHCP server set up on the host is required for BlueField Controller Card to get a private IP from the host for PXE boot process completion. Configure the correct server names and domain names so that the BlueField Controller Card can connect to the network via the host later on.

  1. Get the server/domain names on the host:

    bash-4.2$ cat /etc/resolv.conf
# Generated by NetworkManager search internal.mlnx.com labs.mlnx 
nameserver 10.15.2.29
nameserver 10.15.2.16

    This example shows that the domains are internal.mlnx.com and labs.mlnx, and the names of the servers are 10.15.2.29 and 10.15.2.16.

  2. Set up the DHCP config file accordingly:

    cat >/etc/dhcp/dhcpd.conf <<EOF
allow booting;
allow bootp;
subnet 192.168.100.0 netmask 255.255.255.0 {
range 192.168.100.10 192.168.100.20;
option broadcast-address 192.168.100.255;
option routers 192.168.100.1;
option domain-name-servers 10.15.2.29 10.15.2.16;
# Set the domain search according to the network configuration option domain-search "internal.tilera.com" "mtbu.labs.mlnx"; next-server 192.168.100.1;
filename "/BOOTAA64.EFI";
}
# Specify
the IP address for
this client. host pxe_client {
hardware ethernet 00:1a:ca:ff:ff:01; fixed-address 192.168.100.2;
} EOF

    It is recommended to back up the previous dhcpd.conf file before overwriting it.

Step 5: Set Up the HTTP Server

The TFTP server allows the PXE boot to load the initrd and kernel. The BlueField Controller Card obtains all the other required sources through the network, thus, making it necessary to set up an HTTP.

Setting up the HTTP server requires the HTTP package. If it is not installed, please install it via “yum install httpd” or “apt-get httpd”, depending on your Linux distribution.

To configure the http server to serve the contents of the installation disk, run the following command:

cat >/etc/httpd/conf.d/pxeboot.conf <<EOF Alias /centos7
/mnt
<Directory /mnt>
	Options Indexes FollowSymLinks Require ip 127.0.0.1
	192.168.100.0/24
</Directory> 
EOF
systemctl enable httpd systemctl restart
httpd

Flashing BlueField Controller Card Bootloader Code

Before installing an OS, flash the bootloader code first. The BlueField Controller Card is shipped with an initial bootloader code,  and should be updated with the following instructions.

Opening Terminal Connection to BlueField Controller Card

To open a console window to the BlueField Controller Card, a terminal application is required. The application “minicom” is used for the flow, however, any standard terminal application can work, e.g. “screen”.

Install minicom by running “yum install minicom” or “apt-get install minicom”.

  1. On the host, type “minicom” to open minicom on the current terminal, use “minicom -s” to set it up.
  2. Go to the settings menu by pressing “Ctrl-a + o” (the setting menu opens by default when launching with the “-s” option). Navigate to the “Serial port setup” submenu and set the “Serial Device” to the one connected (should be one of the /dev/ttyUSBx if using the serial- UART cable).
  3. Change the baud rate to 115200 8N1, and ensure that the hardware and software flow control are set to “No”.
    Minicom Settings - Example
  4. Select “Save setup as dfl” in order not to have to set it again in the future.

Using Initial Install Bootstream

  1. On the host side, ensure that the RShim driver is running:

    $ systemctl status rshim

    An RShim device is located under the /dev directory, if you only have one, it should be “rshim0”:

    [root@bu-lab02 ~]# ls /dev/rshim0/ 
boot      console net       rshim

    The boot device is used to push the bootstream to the BlueField Controller Card. Upon writing to it, it automatically resets the Arm cores so that it is booted using the pushed bootstream.

    The console device can be used as a console instead of the serial-USB console. The primary bootloader does not support this device, however, UEFI and Linux support it. In cases where the special UART adapter board is unavailable, this can be used instead.

  2. Push the initial install bootstream to the BlueField Controller Card:

    cat /root/BlueField-1.0.alphaX.XXXXX/sample/install.bfb > \
/dev/rshim0/boot

    On the terminal, various boot messages appear until Linux is loaded. This is the Yocto embedded Linux running off the kernel initramfs pushed in the bootstream.

  3. When prompted, type in “root” to get to the command prompt without any password.
    Yocto Log
     
  4. After Linux is loaded, in the terminal, run the /opt/mlnx/scripts/bfrec script to update the bootloader.


Installing CentOS 7.4 on BlueField Controller Card

If the error “no root is found” appears in the installation process, check or disable the firewall as needed on the server host machine.

Full PXE Boot Installation

  1. Get to the UEFI boot menu.
    1. Reboot the BlueField Controller Card by typing “reboot” on the console. A “UEFI firmware…” message should appear and the screen clears.
    2. Press ESC several times until you enter the UEFI boot menu.

      UEFI Boot Menu

    3. On the host, restart the DHCP and TFTP service:

      systemctl restart dhcpd 
systemctl restart tftp #might be xinetd
    4. Navigate to the Boot Manager.
      UEFI Boot Manager 
    5. Select EFI Network, it will then use the TFTP service on the host to discover all available PXE boot options. Shortly after, a “..Fetching Netboot Image” message will appear enabling CentOS installation.
       Option to Install CentOS

    6. Select CentOS download.

      This process may take few minutes as it fetches data over the USB network. Running “ifconfig” on the host and monitoring the RX/TX packets on the “tmfifo_net0” network indicates that the fetching data process is not complete.

    7. Follow the installation instructions in the configuration menu. Recommended settings are included.

      These configuration inputs are not needed when the kickstart option “-k” is specified when running the setup.sh script.

      ======================================================================================
====================================================================================== VNC

Text mode provides a limited set of installation options. It does not offer custom partitioning for full control over the disk layout. Would you like to use VNC mode instead?

1)	Start VNC

2)	Use text mode

Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: 2
======================================================================================
======================================================================================
Installation:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1
1) [x] Language settings	2) [!] Time settings
(English (United States))	(Timezone is not set.)
3) [!] Installation source	4) [!] Software selection (Processing...)		(Processing...)
5) [!] Installation Destination	6) [x] Kdump
(No disks selected)	(Kdump is enabled)
7) [x] Network configuration	8) [!] Root password
(Wired (eth0) connected)	(Password is not set.)
9) [!] User creation
(No user will be created)
Please make your choice from above ['q' to quit | 'b' to begin installation | 'r' to refresh]: 2
======================================================================================
======================================================================================
Time settings Timezone: not set
NTP servers:not configured

1)	Set timezone
2)	Configure NTP servers
Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: 1
======================================================================================
======================================================================================
Timezone settings Available regions
Available regions
1)	Europe	6)	Pacific	    10)	Arctic
2)	Asia	7)	Australia	11)	US
3)	America	8)	Atlantic	12)	Etc
4)	Africa	9)	Indian		
5)	Antarctica				

Please select the timezone.
Use numbers or type names directly [b to region list, q to quit]: 11
======================================================================================
======================================================================================
Timezone settings

Available timezones in region US
1)	Alaska	4) Eastern	6) Mountain
2)	Arizona	5) Hawaii	7) Pacific
3)	Central
Please select the timezone.
Use numbers or type names directly [b to region list, q to quit]: 4
======================================================================================
======================================================================================
Installation

1) [x] Language settings	2) [x] Time settings (English (United States))		(US/Eastern timezone)
3) [x] Installation source	4) [x] Software selection (http://192.168.100.1/centos7)		(Minimal Install)
5) [!] Installation Destination	6) [x] Kdump
(No disks selected)	(Kdump is enabled)
7) [x] Network configuration	8) [!] Root password
(Wired (eth0) connected)	(Password is not set.)
9) [!] User creation
(No user will be created)
Please make your choice from above ['q' to quit | 'b' to begin installation | 'r' to refresh]: 4
======================================================================================
======================================================================================
Base environment Software selection

Base environment

1)	[x] Minimal Install	6) [ ] Server with GUI
2)	[ ] Compute Node	7) [ ] GNOME Desktop
3)	[ ] Infrastructure Server	8) [ ] KDE Plasma Workspaces
4)	[ ] File and Print Server	9) [ ] Development and Creative
5)	[ ] Basic Web Server	Workstation
Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: 9
======================================================================================
======================================================================================
Base environment Software selection

1)	[ ] Minimal Install	6)	[ ] Server with GUI
2)	[ ] Compute Node	7)	[ ] GNOME Desktop
3)	[ ] Infrastructure Server	8)	[ ] KDE Plasma Workspaces
4)	[ ] File and Print Server	9)	[x] Development and Creative
5) [ ] Basic Web Server	Workstation
Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: c
======================================================================================
======================================================================================
Installation

1) [x] Language settings	2) [x] Time settings (English (United States))		(US/Eastern timezone)
3) [!] Installation source	4) [!] Software selection (Processing...)		(Processing...)
5) [!] Installation Destination	6) [x] Kdump
(No disks selected)	(Kdump is enabled)
7) [x] Network configuration	8) [!] Root password
(Wired (eth0) connected)	(Password is not set.)
9) [!] User creation
(No user will be created)
Please make your choice from above ['q' to quit | 'b' to begin installation | 'r' to refresh]: 5
======================================================================================
======================================================================================
Probing storage...
Installation Destination

[x] 1) : 13.75 GiB (mmcblk0)

1 disk selected; 13.75 GiB capacity; 1007.5 KiB free ...

Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: c
======================================================================================
======================================================================================
Autopartitioning Options

[ ] 1) Replace Existing Linux system(s) [x] 2) Use All Space
[ ] 3) Use Free Space

Installation requires partitioning of your hard drive. Select what space to use for the install target.

Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: c
======================================================================================
======================================================================================
Partition Scheme Options [ ] 1) Standard Partition [ ] 2) Btrfs
[x] 3) LVM
[ ] 4) LVM Thin Provisioning

Select a partition scheme configuration.

Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: 1
======================================================================================
======================================================================================
Partition Scheme Options [x] 1) Standard Partition [ ] 2) Btrfs
[ ] 3) LVM

[ ] 4) LVM Thin Provisioning

Select a partition scheme configuration.

Please make your choice from above ['q' to quit | 'c' to continue | 'r' to refresh]: c Generating updated storage configuration
Checking storage configuration...

======================================================================================
======================================================================================
Installation

1)	[x] Language settings	2) [x] Time settings (English (United States))		(US/Eastern timezone)
3) [x] Installation source	4) [x] Software selection (http://192.168.100.1/centos7)		(Development and Creative
5) [x] Installation Destination		Workstation) (Automatic partitioning	6) [x] Kdump
selected)	(Kdump is enabled)
7) [x] Network configuration	8) [!] Root password
(Wired (eth0) connected)	(Password is not set.)
9) [!] User creation
(No user will be created)
Please make your choice from above ['q' to quit | 'b' to begin installation | 'r' to refresh]: 8
======================================================================================
======================================================================================
Please select new root password. You will have to type it twice.

Password:
Password (confirm):
======================================================================================
======================================================================================
Question

The password you have provided is weak: The password fails the dictionary check
-	it is based on a dictionary word. Would you like to use it anyway?

Please respond 'yes' or 'no': yes
======================================================================================
======================================================================================
Installation

1)	[x] Language settings	2) [x] Time settings (English (United States))		(US/Eastern timezone)
3) [x] Installation source	4) [x] Software selection (http://192.168.100.1/centos7)		(Development and Creative
5) [x] Installation Destination		Workstation) (Automatic partitioning	6) [x] Kdump
selected)	(Kdump is enabled)
7) [x] Network configuration	8) [x] Root password (Wired (eth0) connected)		(Password is set.)
9) [ ] User creation
(No user will be created)
Please make your choice from above ['q' to quit | 'b' to begin installation | 'r' to refresh]: b
    8. Enter “b” and press “enter” to initiate the installation process.

    9. Press “Enter” to reboot into CentOS.
      CentOS Installation Completion Screen

Non-PXE Boot Installation

When the setup script is run with the “-t” option, it generates a nonpxe.bfb file at the directory where the script is run. The directory contains the install kernel and rootfs which are usually loaded by UEFI during the initial PXE boot stage. Thus, if pushing this file, the host TFTP server no longer needs to be used and UEFI would automatically load the install kernel and rootfs from the boot FIFO. Together with the “-k” kickstart option, the host can be configured to initiate non-PXE boot and automatic CentOS installation, as long as the host HTTP and DHCP servers are working. To kick off the installation process, run the following command on the host:

cat nonpxe.bfb > /dev/rshim0/boot; sleep 2; systemctl restart dhcpd

MLNX_OFED Installation

This section is relevant to non-Yocto Operating Systems only.

Installing MLNX_OFED on Arm Cores

Prerequisite Packages for Installing MLNX_OFED

  • MLNX_OFED installation requires some prerequisite packages to be installed on the system.
    Currently, CentOS installed on the BlueField Controller Card has a private network to the host via the USB connection, and it can be used to Secure Copy Protocol (SCP) all the required packages. However, it is recommended for the BlueField Controller Card to have a direct access to the network to use “yum install” to install all the required packages. For direct access to the network, set up the routing on the host via:

    iptables -t nat -o eth0 -A POSTROUTING -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward
systemctl restart dhcpd

    “eth0” is the outgoing network interface on the host. Change this according to your system requirements.

    These commands are not saved in Linux startup script, and might be needed to be applied again after host machine reboots.

  • Reset the BlueField Controller Card network for Internet connection (access to the web) as long as the host is connected:

    [root@localhost
~]# ifdown eth0;
ifup eth0 [root@localhost
~]# ping google.com
PING google.com (172.217.10.142) 56(84)
bytes of data.
64 bytes from lga34s16-in-f14.1e100.net (172.217.10.142): icmp_seq=1 ttl=53 time=19.2 ms
64 bytes from lga34s16-in-f14.1e100.net (172.217.10.142): icmp_seq=2 ttl=53 time=17.7 ms
64 bytes from
lga34s16-in-f14.1e100.net (172.217.10.142): icmp_seq=3 ttl=53 time=15.8 ms

  • Run “yum install” to install all the required MLNX_OFED packages:

    yum install rpm-build
yum group install "Development Tools" yum install kernel-devel-`uname -r`
yum install valgrind-devel libnl3-devel python-devel yum install tcl tk

    Note that this is not needed if you installed CentOS 7 with the kickstart (“-k”) option.

    /auto/sw_mc_soc_project/distro/rhel/kernel-devel-4.11.0-22.el7a.aarch64.rpm

Removing Pre-installed Kernel Module

There are cases where the kernel is shipped with an earlier version of the mlx5_core driver taken from the upstream Linux code. This version does not support the BlueField Arm, but is loaded before the MLNX_OFED driver, and therefore, needs to be removed.

To remove the kernel module from the initramfs, run the following command:

mkdir /boot/tmp 
cd /boot/tmp
gunzip < ../initramfs-4*64.img | cpio -i 
rm -f lib/modules/4*/updates/mlx5_core.ko
rm -f lib/modules/4*/updates/tmfifo*.ko
cp ../initramfs-4*64.img ../initramfs-4.11.0-22.el7a.aarch64.img-bak 
find | cpio -H newc -o | gzip -9 > ../initramfs-4*64.img
rpm -e mlx5_core 
depmod -a

Installing MLNX_OFED on the BlueField Controller Card

  1. Copy the MLNX_OFED image to the BlueField Controller Card via the USB network. The MLNX_OFED images should be provided in the software drop:

    scp MLNX_OFED_LINUX-4.2-1.4.8.0-rhel7.4alternate-aarch64.iso \
root@192.168.100.2:/root

  2. Mount the image on the BlueField Controller Card:

    mount /root/MLNX_OFED_LINUX-4.2-1.4.8.0-rhel7.4alternate-aarch64.iso /mnt

  3. Install MLNX_OFED.
    If the kernel on the BlueField is 4.11.0-22.el7a.aarch64, run:

    cd /mnt
# ./mlnxofedinstall --bluefield

    If the kernel is different than 4.11.0-22.el7a.aarch64, run:

    cd /mnt
# ./mlnxofedinstall --add-kernel-support --skip-repo

    For OFED to support DPDK, use the arguments “--upstream-libs” and “--dpdk”.

    ./mlnxofedinstall --distro rhel7.4alternate --add-kernel-support --upstream-libs
--dpdk

    This step might take longer than expected to be completed. If you are using a different pack- age than the required one, run “yum install”.

    If the date is not set correctly while installing MLNX_OFED, first, set the date (e.g date -s 'Mon Feb 5 15:02:10 EST 2018'), then run the installation.

  4. Restart openibd:

    /etc/init.d/openibd restart

Updating BlueField Controller Card Firmware

The below commands apply to MBF1M616A-CSNAT. The commands vary per OPN.

The below steps demonstrate how to manually update the firmware if the automatic process fails. The firmware image can be found in the BlueField Software package.

  1. Copy the firmware image to the BlueField Arm:

    scp fw-BlueField-rel--XX_XX_XXXX-MBF1M6X6A-CSNA_Ax.ini.bin \
root@192.168.100.2:/root
[root@localhost ~]# mst start
Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success
Loading MST PCI configuration module - Success Create devices
Unloading MST PCI module (unused) - Success [root@localhost ~]# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt41682_pciconf0
- PCI configuration cycles access.
domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92
Chip revision is: 00

  2. The output indicates that the device is “/dev/mst/mt41682_pciconf0”. To update the firmware:

    flint -d /dev/mst/mt41682_pciconf0 b \
-i /root/ fw-BlueField-rel-XX_XX_XXXX-MBF1M6X6A-CSNA_Ax.ini.bin

    When using the mlx and ini files, use the following command instead:

    mlxburn -d /dev/mst/mt41682_pciconf0 -fw fw-BlueField.mlx -c bf.ini

    To burn the firmware which comes with OFED after OFED is installed, run:

    /opt/mellanox/mlnx-fw-updater/firmware/mlxfwmanager_sriov_dis    -force

  3. Power cycle the BlueField Controller Card for the new firmware to take effect.

    root@bluefield:~# flint -i /opt/fw-Bluefield-red-18_99_4608_MBF1M6X6A-CSNA_Ax-Flexboot-3.5.404_UEFI-14.15.20.bin -d /dev/mst/mt41682_pciconf0 b

    Current FW version flash:     18.24.0013
	New FW version:               18.99.4608
	
Burning FW image without signatures - 55%
Burning FW image without signatures - OK
Restoring signature
-I- To load new FW run mlxfwreset or reboot machine.
root@bluefield:~#

    After MLNX_OFED is installed on the Arm cores, use the mlx5_core driver to use the two Ethernet ports on the BlueField Controller Card. If the Ethernet ports on the BlueField Controller Card are connected to the network, there is no need to bridge the host via RShim net to access the network.

To install the kernel modules, please follow the instruction in section RShim Host Driver.