NVIDIA BlueField DPU BSP v3.8.5
1.0

Host-side Interface Configuration

Multiple BlueField cards could connect to the same host machine. Each board has its own device directory (/dev/rshim<N>). The following are some guidelines on how to set up RShim networking properly in this case:

  • Each target should load only one backend (usb, pcie or pcie_lf)

  • The host RShim network interface should have different MAC and IP addresses, which can be configured with ifconfig as shown below, or saved in configuration:

    Copy
    Copied!
                

    $ ifconfig tmfifo_net0 192.168.100.1/24 hw ether 02:02:02:02:02:02

  • The Arm-side tmfifo interface should have unique MAC and IP addresses as well, which can be configured in the console

Multi-board Management Example

This example deals with two BlueField DPUs installed on the same server (the process is similar for more DPUs).

This example assumes that the RShim package has been installed on the host server.

Configuring Host Server Side

Warning

This example is relevant for CentOS/RHEL operating systems only.

  1. Create a bf_tmfifo interface under /etc/sysconfig/network-scripts/. Run:

    Copy
    Copied!
                

    vim /etc/sysconfig/network-scripts/ifcfg-br_tmfifo

  2. Inside ifcfg-br_tmfifo, insert the following content:

    Copy
    Copied!
                

    DEVICE="br_tmfifo" BOOTPROTO="static" IPADDR="192.168.100.1" NETMASK="255.255.255.0" ONBOOT="yes" TYPE="Bridge" NM_CONTROLLED="no"

  3. Create a configuration file for the first BlueField DPU, tmfifo_net0. Run:

    Copy
    Copied!
                

    vim /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0

  4. Inside ifcfg-tmfifo_net0, insert the following content:

    Copy
    Copied!
                

    DEVICE=tmfifo_net0 BOOTPROTO=none ONBOOT=yes NM_CONTROLLED=no BRIDGE=br_tmfifo

  5. Create a configuration file for the second BlueField DPU, tmfifo_net1. Run:

    Copy
    Copied!
                

    DEVICE=tmfifo_net1 BOOTPROTO=none ONBOOT=yes NM_CONTROLLED=no BRIDGE=br_tmfifo

  6. Create the rules for the tmfifo interfaces. Run:

    Copy
    Copied!
                

    vim /etc/udev/rules.d/91-tmfifo_net.rules

  7. Inside 91-tmfifo_net.rules, insert the following content:

    Copy
    Copied!
                

    SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:1a:ca:ff:ff:02", ATTR{type}=="1", NAME="tmfifo_net0", RUN+="/bin/sh -c '/usr/sbin/ifup tmfifo_net0 2>/dev/null || /sbin/ip link set dev tmfifo_net0 up'" SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:1a:ca:ff:ff:04", ATTR{type}=="1", NAME="tmfifo_net1", RUN+="/bin/sh -c '/usr/sbin/ifup tmfifo_net1 2>/dev/null || /sbin/ip link set dev tmfifo_net1 up'"

  8. Restart the network for the changes to take effect. Run:

    Copy
    Copied!
                

    # /etc/init.d/network restart Restarting network (via systemctl): [ OK ]

Configuring BlueField DPU Side

BlueField DPUs arrive with the following factory default configurations for tmfifo_net0.

Address

Value

MAC

00:1a:ca:ff:ff:01

IP

192.168.100.2

Therefore, if you are working with more than one DPU, you must change the default MAC and IP addresses.

Updating MAC Address

Warning

This procedure is relevant for Ubuntu/Debian (sudo needed) and CentOS BFBs.

  1. Open two console screen tabs per BlueField DPU. Run:

    Copy
    Copied!
                

    # sudo screen /dev/rshim<0|1>/console 115200

    Warning

    Screen console RPM must be installed.

    Copy
    Copied!
                

    # sudo yum install screen

    If the /dev/rshim device does not exist, the user should confirm they have RShim drivers installed. Run the following on the x86 host:

    Copy
    Copied!
                

    sudo dpkg -l rshim

  2. Create a configuration file for tmfifo_net0 MAC address. Run:

    Copy
    Copied!
                

    # sudo vi /etc/bf.cfg

  3. Inside bf.cfg, insert the new MAC:

    Copy
    Copied!
                

    NET_RSHIM_MAC=00:1a:ca:ff:ff:03

  4. Apply the new MAC address. Run:

    Copy
    Copied!
                

    sudo bfcfg

  5. Repeat this procedure for the second BlueField DPU (using a different MAC address).

    Note

    Arm must be rebooted for this configuration to take effect. It is recommended to update the IP address before you do that to avoid unnecessary reboots.

Updating IP Address

For Ubuntu:

  1. Access the file 50-cloud-init.yaml and modify the tmfifo_net0 IP address:

    Copy
    Copied!
                

    sudo vim /etc/netplan/50-cloud-init.yaml   tmfifo_net0: addresses: - 192.168.100.2/30 ===>>> 192.168.100.3/30

  2. Reboot the Arm. Run:

    Copy
    Copied!
                

    sudo reboot

  3. Repeat this procedure for the second BlueField DPU (using a different IP address).

    Note

    Arm must be rebooted for this configuration to take effect. It is recommended to update the MAC address before you do that to avoid unnecessary reboots.

For CentOS:

  1. Access the file ifcfg-tmfifo_net0. Run:

    Copy
    Copied!
                

    # vim /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0

  2. Modify the value for IPADDR:

    Copy
    Copied!
                

    IPADDR=192.168.100.3

  3. Reboot the Arm. Run:

    Copy
    Copied!
                

    reboot

    Or perform netplan apply.

  4. Repeat this procedure for the second BlueField DPU (using a different IP address).

    Note

    Arm must be rebooted for this configuration to take effect. It is recommended to update the MAC address before you do that to avoid unnecessary reboots.

Warning

It is assumed that the commands in this section are executed with root (or sudo) permission.

The default MAC address is 00:1a:ca:ff:ff:01. It can be changed using ifconfig or by updating the UEFI variable as follows:

  1. Log into Linux from the Arm console.

  2. Run:

    Copy
    Copied!
                

    $ "ls /sys/firmware/efi/efivars".

  3. If not mounted, run:

    Copy
    Copied!
                

    $ mount -t efivarfs none /sys/firmware/efi/efivars $ chattr -i /sys/firmware/efi/efivars/RshimMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c $ printf "\x07\x00\x00\x00\x00\x1a\xca\xff\xff\x03" > \ /sys/firmware/efi/efivars/RshimMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c

The printf command sets the MAC address to 00:1a:ca:ff:ff:03 (the last six bytes of the printf value). Either reboot the device or reload the tmfifo driver for the change to take effect.

The MAC address can also be updated from the server host side while the Arm-side Linux is running:

  1. Enable the configuration. Run:

    Copy
    Copied!
                

    # echo "DISPLAY_LEVEL 1" > /dev/rshim0/misc

  2. Display the current setting. Run:

    Copy
    Copied!
                

    # cat /dev/rshim0/misc DISPLAY_LEVEL 1 (0:basic, 1:advanced, 2:log) BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap) BOOT_TIMEOUT 300 (seconds) DROP_MODE 0 (0:normal, 1:drop) SW_RESET 0 (1: reset) DEV_NAME pcie-04:00.2 (ro) DEV_INFO BlueField-2(Rev 0) PEER_MAC 00:1a:ca:ff:ff:01 (rw) PXE_ID 0x00000000 (rw) VLAN_ID 0 0 (rw)

  3. Modify the MAC address. Run:

    Copy
    Copied!
                

    $ echo "PEER_MAC xx:xx:xx:xx:xx:xx" > /dev/rshim0/misc

The BlueField-2 OOB interface is a gigabit Ethernet interface which provides TCP/IP network connectivity to the Arm cores. This interface is named oob_net0 and is intended to be used for management traffic (e.g. file transfer protocols, SSH, etc). The Linux driver that controls this interface is named mlxbf_gige.ko, and is automatically loaded upon boot. This interface can be configured and monitored by use of standard tools (e.g. ifconfig, ethtool, etc). The OOB interface is subject to the following design limitations:

  • Only supports 1Gb/s full-duplex setting

  • Only supports GMII access to external PHY device

  • Supports maximum packet size of 2KB (i.e. no support for jumbo frames)

The OOB interface can also be used for PXE boot. This OOB port is not a path for the BlueField-2 boot stream. Any attempt to push a BFB to this port will not work. Please refer to How to use the UEFI boot menu for more information about UEFI operations related to the OOB interface.

OOB Interface MAC Address

The MAC address to be used for the OOB port is burned into Arm-accessible UPVS EEPROM during the manufacturing process. This EEPROM device is different from the SPI Flash storage device used for the NIC firmware and associated NIC MACs/GUIDs. The value of the OOB MAC address is specific to each platform and is visible on the board-level sticker.

Important

It is not recommended to reconfigure the MAC address from the MAC configured during manufacturing.

If there is a need to re-configure this MAC for any reason, follow these steps to configure a UEFI variable to hold new value for OOB MAC.:

Warning

The creation of an OOB MAC address UEFI variable will override the OOB MAC address defined in EEPROM, but the change can be reverted.

  1. Log into Linux from the Arm console.

  2. Issue the command ls /sys/firmware/efi/efivars to show whether efivarfs is mounted. If it is not mounted, run:

    Copy
    Copied!
                

    mount -t efivarfs none /sys/firmware/efi/efivars

  3. Run:

    Copy
    Copied!
                

    chattr -i /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c

  4. Set the MAC address to 00:1a:ca:ff:ff:03 (the last six bytes of the printf value).

    Copy
    Copied!
                

    printf "\x07\x00\x00\x00\x00\x1a\xca\xff\xff\x03" > /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c

  5. Reboot the device for the change to take effect.

To revert this change and go back to using the MAC as programmed during manufacturing, follow these steps:

  1. Log into UEFI from the Arm console, go to "Boot Manager" then "EFI Internal Shell".

  2. Delete the OOB MAC UEFI variable. Run:

    Copy
    Copied!
                

    dmpstore -d OobMacAddr

  3. Reboot the device by running "reset" from UEFI.

  4. Log into Linux from the Arm console.

  5. Issue the command ls /sys/firmware/efi/efivars to show whether efivarfs is mounted. If it is not mounted, run:

    Copy
    Copied!
                

    mount -t efivarfs none /sys/firmware/efi/efivars

  6. Run:

    Copy
    Copied!
                

    chattr -i /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c

  7. Reconfigure the original MAC address burned by the manufacturer in the format aa\bb\cc\dd\ee\ff. Run:

    Copy
    Copied!
                

    printf "\x07\x00\x00\x00\x00\<original-MAC-address>" > /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c

  8. Reboot the device for the change to take effect.

Supported ethtool Options for OOB Interface

The Linux driver for the OOB port supports the handling of some basic ethtool requests: get driver info, get/set ring parameters, get registers, and get statistics.

To use the ethtool options available, use the following format:

Copy
Copied!
            

$ ethtool [<option>] <interface>

Where <option> may be:

  • <no-argument> – display interface link information

  • -i – display driver general information

  • -S – display driver statistics

  • -d – dump driver register set

  • -g – display driver ring information

  • -G – configure driver ring(s)

  • -k – display driver offload information

  • -a – q ueries the specified Ethernet device for pause parameter information

  • -r – re starts auto-negotiation on the specified Ethernet device if auto-negotiation is enabled

For example:

Copy
Copied!
            

$ ethtool oob_net0 Settings for oob_net0: Supported ports: [ TP ] Supported link modes: 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 1000baseT/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 3 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Link detected: yes

Copy
Copied!
            

$ ethtool -i oob_net0 driver: mlxbf_gige version: firmware-version: expansion-rom-version: bus-info: MLNXBF17:00 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: no

Copy
Copied!
            

# Display statistics specific to BlueField-2 design (i.e. statistics that are not shown in the output of "ifconfig oob0_net") $ ethtool -S oob_net0 NIC statistics: hw_access_errors: 0 tx_invalid_checksums: 0 tx_small_frames: 1 tx_index_errors: 0 sw_config_errors: 0 sw_access_errors: 0 rx_truncate_errors: 0 rx_mac_errors: 0 rx_din_dropped_pkts: 0 tx_fifo_full: 0 rx_filter_passed_pkts: 5549 rx_filter_discard_pkts: 4


IP Address Configuration for OOB Interface

The files that control IP interface configuration are specific to the Linux distribution. The udev rules file (/etc/udev/rules.d/92-oob_net.rules) that renames the OOB interface to "oob_net0" and is the same across all three major distributions (Yocto, CentOS, Ubuntu):

Copy
Copied!
            

SUBSYSTEM=="net", ACTION=="add", DEVPATH=="/devices/platform/MLNXBF17:00/net/eth[0-9]", NAME="oob_net0"

The files that control IP interface configuration are slightly different across the two major distributions (CentOS, Ubuntu):

  • CentOS configuration of IP interface:

    • Configuration file for "oob_net0" interface: /etc/sysconfig/network-scripts/ifcfg-oob_net0

    • For example, use the following to enable DHCP:

      Copy
      Copied!
                  

      NAME="oob_net0" DEVICE="oob_net0" NM_CONTROLLED="yes" PEERDNS="yes" ONBOOT="yes" BOOTPROTO="dhcp" TYPE=Ethernet

    • For example, to configure static IP use the following:

      Copy
      Copied!
                  

      NAME="oob_net0" DEVICE="oob_net0" IPV6INIT="no" NM_CONTROLLED="no" PEERDNS="yes" ONBOOT="yes" BOOTPROTO="static" IPADDR="192.168.200.2" PREFIX=30 GATEWAY="192.168.200.1" DNS1="192.168.200.1" TYPE=Ethernet

  • For Ubuntu configuration of IP interface, please refer to section "Network Interface Configuration".

© Copyright 2023, NVIDIA. Last updated on Sep 9, 2023.