BlueField OOB Ethernet Interface
The BlueField OOB interface is a gigabit Ethernet interface which provides TCP/IP network connectivity to the Arm cores. This interface is named oob_net0 and is intended to be used for management traffic (e.g., file transfer protocols, SSH, etc). The Linux driver that controls this interface is named mlxbf_gige.ko, and is automatically loaded upon boot. This interface can be configured and monitored using of standard tools (e.g., ifconfig, ethtool, etc). The OOB interface is subject to the following design limitations:
Only supports 1Gb/s full-duplex setting
Only supports GMII access to external PHY device
Supports maximum packet size of 2KB (i.e., no support for jumbo frames)
The OOB interface can also be used for PXE boot. This OOB port is not a path for the BlueField boot stream. Any attempt to push a BFB to this port would not work. Refer to "How to use the UEFI boot menu" for more information about UEFI operations related to the OOB interface.
The MAC address to be used for the OOB port is burned into Arm-accessible UPVS EEPROM during the manufacturing process. This EEPROM device is different from the SPI Flash storage device used for the NIC firmware and associated NIC MACs/GUIDs. The value of the OOB MAC address is specific to each platform and is visible on the board-level sticker.
It is not recommended to reconfigure the MAC address from the MAC configured during manufacturing.
If there is a need to re-configure this MAC for any reason, follow these steps to configure a UEFI variable to hold new value for OOB MAC.:
The creation of an OOB MAC address UEFI variable will override the OOB MAC address defined in EEPROM, but the change can be reverted.
Log into Linux from the Arm console.
Issue the command ls /sys/firmware/efi/efivars to show whether efivarfs is mounted. If it is not mounted, run:
mount -t efivarfs none /sys/firmware/efi/efivars
Run:
chattr -i /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
Set the MAC address to 00:1a:ca:ff:ff:03 (the last six bytes of the printf value).
printf "\x07\x00\x00\x00\x00\x1a\xca\xff\xff\x03" > /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reboot the device for the change to take effect.
To revert this change and go back to using the MAC as programmed during manufacturing, follow these steps:
Log into UEFI from the Arm console, go to "Boot Manager" then "EFI Internal Shell".
Delete the OOB MAC UEFI variable. Run:
dmpstore -d OobMacAddr
Reboot the device by running "reset" from UEFI.
Log into Linux from the Arm console.
Issue the command ls /sys/firmware/efi/efivars to show whether efivarfs is mounted. If it is not mounted, run:
mount -t efivarfs none /sys/firmware/efi/efivars
Run:
chattr -i /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reconfigure the original MAC address burned by the manufacturer in the format aa\bb\cc\dd\ee\ff. Run:
printf "\x07\x00\x00\x00\x00\<original-MAC-address>" > /sys/firmware/efi/efivars/OobMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reboot the device for the change to take effect.
The Linux driver for the OOB port supports the handling of some basic ethtool requests: get driver info, get/set ring parameters, get registers, and get statistics.
To use the ethtool options available, use the following format:
$ ethtool [<option>] <interface>
Where <option> may be:
<no-argument> – display interface link information
-i – display driver general information
-S – display driver statistics
-d – dump driver register set
-g – display driver ring information
-G – configure driver ring(s)
-k – display driver offload information
-a – query the specified Ethernet device for pause parameter information
-r – restart auto-negotiation on the specified Ethernet device if auto-negotiation is enabled
For example:
$ ethtool oob_net0
Settings for oob_net0:
Supported ports: [ TP ]
Supported link modes: 1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 1000baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 3
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Link detected: yes
$ ethtool -i oob_net0
driver: mlxbf_gige
version:
firmware-version:
expansion-rom-version:
bus-info: MLNXBF17:00
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
# Display statistics specific to BlueField-2 design (i.e. statistics that are not shown in the output of "ifconfig oob0_net")
$ ethtool -S oob_net0
NIC statistics:
hw_access_errors: 0
tx_invalid_checksums: 0
tx_small_frames: 1
tx_index_errors: 0
sw_config_errors: 0
sw_access_errors: 0
rx_truncate_errors: 0
rx_mac_errors: 0
rx_din_dropped_pkts: 0
tx_fifo_full: 0
rx_filter_passed_pkts: 5549
rx_filter_discard_pkts: 4
The files that control IP interface configuration are specific to the Linux distribution. The udev rules file (/etc/udev/rules.d/92-oob_net.rules) that renames the OOB interface to oob_net0 and is the same for Yocto, CentOS, and Ubuntu:
SUBSYSTEM=="net", ACTION=="add", DEVPATH=="/devices/platform/MLNXBF17:00/net/eth[0-9]", NAME="oob_net0"
The files that control IP interface configuration are slightly different for CentOS and Ubuntu:
CentOS configuration of IP interface:
Configuration file for oob_net0: /etc/sysconfig/network-scripts/ifcfg-oob_net0
For example, use the following to enable DHCP:
NAME="oob_net0" DEVICE="oob_net0" NM_CONTROLLED="yes" PEERDNS="yes" ONBOOT="yes" BOOTPROTO="dhcp" TYPE=Ethernet
For example, to configure static IP use the following:
NAME="oob_net0" DEVICE="oob_net0" IPV6INIT="no" NM_CONTROLLED="no" PEERDNS="yes" ONBOOT="yes" BOOTPROTO="static" IPADDR="192.168.200.2" PREFIX=30 GATEWAY="192.168.200.1" DNS1="192.168.200.1" TYPE=Ethernet
For Ubuntu configuration of IP interface, please refer to section "Default Network Interface Configuration".