The NVIDIA® BlueField® family of (Data Processing Unit) DPU devices combines an array of Arm processors coupled with the NVIDIA® ConnectX® interconnect. Standard Linux distributions run on the Arm cores allowing common open source development tools to be used. The SoC can be accessed via USB (external cable) or PCIe driven by our RShim drivers. RShim drivers provides functionalities like resetting the Arm cores, pushing a bootstream image, networking functionality and console functionality.
For further information see RShim Drivers and Usage.
When the adapter is in SmartNIC mode, the following features are controlled from System-On-Chip (SoC) side. For more information on NVIDIA® BlueField® and functionality, please refer to NVIDIA BlueField Family Documentation → BlueField Software Overview.
Encapsulation/Decapsulation – VXLAN/GRE packet encapsulation/decapsulation is done on the SoC side. Please refer to NVIDIA BlueField Family Documentation → Virtual Switch on BlueField SmartNIC
Rate limiting of host PF and VF – For example, users may limit the transmit rate of the PF in the host to 1000mbps and VF to 500 mbps. Please refer to NVIDIA BlueField Family Documentation → QoS Configuration
Offloading VLANs – The OVS can add VLAN tag to all packets sent by network interface running on host PF or VF. Please refer to NVIDIA BlueField Family Documentation → Virtual Switch on BlueField SmartNIC
Bluefield Link Aggregation - configure network bonding on the Arm side in a manner transparent to the host. Under such configuration, the host would only see a single PF. Please refer to NVIDIA BlueField Family Documentation → BlueField Link Aggregation
Setting Host PF and VF Default MAC Address. Please refer to NVIDIA BlueField Family Documentation → Controlling Host PF and VF Parameters
DCQCN and DSCP based congestion control for RoCE. Please refer to: https://support.mellanox.com/s/article/mlnx-qos
QoS – Host settings can be honored or ignored based on settings (changeable using mstpriv tool). Please refer to https://support.mellanox.com/s/article/mlnx-qos
Link speed cannot be changed using user space (mlx5cmd). Please refer to NVIDIA BlueField Family Documentation → Controlling Host PF and VF Parameters
Performance counters – Cannot query Mellanox WinOF-2 PCI Device Diagnostics
Cannot query/modify VF capabilities from Windows host
Dropless mode query/set is not supported from the host side
When performing MlxFwReset (one of our MFT tools), need to disable host network adapters manually and wait until SoC is up before enabling them
Open vSwitch (OVS) running on the Arm cores allows Virtual Machines (VMs) to communicate with each other and with the outside world. For more details on OVS, please refer to NVIDIA BlueField Family Documentation.
OpenvSwitch (OVS) running on the Arm cores supports two modes:
hardware offload mode enabled - With Hardware offload enabled (default mode), the first few packets are processed by the OVS for learning and rule injection which can be processed in parallel thus, test fails because packets go out-of-order to the host (windows driver).
hardware offload mode disabled - With Hardware Offload disabled, all packets go through the Arm core and cannot keep up with heavy network traffic. To overcome this limitation, and to make it easy for customers who want to run certification, we provide two scripts under /opt/mellanox/hlk.
Please execute "/opt/mellanox/hlk/mlnx-pre-hlk" from the SoC before starting the HLK tests and after done, execute "/opt/mellanox/hlk/mlnx-post-hlk" to enable OVS and delete manually programmed rules.
NDIS6.0/6.5 of Windows HLK tests use IPX/SPX protocol for send/receive in quite a few cases. There is no handshake or retransmits. Test keeps track of packet count and ordering.