Port Configurations
This section applies to the following OPNs:
Model | OPNs |
B3140H | 900-9D3D4-00EN-HA0 900-9D3D4-00NN-HA0 |
B3220L | 900-9D3B4-00CV-EA0 900-9D3B4-00SV-EA0 |
BlueField-3 Platforms offer a variety of network port configurations designed to meet the demands of different environments and deployments. This section outlines the available connectivity options for network ports per product model. While the configurations listed here are not exhaustive, they represent the tested and verified options. Customers seeking to support unlisted configurations should contact NVIDIA Support.
The Port Splitting feature allows a single physical networking module (QSFP112 or OSFP) to be split into multiple network ports. This provides flexibility in optimizing port configurations for various network topology use cases.
Key Features
Flexibility and Improved Traffic Segmentation: Port splitting enables network administrators to align device ports with their network infrastructure, enabling traffic segmentation and isolation.
Increased Network Efficiency: By optimizing traffic flow and reducing congestion, port splitting improves overall network efficiency. Specific physical ports can be assigned to handle certain traffic types or applications, allowing control and performance tuning.
Better Utilization of Resources: Instead of needing separate SuperNICs for each logical connection or network interface, port splitting allows a single physical SuperNIC to handle multiple functions. This reduces hardware costs while maintaining high network flexibility.
Use Cases
Load Balancing: Distributing traffic across multiple physical ports can improve network load distribution and balance.
Resiliency : Multiple physical ports can be used for redundancy, ensuring stable network connections even if one port encounters an issue.
Each configuration in this section includes a diagram and instructions on splitting the network port using the mlxconfig or the UEFI tools. The UEFI tool configures the device before the operating system is up, while mlxconfig configures the device once the operating system is up. Select the tool based on your preference; the configuration is saved across reboots.
mlxconfig: The mlxconfig tool allows users to change configurations without burning the firmware. The configuration is also kept after reset. By default, mlxconfig shows the configurations that will be loaded in the next boot. For more information and instructions, refer to mlxconfig.
UEFI: PreBoot drivers initialize the adapter device, check the port protocol type – Ethernet or InfiniBand - and bring up the port. Then it connects to a DHCP server to obtain its assigned IP address and network parameters and obtain the source location of the kernel/OS to boot from. The DHCP server instructs the PreBoot drivers to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For more information and instructions, refer to UEFI.
Before setting the desired configuration, take note of the following warnings:
Any new configuration set by the customer overwrites the previous configuration.
Misconfiguration can potentially damage the system.
It is recommended to establish out-of-band connectivity to the BlueField-3 Platform before setting any configuration for the first time. This allows resetting the NVConfig parameters to their default values in case of misconfiguration.
When switching from one configuration to another (default or non-default), it is advised to first reset the device to its default configuration. This action deletes all current NVConfig settings. For the reset command and further details, refer to the mlxconfig – Changing Device Configuration Tool.
Reboot or power-cycle the SuperNIC for the new configuration to take effect.
The following table summarizes the available port-splitting configurations.
Model | OPNs |
B3140H | 900-9D3D4-00EN-HA0 900-9D3D4-00NN-HA0 |
B3220L | 900-9D3B4-00CV-EA0 900-9D3B4-00SV-EA0 |
Model | OPNs | Configuration | Configuration Description |
B3140H | 900-9D3D4-00EN-HA0 900-9D3D4-00NN-HA0 | Two Ethernet ports, each supporting 200Gbs | |
Four Ethernet ports, each supporting 100Gbs | |||
B3220L | 900-9D3B4-00CV-EA0 900-9D3B4-00SV-EA0 | Two Ethernet ports, each supporting 200Gbs | |
Four Ethernet ports, each supporting 100Gbs | |||
Note1: For higher radix (higher host/HCA connections) with Q3400 XDR Switch. | |||
The B4240 BlueField-3 Platform, featuring a single-port OSFP module, supports up to 800Gb/s. For the reconfiguration to take effect, please perform power cycle.
B4240 Configuration #1 (Default): Two Ethernet Ports, Each Supporting 400GbE
In this configuration, the device can be reconfigured to support two 400GbE Ethernet ports. The configuration is illustrated in the diagram below.
B4240 Configuration #2: Eight Ethernet Ports, Each Supporting 100GbE
In this configuration, the device can be configured to support eight 100GbE Ethernet ports. The configuration is illustrated in the diagram below.
To apply this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set MODULE_SPLIT_M0[0]=1 MODULE_SPLIT_M0[1]=2 MODULE_SPLIT_M0[2]=3 MODULE_SPLIT_M0[3]=4 MODULE_SPLIT_M0[4]=5 MODULE_SPLIT_M0[5]=6 MODULE_SPLIT_M0[6]=7 MODULE_SPLIT_M0[7]=8 MODULE_SPLIT_M0[8..15]=FF
mlxconfig -d <device> set NUM_OF_PF = 8
B4240 Configuration #3: One InfiniBand Port Supporting XDR Lite 400Gb/s
In this configuration, the device can be reconfigured to support one InfiniBand port supporting XDR Lite 400Gb/s (for hosts that only support up to PCIe Gen 5.0). The configuration is illustrated in the diagram below.
To enable this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set MODULE_SPLIT_M0[0..3]=1 MODULE_SPLIT_M0[4..15]=FF
mlxconfig -d <device> set NUM_OF_PF = 1
mlxconfig -d <device> set NUM_OF_PLANES_P1 = 4
mlxconfig -d <device> set LINK_TYPE_P1=1
mlxconfig -d <device> set IB_PROTO_WIDTH_EN_MASK_P1 = 7847937
B4240 Configuration #4: Two InfiniBand Ports Supporting XDR Lite 400Gb/s
In this configuration, the device can be reconfigured to support two InfiniBand ports, each supporting XDR Lite 400Gb/s (for hosts that only support up to PCIe Gen 5.0). The configuration is illustrated in the diagram below.
To apply this configuration, use the following mlxconfig commands:
mmlxconfig -d <device> set MODULE_SPLIT_M0[0..3]=1 MODULE_SPLIT_M0[4..7]=2 MODULE_SPLIT_M0[8..15]=FF
mlxconfig -d <device> set NUM_OF_PF = 2
mlxconfig -d <device> set NUM_OF_PLANES_P1 = 4 NUM_OF_PLANES_P2 = 4
mlxconfig -d <device> set LINK_TYPE_P1=1
mlxconfig -d <device> set LINK_TYPE_P2=1
mlxconfig -d <device> set IB_PROTO_WIDTH_EN_MASK_P1 = 7847937
mlxconfig -d <device> set IB_PROTO_WIDTH_EN_MASK_P2 = 7847937
B4240 Configuration #5: One InfiniBand Port Supporting NDR 400Gb/s
In this configuration, the device can be reconfigured to one InfiniBand port supporting NDR 400Gb/s. The configuration is illustrated in the diagram below.
To apply this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set MODULE_SPLIT_M0[0..3]=1 MODULE_SPLIT_M0[4..15]=FF
mlxconfig -d <device> set LINK_TYPE_P1=1
mlxconfig -d <device> set NUM_OF_PF = 1
B4240 Configuration #6: Two InfiniBand Ports Supporting NDR 400Gb/s
In this configuration, the device can be reconfigured to two InfiniBand port supporting NDR 400Gb/s. The configuration is illustrated in the diagram below.
To apply this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set MODULE_SPLIT_M0[0..3]=1 MODULE_SPLIT_M0[4..7]=2 MODULE_SPLIT_M0[8..15]=FF
mlxconfig -d <device> set LINK_TYPE_P1=1
mlxconfig -d <device> set LINK_TYPE_P2=1
mlxconfig -d <device> set NUM_OF_PF = 2