Port Splitting Configurations
ConnectX-8 SuperNICs offer a variety of network port configurations designed to meet the demands of different environments and deployments. This section outlines the available connectivity options for network ports per product model. While the configurations listed here are not exhaustive, they represent the tested and verified options. Customers seeking to support unlisted configurations should contact NVIDIA Support.
The Port Splitting feature allows a single physical networking module (QSFP112 or OSFP) to be split into multiple network ports. This provides flexibility in optimizing port configurations for various network topology use cases.
Key Features
Flexibility and Improved Traffic Segmentation: Port splitting enables network administrators to align device ports with their network infrastructure, enabling traffic segmentation and isolation.
Increased Network Efficiency: By optimizing traffic flow and reducing congestion, port splitting improves overall network efficiency. Specific physical ports can be assigned to handle certain traffic types or applications, allowing control and performance tuning.
Better Utilization of Resources: Instead of needing separate SuperNICs for each logical connection or network interface, port splitting allows a single physical SuperNIC to handle multiple functions. This reduces hardware costs while maintaining high network flexibility.
Use Cases
Load Balancing: Distributing traffic across multiple physical ports can improve network load distribution and balance.
Resiliency : Multiple physical ports can be used for redundancy, ensuring stable network connections even if one port encounters an issue.
Each configuration in this section includes a diagram and instructions on splitting the network port using the mlxconfig or the UEFI tools. The UEFI tool configures the device before the operating system is up, while mlxconfig configures the device once the operating system is up. Select the tool based on your preference; the configuration is saved across reboots.
mlxconfig: The mlxconfig tool allows users to change configurations without burning the firmware. The configuration is also kept after reset. By default, mlxconfig shows the configurations that will be loaded in the next boot. For more information and instructions, refer to mlxconfig.
UEFI: PreBoot drivers initialize the adapter device, check the port protocol type – Ethernet or InfiniBand - and bring up the port. Then it connects to a DHCP server to obtain its assigned IP address and network parameters and obtain the source location of the kernel/OS to boot from. The DHCP server instructs the PreBoot drivers to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For more information and instructions, refer to UEFI.
Before setting the desired configuration, take note of the following warnings:
Any new configuration set by the customer overwrites the previous configuration.
Misconfiguration can potentially damage the system.
It is recommended to establish out-of-band connectivity to the ConnectX-8 SuperNIC before setting any configuration for the first time. This allows resetting the NVConfig parameters to their default values in case of misconfiguration.
When switching from one non-default configuration to another, it is advised to first reset the device to its default configuration. This action deletes all current NVConfig settings. For the reset command and further details, refer to the mlxconfig – Changing Device Configuration Tool.
The following table summarizes the available port-splitting configurations.
Model | OPNs | Configuration | Configuration Description |
C8180 | 900-9X81E-00NX-ST0 900-9X81E-00EX-ST0 | One InfiniBand port supporting XDR 800Gb/s | |
Two Ethernet ports, each supporting 400GbE | |||
Eight Ethernet ports, each supporting 100GbE | |||
C8240 | 900-9X81Q-00CN-ST0 | Two Ethernet ports, each supporting 400GbE | |
Eight Ethernet ports, each supporting 100GbE |
The C8180 ConnectX-8 SuperNIC, featuring a single-port OSFP module, supports up to 800Gb/s. For the reconfiguration to take effect, please perform power cycle.
C8180 Configuration #1 (Default): One InfiniBand Port, Supporting XDR 800Gb/s
As the default configuration, the device is configured with a single InfiniBand port, providing an InfiniBand XDR rate of 800Gb/s that operates as four times 200G/s . The configuration is illustrated in the diagram below.
C8180 Configuration #2: Two Ethernet Ports, Each Supporting 400GbE
In this configuration, the device can be reconfigured to support two 400GbE Ethernet ports. The configuration is illustrated in the diagram below.
To apply this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set LINK_TYPE_P1=2
mlxconfig -d <device> set NUM_OF_PLANES_P1 = 0
mlxconfig -d <device> set MODULE_SPLIT_M0[0..3]=1 MODULE_SPLIT_M0[4..7]=2 MODULE_SPLIT_M0[8..15]=FF
mlxconfig -d <device> set NUM_OF_PF = 2
C8180 Configuration #3: Eight Ethernet Ports, Each Supporting 100GbE
In this configuration, the device can be configured to support eight 100GbE Ethernet ports. The configuration is illustrated in the diagram below.
To enable this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set LINK_TYPE_P1=2
mlxconfig -d <device> set NUM_OF_PLANES_P1 = 0
mlxconfig -d <device> set MODULE_SPLIT_M0[0]=1 MODULE_SPLIT_M0[1]=2 MODULE_SPLIT_M0[2]=3 MODULE_SPLIT_M0[3]=4 MODULE_SPLIT_M0[4]=5 MODULE_SPLIT_M0[5]=6 MODULE_SPLIT_M0[6]=7 MODULE_SPLIT_M0[7]=8 MODULE_SPLIT_M0[8..15]=FF
mlxconfig -d <device> set NUM_OF_PF = 8
The C8240 ConnectX-8 SuperNIC, featuring dual QSFP modules, supports up to 400Gb/s per port. For the reconfiguration to take effect, please perform power cycle.
C8240 Configuration #1 (Default): Two Ethernet Ports, Each Supporting 400GbE
By default, the device is configured with two Ethernet ports, each supporting a data rate of 400Gb/s. The configuration is illustrated in the diagram below.
C8240 Configuration #2: Eight Ethernet Ports, Each Supporting 100GbE
In this configuration, the device can be configured to support eight 100GbE Ethernet ports, with four ports per module. The configuration is illustrated in the diagram below.
To apply this configuration, use the following mlxconfig commands:
mlxconfig -d <device> set MODULE_SPLIT_M0[0]=1 MODULE_SPLIT_M0[1]=2 MODULE_SPLIT_M0[2]=3 MODULE_SPLIT_M0[3]=4 MODULE_SPLIT_M0[4..15]=FF
mlxconfig -d <device> set MODULE_SPLIT_M1[0]=1 MODULE_SPLIT_M1[1]=2 MODULE_SPLIT_M1[2]=3 MODULE_SPLIT_M1[3]=4 MODULE_SPLIT_M1[4..15]=FF