PCIe Bifurcation Configuration Options

The ConnectX-8 SuperNICs provide a range of configuration scenarios to meet the demands of environments and deployments. This section describes the various connectivity options for peripherals on the PCIe, including scenarios where the SuperNIC acts as the PCIe switch with NVMe SSDs as PCIe endpoints. While this list of scenarios is not exhaustive, it highlights the tested and verified options. Customers seeking to support unlisted configurations should contact NVIDIA Support.

The SuperNIC exposes two x16 PCIe interfaces, with internal PCIe switch architecture. The first interface is exposed via the x16 PCIe Gen6 Goldfinger connector and serves as an endpoint to the host server by default. The additional PCIe x16 interface is exposed through the MCIO connector, featuring programmable bifurcation as a downstream port. The following figure demonstrates the SuperNIC block diagram with the PCIe interfaces.

The various configuration scenarios listed in this section include a diagram and instructions on how to bifurcate the PCIe interface using the mlxconfig tool. For more information on the mlxconfig tool, please refer to mlxconfig – Changing Device Configuration Tool.

Before setting the desired configuration, take note of the following warnings:

• Any customer-set configuration overrides the previous configuration values.

• Misconfiguration may harm the system.

• It is recommended to establish out-of-band connectivity to the SuperNIC before setting any of these configurations for the first time. This enables you to reset the NVConfig parameters to their default values in case of misconfiguration.

The following summarizes the available configuration scenarios.

The default configuration is Gen6 x16.

For the recommended configuration: 2x PCIe x16 supporting Gen5

1. Execute the following commands to configure the PCIe interface:

   Copy

   Copied!

               
   
               
   mlxconfig -d /dev/mst/mt4131_pciconf0 set PCI_BUS00_WIDTH=5
   mlxconfig -d /dev/mst/mt4131_pciconf0 set PCI_BUS00_SPEED=4
   mlxconfig -d /dev/mst/mt4131_pciconf0 set PCI_BUS00_HIERARCHY_TYPE=0
   mlxconfig -d /dev/mst/mt4131_pciconf0 set PCI_BUS10_WIDTH=5
   mlxconfig -d /dev/mst/mt4131_pciconf0 set PCI_BUS10_SPEED=4
   mlxconfig -d /dev/mst/mt4131_pciconf0 set PCI_BUS10_HIERARCHY_TYPE=0
           

2. Reboot or power cycle the system.

To change back to the INI default configuration: 1x PCIe x16 supporting Gen6

1. Execute the following commands to configure the PCIe interface:

   Copy

   Copied!

               
   
               
   mlxconfig -d /dev/mst/mt4131_pciconf0 reset
           

2. Reboot or power cycle the system.