Using mlxconfig
In order to access device in BDF format via configuration cycles, use "pciconf-" as prefix to the device.
Example:
# mlxconfig -d 0000:86:00.0 q
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: 0000:86:00.0
Configurations: Next Boot
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
HOST_CHAINING_MODE DISABLED(0)
HOST_CHAINING_CACHE_DISABLE False(0)
HOST_CHAINING_DESCRIPTORS Array[0..7]
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7]
FLEX_PARSER_PROFILE_ENABLE 0
FLEX_IPV4_OVER_VXLAN_PORT 0
ROCE_NEXT_PROTOCOL 254
ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7]
ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7]
PF_BAR2_SIZE 0
NON_PREFETCHABLE_PF_BAR False(0)
VF_VPD_ENABLE False(0)
STRICT_VF_MSIX_NUM False(0)
VF_NODNIC_ENABLE False(0)
NUM_OF_VFS 0
PARTIAL_WRITE_CACHE_MODE DEVICE_DEFAULT(0)
PF_BAR2_ENABLE False(0)
SRIOV_EN False(0)
PF_LOG_BAR_SIZE 5
.............
In order to set IB/ETH parameters through mlxconfig, use the following command line:
# mlxconfig -d <device> set [LINK_TYPE_P1=<link_type>] [LINK_TYPE_P2=<link_type>]
Example: Configuring both ports as InfiniBand:
# mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot New
LINK_TYPE_P1 ETH(2) IB(1)
LINK_TYPE_P2 ETH(2) IB(1)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
In order to set SR-IOV parameters through mlxconfig, use the following command line:
# mlxconfig -d <device> set [SRIOV_EN=<0|1>] [NUM_OF_VFS=<NUM>]
Example: Turning on SR-IOV and enabling 8 Virtual Functions per Physical Function:
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
Device #1:
----------
Device type: ConnectX4
PCI device: /dev/mst/mt4115_pciconf0
Configurations: Next Boot New
SRIOV_EN 0 1
NUM_OF_VFS 0 8
Apply new Configuration? ? (y/n) [n] : y Applying... Done!
-I- Please reboot machine to load new configurations.
For a full description of the preboot configurable parameters refer to Supported Configurations and their Parameters under "Preboot Settings".
Example: Enable boot option ROM on port 1, set boot retries to 3 and set the boot protocol to PXE (on ConnectX-3 Pro cards only).
# mlxconfig -d /dev/mst/mt4103_pciconf0 set BOOT_OPTION_ROM_EN_P1=1 BOOT_RETRY_CNT_P1=3 LEGACY_BOOT_PROTOCOL_P1=1
Device #1:
----------
Device type: ConnectX3Pro
PCI device: /dev/mst/mt4103_pciconf0
Configurations: Next Boot New
BOOT_OPTION_ROM_EN_P1 False(0) True(1)
BOOT_RETRY_CNT_P1 0 3
LEGACY_BOOT_PROTOCOL_P1 2 1
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Example: Configure VLAN ID to 3 on port 2
# mlxconfig -d /dev/mst/mt4119_pciconf0 set BOOT_VLAN=3
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot New
BOOT_VLAN 1 3
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
The break-out cable is a unique NVIDIA capability, where a single physical quad-lane QSFP port is divided into 2 dual-lane ports. It maximizes the flexibility of the end user to use the NVIDIA switch with a combination of dual-lane and quad-lane interfaces according to the specific requirements of its network. All system ports may be split into 2-lane ports. Splitting a port changes the notation of that port from x/y to x/y/z with “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting sub-physical port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 5 into 2 lanes gives the following new ports: 1/5/1 & 1/5/2.
To enable the port split, the following actions are required:
Step 1. Set the Split Mode in a Remotely Managed Switch.
# mlxconfig -d <device>set SPLIT_MODE=1
Example:
# mlxconfig -d /dev/mst/mt54000_pciconf0 set SPLIT_MODE=1
Device #1:
----------
Device type: Quantum
Name: N/A
Description: N/A
Device: /dev/mst/mt54000_pciconf0
Configurations: Next Boot New
SPLIT_MODE NO_SPLIT_SUPPORT(0) SPLIT_2X(1)
To create a query for the Split Mode parameter using mlxconfig:
# mlxconfig -d <device> q SPLIT_MODE
Example:
# mlxconfig -d /dev/mst/mt54000_pciconf0 q SPLIT_MODE
Device #1:
----------
Device type: Quantum
Name: N/A
Description: N/A
Device: /dev/mst/mt54000_pciconf0
Configurations: Next Boot
SPLIT_MODE SPLIT_2X(1)
Step 2. Split a Port in a Remotely Managed Switch.
To split a specific port for one or more ports of (1-64) using mlxconfig:
# mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=
1WarningPlease note that although the command, is “set SPLIT_PORT[33..64]”, it splits a specific port for one or more ports of the higher ports (33-40).
Please note that the first port is set as 1, e.g., [1], [1..64].
How to turn on the split port for the first port only:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[1]=1How to turn on the split port for the first 32 ports (range of ports):
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[1..32]=1
How to turn on the split port for the last 8 ports:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[57..64]=1
How to turn off the split port for port 40:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[40]=0
How to query the split port for the first 32 ports:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001query SPLIT_PORT[1..32]
Step 3. Reboot the switch.
To disable the port split, the following actions are required:
Step 1. Disable the Split Ports in a Remotely Managed Switch:
To unsplit a specific port for one or more ports (1-32) using mlxconfig:
# mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=
0
Step 2. Disable the Split Mode in a Remotely Managed Switch.
# mlxconfig -d <device> set SPLIT_MODE=0
mlxconfig allows applying raw configuration file for a pre-set configuration. Raw configuration files are intended for advanced users. This document does not cover the generation of such files.
Set the raw configuration file:
# mlxconfig -f ./tlv_file.conf -d /dev/mst/mt4115_pciconf1 set_raw
Raw TLV #1 Info:
Length: 0xc
Version: 0
OverrideEn: 1
Type: 0x00000080
Data: 0xa0000000 0xa0020010 0x00000000
Raw TLV #2 Info:
Length: 0x4
Version: 0
OverrideEn: 1
Type: 0x01020012
Data: 0x0000000b
Raw TLV #3 Info:
Length: 0x8
Version: 0
OverrideEn: 1
Type: 0x00000190
Data: 0x00000010 0x00007d00
Raw TLV #4 Info:
Length: 0x4
Version: 0
OverrideEn: 1
Type: 0x00000082
Data: 0x0000000c
Operation intended for advanced users.
Are you sure you want to apply raw TLV file? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Never apply files from an unreliable source.