Using mlxconfig
In order to access device in BDF format via configuration cycles, use "pciconf-" as prefix to the device.
Example:
# mlxconfig -d 0000:86:00.0 q
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: 0000:86:00.0
Configurations: Next Boot
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
HOST_CHAINING_MODE DISABLED(0)
HOST_CHAINING_CACHE_DISABLE False(0)
HOST_CHAINING_DESCRIPTORS Array[0..7]
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7]
FLEX_PARSER_PROFILE_ENABLE 0
FLEX_IPV4_OVER_VXLAN_PORT 0
ROCE_NEXT_PROTOCOL 254
ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7]
ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7]
PF_BAR2_SIZE 0
NON_PREFETCHABLE_PF_BAR False(0)
VF_VPD_ENABLE False(0)
STRICT_VF_MSIX_NUM False(0)
VF_NODNIC_ENABLE False(0)
NUM_OF_VFS 0
PARTIAL_WRITE_CACHE_MODE DEVICE_DEFAULT(0)
PF_BAR2_ENABLE False(0)
SRIOV_EN False(0)
PF_LOG_BAR_SIZE 5
.............
In order to set IB/ETH parameters through mlxconfig, use the following command line:
# mlxconfig -d <device> set [LINK_TYPE_P1=<link_type>] [LINK_TYPE_P2=<link_type>]
Example: Configuring both ports as InfiniBand:
# mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot New
LINK_TYPE_P1 ETH(2) IB(1)
LINK_TYPE_P2 ETH(2) IB(1)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Setting Auto Negotiation
Auto negotiation modes is displayed at mlxconfig available configuration menu (Using <i> command).
In order to set non-volatile auto negotiation mode through mlxconfig, use the following command line:
# mlxconfig -d <device> set [PHY_AUTO_NEG_P1=<auto_negotiation_mode>] [PHY_AUTO_NEG_P2=<auto_negotiation_mode>]
Example: Configuring both ports with auto negotiation enabled:
# mlxconfig -d /dev/mst/mt4129_pciconf0 set PHY_AUTO_NEG_P1=1 PHY_AUTO_NEG_P2=1
Device #1:
----------
Device type: ConnectX7
Name: CX713106AE-HEA_QP1_Ax
Description: NVIDIA ConnectX-7 HHHL adapter Card; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe 5.0 x16 with x16 PCIe extension option; Crypto Enabled; Secure Boot Capable; IPN for internal use
Device: /dev/mst/mt4129_pciconf0
Configurations: Next Boot New
PHY_AUTO_NEG_P1 DEVICE_DEFAULT(0) AUTO_NEG_ENABLED(1)
PHY_AUTO_NEG_P2 DEVICE_DEFAULT(0) AUTO_NEG_ENABLED(1)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Setting Speed Rate Mask
Speed rate mask enables masking device's available speeds in all link operational modes.
Speed masking requires defining the speed rate mask and enabling speed override configuration.
In order to set non-volatile speed rate mask through mlxconfig, verify first that override mask is enabled using the following command:
# mlxconfig -d <device> set [PHY_RATE_MASK_OVERRIDE_P1=<True|False>] [PHY_RATE_MASK_OVERRIDE_P2=<True|False>]
Example: Enabling both ports' speed mask override:
# mlxconfig -d /dev/mst/mt4129_pciconf0 set PHY_RATE_MASK_OVERRIDE_P1=1 PHY_RATE_MASK_OVERRIDE_P2=1
Device #1:
----------
Device type: ConnectX7
Name: CX713106AE-HEA_QP1_Ax
Description: NVIDIA ConnectX-7 HHHL adapter Card; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe 5.0 x16 with x16 PCIe extension option; Crypto Enabled; Secure Boot Capable; IPN for internal use
Device: /dev/mst/mt4129_pciconf0
Configurations: Next Boot New
PHY_RATE_MASK_OVERRIDE_P1 False(0) True(1)
PHY_RATE_MASK_OVERRIDE_P2 False(0) True(1)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Configure the wanted speed mask using the following command:
# mlxconfig -d <device> set [PHY_RATE_MASK_P1=<rate_mask>] [PHY_RATE_MASK_P2=<rate_mask>]
The rate mask mapping is displayed at mlxconfig available configuration menu (Using <i> command).
Example: Enabling both ports to support CAUI-4 / 100GBASE-CR4 only:
# mlxconfig -d /dev/mst/mt4129_pciconf0 set PHY_RATE_MASK_P1=0x200 PHY_RATE_MASK_P2=0x200
Device #1:
----------
Device type: ConnectX7
Name: CX713106AE-HEA_QP1_Ax
Description: NVIDIA ConnectX-7 HHHL adapter Card; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe 5.0 x16 with x16 PCIe extension option; Crypto Enabled; Secure Boot Capable; IPN for internal use
Device: /dev/mst/mt4129_pciconf0
Configurations: Next Boot New
PHY_RATE_MASK_P1 0 0x200(512)
PHY_RATE_MASK_P2 0 0x200(512)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Setting FEC Mode
Customizes FEC configuration of the port.
Detailed information on FEC modes is displayed at mlxconfig available configuration menu (Using command).In order to set non-volatile FEC mode through mlxconfig, use the following command line:
# mlxconfig -d <device> set [PHY_FEC_OVERRIDE_P1=<fec_mode>] [PHY_FEC_OVERRIDE_P2=<fec_mode>]
Example: Setting FEC mode disabled DME, RS FEC for 25G/50G/100G for both port:
# mlxconfig -d /dev/mst/mt4129_pciconf0 set PHY_FEC_OVERRIDE_P1=0x2 PHY_FEC_OVERRIDE_P2=0x2
Device #1:
----------
Device type: ConnectX7
Name: CX713106AE-HEA_QP1_Ax
Description: NVIDIA ConnectX-7 HHHL adapter Card; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe 5.0 x16 with x16 PCIe extension option; Crypto Enabled; Secure Boot Capable; IPN for internal use
Device: /dev/mst/mt4129_pciconf0
Configurations: Next Boot New
PHY_FEC_OVERRIDE_P1 DEVICE_DEFAULT(0) MODE_2(2)
PHY_FEC_OVERRIDE_P2 DEVICE_DEFAULT(0) MODE_2(2)
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
In order to set SR-IOV parameters through mlxconfig, use the following command line:
# mlxconfig -d <device> set [SRIOV_EN=<0|1>] [NUM_OF_VFS=<NUM>]
Example: Turning on SR-IOV and enabling 8 Virtual Functions per Physical Function:
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
Device #1:
----------
Device type: ConnectX4
PCI device: /dev/mst/mt4115_pciconf0
Configurations: Next Boot New
SRIOV_EN 0 1
NUM_OF_VFS 0 8
Apply new Configuration? ? (y/n) [n] : y Applying... Done!
-I- Please reboot machine to load new configurations.
For a full description of the preboot configurable parameters refer to Supported Configurations and their Parameters under "Preboot Settings".
Example: Enable boot option ROM on port 1, set boot retries to 3 and set the boot protocol to PXE (on ConnectX-3 Pro cards only).
# mlxconfig -d /dev/mst/mt4103_pciconf0 set BOOT_OPTION_ROM_EN_P1=1 BOOT_RETRY_CNT_P1=3 LEGACY_BOOT_PROTOCOL_P1=1
Device #1:
----------
Device type: ConnectX3Pro
PCI device: /dev/mst/mt4103_pciconf0
Configurations: Next Boot New
BOOT_OPTION_ROM_EN_P1 False(0) True(1)
BOOT_RETRY_CNT_P1 0 3
LEGACY_BOOT_PROTOCOL_P1 2 1
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Example: Configure VLAN ID to 3 on port 2
# mlxconfig -d /dev/mst/mt4119_pciconf0 set BOOT_VLAN=3
Device #1:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot New
BOOT_VLAN 1 3
Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
The break-out cable is a unique NVIDIA capability, where a single physical quad-lane HDR or NDR port is divided into 2 dual-lane ports. It maximizes the flexibility of the end user to use the NVIDIA switch with a combination of dual-lane and quad-lane interfaces according to the specific requirements of its network. All system ports may be split into 2-lane ports. Splitting a port changes the notation of that port -
In HDR ports:
From x/y to x/y/z, with “x/y” indicating the previous notation of the port prior to the split, and “z” indicating the number of the resulting sub-physical port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 5 into 2 lanes gives the following new ports: 1/5/1 & 1/5/2.
In NDR ports:
From x/y/z to x/y/z/i, with “x/y/z” indicating the previous notation of the port prior to the split, and “i” indicating the number of the resulting single-lane port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 1/5/1 into 2 lanes results in ports 1/5/1/1 and 1/5/1/2.
To enable the port split, the following actions are required:
Step 1. Set the Split Mode in a Remotely Managed Switch.
# mlxconfig -d <device>set SPLIT_MODE=1
Example:
# mlxconfig -d /dev/mst/mt54000_pciconf0 set SPLIT_MODE=1
Device #1:
----------
Device type: Quantum
Name: N/A
Description: N/A
Device: /dev/mst/mt54000_pciconf0
Configurations: Next Boot New
SPLIT_MODE NO_SPLIT_SUPPORT(0) SPLIT_2X(1)
To create a query for the Split Mode parameter using mlxconfig:
# mlxconfig -d <device> q SPLIT_MODE
Example:
# mlxconfig -d /dev/mst/mt54000_pciconf0 q SPLIT_MODE
Device #1:
----------
Device type: Quantum
Name: N/A
Description: N/A
Device: /dev/mst/mt54000_pciconf0
Configurations: Next Boot
SPLIT_MODE SPLIT_2X(1)
Step 2. Split a Port in a Remotely Managed Switch.
To split a specific port for one or more ports of (1-64) using mlxconfig:
# mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=
1NotePlease note that although the command, is “set SPLIT_PORT[33..64]”, it splits a specific port for one or more ports of the higher ports (33-40).
Please note that the first port is set as 1, e.g., [1], [1..64].
How to turn on the split port for the first port only:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[1]=1How to turn on the split port for the first 32 ports (range of ports):
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[1..32]=1
How to turn on the split port for the last 8 ports:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[57..64]=1
How to turn off the split port for port 40:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001set SPLIT_PORT[40]=0
How to query the split port for the first 32 ports:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001query SPLIT_PORT[1..32]
Step 3. Reboot the switch, or run:
flint -d <device> swreset
To disable the port split, the following actions are required:
Step 1. Disable the Split Ports in a Remotely Managed Switch:
To unsplit a specific port for one or more ports (1-32) using mlxconfig:
# mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=
0
Step 2. Disable the Split Mode in a Remotely Managed Switch.
# mlxconfig -d <device> set SPLIT_MODE=0
mlxconfig allows applying raw configuration file for a pre-set configuration. Raw configuration files are intended for advanced users. This document does not cover the generation of such files.
Set the raw configuration file:
# mlxconfig -f ./tlv_file.conf -d /dev/mst/mt4115_pciconf1 set_raw
Raw TLV #1 Info:
Length: 0xc
Version: 0
OverrideEn: 1
Type: 0x00000080
Data: 0xa0000000 0xa0020010 0x00000000
Raw TLV #2 Info:
Length: 0x4
Version: 0
OverrideEn: 1
Type: 0x01020012
Data: 0x0000000b
Raw TLV #3 Info:
Length: 0x8
Version: 0
OverrideEn: 1
Type: 0x00000190
Data: 0x00000010 0x00007d00
Raw TLV #4 Info:
Length: 0x4
Version: 0
OverrideEn: 1
Type: 0x00000082
Data: 0x0000000c
Operation intended for advanced users.
Are you sure you want to apply raw TLV file? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
Never apply files from an unreliable source.