Using mlxconfig
In order to access device in BDF format via configuration cycles, use "pciconf-" as prefix to the device.
Example:
# mlxconfig -d 0000
:86
:00.0
q
Device #1
:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5
VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0
x16; tall bracket; ROHS R6
Device: 0000
:86
:00.0
Configurations: Next Boot
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1
)
HOST_CHAINING_MODE DISABLED(0
)
HOST_CHAINING_CACHE_DISABLE False(0
)
HOST_CHAINING_DESCRIPTORS Array[0
..7
]
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0
..7
]
FLEX_PARSER_PROFILE_ENABLE 0
FLEX_IPV4_OVER_VXLAN_PORT 0
ROCE_NEXT_PROTOCOL 254
ESWITCH_HAIRPIN_DESCRIPTORS Array[0
..7
]
ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0
..7
]
PF_BAR2_SIZE 0
NON_PREFETCHABLE_PF_BAR False(0
)
VF_VPD_ENABLE False(0
)
STRICT_VF_MSIX_NUM False(0
)
VF_NODNIC_ENABLE False(0
)
NUM_OF_VFS 0
PARTIAL_WRITE_CACHE_MODE DEVICE_DEFAULT(0
)
PF_BAR2_ENABLE False(0
)
SRIOV_EN False(0
)
PF_LOG_BAR_SIZE 5
.............
In order to set IB/ETH parameters through mlxconfig, use the following command line:
# mlxconfig -d <device> set [LINK_TYPE_P1=<link_type>] [LINK_TYPE_P2=<link_type>]
Example: Configuring both ports as InfiniBand:
# mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=1
LINK_TYPE_P2=1
Device #1
:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5
VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0
x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot New
LINK_TYPE_P1 ETH(2
) IB(1
)
LINK_TYPE_P2 ETH(2
) IB(1
)
Apply new
Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new
configurations.
In order to set SR-IOV parameters through mlxconfig, use the following command line:
# mlxconfig -d <device> set [SRIOV_EN=<0
|1
>] [NUM_OF_VFS=<NUM>]
Example: Turning on SR-IOV and enabling 8 Virtual Functions per Physical Function:
# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1
NUM_OF_VFS=8
Device #1
:
----------
Device type: ConnectX4
PCI device: /dev/mst/mt4115_pciconf0
Configurations: Next Boot New
SRIOV_EN 0
1
NUM_OF_VFS 0
8
Apply new
Configuration? ? (y/n) [n] : y Applying... Done!
-I- Please reboot machine to load new
configurations.
For a full description of the preboot configurable parameters refer to Supported Configurations and their Parameters under "Preboot Settings".
Example: Enable boot option ROM on port 1, set boot retries to 3 and set the boot protocol to PXE (on ConnectX-3 Pro cards only).
# mlxconfig -d /dev/mst/mt4103_pciconf0 set BOOT_OPTION_ROM_EN_P1=1
BOOT_RETRY_CNT_P1=3
LEGACY_BOOT_PROTOCOL_P1=1
Device #1
:
----------
Device type: ConnectX3Pro
PCI device: /dev/mst/mt4103_pciconf0
Configurations: Next Boot New
BOOT_OPTION_ROM_EN_P1 False(0
) True(1
)
BOOT_RETRY_CNT_P1 0
3
LEGACY_BOOT_PROTOCOL_P1 2
1
Apply new
Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new
configurations.
Example: Configure VLAN ID to 3 on port 2
# mlxconfig -d /dev/mst/mt4119_pciconf0 set BOOT_VLAN=3
Device #1
:
----------
Device type: ConnectX5
Name: MCX556A-ECA_Ax
Description: ConnectX-5
VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0
x16; tall bracket; ROHS R6
Device: /dev/mst/mt4119_pciconf0
Configurations: Next Boot New
BOOT_VLAN 1
3
Apply new
Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new
configurations.
The break-out cable is a unique NVIDIA capability, where a single physical quad-lane HDR or NDR port is divided into 2 dual-lane ports. It maximizes the flexibility of the end user to use the NVIDIA switch with a combination of dual-lane and quad-lane interfaces according to the specific requirements of its network. All system ports may be split into 2-lane ports. Splitting a port changes the notation of that port -
In HDR ports:
From x/y to x/y/z, with “x/y” indicating the previous notation of the port prior to the split, and “z” indicating the number of the resulting sub-physical port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 5 into 2 lanes gives the following new ports: 1/5/1 & 1/5/2.In NDR ports:
From x/y/z to x/y/z/i, with “x/y/z” indicating the previous notation of the port prior to the split, and “i” indicating the number of the resulting single-lane port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 1/5/1 into 2 lanes results in ports 1/5/1/1 and 1/5/1/2.
To enable the port split, the following actions are required:
Step 1. Set the Split Mode in a Remotely Managed Switch.
# mlxconfig -d <device>set SPLIT_MODE=1
Example:
# mlxconfig -d /dev/mst/mt54000_pciconf0 set SPLIT_MODE=1
Device #1
:
----------
Device type: Quantum
Name: N/A
Description: N/A
Device: /dev/mst/mt54000_pciconf0
Configurations: Next Boot New
SPLIT_MODE NO_SPLIT_SUPPORT(0
) SPLIT_2X(1
)
To create a query for the Split Mode parameter using mlxconfig:
# mlxconfig -d <device> q SPLIT_MODE
Example:
# mlxconfig -d /dev/mst/mt54000_pciconf0 q SPLIT_MODE
Device #1
:
----------
Device type: Quantum
Name: N/A
Description: N/A
Device: /dev/mst/mt54000_pciconf0
Configurations: Next Boot
SPLIT_MODE SPLIT_2X(1
)
Step 2. Split a Port in a Remotely Managed Switch.
To split a specific port for one or more ports of (1-64) using mlxconfig:
# mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=
1
WarningPlease note that although the command, is “set SPLIT_PORT[33..64]”, it splits a specific port for one or more ports of the higher ports (33-40).
Please note that the first port is set as 1, e.g., [1], [1..64].
How to turn on the split port for the first port only:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001
set SPLIT_PORT[1
]=1
How to turn on the split port for the first 32 ports (range of ports):
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001
set SPLIT_PORT[1
..32
]=1
How to turn on the split port for the last 8 ports:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001
set SPLIT_PORT[57
..64
]=1
How to turn off the split port for port 40:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001
set SPLIT_PORT[40
]=0
How to query the split port for the first 32 ports:
mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-
0x0001
query SPLIT_PORT[1
..32
]
Step 3. Reboot the switch, or run:
flint -d <device> swreset
To disable the port split, the following actions are required:
Step 1. Disable the Split Ports in a Remotely Managed Switch:
To unsplit a specific port for one or more ports (1-32) using mlxconfig:
# mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=
0
Step 2. Disable the Split Mode in a Remotely Managed Switch.
# mlxconfig -d <device> set SPLIT_MODE=0
mlxconfig allows applying raw configuration file for a pre-set configuration. Raw configuration files are intended for advanced users. This document does not cover the generation of such files.
Set the raw configuration file:
# mlxconfig -f ./tlv_file.conf -d /dev/mst/mt4115_pciconf1 set_raw
Raw TLV #1
Info:
Length: 0xc
Version: 0
OverrideEn: 1
Type: 0x00000080
Data: 0xa0000000
0xa0020010
0x00000000
Raw TLV #2
Info:
Length: 0x4
Version: 0
OverrideEn: 1
Type: 0x01020012
Data: 0x0000000b
Raw TLV #3
Info:
Length: 0x8
Version: 0
OverrideEn: 1
Type: 0x00000190
Data: 0x00000010
0x00007d00
Raw TLV #4
Info:
Length: 0x4
Version: 0
OverrideEn: 1
Type: 0x00000082
Data: 0x0000000c
Operation intended for
advanced users.
Are you sure you want to apply raw TLV file? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new
configurations.
Never apply files from an unreliable source.