Using mlxconfig

In order to access device in BDF format via configuration cycles, use "pciconf-" as prefix to the device.

Example:

Copy
Copied!
            

# mlxconfig -d 0000:86:00.0 q   Device #1: ---------- Device type: ConnectX5 Name: MCX556A-ECA_Ax Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6 Device: 0000:86:00.0   Configurations: Next Boot MEMIC_BAR_SIZE 0 MEMIC_SIZE_LIMIT _256KB(1) HOST_CHAINING_MODE DISABLED(0) HOST_CHAINING_CACHE_DISABLE False(0) HOST_CHAINING_DESCRIPTORS Array[0..7] HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7] FLEX_PARSER_PROFILE_ENABLE 0 FLEX_IPV4_OVER_VXLAN_PORT 0 ROCE_NEXT_PROTOCOL 254 ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7] ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7] PF_BAR2_SIZE 0 NON_PREFETCHABLE_PF_BAR False(0) VF_VPD_ENABLE False(0) STRICT_VF_MSIX_NUM False(0) VF_NODNIC_ENABLE False(0) NUM_OF_VFS 0 PARTIAL_WRITE_CACHE_MODE DEVICE_DEFAULT(0) PF_BAR2_ENABLE False(0) SRIOV_EN False(0) PF_LOG_BAR_SIZE 5 .............

In order to set IB/ETH parameters through mlxconfig, use the following command line:

Copy
Copied!
            

# mlxconfig -d <device> set [LINK_TYPE_P1=<link_type>] [LINK_TYPE_P2=<link_type>]

Example: Configuring both ports as InfiniBand:

Copy
Copied!
            

# mlxconfig -d /dev/mst/mt4119_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1   Device #1: ---------- Device type: ConnectX5 Name: MCX556A-ECA_Ax Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6 Device: /dev/mst/mt4119_pciconf0   Configurations: Next Boot New LINK_TYPE_P1 ETH(2) IB(1) LINK_TYPE_P2 ETH(2) IB(1)   Apply new Configuration? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

In order to set SR-IOV parameters through mlxconfig, use the following command line:

Copy
Copied!
            

# mlxconfig -d <device> set [SRIOV_EN=<0|1>] [NUM_OF_VFS=<NUM>]

Example: Turning on SR-IOV and enabling 8 Virtual Functions per Physical Function:

Copy
Copied!
            

# mlxconfig -d /dev/mst/mt4115_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8   Device #1: ---------- Device type: ConnectX4 PCI device: /dev/mst/mt4115_pciconf0 Configurations: Next Boot New SRIOV_EN 0 1 NUM_OF_VFS 0 8   Apply new Configuration? ? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

For a full description of the preboot configurable parameters refer to Supported Configurations and their Parameters under "Preboot Settings".

Example: Enable boot option ROM on port 1, set boot retries to 3 and set the boot protocol to PXE (on ConnectX-3 Pro cards only).

Copy
Copied!
            

# mlxconfig -d /dev/mst/mt4103_pciconf0 set BOOT_OPTION_ROM_EN_P1=1 BOOT_RETRY_CNT_P1=3 LEGACY_BOOT_PROTOCOL_P1=1   Device #1: ---------- Device type: ConnectX3Pro PCI device: /dev/mst/mt4103_pciconf0 Configurations: Next Boot New BOOT_OPTION_ROM_EN_P1 False(0) True(1) BOOT_RETRY_CNT_P1 0 3 LEGACY_BOOT_PROTOCOL_P1 2 1   Apply new Configuration? ? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

Example: Configure VLAN ID to 3 on port 2

Copy
Copied!
            

# mlxconfig -d /dev/mst/mt4119_pciconf0 set BOOT_VLAN=3   Device #1: ---------- Device type: ConnectX5 Name: MCX556A-ECA_Ax Description: ConnectX-5 VPI adapter card; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6 Device: /dev/mst/mt4119_pciconf0   Configurations: Next Boot New BOOT_VLAN 1 3   Apply new Configuration? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

The break-out cable is a unique NVIDIA capability, where a single physical quad-lane QSFP port is divided into 2 dual-lane ports. It maximizes the flexibility of the end user to use the NVIDIA switch with a combination of dual-lane and quad-lane interfaces according to the specific requirements of its network. All system ports may be split into 2-lane ports. Splitting a port changes the notation of that port from x/y to x/y/z with “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting sub-physical port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 5 into 2 lanes gives the following new ports: 1/5/1 & 1/5/2.

image2019-3-19_16-14-40.png

To enable the port split, the following actions are required:

Step 1. Set the Split Mode in a Remotely Managed Switch.

Copy
Copied!
            

# mlxconfig -d <device>set SPLIT_MODE=1

Example:

Copy
Copied!
            

# mlxconfig -d /dev/mst/mt54000_pciconf0 set SPLIT_MODE=1   Device #1: ---------- Device type: Quantum Name: N/A Description: N/A Device: /dev/mst/mt54000_pciconf0 Configurations: Next Boot New SPLIT_MODE NO_SPLIT_SUPPORT(0) SPLIT_2X(1)

To create a query for the Split Mode parameter using mlxconfig:

Copy
Copied!
            

# mlxconfig -d <device> q SPLIT_MODE

Example:

Copy
Copied!
            

# mlxconfig -d /dev/mst/mt54000_pciconf0 q SPLIT_MODE   Device #1: ---------- Device type: Quantum Name: N/A Description: N/A Device: /dev/mst/mt54000_pciconf0 Configurations: Next Boot SPLIT_MODE SPLIT_2X(1)

Step 2. Split a Port in a Remotely Managed Switch.

  • To split a specific port for one or more ports of (1-64) using mlxconfig:

    Copy
    Copied!
                

    # mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=1

    Warning

    Please note that although the command, is “set SPLIT_PORT[33..64]”, it splits a specific port for one or more ports of the higher ports (33-40).

    Please note that the first port is set as 1, e.g., [1], [1..64].

  • How to turn on the split port for the first port only:

    Copy
    Copied!
                

    mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-0x0001 set SPLIT_PORT[1]=1

  • How to turn on the split port for the first 32 ports (range of ports):

    Copy
    Copied!
                

    mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-0x0001 set SPLIT_PORT[1..32]=1

  • How to turn on the split port for the last 8 ports:

    Copy
    Copied!
                

    mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-0x0001 set SPLIT_PORT[57..64]=1

  • How to turn off the split port for port 40:

    Copy
    Copied!
                

    mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-0x0001 set SPLIT_PORT[40]=0

  • How to query the split port for the first 32 ports:

    Copy
    Copied!
                

    mlxconfig -d /dev/mst/SW_MT54000_EVB-SX-1_L00_lid-0x0001 query SPLIT_PORT[1..32]

Step 3. Reboot the switch.

To disable the port split, the following actions are required:

Step 1. Disable the Split Ports in a Remotely Managed Switch:

  • To unsplit a specific port for one or more ports (1-32) using mlxconfig:

    Copy
    Copied!
                

    # mlxconfig -d <device> set SPLIT_PORT[<port_num>/<port_range>]=0

Step 2. Disable the Split Mode in a Remotely Managed Switch.

Copy
Copied!
            

# mlxconfig -d <device> set SPLIT_MODE=0

mlxconfig allows applying raw configuration file for a pre-set configuration. Raw configuration files are intended for advanced users. This document does not cover the generation of such files.

Set the raw configuration file:

Copy
Copied!
            

# mlxconfig -f ./tlv_file.conf -d /dev/mst/mt4115_pciconf1 set_raw   Raw TLV #1 Info: Length: 0xc Version: 0 OverrideEn: 1 Type: 0x00000080 Data: 0xa0000000 0xa0020010 0x00000000   Raw TLV #2 Info: Length: 0x4 Version: 0 OverrideEn: 1 Type: 0x01020012 Data: 0x0000000b   Raw TLV #3 Info: Length: 0x8 Version: 0 OverrideEn: 1 Type: 0x00000190 Data: 0x00000010 0x00007d00   Raw TLV #4 Info: Length: 0x4 Version: 0 OverrideEn: 1 Type: 0x00000082 Data: 0x0000000c Operation intended for advanced users.   Are you sure you want to apply raw TLV file? ? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations.

Warning

Never apply files from an unreliable source.

© Copyright 2023, NVIDIA. Last updated on Oct 11, 2023.