The mlxlink tool is used to check and debug link status and issues related to them. The tool can be used on different links and cables (passive, active, transceiver and backplane).
In order for mlxlink to function properly, make sure to update the firmware version to the latest version.
mlxlink is intended for advanced users with appropriate technical background.
Do not use mlxlink to disable the port connecting between the host and the unmanaged switch using (“--port_state dn”) flag.
mlxlink errors, warnings and notes are printed on stderr console.
Setting the speeds (50GbE, 100GbE and 200GbE) for the new devices (NVIDIA Connect-X 6 and above, NVIDIA Quantum switches and above) requires specifying the number of lanes for the speed: mlxlink -d <dev> --speeds [50G_2X | 50G_1X | 100G_2X | 100G_4X | 200G_4X].
To run mlxlink:
mlxlink [OPTIONS]
where:
Options:
-h |--help | Display help message. |
-v |--version | Display version info. |
-d |--device <device> | Perform operation for a specified mst device |
-p |--port <port_number> | Port Number |
--port_type <port_type> | Port Type [NETWORK(Default)/PCIE/OOB] |
--depth <depth> | Depth level of the DUT of some hierarchy (valid for PCIe port type only) |
--pcie_index <pcie_index> | PCIe index number (Internal domain index) (valid for PCIe port type only) |
--node <node> | The node within each depth (valid for PCIe port type only) |
--json | Print the output in JSON format |
Queries:
--show_links | Show valid PCIe links (valid for PCIe port type only) |
-m |--show_module | Show Module Info |
-c |--show_counters | Show Physical Counters and BER Info |
-e |--show_eye | Show Eye Opening Info |
--show_fec | Show FEC Capabilities |
--show_serdes_tx | Show Transmitter Info |
--show_tx_group_map <group_num> | Display all label ports mapped to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices). |
--show_device | General Device Info |
--show_ber_monitor | Show BER Monitor Info. Note: The flag is not supported in HCAs. |
--show_external_phy | Show External PHY Info Note: The flag is supported in NVIDIA Spectrum switch systems only. |
Commands:
-a |--port_state <port_state> | Configure Port State [UP(up)/DN(down)/TG(toggle)] |
-s |--speeds <speeds> | Configure Speeds [speed1,speed2,...] |
--link_mode_force | Configure Link Mode Force (Disable AN) |
-l |--loopback <loopback> | Configure Loopback Mode [NO(No Loopback)/PH(phy loopback)/EX(external loopback)] |
-k |--fec <fec_override> | Configure FEC [AU(Auto)/NF(No-FEC)/FC(FireCode FEC)/ RS(RS-FEC)] |
--fec_speed <fec_speed> | Speed to Configure FEC [100G/50G/25G/...] (Default is Active Speed) |
--serdes_tx <params> | Configure Transmitter Parameters [polarity,ob_tap0,...] |
--serdes_tx_lane <transmitter_lane> | Transmitter Lane to Set (Optional - Default All Lanes) |
--database | Save Transmitter Configuration for Current Speed Permanently (Optional) |
--tx_group_map <group_num> | Map ports to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices) |
--ports <ports> | Ports to be mapped [1,2,3,4..] |
--test_mode <prbs_mode> | Physical Test Mode Configuration [EN(enable)/DS(disable)/TU(perform tuning)] |
--rx_prbs <rx_prbs_mode> | RX PRBS Mode [PRBS31(Default)/PRBS7/...] (Optional - Default PRBS31) |
--tx_prbs <tx_prbs_mode> | TX PRBS Mode [PRBS31(Default)/PRBS7/...] (Optional - Default PRBS31) |
--rx_rate <rx_lane_rate> | RX Lane Rate [EDR(Default)/25G/10G/...] (Optional - Default 25G) |
--tx_rate <tx_lane_rate> | TX Lane Rate [EDR(Default)/25G/10G/...] (Optional - Default 25G) |
--invert_tx_polarity | PRBS TX polarity inversion (Optional - Default No Inversion) |
--invert_rx_polarity | PRBS RX polarity inversion (Optional - Default No Inversion) |
--lanes | PRBS lanes to set (one or more lane separated by comma)[0,1,2,...] Optional: Default all lanes |
-b |--ber_collect <csv_file> | Port Extended Information Collection [CSV File] |
--amber_collect <csv_file> | AmBER Port Extended Information Collection For 16nm Products and Later [CSV File] |
--ber_limit <limit_criteria> | BER Limit Criteria [Nominal(Default)/Corner/Drift] (Optional - Default Nominal) |
--iteration <iteration> | Iteration Number of BER Collection |
--pc | Clear Counters |
--set_external_phy | Set External PHY Note: The flag is supported in NVIDIA Spectrum switch systems only. |
--twisted_pair_force_mode <twisted_pair_force_mode> | Twisted Pair Force Mode [MA(Master)/SL(Slave)] |
--cable | Perform operations on the cables |
--dump | Dump cable pages in raw format |
--ddm | Get cable Digital Diagnostic Monitoring information |
--read | Perform read operation from specific page |
--length <length> | Length of data to read in bytes (Optional - Default 1 byte) |
--page <pageNum> | Specific page number to read/write |
--offset <offset> | Specific page offset to read/write |
--write <bytes> | Perform write operation with specific data (list of bytes, separated by ',') |
--margin | Read the SerDes eye margins per lane |
--measure_time <time> | Measure time in seconds for single eye [10, 30, 60, 90, 120, 240, 480, 600 and 900] (Optional - Default 60 for PCIe and 30 for Network ports) |
--eye_select <eye_sel> | Eye selection for PAM4 [UP, MID, DOWN, ALL] (Default ALL) |
--lane <lane_index> | Run eye for specific lane index (Default all lanes) |
--rx_error_injection | Enable the RX link deterioration |
--mixer_offset0 <value> | Fine change to the center of the eye [0x0 to 0x7ff] |
--mixer_offset1 <value> | Coarse change to the center of the eye [0x0 to 0x3ff] |
--show_mixers_offset | Show mixer offset 0 and mixer offset 1 |
--rx_fec_histogram | Provide histogram of FEC errors. The result is divided to bins. Each bin is holding different number of errored bit within FEC protected block |
--show_histogram | Show FEC errors histogram |
--clear_histogram | Clears FEC errors histograms |
--yes | Non-interactive mode, answer yes to all questions |
Examples:
Get info of <device>, <port_number>:
mlxlink -d <device> -p <port_number>
Get info of <device>, <port_number> and BER Counters:
mlxlink -d <device> -p <port_number> -c
Get info of <device>, <port_number> and Transmitter Parameters:
mlxlink -d <device> -p <port_number> --show_serdes_tx
Configure Port State:
mlxlink -d <device> -p <port_number> --port_state UP
Configure Port Speeds:
mlxlink -d <device> -p <port_number> --speeds 25G,50G,100G
Configure FEC:
mlxlink -d <device> -p <port_number> --fec RS
Configure Port for Physical Test Mode:
mlxlink -d <device> -p <port_number> --test_mode EN (--rx_prbs PRBS31 --rx_rate 25G --tx_prbs PRBS7 --tx_rate 10G --invert_rx_polarity --invert_tx_polarity)
Perform PRBS Tuning:
mlxlink -d <device> -p <port_number> --test_mode TU
RX and TX lane rates for new devices includes the PAM4 speeds (50G_1X and 100G_2X)
eg: mlxlink -d <device> --test_mode EN --rx_rate [normal speeds | 50G_1X | 100G_2X] --tx_rate [normal speeds | 50G_1X | 100G_2X]
The PRBS pattern that is configured in PAM4 rates is PRBSQ.
Cable operations:
mlxlink -d <device> --cable [Options]
Dump cable EEPROM pages:
mlxlink -d <device> --cable --dump
Get cable DDM info:
mlxlink -d <device> --cable --ddm
Read from cable:
mlxlink -d <device> --cable --read --page <page number> --offset <bytes offset> --length <number of bytes>
Write to cable:
mlxlink -d <device> --cable --write <bytes separated by comma> --page <page number> --offset <bytes offset>
Configure Transmitter Parameters (on lane, to database):
mlxlink -d <device> -p <port_number> --serdes_tx <polar- ity>,<ob_tap0>,<ob_tap1>,<ob_tap2>,<ob_bias>,<ob_preemp_mode>,<ob_reg>,<ob_leva> (--serdes_tx_lane <lane number>) (--database)
Configure Transmitter Parameters for 16nm devices:
mlxlink -d <device> -p <port_number> --serdes_tx <pre_2_tap>,<pre_tap>,<main_tap>,<post_tap>,<ob_m2lp>,<ob_amp>
Getting PCIe links info:
mlxlink -d /dev/mst/mt41682_pciconf0 --port_type PCIE --show_links
Valid PCIe Links
----------------
: depth, pcie_index, node, port
Link 1 : 3, 0, 0, 60
Link 2 : 3, 0, 1, 61
Link 3 : 3, 0, 2, 62
..
To query information for a specific link, the depth, pcie_index and node for the link must be specified:
mlxlink -d /dev/mst/mt41682_pciconf0 --port_type PCIE --depth 3 --pcie_index 0 --node 1 --show_serdes_tx --show_eye
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 1
Link Speed Active (Enabled) : 8G-Gen 3 (16G-Gen 4)
Link Width Active (Enabled) : 2X (16X)
Device Status : N/A
EYE Opening Info (PCIe)
-----------------------
Physical Grade : 84, 84
Height Eye Opening [mV] : 1194, 1194
Phase Eye Opening [psec] : 84, 84
Serdes Tuning Transmitter Info (PCIe)
-------------------------------------
: Pol,tap0,tap1,tap2,bias,preemp_mode,reg,leva
Lane 0 : 1,0,114,6,15,1,11,9
Lane 1 : 0,0,116,4,15,1,11,9
To print the output in JSON format:
mlxlink -d <device> --show_module --json
To show ports group map (for NVIDIA Quantum and NVIDIA Spectrum-2):
mlxlink –d<device> --show_tx_group_map 0
To assign ports to a specific group on NVIDIA Quantum and NVIDIA Spectrum-2
mlxlink –d <device> --tx_group_map 1 –ports 1,2,3,5,4,8,7,8,9,10,11
To show histogram of FEC errors:
mlxlink -d /dev/mst/mt4125_pciconf0 --rx_fec_histogram --show_histogram
To clear histogram:
mlxlink -d /dev/mst/mt4125_pciconf0 --rx_fec_histogram --clear_histogram
The margin scan tool is used for scanning PCIe or Network ports [EDR\25G or HDR\PAM4 speeds].
If the margin scan fails with this message (Eye scan not completed), perform a reboot and run the scan again.
To enable the margin scan with measure time 10 seconds:
mlxlink –d <device> --port_type PCIE –margin –measure_time 10
To enable the margin scan for Multi-host or Socket Direct systems through:
depth, pcie_index and node:
Mlxlink –d <device> --port_type PCIE –depth
0–pcie_index1–node0–margin –measure_time30The local port (it can be shown by the –show_links command):
Mlxlink –d <device> --port_type PCIE –port
1–margin –measure_time10
Allows modifying the Eye Center capability by changing the mixer_offset0 (fine change) and mixer_offset1 (coarse change) flags for 28nm products to produce RX errors.
Flags Usage
To change the values of mixers:
mlxlink -d /dev/mst/mt4117_pciconf0 --rx_error_injection --mixer_offset0
0x200--mixer_offset10x305WarningModifying mixer_offset0 and mixer_offset1 flags can change the Eye Center and might cause link degradation.
To query the mixers values:
mlxlink -d /dev/mst/mt4117_pciconf0 --rx_erro r_injection --show_mixers_offset
When using mlxlink tool with NIC, notice that the "label_port" flag -p should not be used. To address different ports please use different mst devices.
For example:
To address port 1 in ConnectX-4 use:
mlxlink -d /dev/mst/mt4115_pciconf0
To address port 2 use:
mlxlink -d /dev/mst/mt4115_pciconf0.1
Any mlxlink command for switch should include the "-p" flag to address the specific port in the switch.
When working with the NIC, if an MTUSB is used for communication with the NVIDIA NIC, to address port 2, use mlxlink -d /dev/mst/mt4115_pciconf0 --gvmi_address
<0xAddress>.
Using mlxlink on NVIDIA Quantum HDR based switch systems split ports if the split port number is not provided by the ibdiagnet tool:
Mlxlink -d lid-<LID> -p <formula>
Formula:
In case of 2X port:
1- port_num = round_down[(Iblinkinfo_port_num + 1)*0.5]
2- if (Iblinkinfo_port_num + 1) modulo 2 =1 then append ‘/2’ to port_num
In case of 4X port, use only item #1 above
Example:
43 23[ ] ==( 2X 53.125 Gbps Active/ LinkUp)==> mlxlink -d lid-43 -p 12
43 24[ ] ==( 2X 53.125 Gbps Active/ LinkUp)==> mlxlink -d lid-43 -p 12/2
Link Speed and Width
For PCIe link speed and width use the following flag: --port_type PCIE
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : [Depth, pcie index, node]
Link Speed Active (Enabled) : [Freq – Gen]
Link Width Active (Enabled) : [Width]
Device Status : [Device Status]
PCIe Switch
For NVIDIA® ConnectX®-5 and newer devices, the PCIe interface can be configured for PCIe switch. When the PCIe switch is enabled, the Depth, pcie_index and node parameters are needed to specify the PCIe port where the requested information (such as counters or eye info) is gathered from.
Parameters | Description |
Depth | This defines the number of layers from the Root Complex to the specific port.
|
Pcie_index | This defines the root complex ID or host index.
|
Node | This defines the specific pcie port.
Note: For NVIDIA® BlueField/BlueField-2 SmartNIC mode, the PCIe link information can only be gathered from the external host. The PCIe interface status cannot be retrieved from the Arm side. When retrieving the PCIe link information from the external host, there is no need to specify the depth, pcie_index and node. |
Example: NVIDIA® BlueField JBoF Mode
# mlxlink -d /dev/mst/mt41682_pciconf0 --port_type pcie --depth 3 --pcie_index 0 --node 4 -c
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 4
Link Speed Active (Enabled) : 8G-Gen 3 (16G-Gen 4)
Link Width Active (Enabled) : 2X (2X)
Device Status : N/A
Management PCIe Timers Counters Info
------------------------------------
dl down : 0
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : 0
TX Errors : 0
CRC Error dllp : 0
CRC Error tlp : 0
Link Counters
For PCIe counters information use the following flag: --port_type PCIE –c
Management PCIe Timers Counters Info
------------------------------------
dl down : [link down counter]
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : [Rx Errors]
TX Errors : [Tx Errors]
CRC Error dllp : [CRC Errors dllp]
CRC Error tlp : [CRC Errors tlp]
RX errors: indicate number of transitions to recovery due to Framing errors and CRC (dlp and tlp) errors.
TX errors: indicate number of transitions to recovery due to EIEOS and TS errors.
CRC Error dllp: indicate CRC error in Data Link Layer Packets
CRC Error tlp: indicate CRC error in Transaction Layer Packet
Example:
# mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -c
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
Device Status : Correctable Error detected, Unsupported Request detected.
Management PCIe Timers Counters Info
------------------------------------
dl down : 3
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : 0
TX Errors : 16
CRC Error dllp : 0
CRC Error tlp : 0
Link Eye Opening and Grade
For PCIe link physical grade and eye opening information use the following flag: --port_type PCIE –e
EYE Opening Info (PCIE)
------------------------
Physical Grade : [Grade0, Grade1, Grade2, Grade3, Grade4, Grade5, Grade6, Grade7, Grade8, Grade9, Grade10, Grade11, Grade12, Grade13, Grade14, Grade15]
Height Eye Opening [mV] : [Height0, Height1, Height2, Height3, Height4, Height5, Height6, Height7, Height8, Height9, Height10, Height11, Height12, Height13, Height14, Height15]
Phase Eye Opening [psec] : [Phase0, Phase1, Phase2, Phase3, Phase4, Phase5, Phase6, Phase7, Phase8, Phase9, Phase10, Phase11, Phase12, Phase13, Phase14, Phase15]
Example:
# mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -e
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
Device Status : Correctable Error detected, Unsupported Request detected.
EYE Opening Info (PCIe)
-----------------------
Physical Grade : 57279, 56340, 59340, 61824, 55140, 60501, 61530, 57392, 61573, 58930, 62752, 60421, 57188, 59796, 60066, 60847
Height Eye Opening [mV] : 292, 288, 314, 325, 278, 310, 319, 299, 316, 318, 343, 323, 310, 311, 335, 318
Phase Eye Opening [psec] : 30, 30, 30, 30, 30, 30, 30, 30, 30, 28, 28, 28, 28, 30, 28, 30
Pass / Fail Criteria
SLRED (ConnectX-6/ConnectX-6 Dx/ConnectX-6 Lx)
mlxlink -d [device] --port_type PCIE --margin
Gen3
Gen3 | |
Eye Grade | Figure of Merit (FOM) |
0 < Eye Grade < 700 | FAIL |
700 < Eye Grade < 2300 | gray area |
2300 < Eye Grade | PASS |
Gen4
Gen4 | |
Eye Margin | FOM |
0 < Eye Grade < 150 | FAIL |
150 < Eye Grade < 400 | gray area |
400 < Eye Grade | PASS |