mlxlink Utility

NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

The mlxlink tool is used to check and debug link status and related issues. The tool can be used on different links and cables (passive, active, transceiver and backplane).

Warning
  • In order for mlxlink to function properly, make sure to update the firmware version to the latest version.

  • mlxlink is intended for advanced users with appropriate technical background.

  • Do not use mlxlink to disable the port connecting between the host and the unmanaged switch using (“--port_state dn”) flag.

  • mlxlink errors, warnings and notes are printed on stderr console.

  • Setting the speeds (50GbE, 100GbE and 200GbE) for the new devices (NVIDIA ConnectX-6 and above, NVIDIA Quantum switches and above) requires specifying the number of lanes for the speed: mlxlink -d <dev> --speeds [50G_2X | 50G_1X | 100G_2X | 100G_4X | 200G_4X ].

To run mlxlink:

Copy
Copied!
            

mlxlink [OPTIONS]

where:

Options:

-h |--help

Display help message.

-v |--version

Display version info.

-d |--device <device>

Perform operation for a specified mst device

-p |--port <port_number>

Port Number

--port_type <port_type>

Port Type [NETWORK(Default)/PCIE/OOB]

--depth <depth>

Depth level of the DUT of some hierarchy (valid for PCIe port type only)

--pcie_index <pcie_index>

PCIe index number (Internal domain index) (valid for PCIe port type only)

--node <node>

The node within each depth (valid for PCIe port type only)

--json

Print the output in JSON format

Queries:

--show_links

Show valid PCIe links (valid for PCIe port type only)

-m |--show_module

Show Module Info

-c |--show_counters

Show Physical Counters and BER Info

-e |--show_eye

Show Eye Opening Info

--show_fec

Show FEC Capabilities

--show_serdes_tx

Show Transmitter Info

--show_tx_group_map <group_num>

Display all label ports mapped to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices).

--show_device

General Device Info

--show_ber_monitor

Show BER Monitor Info.

Note: The flag is not supported in HCAs.

--show_external_phy

Show External PHY Info

Note: The flag is supported in NVIDIA Spectrum switch systems only.

Commands:

-a |--port_state <port_state>

Configure Port State [UP(up)/DN(down)/TG(toggle)]

-s |--speeds <speeds>

Configure Speeds [speed1,speed2,...]

--link_mode_force

Configure Link Mode Force (Disable AN)

-l |--loopback <loopback>

Configure Loopback Mode [NO(no loopback)/RM(phy remote Rx-to-Tx loopback)/PH(internal phy Tx-to-Rx loopback)/EX(external loopback connector needed)]

-k |--fec <fec_override>

Configure FEC [AU(Auto)/NF(No-FEC)/FC(FireCode FEC)/ RS(RS-FEC)]/LL(LL-RS-FEC)/DF-RS(Interleaved_RS-FEC)/DF-LL(Interleaved_LL_RS-FEC)]

--fec_speed <fec_speed>

Speed to Configure FEC [100G/50G/25G/...] (Default is Active Speed)

--serdes_tx <params>

Configure Transmitter Parameters [polarity,ob_tap0,...]

--serdes_tx_lane <transmitter_lane>

Transmitter Lane to Set (Optional - Default All Lanes)

--database

Save Transmitter Configuration for Current Speed Permanently (Optional)

--tx_params_override

Set the parameters according to Data Base only, otherwise it will be set according to the best possible configuration chosen by the system (e.g. KR-startup) (Optional)

--tx_group_map <group_num>

Map ports to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices)

--ports <ports>

Ports to be mapped [1,2,3,4..]

--test_mode <prbs_mode>

Physical Test Mode Configuration [EN(enable)/DS(disable)/TU(perform tuning)]

--rx_prbs <rx_prbs_mode>

RX PRBS Mode [PRBS31(Default)/PRBS7/...] (Optional - Default PRBS31)

--tx_prbs <tx_prbs_mode>

TX PRBS Mode [PRBS31(Default)/PRBS7/...] (Optional - Default PRBS31)

--rx_rate <rx_lane_rate>

RX Lane Rate [EDR(Default)/25G/10G/...] (Optional - Default 25G)

--tx_rate <tx_lane_rate>

TX Lane Rate [EDR(Default)/25G/10G/...] (Optional - Default 25G)

--invert_tx_polarity

PRBS TX polarity inversion (Optional - Default No Inversion)

--invert_rx_polarity

PRBS RX polarity inversion (Optional - Default No Inversion)

--lanes

PRBS lanes to set (one or more lane separated by comma)[0,1,2,...]

Optional: Default all lanes

-b |--ber_collect <csv_file>

Port Extended Information Collection [CSV File]

--amber_collect <csv_file>

AmBER Port Extended Information Collection For 16nm Products and Later [CSV File]

--ber_limit <limit_criteria>

BER Limit Criteria [Nominal(Default)/Corner/Drift] (Optional - Default Nominal)

--iteration <iteration>

Iteration Number of BER Collection

--pc

Clear Counters

--set_external_phy

Set External PHY

Note: The flag is supported in NVIDIA Spectrum switch systems only.

--twisted_pair_force_mode <twisted_pair_force_mode>

Twisted Pair Force Mode [MA(Master)/SL(Slave)]

--cable

Perform operations on the cables

--dump

Dump cable pages in raw format

--ddm

Get cable Digital Diagnostic Monitoring information

--read

Perform read operation from specific page

--length <length>

Length of data to read in bytes (Optional - Default 1 byte)

--page <pageNum>

Specific page number to read/write

--offset <offset>

Specific page offset to read/write

--write <bytes>

Perform write operation with specific data (list of bytes, separated by ',')

--prbs_select <side>

Module PRBS test mode side selector [MEDIA, HOST]

--prbs_mode <cmd>

Perform PRBS test mode on the Module [EN(Enable),DS(Disable)]

--generator_pattern <pattern>

Set PRBS generator pattern [PRBS31(default),PRBS23,PRBS7,PRBS11,PRBS9,PRBS13,SSPR,SSPRQ]

--swap_generator

Enable PAM4 MSB <-> LSB generator swapping (Optional)

--invert_generator

Enable PRBS generator inversion (Optional)

--generator_lanes <lanes>

PRBS generator lanes to set (one or more lane separated by comma)[0,1,2,3,4,5,6,7] (Optional - Default all lanes)

--checker_pattern <pattern>

Set PRBS checker pattern [PRBS31(default),PRBS23,PRBS7,PRBS11,PRBS9,PRBS13,SSPR,SSPRQ]

--swap_checker

Enable PAM4 MSB <-> LSB checker swapping (Optional)

--invert_checker

Enable PRBS checker inversion (Optional)

--checker_lanes <lanes>

PRBS checker lanes to set (one or more lane separated by comma)[0,1,2,3,4,5,6,7] (Optional - Default all lanes)

--lane_rate <rate>

Set PRBS checker and generator lane rate [HDR(50G)(default),1.25G,SDR(2.5G),10.3125G,FDR(14G),EDR(25G),NDR(100G)]

--show_diagnostic_info

Show PRBS diagnostic counters information

--clear_diagnostic_info

Clear PRBS diagnostic counters

--control_parameters

Show Module Control Parameters

--tx_equalization <value>

Set Module Tx Input Equalization in dB [NE(No Equalization),1,2,3,4,5,6,7,8, 9,10,11,12]

--rx_emphasis <value>

Set Module RX Output Emphasis in dB. for CMIS, pre-emphasis value will be set [NE(No Equalization),0.5,1,1.5,2,2.5,3,3.5,4,5,6,7]

--rx_post_emphasis <value>

Set Module Rx Post Emphasis in dB [NE(No Equalization),1,2,3,4,5,6,7]

--rx_amplitude <value>

Set Module Rx Output Amplitude [0(100-400mV),1(300-600mV),2(400-800mV),3(600-1200mV)]

--margin

Read the SerDes eye margins per lane

--measure_time <time>

Measure time in seconds for single eye [10, 30, 60, 90, 120, 240, 480, 600 and 900] (Optional - Default 60 for PCIe and 30 for Network ports)

--eye_select <eye_sel>

Eye selection for PAM4 [UP, MID, DOWN, ALL] (Default ALL)

--lane <lane_index>

Run eye for specific lane index (Default all lanes)

--rx_error_injection

Enable the RX link deterioration

--mixer_offset0 <value>

Fine change to the center of the eye [0x0 to 0x7ff]

--mixer_offset1 <value>

Coarse change to the center of the eye [0x0 to 0x3ff]

--show_mixers_offset

Show mixer offset 0 and mixer offset 1

--rx_fec_histogram

Provide histogram of FEC errors. The result is divided to bins. Each bin is holding different number of errored bit within FEC protected block

--show_histogram

Show FEC errors histogram

--clear_histogram

Clears FEC errors histograms

--pcie_error_injection

Start/show PCIe error injection

--error_type <type>

PCIe error type [ABORT(0),BAD_DLLP_LCRC(1),BAD_TLP_LCRC(2),BAD_TLP_ECRC(3),ERR_MSG(4),MALFORMED_TLP(5),POISONED_TLP(6),UNEXPECTED_CPL(7),ACS_VIOLATION(8),SURPRISE_LINK_DOWN(100),RECEIVER_ERROR(101)]

--error_duration <duration>

Error duration, depend on the error type

--injection_delay <delay>

Delay in micro-seconds before applying the error (Optional)

--error_parameters <params>

Comma-separated parameters for selected error type (param0,param1,param2,param3)

--dbdf <dbdf>

Destination bus device function, e.g af:00.0 (Optional used for specific error type)

--yes

Non-interactive mode, answer yes to all questions

Examples:

Get info of <device>, <port_number>:

Copy
Copied!
            

mlxlink -d <device> -p <port_number>

Get info of <device>, <port_number> and BER Counters:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> -c

Get info of <device>, <port_number> and Transmitter Parameters:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --show_serdes_tx

Configure Port State:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --port_state UP

Configure Port Speeds:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --speeds 25G,50G,100G

Configure FEC:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --fec RS

Configure Port for Physical Test Mode:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --test_mode EN (--rx_prbs PRBS31 --rx_rate 25G --tx_prbs PRBS7 --tx_rate 10G --invert_rx_polarity --invert_tx_polarity)

Perform PRBS Tuning:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --test_mode TU

Warning

RX and TX lane rates for new devices include the PAM4 speeds (50G_1X and 100G_2X).
eg: mlxlink -d <device> --test_mode EN --rx_rate [normal speeds | 50G_1X | 100G_2X] --tx_rate [normal speeds | 50G_1X | 100G_2X]

Warning

The PRBS pattern configured in PAM4 rates is PRBSQ.

Cable operations:

Copy
Copied!
            

mlxlink -d <device> --cable [Options]

Dump cable EEPROM pages:

Copy
Copied!
            

mlxlink -d <device> --cable --dump

Get cable DDM information:

Copy
Copied!
            

mlxlink -d <device> --cable --ddm

Read from cable:

Copy
Copied!
            

mlxlink -d <device> --cable --read --page <page number> --offset <bytes offset> --length <number of bytes>

Write to cable:

Copy
Copied!
            

mlxlink -d <device> --cable --write <bytes separated by comma> --page <page number> --offset <bytes offset>

Configure Transmitter Parameters (on lane, to database):

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --serdes_tx <polar- ity>,<ob_tap0>,<ob_tap1>,<ob_tap2>,<ob_bias>,<ob_preemp_mode>,<ob_reg>,<ob_leva> (--serdes_tx_lane <lane number>) (--database)

Configure Transmitter Parameters for 16nm devices:

Copy
Copied!
            

mlxlink -d <device> -p <port_number> --serdes_tx <pre_2_tap>,<pre_tap>,<main_tap>,<post_tap>,<ob_m2lp>,<ob_amp>

Getting PCIe links info:

Copy
Copied!
            

mlxlink -d /dev/mst/mt41682_pciconf0 --port_type PCIE --show_links Valid PCIe Links ---------------- : depth, pcie_index, node, port Link 1 : 3, 0, 0, 60 Link 2 : 3, 0, 1, 61 Link 3 : 3, 0, 2, 62 ..

To query information for a specific link, the depth, pcie_index and node for the link must be specified:

Copy
Copied!
            

mlxlink -d /dev/mst/mt41682_pciconf0 --port_type PCIE --depth 3 --pcie_index 0 --node 1 --show_serdes_tx --show_eye   PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node : 3, 0, 1 Link Speed Active (Enabled) : 8G-Gen 3 (16G-Gen 4) Link Width Active (Enabled) : 2X (16X)   EYE Opening Info (PCIe) ----------------------- Physical Grade : 84, 84 Height Eye Opening [mV] : 1194, 1194 Phase Eye Opening [psec] : 84, 84    Serdes Tuning Transmitter Info (PCIe) ------------------------------------- Serdes TX parameters : Pol ,tap0 ,tap1 ,tap2 ,bias ,preemp_mode ,reg ,leva Lane 0 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 1 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 2 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 3 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 4 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 5 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 6 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9 Lane 7 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9 

To print the output in JSON format:

Copy
Copied!
            

mlxlink -d <device> --show_module --json

To show ports group map (for NVIDIA Quantum and NVIDIA Spectrum-2):

Copy
Copied!
            

mlxlink –d<device> --show_tx_group_map 0

To assign ports to a specific group on NVIDIA Quantum and NVIDIA Spectrum-2:

Copy
Copied!
            

mlxlink –d <device> --tx_group_map 1 –ports 1,2,3,5,4,8,7,8,9,10,11

To show histogram of FEC errors:

Copy
Copied!
            

mlxlink -d /dev/mst/mt4125_pciconf0 --rx_fec_histogram --show_histogram

To clear histogram:

Copy
Copied!
            

mlxlink -d /dev/mst/mt4125_pciconf0 --rx_fec_histogram --clear_histogram

The margin scan tool is used for scanning PCIe or Network ports (in EDR\25G or HDR\PAM4 speeds).

Warning

If the margin scan fails with an "eye scan not completed" message, perform a reboot and run the scan again.

To enable the margin scan with a measure time of 10 seconds:

Copy
Copied!
            

mlxlink –d <device> --port_type PCIE –margin –measure_time 10

To enable the margin scan for Multi-host or Socket Direct systems through:

  • depth, pcie_index and node:

    Copy
    Copied!
                

    mlxlink –d <device> --port_type PCIE –depth 0 –pcie_index 1 –node 0 –margin –measure_time 30

  • The local port (can be shown by the –show_links command):

    Copy
    Copied!
                

    mlxlink –d <device> --port_type PCIE –port 1 –margin –measure_time 10

Allows modifying the Eye Center capability by changing the mixer_offset0 (fine change) and mixer_offset1 (coarse change) flags for 28nm products to produce RX errors.

Flags Usage

  • To change the mixers values:

    Copy
    Copied!
                

    mlxlink -d /dev/mst/mt4117_pciconf0 --rx_error_injection --mixer_offset0 0x200 --mixer_offset1 0x305

    Warning

    Modifying mixer_offset0 and mixer_offset1 flags can change the Eye Center and might cause link degradation.

  • To query the mixers values:

    Copy
    Copied!
                

    mlxlink -d /dev/mst/mt4117_pciconf0 --rx_erro r_injection --show_mixers_offset

The module PRBS test mode can be performed by using the new flags under the --cable command.

Note
  • This feature supports Active/Optical CMIS modules only.

  • Either the media or host side can run with PRBS mode.

  • To enable the PRBS test mode, the module should be plugged in and active.

Enabling\Disabling The Module PRBS Test Mode

To enable the module PRBS test mode, the side of the module should be selected using the --prbs_select flag. After providing the --cable flag, either the HOST or the MEDIA side should be selected, so the --prbs_mode <EN\DS> can be used to enable or disable the PRBS test mode process.

E.g.: the following command will enable the PRBS test mode on the HOST side of the module:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --prbs_mode EN

The command above will put the HOST side of the module in PRBS test mode with default Checker and Generator parameters.

The Checker and Generator parameters can be overridden while enabling the PRBS test mode according to their related flags in the help menu:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --prbs_mode EN --checker_pattern PRBS13 --invert_checker --generator_pattern PRBS31 --swap_generator --lane_rate HDR

To disable the PRBS test mode, the following command can be executed:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --prbs_mode DS


PRBS Diagnostic Counters Information

After performing the PRBS test mode, the module counters can be queried by using the following command:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --show_diagnostic_info

The module PRBS test mode counters can be cleared by using the following command, which will clear the diagnostic counters on the HOST side only:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --clear_diagnostic_info


Some of the module parameters can be controlled by mlxlink after providing the --control_paramenter flag, which can be executed under the --cable flag.

The possible parameters can be controlled as follows:

  • Reading and configuring Tx Equalization

  • Reading and configuring Rx Emphasis (PreCursor & PostCursor)

  • Reading and configuring Rx Amplitude

Note
  • To apply the changes, the link should be disabled first.

  • After configuring a new parameter, the link should be raised again to allow the firmware to load the new configuration.

  • Cable control parameters are valid for active\optical modules only.

Querying and Configuring The Module Control Parameters

To query the currently configured module control parameters, the --control_parameters flag can be used under the --cable flag as follows:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --cable --control_parameters ... Module Control Parameters ------------------------- TX Equalization : 1dB RX Emphasis (pre) : 2.5dB RX Post Emphasis : No Equalization RX Amplitude : 600-1200 mV (P-P)

To configure the module control parameters, the following command can be executed:

Copy
Copied!
            

mlxlink -d /dev/mst/mt53104_pciconf0 --cable --control_parameters --tx_equalization 2 --rx_amplitude 1 


When using the mlxlink tool with an adapter card, notice that the "label_port" -p flag should not be used. To address different ports, please use different MST devices.

For example:

To address port 1 when using ConnectX-4:

Copy
Copied!
            

mlxlink -d /dev/mst/mt4115_pciconf0

To address port 2, use:

Copy
Copied!
            

mlxlink -d /dev/mst/mt4115_pciconf0.1

Warning
  • Any mlxlink command for a switch should include the "-p" flag to address the specific port in the switch.

  • When working with an adapter card, if an MTUSB is used for communication with the NVIDIA NIC, to address port 2, use mlxlink -d /dev/mst/mt4115_pciconf0 --gvmi_address<0xAddress>.

If the split port number is not provided by the ibdiagnet tool, to use mlxlink on NVIDIA Quantum HDR based switch systems split ports, run:

Copy
Copied!
            

mlxlink -d lid-<LID> -p <formula>

Formula:

In case of 2X port:

  • 1- port_num = round_down[(Iblinkinfo_port_num + 1)*0.5]

  • 2- if (Iblinkinfo_port_num + 1) modulo 2 =1 then append ‘/2’ to port_num

In case of 4X port, use only item #1 above.

Example:

Copy
Copied!
            

43 23[ ] ==( 2X 53.125 Gbps Active/ LinkUp)==> mlxlink -d lid-43 -p 12 43 24[ ] ==( 2X 53.125 Gbps Active/ LinkUp)==> mlxlink -d lid-43 -p 12/2

In NVIDIA Quantum-2 NDR switch systems, there are 32-OSFP cages (8x), where each one holds 2 (4x) ports instead of 1, and each port can be accessed by providing the cage number and the port in the cage - “Cage/Port”.

Copy
Copied!
            

mlxlink -d <mst deivce> -p <Cage>/<Port>

If the split profile is ready, it is possible to access the split ports by providing the number of split to the port flag, e.g.:

  • To access the main port of 15/2:

    Copy
    Copied!
                

    mlxlink -d <mst deivce> -p 15/2

  • To access the split port of 15/2:

    Copy
    Copied!
                

    mlxlink -d <mst deivce> -p 15/2/2

Link Speed and Width

For PCIe link speed and width, use the following flag: --port_type PCIE.

Copy
Copied!
            

PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node     : [Depth, pcie index, node] Link Speed Active (Enabled) : [Freq – Gen] Link Width Active (Enabled) : [Width]


PCIe Switch

When using NVIDIA ConnectX-5 and newer devices, the PCIe interface can be configured for a PCIe switch. When the PCIe switch is enabled, the depth, pcie_index and node parameters are needed in order to specify the PCIe port from which the requested information (such as counters or eye info) is gathered.

Parameters

Description

Depth

This parameter defines the number of layers from the Root Complex to the specific port.

  • For NVIDIA ConnectX adapter cards multi-host mode, the depth should be set to 0.

  • For NVIDIA BlueField/BlueField-2 JBoF, the depth should be set to 3.

Pcie_index

This parameter defines the root complex ID or host index.

  • For NVIDIA ConnectX adapter cards multi-host mode, the pcie_index is the host index (0–3).

  • For NVIDIA BlueField/BlueField-2 JBoF, the pcie_index is always 0.

Node

This parameter defines the specific PCIe port.

  • For NVIDIA ConnectX adapter cards multi-host mode, the node is always 0 for each host_index.

  • For NVIDIA BlueField JBoF mode, this parameter range is 0x0–0xF, which amounts for up to 16 possible ports for BlueField JBoF.

  • For NVIDIA BlueField-2, this parameter's range is 0x0–0x7.

Note: For NVIDIA BlueField/BlueField-2 SmartNIC mode, the PCIe link information can only be gathered from the external host. The PCIe interface status cannot be retrieved from the Arm side. When retrieving the PCIe link information from the external host, there is no need to specify the depth, pcie_index and node.

Example: NVIDIA BlueField JBoF Mode

Copy
Copied!
            

# mlxlink -d /dev/mst/mt41682_pciconf0 --port_type pcie --depth 3 --pcie_index 0 --node 4 -c     PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node         : 3, 0, 4 Link Speed Active (Enabled)     : 8G-Gen 3 (16G-Gen 4) Link Width Active (Enabled)     : 2X (2X)   Management PCIe Timers Counters Info ------------------------------------ dl down                         : 0   Management PCIe Performance Counters Info ----------------------------------------- RX Errors                       : 0 TX Errors                       : 0 CRC Error dllp                  : 0 CRC Error tlp                   : 0


Link Counters

For PCIe counters information, use the --port_type PCIE –c flag.

Copy
Copied!
            

Management PCIe Timers Counters Info ------------------------------------ dl down                         : [link down counter]   Management PCIe Performance Counters Info ----------------------------------------- RX Errors                       : [Rx Errors] TX Errors                       : [Tx Errors] CRC Error dllp                  : [CRC Errors dllp] CRC Error tlp                   : [CRC Errors tlp]

  • RX Errors: indicate the number of transitions to recovery required due to framing errors and CRC (dlp and tlp) errors.

  • TX Errors: indicate the number of transitions to recovery required due to EIEOS and TS errors.

  • CRC Error dllp: indicate CRC error in Data Link Layer Packets.

  • CRC Error tlp: indicate CRC error in Transaction Layer Packet.

Example:

Copy
Copied!
            

# mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -c   PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node         : 0, 0, 0 Link Speed Active (Enabled)     : 16G-Gen 4 (16G-Gen 4) Link Width Active (Enabled)     : 16X (16X)   Management PCIe Timers Counters Info ------------------------------------ dl down                         : 3   Management PCIe Performance Counters Info ----------------------------------------- RX Errors                       : 0 TX Errors                       : 16 CRC Error dllp                  : 0 CRC Error tlp                   : 0


Link Eye Opening and Grade

For PCIe link physical grade and eye opening information, use the --port_type PCIE –e flag.

Copy
Copied!
            

EYE Opening Info (PCIE) ------------------------ Physical Grade : [Grade0, Grade1, Grade2, Grade3, Grade4, Grade5, Grade6, Grade7, Grade8, Grade9, Grade10, Grade11, Grade12, Grade13, Grade14, Grade15] Height Eye Opening [mV] : [Height0, Height1, Height2, Height3, Height4, Height5, Height6, Height7, Height8, Height9, Height10, Height11, Height12, Height13, Height14, Height15] Phase Eye Opening [psec] : [Phase0, Phase1, Phase2, Phase3, Phase4, Phase5, Phase6, Phase7, Phase8, Phase9, Phase10, Phase11, Phase12, Phase13, Phase14, Phase15]

Example:

Copy
Copied!
            

# mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -e   PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node         : 0, 0, 0 Link Speed Active (Enabled)     : 16G-Gen 4 (16G-Gen 4) Link Width Active (Enabled)     : 16X (16X)   EYE Opening Info (PCIe) ----------------------- Physical Grade                  :  57279, 56340, 59340, 61824, 55140, 60501, 61530, 57392, 61573, 58930, 62752, 60421, 57188, 59796, 60066, 60847 Height Eye Opening [mV]         :    292,   288,   314,   325,   278,   310,   319,   299,   316,   318,   343,   323,   310,   311,   335,   318 Phase  Eye Opening [psec]       :     30,    30,    30,    30,    30,    30,    30,    30,    30,    28,    28,    28,    28,    30,    28,    30


Pass/Fail Criteria

SLRED (ConnectX-6/ConnectX-6 Dx/ConnectX-6 Lx)

Copy
Copied!
            

mlxlink -d [device] --port_type PCIE --margin

Gen3

Gen3

Eye Grade

Figure of Merit (FOM)

0 < Eye Grade < 700

FAIL

700 < Eye Grade < 2300

Gray area

2300 < Eye Grade

PASS

Gen4

Gen4

Eye Margin

FOM

0 < Eye Grade < 150

FAIL

150 < Eye Grade < 400

Gray area

400 < Eye Grade

PASS

PCIE Error Injection

This test feature allows errors injection over the PCI links. It is used to verify that the system can handle the PCIe errors, which rarely occur in regular usage.

The ConnectX-7 device includes testability features that can be configured to act as an error injection ‘exerciser’ in order to test other components in the system. This is supported when the ConnectX-7 is used as a PCIe switch.

Warning
  • This is a PCIe related feature that should be run over PCIe links only (--port_type PCIE) with specific depth, PCIe index and node (DPN).

  • If the DPN is not provided, the tool will take the default values - 0,0 and 0, respectively.

  • The mapping between the BDF and its DPN can be found by executing the show_links command (see example below).

Error Types

ID

Error Type

Description

Unit

Additional Parameters

(--errors_parameters)

Advanced Error Reporting Flag Set by This Error

0

ABORT

Cancels the current pending error, if exists.

NA

NA

NA

1

BAD_DLLP_LCRC

Flips a bit in the LCRC of the next “error_duration” DLLPs that are transmitted through the port.

Packets

NA

Bad DLLP Status

2

BAD_TLP_LCRC

Flips a bit in the LCRC of the next “error_duration” TLPs that are transmitted

through the port.

The packets are VDM TLPs that are sent by the port to the destination BDF - “dbdf”.

Packets

NA

Bad TLP Status

3

BAD_TLP_ECRC

Flips a bit in the ECRC of the next “error_duration” TLPs that are transmitted

through the port.

The packets are VDM TLPs that are sent by the port to the destination BDF - “dbdf”.

Packets

NA

ECRC Error Status

4

ERR_MSG

Sends an error signaling message to the RC.

Packets

Parameter 0: message type

0 - Correctable

1 - Nonfatal

2 - Fatal

ERR_COR Received /

Non-Fatal Error Messages Received /

Fatal Error Messages Received

5

MALFORMED_ TLP

Sends an “error_duration” PM_ACTIVE_STATE_NACK message to the destination BDF - “dbdf” with TC=1 instead of 0.

Packets

NA

Malformed TLP Status

6

POISONED_TLP

Sends an “error_duration” VDMs with data to the destination BDF - “dbdf” with EP = 1.

Packets

NA

Poisoned TLP Received

7

UNEXPECTED_CPL

Sends “error_duration” completions to the destination BDF - “dbdf” with 0xff tag.

Packets

NA

Unexpected Completion Status

8

ACS_VIOLATION

Sends “error_duration” VDMs to the destination BDF - “dbdf” with source_bdf=0.

Packets

NA

ACS Violation Status

100

SURPRISE_LINK_DOWN

Sets a port state to DETECT.

NA

NA

Surprise Down Error Status

101

RECEIVER_ERROR

Sends a clock instead of data for “error_duration” usecs.

A value of 0 in ‘error_duration’ means that this error must be toggled by the firmware as fast as possible.

uSec

NA

Receiver Error Status


PCIe Error Injection Inputs

The following values should be provided in the error injection command line. Some values may be optional according to the error type.

Input

Command Line Flag

Description

Obligatory

Default

Error Type

--error_type

Error type according to the table above.

Yes

-

Error Duration

--duration

The minimal number of packets with this error that will be sent, or the minimal amount of time that this error state would be applied.

No

1

Injection Delay

--injection_delay

Delay in microseconds before the error is applied.

This allows time for the completion to return to the tool caller correctly. A higher value can be used to allow the system to get to a lower power state.

No

0

Destination BDF

--dbdf

Destination BDF. Relevant for some of the errors that require packet generation. See error table above.

No

0:00.0

Additional Parameters

--errors_parameters

Additional parameters according to the error type.

See error table above.

No

0

Mlxlink will trigger the firmware to start the error injection process by providing the --pcie_error_injection flag with the requested configuration parameters.

Note that the command returns immediately, but the error injection can take longer to complete (according to the error duration and injection delay inputs).

When the tool is run without the parameters above, it will query the error injection state – Whether it is ready to start a new error injection, or it is in the middle of the previous injection.

Usage Example

Start the process by performing error injection with error type UNEXPECTED_CPL.

This example shows how to start the error injection by sending 5 unexpected completion packets. The packets (of error type UNEXPECTED_CPL (id 7)) are directed from BDF 05:00.0 to BDF 06:0.0 after 500µs of sending the command in the following environment:

PCIe Component

BDF

Root port

00:01.0

(exerciser) PCIe Switch Upstream port

01:00.0

(exerciser) PCIe Switch Downstream port

05:00.0

Endpoint

06:00.0

To get the related depth, pcie_index and node for the specific BDF 05:00.0, the show_links command should be executed as follows:

Show Links

Copy
Copied!
            

mlxlink -d /dev/mst/mt4129_pciconf0 --port_type PCIE --show_links   Valid PCIe Links ---------------- Legend : depth ,pcie_index ,node ,port ,bdf Link 1 : 0 ,0 ,0 ,0 ,01:00.0 Link 2 : 3 ,0 ,0 ,60 ,05:00.0

In this case, the depth, pcie_index, and node flags for the downstream port should be 3, 0, and 0, respectively.

Then, the following command can be executed to start the process:

Start PCIe Error Injection

Copy
Copied!
            

mlxlink -d /dev/mst/mt4129_pciconf0 --port_type PCIE --depth 3 --pcie_index 0 --node 0 --pcie_error_injection --error_type UNEXPECTED_CPL --error_duration 5 --dbdf 06:00.0 --injection_delay 500   PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node : 3, 0, 0 Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4) Link Width Active (Enabled) : 16X (16X)   PCIe error injection might cause a PCIe bus failures or a system hang Do you want to continue? yes Starting PCIe Error Injection...  

Query Error Injection Status

After sending the configuration command, the progress of the process can be checked by executing the tool with the pcie_error_injection flag only:

Query PCIe Error Injection Status 1

Copy
Copied!
            

mlxlink -d /dev/mst/mt4129_pciconf0 –-port_type PCIE --depth 3 --pcie_index 0 --node 0 –-pcie_error_injection   PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node : 3, 0, 0 Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4) Link Width Active (Enabled) : 16X (16X)   PCIe Error Injection Info ------------------------- Error Injection Status : In progress Error Injection Type : UNEXPECTED_CPL Error Injection Duration : 5 Packets

Once the process is complete, the output will be changed to "ready". This means that another error injection request can be submitted:

Query PCIe Error Injection Status 2

Copy
Copied!
            

mlxlink -d /dev/mst/mt4129_pciconf0 –-port_type PCIE --depth 3 --pcie_index 0 --node 0 –-pcie_error_injection   PCIe Operational (Enabled) Info ------------------------------- Depth, pcie index, node : 3, 0, 0 Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4) Link Width Active (Enabled) : 16X (16X)   PCIe Error Injection Info ------------------------- Error Injection Status : Ready Error Injection Type : N/A Error Injection Duration : N/A

© Copyright 2023, NVIDIA. Last updated on Oct 11, 2023.