The tool can support new devices only once the tool is upgraded to its latest version.
mlxfwstress enables/disables various firmware stress flows. It can work in multiple modes:
- Enable/disable a specific set of stress types
- Clear all stress types
- Random mode:
- Single mode - choose one stress type in each iteration and enable/disable it
- Wild-mode- choose multiple stress types in each iteration and enable/disable them
Each time a stress type is chosen in a random iteration, the opposite operation is done on it (e.g., if a stress type is turned on, in the next iteration it will be turned off and vice versa).
- Toggle mode:
Turns on and off the list of stress types alternating. Can be used with iterations.
To disable a stressor while in toggling mode, first you must disable the mlxfwstress tool, and only after that disable the stressor.
- Clear semaphore:
Note: This functionality is supported in ConnectX-3 Pro adapter cards only.
mlxfwstress Synopsis
# mlxfwstress [-d|--dev <DeviceName>] [-h|--help] [-v|--version] [-o|--operation <Operation>] [--rand-mode <Random mode>] [-t|--stress-type <Stress type>] [--iterations <Iterations>] [--stress-delay <Stress delay>] [--max-rand-on <Max rand on>] [--hang-type <Hang type>] [--seed <seed>] [--toggle-time <x,y>]
where:
-d|--dev <DeviceName> | Perform operation for a specified device |
-h|--help | Show this message and exit |
-v|--version | Show the executable version and exit |
-o|--operation <Operation> | Choose operation: on, off, clear_all, random query, clear_semaphore |
--rand-mode <Random mode> | Choose a random mode: single, wild |
-t|--stress-type <Stress type> | Specify a list of stress types separated by comma. (See Stress Types.) |
--iterations <Iterations> | Specify the number of iterations. |
--stress-delay <Stress delay> | Specify the stress delay in seconds (can be float). Recommended values: 0-1 |
--max-rand-on <Max rand on> | Specify the maximal time a stress is allowed to be on in random mode in seconds. Recommended values (0,1] Default is 1 |
--hang-type <Hang type> | Specify a list of hang types separated by comma. (See Hang Types.) |
--seed <seed> | Specify the seed for the random. |
--toggle-time <x,y> | Toggle time after off, both in seconds (can be float). If y is not supplied the tool will use equal values for x and y |
Stress Types
ConnectX-4/ConnectX-4 Lx/ConnectX-5 Adapter Cards Stress Types
The following are the stress types available for ConnectX-4/ConnectX-4 Lx/ConnectX-5 adapter cards:
| Category | Stress Type | Description | Notes |
|---|---|---|---|
| Transparent | PAUSE_STORM_GENERATION | Generates pause frames from the device toward the network | |
| INVALIDATE_INTERNAL_CACHE_RX_1 | Invalidates STE cache | ||
| INVALIDATE_INTERNAL_CACHE_RX_2 | Invalidates qp L0 cache (RX) | ||
| INVALIDATE_INTERNAL_CACHE_RX_3 | Invalidates dct L0 cache (RX) | ||
| INVALIDATE_INTERNAL_CACHE_RX_4 | Invalidates scatter list cache in RX | ||
| INVALIDATE_INTERNAL_CACHE_CQ | Invalidates CQC cache | ||
| INVALIDATE_INTERNAL_CACHE_SX1 | Invalidates SXDC cache | ||
| INVALIDATE_INTERNAL_CACHE_RX_5 | Invalidates LDB cache | ||
| INVALIDATE_INTERNAL_CACHE_GENERAL_1 | Invalidates RO caches | ||
| INVALIDATE_INTERNAL_CACHE_SX2 | Invalidates pkey cache (SX) | ||
| INVALIDATE_INTERNAL_CACHE_SX3 | Invalidates guid cache (SX) | ||
| INVALIDATE_INTERNAL_CACHE_QP | Invalidates QPC (main QP cache unit) | ||
| Hang FW/HW | PACKET_DROP | Drops N packets on portx | This type requires the following extra flags:
|
ConnectX-3 Pro Adapter Cards Stress Types
The following are the stress types available for ConnectX-3 Pro adapter cards:
Stressors in "Transparent" category that are active for more than 100 msec, may cause resiliency.
| Category | Stress Type | Description |
|---|---|---|
| Transparent | STOP_CE_INSTAGE_EQE | Stops sending EQEs created by the hardware (not the ones created by the firmware). |
| STOP_EDBH | Stops the handling of external doorbells. | |
| STOP_IDBH | Stops the handling of internal doorbells. | |
STOP_QPC_MISS_MACHINE_0 STOP_QPC_MISS_MACHINE_1 STOP_QPC_MISS_MACHINE_2 STOP_QPC_MISS_MACHINE_3 | Spots reading a QPC from the ICM on a miss-blocking hardware/firmware that accesses the QPC | |
| LOCK_CEGW | Locks the CQE gateway. | |
LOCK_OBGW_TPT LOCK_OBGW_TCU LOCK_OBGW_SXD | Locks the OBGW (access to the host memory gateway). | |
| LOCK_QPCGW_RX | Locks QPCGW. | |
LOCK_SEMAPHORE_IPC_RX0 LOCK_SEMAPHORE_IPC_RX1 LOCK_SEMAPHORE_IPC_LDB LOCK_SEMAPHORE_IPC_SX1 | Locks the IPC semaphore. | |
| INVALIDATE_CACHES | Invalidates caches. | |
| Performance | STOP_SXP_VL_ARB_PORT1 STOP_SXP_VL_ARB_PORT2 | Stops transmission of packets to the wire. Causes head-of-line packet drop (HLL) if enabled. |
RX_BACKPRESSURE | Stops the RX pipe - back-pressure to wire- sending tx pauses. | |
| DROP_PACKETS_TX | Drops packets on the TX side. |
Turning On Stress Types
To turn on a specific stress type:
mlxfwstress -d mt4103_pciconf0 -o on -t STOP_CE_INSTAGE_EQE ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON stress type: stop_ce_instage_eqe -PASSED
To turn on a set of stress types:
mlxfwstress -d mt4103_pciconf0 -o on -t STOP_CE_INSTAGE_EQE,STOP_QPC_MISS_MACHINE_3,LOCK_SEMAPHORE_IPC_RX1 ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON stress type: stop_ce_instage_eqe -PASSED Turning ON stress type: stop_qpc_miss_machine_3 -PASSED Turning ON stress type: lock_semaphore_ipc_rx1 -PASSED
To turn on all the available stress types:
mlxfwstress -d mt4119_pciconf0 -t ALL -o on Random seed: [1587969653] ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON stress type: INVALIDATE_INTERNAL_CACHE_CQ -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_GENERAL_1 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_QP -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_2 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_3 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_4 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_5 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_SX1 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_SX2 -PASSED Turning ON stress type: INVALIDATE_INTERNAL_CACHE_SX3 -PASSED
Turning Off Stress Types
To turn off a specific stress type:
mlxfwstress -d mt4103_pciconf0 -o off -t STOP_CE_INSTAGE_EQE ------------------------------------------------- Operation: [OFF] ------------------------------------------------- Turning OFF stress type: stop_ce_instage_eqe -PASSED
To turn off a set of stress types:
mlxfwstress -d mt4103_pciconf0 -o off -t STOP_CE_INSTAGE_EQE,STOP_QPC_MISS_MACHINE_3,LOCK_SEMAPHORE_IPC_RX1 ------------------------------------------------- Operation: [OFF] ------------------------------------------------- Turning OFF stress type: stop_ce_instage_eqe -PASSED Turning OFF stress type: stop_qpc_miss_machine_3 -PASSED Turning OFF stress type: lock_semaphore_ipc_rx1 -PASSED
Querying the Stress Types
To query the state of all stress types:
mlxfwstress -d mt4117_pciconf0 -o query -t ALL ------------------------------------------------- Operation: [QUERY] ------------------------------------------------- Querying stress type: INVALIDATE_INTERNAL_CACHE_CQ -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_GENERAL_1 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_QP -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_2 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_3 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_4 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_5 -NOT SUPPORTED Querying stress type: INVALIDATE_INTERNAL_CACHE_SX1 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_SX2 -ENABLED Querying stress type: INVALIDATE_INTERNAL_CACHE_SX3 -ENABLED
Hang Types
ConnectX-4/ConnectX-4 Lx/ConnectX-5 Adapter Cards Hang Types
The following are the hang types available for ConnectX-4/ConnectX-4 Lx/ConnectX-5 adapter cards:
| Category | Stress Type | Description | Notes |
|---|---|---|---|
| Hang FW/HW | STOP_RX_PER_PRIO1 | This type requires the following extra flags:
|
mlxfwstress -d mt4115_pciconf0 -o on --hang-type STOP_RX_PER_PRIO --extra %STOP_RX_PER_PRIO[0x00100FF] Random seed: [1588056318] ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON hang type: STOP_RX_PER_PRIO -PASSED
To turn this Hang Type, the command must be executed in the following format:
Example:
mlxfwstress -d mt4115_pciconf0 -o on --hang-type STOP_RX_PER_PRIO --extra % STOP_RX_PER_PRIO [0x000100FF] output: Random seed: [1573642282] ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON hang type: STOP_RX_PER_PRIO-PASSED
Turning On Hang Types
To turn on a specific hang type:
mlxfwstress -d mt4103_pciconf0 -o on --hang-type HANG_SX1 ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON hang type: Sx1 -PASSED
To turn on a set of hang types:
mlxfwstress -d mt4103_pciconf0 -o on --hang-type HANG_SX1,HANG_RX1 ------------------------------------------------- Operation: [ON] ------------------------------------------------- Turning ON hang type: Sx1 -PASSED Turning ON hang type: Rx1 -PASSED
Turning Off Hang Types
To turn off a specific hang type:
mlxfwstress -d mt4103_pciconf0 -o off --hang-type HANG_SX1 ------------------------------------------------- Operation: [OFF] ------------------------------------------------- Turning OFF hang type: Sx1 -PASSED
To turn off a set of hang types:
mlxfwstress -d mt4103_pciconf0 -o off --hang-type HANG_SX1,HANG_RX1 ------------------------------------------------- Operation: [OFF] ------------------------------------------------- Turning OFF hang type: Sx1 -PASSED Turning OFF hang type: Rx1 -PASSED
Querying the Hang Types
To query the state of all hang types:
mlxfwstress -d mt4103_pciconf0 -o query --hang-type ALL ------------------------------------------------- Operation: [QUERY] ------------------------------------------------- Querying hang type: Sx1 -ENABLED Querying hang type: Rx1 -ENABLED Querying hang type: Tx -ENABLED Querying hang type: Rx -ENABLED
Clearing all Stress/Hang Types
To clear all stress/hang types:
mlxfwstress - d mt4103_pciconf0 -o clear_all ------------------------------------------------- Operation: [CLEAR_ALL] ------------------------------------------------- Turning OFF hang type: Sx1 -PASSED Turning OFF hang type: Rx1 -PASSED Turning OFF hang type: Tx -PASSED Turning OFF hang type: Rx -PASSED Turning OFF stress type: stop_ce_instage_eqe -PASSED Turning OFF stress type: stop_sxp_vl_arb_port1 -PASSED Turning OFF stress type: stop_sxp_vl_arb_port2 -PASSED Turning OFF stress type: stop_edbh -PASSED Turning OFF stress type: stop_idbh -PASSED Turning OFF stress type: stop_qpc_miss_machine_0 -PASSED Turning OFF stress type: stop_qpc_miss_machine_1 -PASSED Turning OFF stress type: stop_qpc_miss_machine_2 -PASSED Turning OFF stress type: stop_qpc_miss_machine_3 -PASSED Turning OFF stress type: lock_cegw -PASSED Turning OFF stress type: lock_obgw_tpt -PASSED Turning OFF stress type: lock_obgw_tcu -PASSED Turning OFF stress type: lock_obgw_sxd -PASSED Turning OFF stress type: lock_qpcgw_rx -PASSED Turning OFF stress type: lock_semaphore_ipc_sx1 -PASSED Turning OFF stress type: lock_semaphore_ipc_rx0 -PASSED Turning OFF stress type: lock_semaphore_ipc_rx1 -PASSED Turning OFF stress type: lock_semaphore_ipc_ldb -PASSED Turning OFF stress type: invalidate_caches -PASSED
Clearing the Semaphore
To clear the semaphore:
mlxfwstress -d mt4103_pciconf0 -o clear_semaphore ------------------------------------------------- Operation: [CLEAR_SEMAPHORE] ------------------------------------------------- Semaphore was cleared successfully
Random Operation
There are two random modes you can choose from:
- Single - gives a set of stress types, in each iteration one stress type is chosen an toggled ON/OFF according to his current state
- Wild - gives a set of stress types, in each iteration a random subset of stress types is chosen and toggled ON/OFF according to their current state
Setting the Random Mode for the Stress Types
To set the Single Mode:
mlxfwstress -d mt4103_pciconf0 -o random --rand-mode single -t STOP_CE_INSTAGE_EQE --stress-delay 0.2 --iterations 10 ------------------------------------------------- Operation: [RANDOM] ------------------------------------------------- ############################################# Random: Iterations delay: 0.2 [sec] Iterations number: 10 Max on time: 1 [sec] ############################################# RANDOM ITERATION: [1] [stop_ce_instage_eqe]: [ON] , duration since last operation: 0 [ms] RANDOM ITERATION: [2] [stop_ce_instage_eqe]: [OFF], duration since last operation: 200 [ms] RANDOM ITERATION: [3] [stop_ce_instage_eqe]: [ON] , duration since last operation: 201 [ms] RANDOM ITERATION: [4] [stop_ce_instage_eqe]: [OFF], duration since last operation: 200 [ms] RANDOM ITERATION: [5] [stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms] RANDOM ITERATION: [6] [stop_ce_instage_eqe]: [OFF], duration since last operation: 201 [ms] RANDOM ITERATION: [7] [stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms] RANDOM ITERATION: [8] [stop_ce_instage_eqe]: [OFF], duration since last operation: 201 [ms] RANDOM ITERATION: [9] [stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms] Turning OFF stress type: stop_ce_instage_eqe RANDOM ITERATION: [10] [stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms] ======================================================= Turning off all stress types after random: Turning OFF stress type: stop_ce_instage_eqe
- As seen in the example above, after the specified number of iterations, the tool turns off all the stress types.
- The default value for stress-delay is 1 second.
- If no number of iterations was supplied then the user is expected to stop the tool with ctrl+c. The tool turns off all the stress types.
To set the Wild Mode:
mlxfwstress -d mt4103_pciconf0 -o random --rand-mode wild -t ALL --stress-delay 0.2 --max-rand-on 1 --iterations 5 ------------------------------------------------- Operation: [RANDOM] ------------------------------------------------- ############################################# Random: Iterations delay: 0.2 [sec] Iterations number: 5 Max on time: 1 [sec] ############################################# RANDOM ITERATION: [1] [stop_ce_instage_eqe]: [ON] , duration since last operation: 0 [ms] [stop_sxp_vl_arb_port2]: [ON] , duration since last operation: 0 [ms] [stop_edbh]: [ON] , duration since last operation: 0 [ms] [stop_idbh]: [ON] , duration since last operation: 0 [ms] [stop_qpc_miss_machine_0]: [ON] , duration since last operation: 0 [ms] [stop_qpc_miss_machine_3]: [ON] , duration since last operation: 0 [ms] [lock_cegw]: [ON] , duration since last operation: 0 [ms] [lock_obgw_tcu]: [ON] , duration since last operation: 0 [ms] [lock_qpcgw_rx]: [ON] , duration since last operation: 0 [ms] [lock_semaphore_ipc_sx1]: [ON] , duration since last operation: 0 [ms] RANDOM ITERATION: [2] [stop_sxp_vl_arb_port1]: [ON] , duration since last operation: 0 [ms] [stop_edbh]: [OFF], duration since last operation: 203 [ms] [stop_idbh]: [OFF], duration since last operation: 203 [ms] [stop_qpc_miss_machine_3]: [OFF], duration since last operation: 202 [ms] [lock_cegw]: [OFF], duration since last operation: 202 [ms] [lock_obgw_tpt]: [ON] , duration since last operation: 0 [ms] [lock_obgw_tcu]: [OFF], duration since last operation: 203 [ms] [lock_semaphore_ipc_rx0]: [ON] , duration since last operation: 0 [ms] [lock_semaphore_ipc_rx1]: [ON] , duration since last operation: 0 [ms] [lock_semaphore_ipc_ldb]: [ON] , duration since last operation: 0 [ms] RANDOM ITERATION: [3] [stop_ce_instage_eqe]: [OFF], duration since last operation: 406 [ms] [stop_sxp_vl_arb_port2]: [OFF], duration since last operation: 406 [ms] [stop_edbh]: [ON] , duration since last operation: 203 [ms] [stop_idbh]: [ON] , duration since last operation: 203 [ms] [stop_qpc_miss_machine_0]: [OFF], duration since last operation: 406 [ms] [stop_qpc_miss_machine_2]: [ON] , duration since last operation: 0 [ms] [lock_obgw_tpt]: [OFF], duration since last operation: 203 [ms] [lock_obgw_sxd]: [ON] , duration since last operation: 0 [ms] [lock_semaphore_ipc_sx1]: [OFF], duration since last operation: 405 [ms] [lock_semaphore_ipc_ldb]: [OFF], duration since last operation: 203 [ms] RANDOM ITERATION: [4] [stop_sxp_vl_arb_port2]: [ON] , duration since last operation: 203 [ms] [stop_edbh]: [OFF], duration since last operation: 202 [ms] [stop_idbh]: [OFF], duration since last operation: 202 [ms] [stop_qpc_miss_machine_1]: [ON] , duration since last operation: 0 [ms] [stop_qpc_miss_machine_3]: [ON] , duration since last operation: 406 [ms] [lock_obgw_tpt]: [ON] , duration since last operation: 202 [ms] [lock_obgw_tcu]: [ON] , duration since last operation: 406 [ms] [lock_obgw_sxd]: [OFF], duration since last operation: 203 [ms] [lock_semaphore_ipc_sx1]: [ON] , duration since last operation: 203 [ms] [lock_semaphore_ipc_rx1]: [OFF], duration since last operation: 406 [ms] [invalidate_caches]: [ON] , duration since last operation: 0 [ms] Turning OFF stress type: stop_sxp_vl_arb_port1 Turning OFF stress type: stop_sxp_vl_arb_port2 Turning OFF stress type: stop_qpc_miss_machine_1 Turning OFF stress type: stop_qpc_miss_machine_2 Turning OFF stress type: stop_qpc_miss_machine_3 Turning OFF stress type: lock_obgw_tpt Turning OFF stress type: lock_obgw_tcu Turning OFF stress type: lock_qpcgw_rx Turning OFF stress type: lock_semaphore_ipc_sx1 Turning OFF stress type: lock_semaphore_ipc_rx0 Turning OFF stress type: invalidate_caches RANDOM ITERATION: [5] [stop_sxp_vl_arb_port2]: [ON] , duration since last operation: 202 [ms] [stop_idbh]: [ON] , duration since last operation: 322 [ms] [lock_obgw_tpt]: [ON] , duration since last operation: 202 [ms] [lock_obgw_tcu]: [ON] , duration since last operation: 202 [ms] [lock_qpcgw_rx]: [ON] , duration since last operation: 202 [ms] [invalidate_caches]: [ON] , duration since last operation: 202 [ms] ======================================================= Turning off all stress types after random: Turning OFF stress type: stop_sxp_vl_arb_port2 Turning OFF stress type: stop_idbh Turning OFF stress type: lock_obgw_tpt Turning OFF stress type: lock_obgw_tcu Turning OFF stress type: lock_qpcgw_rx Turning OFF stress type: invalidate_caches
ConnectX-3/ConnectX-3 Pro Adapter Cards Hang Types
The following are the hang types available for ConnectX-3/ConnectX-3 Pro adapter cards:
| Category | Stress Type | Description | Notes |
|---|---|---|---|
| Hang FW/HW | HANG_SX1 | ||
| HANG_RX1 | |||
| HANG_TX | |||
| HANG_RX | |||
| ALL | Hang types that require extra flags are not supported when running with the 'ALL' option. |
