NVIDIA Firmware Tools (MFT) Documentation v4.20.0
NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

fwtrace Utility

The fwtrace utility extracts and prints trace messages generated by the firmware running on 5th generation (Group II) devices iRISCs.

These trace messages inform developers of software drivers about internal status, events, critical errors, etc. Trace messages generated by iRISCs are stored in the trace buffer. The trace buffer is located in host memory. The tool also supports mem free mode where it uses a device internal small buffer.

By default, the firmware does not print trace messages. Please contact your FAE for more details on how to enable firmware tracing.

Warning

When using secure firmware, the user needs to validate that the value "1" is set to /sys/kernel/debug/tracing/events/mlx5/fw_tracer/enable.

Warning

Memory mode on 5th generation (Group II) devices is supported only by PCI mst devices.

Warning

For the tool to properly work with Inband devices, both the MFT and the Firmware must be updated to the latest (MFT v4.18.0 & firmware vXX.32.1xxx).

Warning

If ConnectX-4 adapter card is used as an Inband device, for the tool to work properly, you need to use MFT 4.17.0.

  1. Start the mst driver (mst start or mst restart)

  2. Enter the following command:

    Copy
    Copied!
                

    # fwtrace [options...]

where

-h|--help

Print this help message and exit

-d|--device

mst device name

-f|--fw_strings

Fw strings db file containing the FW strings

--tracer_mode

Tracer mode [FIFO | MEM]

--real_ts

Print real timestamps in [hh:mm:ss:nsec] format

-i|--irisc

iRISC name (See below for full list of irisc names)

-s|--stream

Run in streaming mode

-c|--cfg

HW tracer events cfg file

-n|--snapshot

Take events snapshot - this assumes previous FW configurations

-S|--buf_size

HW tracer MEM buffer size in [MB]

--dump

Dump file name

-m|--mask

Trace class mask, use "+" to enable multiple classes or use integer format, e.g: -m class1+class2+... or 0xff00ff00

-l|--level

Trace level

v|--version

Print tool's version and exit

--gvmi

Global virtual machine interface

--ignore_old_events

Ignore collecting old events

Device Specific Info:

  • Connect-IB, ConnectX-4, ConnectX-4 Lx, ConnectX-6 Lx, Switch-IB, Switch-IB 2, Spectrum, Spectrum-2, Quantum:
    iRISC names: [i0, iron, i2, i3, i4, i5, i6, i7, all]

  • Spectrum-3, Quantum-2:
    iRISC names: [i0, iron, i2, i3, i4, i5, i6, i7, i8, i9, all]

  • ConnectX-5, BlueField, ConnectX-6, ConnectX-6 Dx, BlueField-2
    iRISC names: [i0, iron, i2, i3, i4, i5, i6, i7, i8, i9, i10, all]

  • Trace classes:
    DEBUG_INIT, INIT, ICM, ICM_FREE_LIST, HOST_MNG, CMD_IF, PHY_IB, PHY_RX_ADAP, LIBFHI, PHY_COMMON, PHY_MANAGER, PWR, FLR, ICM_ACCESS, MAD, RXT_CHECKS, I2C, TRANSPORT, FW_LL, RX_ERRORS, CMD_DRIVER, PROFILING, MANAGEMENT, FLASH, STEERING, IFARM, ICMD, PCI, DC_CLEANUP, PHY_ETH, VIRT

    Example:

    Copy
    Copied!
                

    # fwtrace -d mlx5_0 -i all -s -I- Found FW string db cache file, going to use it mlxtrace -d mlx5_0 -m MEM -c /tmp/itrace_8153.cfg -S -I- Tracer Configuration: -I- ===================== -I- Mode : Collector -I- Activation Mode : Memory Mode -I- Memory Access Method : NA -I- Configuration File Path : /tmp/itrace_8153.cfg -I- Output file (Trace File) Path : mlxtrace.trc -I- User Buffer Size : NA[MBytes] -I- Use Stream Mode : YES -I- Configure Only : NO -I- Only Snapshot (Skip Configuration Stage) : NO -I- Continuous fill : NO -I- Print timestamp in [hh:mm:ss:nsec] format : NO -I- Output file for streaming : STDOUT -I- Delay between samples : 0[usec] -I- =============================================== -I- Device is: cib -I- Configuring Tracer... -I- Invalidating kernel buffer... (Press ^C to skip) -I- Done -I- Tracer was configured successfully Device frequency: 276MHz -I- Starting event streaming... Reading new events... 774774193803 I2 Mad received on port 1 - QP 0 774774215444 I2 process set_get_pkey_table on port=1 set_get_=0 block=1 774775079296 I2 Mad received on port 1 - QP 0 774775120645 I2 port_state changed from INIT to ARM 774775166315 I2 process set_get_port_info on port 1 set_get_: 1 status:0x0 774775335890 I2 Mad received on port 1 - QP 0 774775367205 I2 port_state changed from ARM to ACTIVE 774775410880 I2 process set_get_port_info on port 1 set_get_: 1 status:0x0 774786733806 I3 process MAD_IFC on port 1 774786744859 I3 process set_get_port_info on port 1 set_get_: 0 status:0x0 . . .

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.