NVIDIA Firmware Tools (MFT) Documentation v4.30.1 LTS

fwtrace Utility

The fwtrace utility extracts and prints trace messages generated by the firmware running on 5th generation (Group II) devices iRISCs.

These trace messages inform developers of software drivers about internal status, events, critical errors, etc. Trace messages generated by iRISCs are stored in the trace buffer. The trace buffer is located in host memory. The tool also supports mem free mode where it uses a device internal small buffer.

By default, the firmware does not print trace messages. Please contact your FAE for more details on how to enable firmware tracing.

Note

When using secure firmware, the user needs to validate that the value "1" is set to /sys/kernel/debug/tracing/events/mlx5/fw_tracer/enable.

As of MFT v4.30.0, fwtrace can extract events from secure devices (for supported Firmware versions) using mlxptrace utility and without the need of mlx5 driver. fwtrace will automatically use the new capabilities if the device supports it.

Note

Memory mode on 5th generation (Group II) devices is supported only by PCI mst devices.

Note

For the tool to properly work with Inband devices, both the MFT and the Firmware must be updated to the latest (MFT version 4.18.0 and above, and Firmware version XX.32.1xxx and above).

  1. Start the mst driver (mst start or mst restart)

  2. Enter the following command:

    Copy
    Copied!
                

    # fwtrace [options...]

where

-h|--help

Print this help message and exit

-d|--device

mst device name

-f|--fw_strings

Fw strings db file containing the FW strings

--tracer_mode

Tracer mode [FIFO | MEM]

--real_ts

Print real timestamps in [hh:mm:ss:nsec] format

-i|--irisc

iRISC name (See below for full list of irisc names)

-t|--tile

Allow the user to specify which tile events he would like to see.

# "1.all" will enable risc1 in all tiles.

# "1" will be treated like "1.all"

# "all" will be treated like all.all

# "all.1" will enable all iriscs in tile 1

# .1 is invalid input.

-a|--apu

If enabled, tool will show APU events.

--include_phy_uc

If enabled, tool will also show phy_uc events related to the enabled riscs (main / tiles)

--keep_cfg

If enabled, the FWtrace will not delete the temporary cfg file (if one was created during the current run)

-s|--stream

Run in streaming mode

-c|--cfg

Hardware tracer events cfg file

-n|--snapshot

Take events snapshot - this assumes previous FW configurations

-S|--buf_size

Hardware tracer MEM buffer size in [MB]

--dump

Dump file name

-m|--mask

Trace class mask, use "+" to enable multiple classes or use integer format, e.g: -m class1+class2+... or 0xff00ff00

-l|--level

Trace level

--log_delay

Firmware tracer log delay in uSec

v|--version

Print tool's version and exit

--gvmi

Global virtual machine interface

--ignore_old_events

Ignore collecting old events

--mem_access

Memory access method: OB_GW, VMEM, UDRIVER

--keep_running

Keep the hardware tracer unit running after exit

--config_only

Configure tracer and exit

--fw_cfg_only

Skip HW config and only configure FW events (default=off)

Device Specific Info:

To view device specific information, run "fwtrace -d <mst_dev> -h".

Example:

Copy
Copied!
            

fwtrace -d /dev/mst/mt4123_pciconf0 -i all -s -l 0 -m 0xFFFFFFFF -I- Found FW string db cache file, going to use it Got ownership successfully! mlxtrace -d /dev/mst/mt4123_pciconf0 -m MEM --skip_ownership -c /tmp/itrace_2946580.cfg -S -I- Tracer Configuration: -I- ===================== -I- Mode : Collector -I- Activation Mode : Memory Mode -I- Memory Access Method : NA -I- Configuration File Path : /tmp/itrace_2946580.cfg -I- Output file (Trace File) Path : mlxtrace.trc -I- User Buffer Size : NAMBytes -I- Use Stream Mode : YES -I- Configure Only : NO -I- Only Snapshot (Skip Configuration Stage) : NO -I- Ignore old events : NO -I- Continuous fill : NO -I- Print timestamp in hh:mm:ss:nsec format : NO -I- Output file for streaming : STDOUT -I- Delay between samples : 0usec -I- Stop tracer after exit : YES -I- =============================================== -I- Tracer ownership is already taken or skipped -I- Device is: ConnectX6 -I- Configuring Tracer... -I- Invalidating kernel buffer... (Press ^C to skip) -I- Done -I- Tracer was configured successfully Device frequency: 430MHz -I- Starting event streaming... #time stamp, unreliable, lost, missing TS, event name Reading new events... 512458354925 I3 in pll_management_handler. freq_in = 3. 512458355201 I3 in pll_management_handler. set pll 0 in center pll 0 to state 0. 512458355749 I3 in pll_management_handler. freq_in = 3. 512458356030 I3 in pll_management_handler. set pll 1 in center pll 0 to state 0. 512458356583 I3 in pll_management_handler. freq_in = 3. 512458356865 I3 in pll_management_handler. set pll 2 in center pll 0 to state 3. . . .

© Copyright 2024, NVIDIA. Last updated on Dec 5, 2024.