mlxtrace Utility

The mlxtrace utility is used to configure and extract HW events generated by different units in NVIDIA devices. The utility generates a dump ".trc" file which contains HW events that assist us with debug, troubleshooting and performance analysis. Events can be stored in host memory if driver is up or in a small on-chip buffer (always available) depending on the utility running mode. In order to run the utility it's required to have a configuration file first, this file should be provided by the NVIDIA representative.

A dump file "mlxtrace.trc" will be generated by end of run (file name can be controlled by "-o" flag), this file should be sent to the NVIDIA representative for further diagnostics/troubleshooting.

Warning

Memory mode on 5th generation (Group II) devices is supported only by PCI mst devices. Memory mode is supported in Windows, as well as in Linux.

Warning

For the tool to properly work with Inband devices, both the MFT and the Firmware must be updated to the latest (MFT v4.18.0 & firmware vXX.32.1xxx).

Warning

If ConnectX-4 adapter card is used as an Inband device, for the tool to work properly, you need to use MFT 4.17.0.

  1. The mst driver must be started prior to running the mlxtrace tool.

  2. For MEM buffer mode driver must be "loaded" also.

  3. Enter the following command:

    Copy
    Copied!
                

    mlxtrace [options]

Options

-h, --help

Print help and exit

-v, --version

Print version (default=off)

-p, --parse

Move to parser mode (default=off)

Mode: CollectMode

-d, --device=MstDev

Mst device

-m, --mode=Mode

Activation mode: FIFO - HW BUFFER , MEM - KERNEL BUFFER (possible values="FIFO", "MEM")

-a, --mem_access=MemMethod

Memory access method: OB_GW, MEM, DMEM, FMEM, VMEM (possible values="OB_GW", "MEM", "DMEM", "FMEM", "VMEM")

Note: As of MFT v4.21.0, the following values will be deprecated: MEM, DMEM, FMEM.

--gvmi

Global virtual machine interface

-c, --cfg=CfgFile

Mlxtrace configuration file

-o, --trc_file=TrcPath

Output TRC file path (default=`mlxtrace.trc')

-C, --config_only

Configure tracer and exit (default=off)

-n, --snapshot

Take events snapshot - this assumes previous run with -- config_only (default=off)

-s, --buf_size=BufSize

User buffer size [MB] (default=`1')

-S, --stream

Don't save events to file parse it immediately (default=off)

--ignore_old_events

Ignore collecting old events in MEM mode (default=off)

-g, --continuous_fill

Do not stop recording (stopping only with ^C), keep filling user's buffer cyclically (default=off)

--sample_delay=Delay

Delay between samples when polling new events in [usec] (default=`0')

--keep_running

Keep the HW tracer unit running after exit (default=off)

--enable_limiting_every_chunk

Limit the HW tracer after reading every chunk (default=off)

--skip_ownership

Skip taking ownership (default=off)

Mode: ParseMode

-i, --input=TrcFile

Input file (default=`mlxtrace.trc')

--csv_mode

Enable csv output format (default=off)

--print_ts

Print timestamp events (default=off)

-r, --real_ts

Print real timestamps in [hh:mm:ss.nsec] format (default=off)

--print_raw

Print event bytes in each line header (default=off)

--ts_format=format

Choose printed TS format hex/dec (possible values="hex", "dec" default=`dec')

--print_delta

Enable printing delta between events (in cycles) (default=off)

-f, --print_file=FilePath

Print parsed event to the given file and not to stdout

--enable_db_check

Enable events DB checks (default=off)

Examples:

Choose the suitable .cfg file depending on the device you are using, and run the following command to generate a .trc file:

Copy
Copied!
            

# mlxtrace -d /dev/mst/mt4099_pci_cr0 -c connectx3.cfg -m MEM -o connectx3.trc

To generate a .trc file with a maximal size of 100 MB, run the following command:

Copy
Copied!
            

# mlxtrace -d /dev/mst/mt4099_pci_cr0 -c connectx3.cfg -m MEM -s 100 -o connectx3.trc

© Copyright 2023, NVIDIA. Last updated on Oct 11, 2023.