image image image image image

On This Page

The mlxtrace utility is used to configure and extract HW events generated by different units in NVIDIA devices. The utility generates a dump ".trc" file which contains HW events that assist us with debug, troubleshooting and performance analysis. Events can be stored in host memory if driver is up or in a small on-chip buffer (always available) depending on the utility running mode. In order to run the utility it's required to have a configuration file first, this file should be provided by the NVIDIA representative.

A dump file "mlxtrace.trc" will be generated by end of run (file name can be controlled by "-o" flag), this file should be sent to the NVIDIA representative for further diagnostics/troubleshooting.

Memory mode on 5th generation (Group II) devices is supported only by PCI mst devices. Memory mode is supported in Windows, as well as in Linux.

For the tool to properly work with Inband devices, both the MFT and the Firmware must be updated to the latest (MFT v4.18.0 & firmware vXX.32.1xxx). 

If ConnectX-4 adapter card is used as an Inband device, for the tool to work properly, you need to use MFT 4.17.0.  

mlxtrace Usage

  1. The mst driver must be started prior to running the mlxtrace tool.
  2. For MEM buffer mode driver must be "loaded" also.
  3. Enter the following command:

    mlxtrace [options]


-h, --help

Print help and exit

-v, --version

Print version (default=off)

-p, --parse

Move to parser mode (default=off)

Mode: CollectMode

-d, --device=MstDev

Mst device

-m, --mode=Mode

Activation mode: FIFO - HW BUFFER , MEM - KERNEL BUFFER (possible values="FIFO", "MEM")

-a, --mem_access=MemMethod

Memory access method: OB_GW, MEM, DMEM, FMEM, VMEM (possible values="OB_GW", "MEM", "DMEM", "FMEM", "VMEM")

Note: As of MFT v4.21.0, the following values will be deprecated: MEM, DMEM, FMEM.

-c, --cfg=CfgFile

Mlxtrace configuration file

-o, --trc_file=TrcPath

Output TRC file path (default=`mlxtrace.trc')

-C, --config_only

Configure tracer and exit (default=off)

-n, --snapshot

Take events snapshot - this assumes previous run with -- config_only (default=off)

-s, --buf_size=BufSize

User buffer size [MB] (default=`1')

-S, --stream

Don't save events to file parse it immediately (default=off)


Ignore collecting old events in MEM mode (default=off)

-g, --continuous_fill

Do not stop recording (stopping only with ^C), keep filling user's buffer cyclically (default=off)


Delay between samples when polling new events in [usec] (default=`0')


Keep the HW tracer unit running after exit (default=off)

--enable_limiting_every_chunkLimit the HW tracer after reading every chunk  (default=off)
--skip_ownershipSkip taking ownership  (default=off)

Mode: ParseMode

-i, --input=TrcFile

Input file (default=`mlxtrace.trc')


Enable csv output format (default=off)


Print timestamp events (default=off)

-r, --real_ts

Print real timestamps in [hh:mm:ss.nsec] format (default=off)


Print event bytes in each line header (default=off)


Choose printed TS format hex/dec (possible values="hex", "dec" default=`dec')


Enable printing delta between events (in cycles) (default=off)

-f, --print_file=FilePath

Print parsed event to the given file and not to stdout


Enable events DB checks (default=off)


Choose the suitable .cfg file depending on the device you are using, and run the following command to generate a .trc file:

# mlxtrace -d /dev/mst/mt4099_pci_cr0 -c connectx3.cfg -m MEM -o connectx3.trc

To generate a .trc file with a maximal size of 100 MB, run the following command:

# mlxtrace -d /dev/mst/mt4099_pci_cr0 -c connectx3.cfg -m MEM -s 100 -o connectx3.trc