mlxtrace Utility
The mlxtrace utility is used to configure and extract HW events generated by different units in NVIDIA devices. The utility generates a dump ".trc" file which contains HW events that assist us with debug, troubleshooting and performance analysis. Events can be stored in host memory if driver is up or in a small on-chip buffer (always available) depending on the utility running mode. In order to run the utility it's required to have a configuration file first, this file should be provided by the NVIDIA representative.
A dump file "mlxtrace.trc" will be generated by end of run (file name can be controlled by "-o" flag), this file should be sent to the NVIDIA representative for further diagnostics/troubleshooting.
Memory mode on 5th generation (Group II) devices is supported only by PCI mst devices. Memory mode is supported in Windows, as well as in Linux.
For the tool to properly work with Inband devices, both the MFT and the Firmware must be updated to the latest (MFT v4.18.0 & firmware vXX.32.1xxx).
If ConnectX-4 adapter card is used as an Inband device, for the tool to work properly, you need to use MFT 4.17.0.
The mst driver must be started prior to running the mlxtrace tool.
For MEM buffer mode driver must be "loaded" also.
Enter the following command:
mlxtrace [options]
Options
|
-h, --help |
Print help and exit |
|
-v, --version |
Print version (default=off) |
|
-p, --parse |
Move to parser mode (default=off) |
Mode: CollectMode
|
-d, --device=MstDev |
Mst device |
|
-m, --mode=Mode |
Activation mode: FIFO - HW BUFFER , MEM - KERNEL BUFFER (possible values="FIFO", "MEM") |
|
-a, --mem_access=MemMethod |
Memory access method: OB_GW, MEM, DMEM, FMEM, VMEM (possible values="OB_GW", "MEM", "DMEM", "FMEM", "VMEM") Note: As of MFT v4.21.0, the following values will be deprecated: MEM, DMEM, FMEM. |
|
--gvmi |
Global virtual machine interface |
|
-c, --cfg=CfgFile |
Mlxtrace configuration file |
|
-o, --trc_file=TrcPath |
Output TRC file path (default=`mlxtrace.trc') |
|
-C, --config_only |
Configure tracer and exit (default=off) |
|
-b, --backpressure |
Enable "Allow_backpressure" for all supported hardware events (default=off) |
|
-n, --snapshot |
Take events snapshot - This assumes previous run with -- config_only (default=off) |
|
-s, --buf_size=BufSize |
User buffer size [MB] (default=`1') |
|
-S, --stream |
Don't save events to file parse it immediately (default=off) |
|
--ignore_old_events |
Ignore collecting old events in MEM mode (default=off) |
|
-g, --continuous_fill |
Do not stop recording (stopping only with ^C), keep filling user's buffer cyclically (default=off) |
|
--sample_delay=Delay |
Delay between samples when polling new events in [usec] (default=`0') |
|
--keep_running |
Keep the HW tracer unit running after exit (default=off) |
|
--enable_limiting_every_chunk |
Limit the HW tracer after reading every chunk (default=off) |
|
--skip_ownership |
Skip taking ownership (default=off) |
Mode: ParseMode
|
-i, --input=TrcFile |
Input file (default=`mlxtrace.trc') |
|
--csv_mode |
Enable csv output format (default=off) |
|
--print_ts |
Print timestamp events (default=off) |
|
-r, --real_ts |
Print real timestamps in [hh:mm:ss.nsec] format (default=off) |
|
--print_raw |
Print event bytes in each line header (default=off) |
|
--ts_format=format |
Choose printed TS format hex/dec (possible values="hex", "dec" default=`dec') |
|
--print_delta |
Enable printing delta between events (in cycles) (default=off) |
|
-f, --print_file=FilePath |
Print parsed event to the given file and not to stdout |
|
--enable_db_check |
Enable events DB checks (default=off) |
Examples:
Choose the suitable .cfg file depending on the device you are using, and run the following command to generate a .trc file:
# mlxtrace -d /dev/mst/mt4099_pci_cr0 -c connectx3.cfg -m MEM -o connectx3.trc
To generate a .trc file with a maximal size of 100 MB, run the following command:
# mlxtrace -d /dev/mst/mt4099_pci_cr0 -c connectx3.cfg -m MEM -s 100 -o connectx3.trc