itrace Utility

NVIDIA ConnectX-5 Adapter Cards Firmware Release Notes v16.35.3502 LTS

The itrace utility extracts and prints trace messages generated by the firmware of a ConnectX-2/ ConnectX-3/ConnectX-3 Pro adapter cards. These trace messages inform developers of software drivers about internal status, events, critical errors, etc., for each iRISC. Trace messages generated by iRISCs are stored in the trace buffer. The trace buffer is located in host memory for MemFree adapter cards (i.e., without on-board memory), and in adapter memory for adapter cards with on-board memory.

The utility is a command line application controlled by command line parameters. It prints trace messages in text format to the console.

In order to print the firmware traces, the following are required:

  • Debug firmware is burnt and loaded on the device

  • The driver is up, meaning:

    • For adapters with on-board memory: The SYS_ENABLE command has been executed

    • For adapters without on-board memory (MemFree): The RUN_FW command has been executed

  • The desired trace mask is set (see the -m flag below)

The mst driver must be started prior to running itrace tool. To start itrace:

  1. Start the mst driver (mst start or mst restart).

  2. Enter the following command:


    # itrace [options...] IRISC_NAME



The iRISC for which traces are to be printed. This can be specified once anywhere in the command line as a special option without the leading hyphen. Run ‘itrace -h’ to get a list of iRISC names for each adapter device.

-h, --help

Displays help about itrace usage.

-m --mask=TRACE_MASK

Sets the Trace Mask.

To enable generating trace messages for an iRISC, the trace_mask register must be set according to the specifications in the device’s Programmer’s Reference Manual. Setting or clearing bits of the trace_mask register enables or disables, respectively, the generation of specific types of trace messages.

The trace_mask parameter must be either a hexadecimal or a decimal number and its value will be written into the trace_mask register. Changing the trace_mask parameter will not change or remove messages previously stored in the trace buffer, so disabled types of messages can still be displayed by itrace if they were previously generated.

-w, --wait

Runs itrace in wait mode. itrace will exit only if you press <Ctrl-C>. This is not the default behavior of itrace. Without the -w option, itrace will exit if there have been no
new traces in the last 0.5 seconds.

-d, --device=DEVICE

Specifies the name of the mst device driver for accessing the cr-space. The default value is:/dev/mst/mt4099_pci_cr0.
To run itrace via the I2C interface, use this option to specify the following:
-d=DEVICE, where the device is an I2C device (such as mtusb-1)

-l, --nolock

Ignore NSI GW lock


Sets the itrace to not access memory directly (via memory mapping) for reading the trace buffer, but to use the adapter memory access Gateway instead. By default, itrace
accesses the memory directly. If the cr-space device specified by the -d parameter is one of the I2C devices, -nomap is switched on.


Sets the itrace not to animate the propeller in wait mode (-w option). By default, animation is enabled.

-v, --version

Prints the MFT version and exits.

-c, --color

Enables color in trace output.

-D, --dump

Dumps the trace buffer and exits. This option is useful for debugging itrace; it dumps the contents of the trace buffer in row format.


Sets first message number to display.


Control trace. See: --help-debug.


Prints trace usage.


For Linux, device names should be listed with the /dev/mst prefix. For Windows, no prefix is required.



itrace -d /dev/mst/mt4099_pci_cr0 sx1 itrace: read memory (174712 bytes), each dot denotes 2048 bytes: [.....................................................................................] IRISC Trace Viewer (Mellanox ConnectX), mft 4.1.0-26, built on Aug 16 2015, 17:32:24. Git SHA Hash: dd3f359 FW Version: 2.34.8420 09/08/2015 19:44:8 (00000003 c1b59e4e) SCHD: exeqpc_valid2freed(0x0) vec_busy_valid=0x00000010 (00000004 dda895e4) SCHD: SQP:0x000400 exes_super_scheduler:busy_done (00000005 dda89760) SCHD: writing QpState SQPSTATE_GOOD_IDLE!!!! (00000006 dda89868) SCHD: exeqpc_valid2freed(0x0) vec_busy_valid=0x00000010 (00000007 dda97ccf) SCHD: SQP:0x000400 exes_super_scheduler:busy_ (00000008 dda97e47) SCHD: writing QpState SQPSTATE_GOOD_IDLE!!!! (00000009 dda97f4f) SCHD: exeqpc_valid2freed(0x0) vec_busy_valid=0x00000010 (0000000a dda9a8f6) SCHD: SQP:0x000400 exes_super_scheduler:busy (0000000b dda9aa6e) SCHD: writing QpState SQPSTATE_GOOD_IDLE!!!! (0000000c dda9ab79) SCHD: exeqpc_valid2freed(0x0) vec_busy_valid=0x00000010 (0000000d ddaaadc1) SCHD: SQP:0x000400 exes_super_scheduler:busy_ (00000029 ddaee521) INFO: IPCdata[00]=0x01abcd0a(0000002a ddaee60c) INFO: IPCdata[01]=0x00000014 (0000002b ddaee8ce) MAD: exes_mad: QPN=0x000000, nda_nds=0x7c58d014 (0000002c ddaee9f2) SCHD: SQP:0x000000 sqpc_access_db_algorithm: INC (0000002d ddaef0d5) SCHD: exes_scheduler: try to insert (0000002e ddaef2d9) SCHD: SQP:0x000000 exes_scheduler chosen (0000002f ddaef6aa) SCHD: EXES_GO(0x0)..

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.