Program Trace Macrocell

Applies to: Jetson Nano devices, Jetson TX2 series, and Jetson TX1 only
A Program Trace Macrocell (PTM) is a real-time trace module providing instruction tracing of an ARM® core. It traces way-points in:
All indirect branches conditional
Unconditional direct branches
All exceptions
PTM also generates processing elements. This way the PTM module is triggered when there is a deviation from the normal code flow. The PTM v4 and protocols use waypoints and correlate the trace to the code image, vmlinux. These PTM packets can be decoded later using mem_parser to obtain the complete CPU instruction flow.
PTM is advantageous may be necessary for:
Debugging - On many production systems it is not possible to connect JTAG devices. In these cases you can use PTM to check which instructions ran before the CPU crashed.
Code Profiling - Because there is no Watchdog Timer (WDT) reset, code profiling checks are possible, provided a start and end address.
Performance Analysis - PTM has no performance overhead as compared with ftrace. PTM can also provide instruction counts and timestamps that can be used for performance analysis.
PTM is a component of the ARM CoreSight framework. Detailed information about the framework is available at:
https://www.kernel.org/doc/Documentation/trace/coresight.txt
To configure PTM drivers
1. Verify that the CoreSight drivers have been enabled in the kernel as follows:
# zcat /proc/config.gz | grep -i coresight | grep "=y"
CONFIG_CORESIGHT=y
CONFIG_CORESIGHT_LINKS_AND_SINKS=y
CONFIG_CORESIGHT_LINK_AND_SINK_TMC=y
CONFIG_CORESIGHT_SINK_TPIU=y
CONFIG_CORESIGHT_SINK_ETBV10=y
CONFIG_CORESIGHT_SOURCE_ETM4X=y
2. Verify that the devices defined in the DTB exist in:
/sys/bus/coresight/devices/.
3. Execute the commands:
# ll /sys/bus/coresight/devices/ | cut -d " " -f10,11,12
8010000.funnel_major -> ../../../devices/8010000.funnel_major/8010000.funnel_major/
8030000.etf -> ../../../devices/8030000.etf/8030000.etf/
8040000.replicator -> ../../../devices/8040000.replicator/8040000.replicator/
8050000.etr -> ../../../devices/8050000.etr/8050000.etr/
8060000.tpiu -> ../../../devices/8060000.tpiu/8060000.tpiu/
8820000.funnel_minor -> ../../../devices/8820000.funnel_minor/8820000.funnel_minor/
9010000.funnel_bccplex -> ../../../devices/9010000.funnel_bccplex/9010000.funnel_bccplex/
9840000.ptm -> ../../../devices/9840000.ptm/9840000.ptm/
9940000.ptm -> ../../../devices/9940000.ptm/9940000.ptm/
9a40000.ptm -> ../../../devices/9a40000.ptm/9a40000.ptm/
9b40000.ptm -> ../../../devices/9b40000.ptm/9b40000.ptm/

Using the PTM

You can enable and disable trace capture at runtime. The content of the ETF/ETR on Jetson TX2 or Jetson TX1 devices, during the period that trace capture is enabled, can be read back to parse.
Note:
PTM is not supported for Denver cores.
To collect PTM trace on A57 Core 0
On the target, execute the command:
# echo 1 > /sys/bus/coresight/devices/8030000.etf/enable_sink
# echo 1 > /sys/bus/coresight/devices/9840000.ptm/enable_source
[ 246.479271] coresight-tmc 8030000.etf: TMC enabled
[ 246.484362] coresight-funnel 8010000.funnel_major: FUNNEL inport 0 enabled
[ 246.491890] coresight-funnel 9010000.funnel_bccplex: FUNNEL inport 1 enabled
[ 246.499761] coresight-etm4x 9840000.ptm: ETM tracing enabled
# cat /sys/bus/coresight/devices/8030000.etf/status
Depth: 0x2000
Status: 0x1
RAM read ptr: 0x5b60
RAM wrt ptr: 0x5b60
Trigger cnt: 0x0
Control: 0x1
Flush status: 0x0
Flush ctrl: 0x133
Mode: 0x0
PSRC: 0x0
DEVID: 0x580
# echo 0 > /sys/bus/coresight/devices//9840000.ptm/enable_source
[ 264.323280] coresight-etm4x 9840000.ptm: ETM tracing disabled
[ 264.329400] coresight-funnel 9010000.funnel_bccplex: FUNNEL inport 1 disabled
[ 264.336998] coresight-funnel 8010000.funnel_major: FUNNEL inport 0 disabled
[ 264.360311] coresight-tmc 8030000.etf: TMC disabled
# dd if=/dev/8030000.etf of=~/cstrace-t186.bin
On the host, execute the command:
$ ./mem_parser --formatter --etb <dumped_file_with_trace buffer> --elf <vmlinux_with_path> > <output_file_with_instructions>
For example:
$ ./mem_parser --formatter --etb ~/cstrace-t186.bin --elf $TEGRA_TOP/out/l4t-t186ref-debug-aarch64/nvidia/kernel/vmlinux > ./cstrace-t186.disassemble.txt
To collect PTM trace on other three A57 cores:
The correspond commands for other 3 cores in A57 are as follows.
Core 1 on A57:
echo 1 > /sys/bus/coresight/devices/8030000.etf/enable_sink
echo 1 > /sys/bus/coresight/devices/9940000.ptm/mode [set EXC bit i.e. bit 24 in ETMTECR1 to trace all memory. Refer *etm*architecture*.pdf, ignore this if you don't know the result.]
echo 1 > /sys/bus/coresight/devices/9940000.ptm/enable_source (enable tracing)
cat /sys/bus/coresight/devices/8030000.etf/status [Will show RAM wrt ptr moving]
echo 0 > /sys/bus/coresight/devices//9940000.ptm/enable_source (disable tracing)
Core 2 on A57:
echo 1 > /sys/bus/coresight/devices/8030000.etf/enable_sink
echo 1 > /sys/bus/coresight/devices/9a40000.ptm/mode [set EXC bit i.e. bit 24 in ETMTECR1 to trace all memory. Refer *etm*architecture*.pdf, ignore this if you don't know the result.]
echo 1 > /sys/bus/coresight/devices/9a40000.ptm/enable_source (enable tracing)
cat /sys/bus/coresight/devices/8030000.etf/status [Will show RAM wrt ptr moving]
echo 0 > /sys/bus/coresight/devices//9a40000.ptm/enable_source (disable tracing)
Core 3 on A57:
echo 1 > /sys/bus/coresight/devices/8030000.etf/enable_sink
echo 1 > /sys/bus/coresight/devices/9b40000.ptm/mode [set EXC bit i.e. bit 24 in ETMTECR1 to trace all memory. Refer *etm*architecture*.pdf, ignore this if you don't know the result.]
echo 1 > /sys/bus/coresight/devices/9b40000.ptm/enable_source (enable tracing)
cat /sys/bus/coresight/devices/8030000.etf/status [Will show RAM wrt ptr moving]
echo 0 > /sys/bus/coresight/devices//9b40000.ptm/enable_source (disable tracing)

Decode Trace Example

The following example is for mem_parser Version 2.0.9
Parsing Pogram Trace Macrocell buffer (ETM4.x) ......
etb buffer size: 32768
>>>>>>>>>>>>>>>>>>>>>>>>>>> CPU: ID = 0x20 start:
etb buffer size: 30404
..........
..........
ATOM packet format 2:
ATOMS: N, E,
<0fc7:da >
<__do_softirq+0x130>
ffffffc0000a8b98: 910022b5
ffffffc0000a8b9c: 1ad326d6
ffffffc0000a8ba0: 5ac002d3
ffffffc0000a8ba4: 6b1f02df
ffffffc0000a8ba8: 5ac01273
ffffffc0000a8bac: 1a9307f3
E ffffffc0000a8bb0: 35fffb73 cbnz x19, ffffffc0000a8b1c <__do_softirq+0xb4>
<__do_softirq+0xb4>
ffffffc0000a8b1c: 928000e0
ffffffc0000a8b20: 8b33cc00
ffffffc0000a8b24: 8b0002b5
ffffffc0000a8b28: 91080357<__do_softirq+0xb4>