Debugging the Kernel on Jetson Devices

Applies to: Jetson TX2 series devices only
This topic describes how to use the Linux kernel source-level debugger (KGDB) for debugging the Linux kernel on NVIDIA® Jetson™ TX2 series devices.
Detailed information about KGDB is available at:
Using KGDB requires a host system with a serial connection to the target device to be debugged. This allows live source-level debugging of kernel operations, setting breakpoints, and tracing errors back through function calls.
You cannot use KGDB to trace system calls and other types of processes. Use a tracer for that type of debugging.

Reconfiguring and Rebuilding the Kernel

Before you use KDGB you must determine whether the KDGB feature is enabled in the kernel. If it is not enabled, you must reconfigure and rebuild the kernel to connect to GDB and use a serial port terminal for debugging.
Note:
To debug a device with a Jetson  processor, NVIDIA recommends using GDB 8.0 to connect to KGDB. GDB 7.0 has been used in the past but has not been tested with the current release.
To determine whether the kernel configuration must be modified
1. Execute the command:
$ zcat /proc/config.gz | egrep 'CONFIG_KGDB|KDGB|CONFIG_FRAME_POINTER|CONFIG_WATCHDOG|IKCONFIG'
If your kernel currently has all of the requisite macros enabled, the command displays exactly the following output:
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_WATCHDOG is not set
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
# CONFIG_KGDB_KDB is not set
CONFIG_KGDB_KDB=y
CONFIG_KDB_DEFAULT_ENABLE=0x1
CONFIG_KDB_KEYBOARD=y
CONFIG_KDB_CONTINUE_CATASTROPHIC=0
2. If the output indicates that /proc/config.gz does not exist, your kernel was not compiled with CONFIG_IKCONFIG_PROC set. In that case, recompile the kernel with CONFIG_IKCONFIG_PROC set.
General instructions for reconfiguring and rebuilding the Linux kernel for KGDB are available at:

Modifying the Kernel Configuration File

You may have to modify the kernel configuration file if any of the above-mentioned macros are not set correctly.
1. Select the appropriate configuration file for your target device.
For example, the configuration file for a T186 board is tegra_defconfig,
2. Set up the Linux build environment and execute these commands within the kernel directory:
$ cd $TEGRA_TOP/kernel/kernel-4.9
$ export ARCH=arm64
$ make DEFCONFIG_PATH=arch/arm64/configs tegra_defconfig
$ make menuconfig
<<Update config options>>
$ make savedefconfig
$ cp -v defconfig arch/arm64/configs/tegra_defconfig
$ make mrproper
3. Rebuild the kernel using the new configuration file by executing the command:
$ tmp kernel
4. Flash the new kernel on the target Jetson device to be debugged.

Using KGDB

To use KGDB, you must attach it to GDB. You must set up KGDB to be attached using one of two methods, depending on the state of the system:
At boot: Use the kernel command line.
At runtime with sysfs node: Use a system request trigger to KGDB to wait for a GDB connection.

Starting KDGB at Boot from the Kernel Command Line

A custom kernel boot command line for KGDB must specify:
The terminal device used for debugging
The baud rate of the serial connection between the host and the target device
The kgdbwait command
To start KGDB at boot
1. Check whether the console parameter is set for using KGDB at boot in the kernel command line.
2. If the console parameter is not set, add the following arguments to the kernel command line:
console=<device>,<bpc> kgdboc=<device>,<baudrate> kgdbwait
Where:
<device> is the serial device for debugging, e.g. ttyS0.
<bpc> specifies the baud rate, parity, and character length of the serial connection between the host and the target device, e.g. 115200n8.
<baudrate> is the baud rate of the serial connection, e.g. 115200.
This is a specific example of console parameters:
console=ttyS0,115200n8 kgdboc=ttyS0,115200 kgdbwait
 
Note:
Verify that the baud rate is the same for the host and target device. You can use any type of connection that can work like an RS-232 serial connection, including connection through a USB bridge cable or a network connection that can tunnel serial data.
Determine which host serial port or USB port receives boot messages from the device. Use the kgdboc parameter to direct the GDB target device to the same port.
3. Once the kernel command line is correctly set, there are two ways you can connect to GDB:
If the CONFIG_KDB* macros are enabled in defconfig, the kernel waits for KGDB during boot. It opens a terminal window showing the KDB console and displays these messages:
[ 3.600430] console [ttyS0] enabled
[ 3.604746] KGDB: Registered I/O driver kgdboc
[ 3.626509] KGDB: Waiting for connection from remote gdb...
Entering kdb (current=0xffffffc1ecb30000, pid 1) on processor 2 due to Keyboard Entry
[2]kdb>
This ends the procedure. You can now debug through the console by entering standard KDB commands at the prompt.
If the CONFIG_KDB* macros are not enabled in defconfig, the kernel displays these messages and waits for a connection from the remote GDB:
[ 2.368148] console [ttyS0] enabled
[ 2.368148] console [ttyS0] enabled
[ 2.375339] bootconsole [uart0] disabled
[ 2.375339] bootconsole [uart0] disabled
[ 2.383940] KGDB: Registered I/O driver kgdboc
[ 2.402324] KGDB: Waiting for connection from remote gdb...
Go on to the next step.
4. Perform the remaining steps only if the kernel is waiting for a connection from KGDB, i.e., the CONFIG_KDB* macros are not enabled.
If you are using same serial line for console and debugging, you must disconnect the terminal application for the console (for example, minicom) at this point.
5. Start GDB in the usual way, by executing these commands on host:
$ <path_to_toolchain>/linaro/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gdb $TEGRA_TOP/out/l4t-t186ref-debug-aarch64/nvidia/kernel/vmlinux
For example:
(gdb) target remote /dev/ttyUSB2
 
6. To connect GDB, enter this command after the gdb debugger is started:
(gdb) target remote /dev/<tty_device>
Where <tty_device> is the host output device configured for GDB.
For example:
(gdb) target remote /dev/ttyUSB2
In this example, the terminal is connected through a USB port at /dev/ttyUSB2.
To connect through a standard serial port, replace the USB device number with a serial port number such as /dev/ttyS0. Typically you would use the same port you use to receive boot messages.
To start KDGB at runtime with sysfs node
You can also start KDGB after the target system has started, as described in this section.
1. To trigger KGDB to wait for GDB connection on a running system:
1. Open a terminal window.
2. Enable the KGDB console on the configured terminal device at the configured baud rate.
On the target, enter the command:
$ sudo su
# echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
 
Note:
This sets the target device to use ttyS0. On the host, you must identify the serial port or USB port on which you are receiving boot messages and use the same port for your GDB host.
If you were already connected to the serial console with minicom or screen, the kernel displays this:
[ 60.206779] kgdb: Registered I/O driver kgdboc.
2. To signal KGDB to break and use KGDB:
1. Enter the command:
# echo -n g > /proc/sysrq-trigger
If the CONFIG_KDB* macros are enabled in defconfig, the command displays this:
[ 54.299912] sysrq: SysRq : DEBUG
Entering kdb (current=0xffffffc1e9259c00, pid 4362) on processor 2 due to Keyboard Entry
[2]kdb> help
This ends the procedure. You can now debug through the console by entering standard KDB commands at the prompt.
If the CONFIG_KDB* macros are not enabled in defconfig, the command displays this:
[ 68.398684] sysrq: SysRq : DEBUG
[ 68.402103] KGDB: Entering KGDB
Go on to the next step.
2. Perform the remaining steps only if the kernel is waiting for a connection from KGDB, i.e., the CONFIG_KDB* macros are not enabled.
On the host, execute the command:
$ sudo `which <gdb_name>` $TEGRA_TOP/out/l4t-t186ref-debug-aarch64/nvidia/kernel/vmlinux
$ set remotebaud 115200
$ target remote /dev/<tty_device>
Where <tty_device> is the host output device configured for GDB.
For example:
$ (gdb) target remote /dev/ttyUSB2
In this example, the terminal is connected through the USB port /dev/ttyUSB2.
To connect through a standard serial port, replace the USB device number with a serial port number such as /dev/ttyS0. Typically you would use the same port you use to receive boot messages.
To save typing each time you connect GDB to KGDB, define a dbgattach alias for them in a .gdbinit file like this:
define dbgattach
echo set remotebaud 115200 \n
set remotebaud 115200
echo target remote /dev/<tty_device> \n
target remote /dev/<tty_device>
end