Debugging the Kernel¶

This topic describes how to use the Linux kernel source-level debugger (KGDB) for debugging the Linux kernel on NVIDIA^® Jetson™ series devices.

Detailed information about KGDB is available on the page Using kgdb, kdb and the kernel debugger internals in the Linux Kernel Archives.

Using KGDB requires a host system with a serial connection to the target device to be debugged. This allows live source-level debugging of kernel operations, setting breakpoints, and tracing errors back through function calls.

You cannot use KGDB to trace system calls and other types of processes. Use a tracer for that type of debugging.

Reconfiguring and Rebuilding the Kernel¶

Before you use KDGB you must determine whether the KDGB feature is enabled in the kernel. If it is not enabled, you must reconfigure and rebuild the kernel to connect to GDB and use a serial port terminal for debugging.

To determine whether the kernel configuration must be modified¶

Execute the command:

$ zcat /proc/config.gz | egrep 'CONFIG_KGDB|KDGB|CONFIG_FRAME_POINTER|CONFIG_WATCHDOG|IKCONFIG'

If your kernel currently has all of the requisite macros enabled, the command displays the similar output:

CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_WATCHDOG_NOWAYOUT=y
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
CONFIG_WATCHDOG_OPEN_TIMEOUT=0
# CONFIG_WATCHDOG_SYSFS is not set
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_HONOUR_BLOCKLIST=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
CONFIG_KGDB_KDB=y
CONFIG_KDB_DEFAULT_ENABLE=0x1
CONFIG_KDB_KEYBOARD=y
CONFIG_KDB_CONTINUE_CATASTROPHIC=0

If the output indicates that /proc/config.gz does not exist, you must recompile the kernel with CONFIG_IKCONFIG_PROC set.

General instructions for reconfiguring and rebuilding the Linux kernel for KGDB are available on the page Compiling a kernel in the Linux Kernel Archives.

Modifying the Kernel Configuration File¶

You must modify the kernel configuration file if any of the macros described above are not set correctly.

Go to Kernel Customization for information about downloading the kernel source and building the kernel.
Select the appropriate configuration file for your target device.

For example, the configuration file for a T234 board is tegra_defconfig.
Find tegra_defconfig file inside kernel source directory and enable the missing configs required for KGDB as mentioned above.
Build kernel.
Flash the new kernel on the target Jetson device to be debugged.

Using KGDB¶

You must set up KGDB to be attached using one of two methods, depending on the state of the system:

At boot: Use the kernel command line.
At runtime with sysfs node: Use a system request trigger to KGDB to wait for a GDB connection.

Starting KDGB at Boot from the Kernel Command Line¶

A custom kernel boot command line for KGDB must specify:

The terminal device used for debugging
The baud rate of the serial connection between the host and the target device
The kgdbwait command

To start KGDB at boot¶

Check whether the console parameter is set for using KGDB at boot in the kernel command line.
If the console parameter is not set, add the following arguments to the kernel command line:
```
console=<device>,<bpc> kgdboc=<device>,<baudrate> kgdbwait
```
Where:
- <device> is the serial device for debugging, e.g. on Tegra234: ttyAMA0.
- <bpc> specifies the baud rate, parity, and character length of the serial connection between the host and the target device, e.g. 115200n8.
- <baudrate> is the baud rate of the serial connection, e.g. 115200.
This is a specific example of console parameters:
```
console=ttyTCU0,115200 console=ttyAMA0,115200n8 kgdboc=ttyAMA0,115200 kgdbwait
```
Note

Verify that the baud rate is the same for the host and target device. You can use any type of connection that can work like an RS-232 serial connection, including connection through a USB bridge cable or a network connection that can tunnel serial data.

Determine which host serial port or USB port receives boot messages from the device. Use the kgdboc parameter to direct the GDB target device to the same port.
Once the kernel command line is correctly set, there are two ways you can connect to GDB:
- If the CONFIG_KDB* macros are enabled in tegra_defconfig, the kernel waits for KGDB during boot. It opens a terminal window showing the KDB console and displays these messages:
```
[ 2.368148] console [ttyAMA0] enabled
[ 2.368148] console [ttyTCU0] enabled
[ 3.604746] KGDB: Registered I/O driver kgdboc
[ 3.626509] KGDB: Waiting for connection from remote gdb...
Entering kdb (current=0xffffffc1ecb30000, pid 1) on processor 2 due to Keyboard Entry
[2]kdb>
```
  This ends the procedure. You can now debug through the console by entering standard KDB commands at the prompt.
- If the CONFIG_KDB* macros are not enabled in tegra_defconfig, the kernel displays these messages and waits for a connection from the remote GDB:
```
[ 2.368148] console [ttyAMA0] enabled
[ 2.368148] console [ttyTCU0] enabled
[ 2.383940] KGDB: Registered I/O driver kgdboc
[ 2.402324] KGDB: Waiting for connection from remote gdb...
```
  Go on to the next step.
Perform the remaining steps only if the kernel is waiting for a connection from KGDB, i.e., the CONFIG_KDB* macros are not enabled.

If you are using same serial line for console and debugging, you must disconnect the terminal application for the console (for example, minicom) at this point.

Start GDB in the usual way, by executing these commands on host:

$ <path_to_toolchain>/bin/aarch64-linux-gnu-gdb <path_to_kernel_out_directory>vmlinux

To connect GDB, enter this command after the GDB debugger is started:
```
(gdb) target remote /dev/<tty_device>
```
Where <tty_device> is the host output device configured for GDB.

For example:
```
(gdb) target remote /dev/ttyACM1
```
In this example, the terminal is connected through a USB port at /dev/ttyACM1.

To connect through a standard serial port, replace the tty device number with a serial port number such as /dev/ttyS0. Typically you would use the same port you use to receive boot messages.

To start KDGB at runtime with sysfs node¶

You can also start KDGB after the target system has started, as described in this section.

To trigger KGDB to wait for GDB connection on a running system:
1. Open a terminal window.
2. Enable the KGDB console on the configured terminal device at the configured baud rate.
  
  On the target, enter the command:
```
$ sudo su
$ echo ttyAMA0,115200 > /sys/module/kgdboc/parameters/kgdboc
```
  Note
  
  This sets the target device to use ttyAMA0. On the host, you must identify the serial port or USB port on which you are receiving boot messages and use the same port for your GDB host.
  
  If you were already connected to the serial console with minicom or screen, the kernel displays this:
```
[ 60.206779] kgdb: Registered I/O driver kgdboc.
```
To signal KGDB to break and use KGDB:
1. Enter the command:
```
$ echo -n g > /proc/sysrq-trigger
```
  - If the CONFIG_KDB* macros are enabled in tegra_defconfig, the command displays this:
    [ 54.299912] sysrq: SysRq : DEBUG Entering kdb (current=0xffffffc1e9259c00, pid 4362) on processor 2 due to Keyboard Entry [2]kdb> help
    This ends the procedure. You can now debug through the console by entering standard KDB commands at the prompt.
  - If the CONFIG_KDB* macros are not enabled in tegra_defconfig, the command displays this:
    [ 68.398684] sysrq: SysRq : DEBUG [ 68.402103] KGDB: Entering KGDB
    Go on to the next step.
2. Perform the remaining steps only if the kernel is waiting for a connection from KGDB, i.e., the CONFIG_KDB* macros are not enabled.
  
  On the host, execute the command:
```
$ <path_to_toolchain>/bin/aarch64-linux-gnu-gdb <path_to_kernel_out_directory>vmlinux
$ set remotebaud 115200
$ target remote /dev/<tty_device>
```
  Where <tty_device> is the host output device configured for GDB.
  
  For example:
```
$ (gdb) target remote /dev/ttyACM1
```
  In this example, the terminal is connected through the USB port /dev/ttyACM1.
  
  To connect through a standard serial port, replace the USB device number with a serial port number such as /dev/ttyS0. Typically you would use the same port you use to receive boot messages.

To save typing each time you connect GDB to KGDB, define a dbgattach alias for them in a .gdbinit file like this:

define dbgattach
    echo set remotebaud 115200 \n
         set remotebaud 115200
    echo target remote /dev/<tty_device> \n
         target remote /dev/<tty_device>
end