VPU Debugging with LLDB (NVVPU Toolchain)#

This tutorial walks through debugging VPU programs compiled with the NVVPU toolchain using LLDB, on target hardware and in EMU. Two debugger clients are covered:

  • Command-line — an interactive lldb session

  • GUI — VS Code debugger connected via the Debug Adapter Protocol (lldb-dap)

For an overview of how VPU debugging works and the supported platforms, see VPU Debugging.

Prerequisites#

  1. The PVA SDK and NVVPU installed as described in Installation.

  2. A cuPVA application built with NVVPU for EMU or L4T.

  3. The path to the VPU .elf file you want to debug. The file is in the application’s build directory.

  4. For hardware, SSH access from the development host to the target board (passwordless or via the DOIP_USER_PASSWORD environment variable).

Inserting a Software Breakpoint#

On hardware, the VPU program must be halted before the debugger connects. Insert a swbrk() call at the point you want to begin debugging:

#define CUPVA_ALLOW_NONSAFETY 1
#include <cupva_device_debug.h>

CUPVA_VPU_MAIN()
{
    // ... setup code ...

    swbrk();   // VPU halts here; LLDB connects while it is stopped
    // ... code to debug ...
}

Rebuild and deploy the updated binary to the target. EMU waits for LLDB at the beginning of a debug-enabled run, so it does not require an initial swbrk(). You can still use swbrk() for later stopping points.

Command-Line Debugging on Hardware#

  1. Start the host application on the target, e.g.:

    ssh ubuntu@<target-ip> "nohup ./my_cupva_application >/tmp/my_cupva_application.log 2>&1 &"

    The application runs until the VPU hits swbrk() and stalls.

  2. Launch LLDB through the DoIP utility on the host:

    pva-sdk-2.10-doip --nvvpu --target-ip=<target-ip>

    The utility automatically:

    • Detects the target OS.

    • Detects the board generation (Orin/GEN2 or Thor/GEN3).

    • Copies the appropriate lldb_rsp_server_* binary to /tmp on the target.

    • Starts the RSP server on the target (with sudo on L4T).

    • Waits for the server to start listening.

    • Launches an interactive lldb session connected to the target.

    Optional flags:

    --target-ip=<ip>      IP address of the target (default: 192.168.55.1)
--target-port=<port>  TCP port for the RSP connection (default: 2345)
--target-ssh-port=<port>
                      SSH TCP port (default: 22)
--target-user=<user>  SSH username (default: ubuntu)
--vpu-core=<core>     VPU core to debug (0 or 1; default: 0)
--rsp-verbosity=<n>   RSP server verbosity (0-5; default: 0)

    The board generation normally requires no option. Supply --pva-version=GEN2 or --pva-version=GEN3 to select it explicitly when board detection is unavailable or to override the detected generation; the utility warns if the explicit value does not match.

  3. Debug in lldb. Once connected, the VPU process is displayed as stopped. Useful commands:

    (lldb) bt                      # backtrace
(lldb) frame select 0          # select innermost frame
(lldb) list                    # show source around current PC
(lldb) p my_variable           # print a variable
(lldb) register read           # dump scalar registers
(lldb) n                       # step over (next)
(lldb) s                       # step into
(lldb) c                       # continue
(lldb) br set -f file.c -l 42  # set a breakpoint at file.c line 42
(lldb) quit                    # exit (RSP server is cleaned up automatically)

Command-Line Debugging with EMU#

EMU contains the RSP server in the emulated VPU runtime. Enable it when starting the host application by setting PVA_EMU_DEBUG_PORT and select the VPU engine with PVA_EMU_DEBUG_ENGINE:

PVA_GEN=GEN2 \
PVA_EMU_DEBUG_PORT=2345 \
PVA_EMU_DEBUG_ENGINE=0 \
./my_host_application

The selected VPU engine waits at the beginning of its run until LLDB connects. Do not run pva-sdk-2.10-doip for EMU. In another terminal, connect the LLDB provided by the PVA SDK directly to the local RSP port:

/opt/nvidia/pva-sdk-2.10/tools/nvvpu/bin/lldb ./my_vpu_app__gen2.elf \
    -o "gdb-remote localhost:2345"

Use PVA_GEN=GEN3 with the matching __gen3.elf for GEN3. If PVA_EMU_DEBUG_PORT=0 is used, EMU selects a free port and prints it; use that port in the LLDB command. The RSP server listens on loopback by default.

Terminal UI Debugging#

The bundled LLDB also provides its curses-based terminal UI. After connecting to a hardware or EMU VPU as described above, enter it from the LLDB prompt:

(lldb) gui

The terminal UI displays the source or disassembly, variables, threads, and process state in separate panes. Press Tab to move between panes, h for the active pane’s key bindings, and Esc to return to the LLDB prompt.

The nvvpu Debian package recommends the required ncurses runtime. Ensure libncurses6 is installed if apt recommendations are disabled. The gui command must be launched from an interactive terminal with a valid TERM value. LLDB’s standard command interpreter and batch mode continue to support redirected or non-terminal input.

GUI Debugging with VS Code#

VS Code can be connected to the same RSP server via lldb-dap (the LLDB Debug Adapter Protocol bridge). This gives you a full graphical debugging experience: source and disassembly views, variable inspector, watch expressions, breakpoints, and call stack.

Step 1 — Install and Configure LLDB DAP#

Install the official LLDB DAP extension from the VS Code Marketplace. Its extension ID is llvm-vs-code-extensions.lldb-dap. For example:

code --install-extension llvm-vs-code-extensions.lldb-dap

Then open the VS Code user or workspace settings.json and make the extension use the lldb-dap supplied by this PVA SDK:

{
    "lldb-dap.executable-path": "/opt/nvidia/pva-sdk-2.10/tools/nvvpu/bin/lldb-dap"
}

This extension and the packaged executable above are the supported configuration. Do not use CodeLLDB or another LLDB installation, because it may not contain NVVPU target support.

Step 2 — Start the Host Application#

For hardware, start the application that loads the VPU binary and leaves the VPU halted at swbrk(), as described in the hardware command-line workflow. For EMU, start the application with PVA_EMU_DEBUG_PORT as described in the EMU command-line workflow. Keep the application running while debugging.

Step 3 — Configure Hardware DoIP as a VS Code Task#

This step applies only to hardware. VS Code can start the DoIP utility automatically before lldb-dap attaches. Create .vscode/tasks.json with a background task:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Start NVVPU DoIP",
            "type": "shell",
            "command": "pva-sdk-2.10-doip",
            "args": [
                "--nvvpu",
                "--skip-lldb",
                "--target-ip=<target-ip>"
            ],
            "isBackground": true,
            "problemMatcher": {
                "pattern": {
                    "regexp": ".*"
                },
                "background": {
                    "activeOnStart": true,
                    "beginsPattern": ".*starting RSP server on target.*",
                    "endsPattern": ".*RSP server started on .*Waiting for an external LLDB client.*"
                }
            },
            "presentation": {
                "reveal": "always",
                "panel": "dedicated"
            }
        }
    ]
}

Replace <target-ip> with the board’s IP. The background matcher lets VS Code wait until the utility has detected the board and OS, deployed the matching server, and reported that RSP is ready. The utility remains active during debugging and exits when the RSP connection closes. If debugging is cancelled before LLDB connects, terminate the task from Terminal → Terminate Task. If the target SSH connection is forwarded through a non-default local port, add --target-ssh-port=<port> to the task arguments.

EMU does not need this task because VS Code connects directly to the RSP server embedded in EMU.

Step 4 — Configure VS Code launch.json#

Create or update .vscode/launch.json in your workspace. The hardware configuration uses preLaunchTask to start the task above; the EMU configuration attaches directly:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "NVVPU Debug (Hardware)",
            "type": "lldb-dap",
            "request": "attach",
            "program": "${workspaceFolder}/build_l4t/tests/debugger/my_vpu_app__gen2.elf",
            "sourcePath": "${workspaceFolder}/path/to/vpu/source",
            "stopOnEntry": true,
            "preLaunchTask": "Start NVVPU DoIP",
            "gdb-remote-hostname": "<target-ip>",
            "gdb-remote-port": 2345
        },
        {
            "name": "NVVPU Debug (EMU)",
            "type": "lldb-dap",
            "request": "attach",
            "program": "${workspaceFolder}/build_emu/tests/debugger/my_vpu_app__gen2.elf",
            "sourcePath": "${workspaceFolder}/path/to/vpu/source",
            "stopOnEntry": true,
            "gdb-remote-hostname": "localhost",
            "gdb-remote-port": 2345
        }
    ]
}

Replace <target-ip> with the board’s IP in both tasks.json and launch.json. Adjust each program entry to the unstripped NVVPU ELF that corresponds to the application build. Set sourcePath to the directory containing the VPU sources. This maps relative source paths in the ELF’s debug information to absolute workspace paths so VS Code can open source files and set source breakpoints. For a non-default port, add --target-port=<port> to the hardware task or set PVA_EMU_DEBUG_PORT=<port> for EMU, then use the same gdb-remote-port value. Keep stopOnEntry enabled so that LLDB DAP does not resume the VPU during configuration before VS Code displays its initial stopped state.

Step 5 — Start the Debug Session#

Press F5 (or Run → Start Debugging) in VS Code. The debugger connects to the RSP server. On hardware, the VPU remains at the swbrk() breakpoint; EMU initially remains at its entry point.

From here you can:

  • View source — the call stack shows the current VPU frame with source.

  • Inspect variables — hover over variables or use the Variables pane.

  • Set breakpoints — click in the editor gutter or use the Breakpoints pane.

  • Step — use F10 (step over), F11 (step into), F5 (continue).

  • View disassembly — right-click the active stack frame or source editor and select Open Disassembly View. The view supports instruction stepping and instruction breakpoints.

  • Evaluate expressions — use the Debug Console.

Troubleshooting#

lldb_rsp_server fails to start

Check /tmp/lldb_rsp_server_<port>.log on the target, where <port> is 2345 by default. Common causes:

  • The VPU OCD debug filesystem is not mounted (check /sys/kernel/debug/pva0/).

  • The RSP server requires sudo on L4T; ensure the SSH user has sudo access.

  • Port 2345 is already in use — use --target-port=<other_port>.

lldb shows “connection refused”

On hardware, keep pva-sdk-2.10-doip running for the whole debug session. The utility reports an error and target diagnostics if its server does not begin listening. On EMU, verify that the host application was started with PVA_EMU_DEBUG_PORT and is still running.

lldb cannot find source files

Set the source map in lldb to remap build paths to your local workspace:

(lldb) settings set target.source-map /build/path /local/path

VS Code shows “Failed to attach”

For hardware, ensure pva-sdk-2.10-doip --nvvpu --skip-lldb is running before starting the VS Code debug session. For EMU, ensure the host application is waiting on the port in PVA_EMU_DEBUG_PORT. Also verify that lldb-dap.executable-path names the executable from this PVA SDK and that the launch configuration uses "type": "lldb-dap".