DocumentationThe Holoscan SDK is designed to streamline the debugging process for developers working on advanced applications.
This comprehensive guide covers the SDK’s debugging capabilities, with a focus on Visual Studio Code integration, and provides detailed instructions for various debugging scenarios.
It includes methods for debugging both the C++ and Python components of applications, utilizing tools like GDB, UCX, and Python-specific debuggers.
The Holoscan SDK can be effectively developed using Visual Studio Code, leveraging the capabilities of a development container. This container, defined in the .devcontainer folder, is pre-configured with all the necessary tools and libraries, as detailed in Visual Studio Code’s documentation on development containers.
./run vscode command to launch Visual Studio Code in a development container (-j <# of workers> or --parallel <# of workers> can be used to specify the number of parallel jobs to run during the build process). If Cursor is installed and available, it will be launched automatically instead of VSCode. For more information, refer to the instructions from ./run vscode -h../run vscode_remote. Additional instructions can be accessed via ./run vscode_remote -h.Upon launching Visual Studio Code (or Cursor), the development container will automatically be built. This process also involves the installation of recommended extensions and the configuration of CMake.
The ./run vscode command now supports multiple IDE options:
--ide <ide_name> to specify which IDE to use:
--ide vscode - Launch VSCode--ide vscode-insiders - Launch VSCode Insiders--ide cursor - Launch Cursor--code - Force launch VSCode--cursor - Force launch Cursor--cmd <path> to specify a custom IDE binary pathExamples:
For manual adjustments to the CMake configuration:
Ctrl + Shift + P).CMake: Configure command.To build the source code within the development container:
Ctrl + Shift + B.Ctrl + Shift + P) and run Tasks: Run Build Task.For debugging the source code:
Run and Debug view in VSCode (Ctrl + Shift + D).F5 to start the debugging session.The launch configurations are defined in .vscode/launch.json(link).
Please refer to Visual Studio Code’s documentation on debugging for more information.
The Holoscan SDK facilitates seamless debugging of both C++ and Python components within your applications. This is achieved through the integration of the Python C++ Debugger extension in Visual Studio Code, which can be found here.
This powerful extension is specifically designed to enable effective debugging of Python operators that are executed within the C++ runtime environment. Additionally, it provides robust capabilities for debugging C++ operators and various SDK components that are executed via the Python interpreter.
To utilize this feature, debug configurations for Python C++ Debug should be defined within the .vscode/launch.json file, available here.
Here’s how to get started:
examples/ping_vector/python/ping_vector.py.Run and Debug view of Visual Studio Code, select the Python C++ Debug debug configuration.F5.Upon starting the session, two separate debug terminals will be launched; one for Python and another for C++. In the C++ terminal, you will encounter a prompt regarding superuser access:
Respond with y to proceed.
Following this, the Python application initiates, and the C++ debugger attaches to the Python process. This setup allows you to simultaneously debug both Python and C++ code. The CALL STACK tab in the Run and Debug view will display Python: Debug Current File and (gdb) Attach, indicating active debugging sessions for both languages.
By leveraging this integrated debugging approach, developers can efficiently troubleshoot and enhance applications that utilize both Python and C++ components within the Holoscan SDK.
On interactive debugging, it may be desirable to set the VSCode “RUN AND DEBUG” menu’s “BREAKPOINTS” settings to uncheck the “Raised Exceptions” box. Otherwise, the debugger may break at an expected PackageNotFoundException handled by a try/except statement from the top-level holoscan __init__.py. If the debugger stops here, hitting “continue” will proceed to the application script of interest. By deselecting “Raise Exceptions” from the options, the debugger should not stop in the top-level __init__.py.
When performing interactive debugging within the __init__ method of a Python operator (inheriting from holoscan.core.Operator), the self variable cannot be inspected from a breakpoint in the debugger until after the parent class’s super().__init__ (Operator.__init__) method has been called. The class will be an in incomplete state until that time and inspecting it in the debugger (or trying to print via print(self)) will fail. The same restriction will also apply to the __init__ method of Python condition classes inheriting from holoscan.core.Condition or Python resource classes inheriting from holoscan.core.Resource.
This section outlines the procedures for debugging an application crash.
In the event of an application crash, you might encounter messages like Segmentation fault (core dumped) or Aborted (core dumped). These indicate the generation of a core dump file, which captures the application’s memory state at the time of the crash. This file can be utilized for debugging purposes.
There are instances where core dumps might be disabled or not generated despite an application crash.
To activate core dumps, it’s necessary to configure the ulimit setting, which determines the maximum size of core dump files. By default, ulimit is set to 0, effectively disabling core dumps. Setting ulimit to unlimited enables the generation of core dumps.
Additionally, configuring the core_pattern value is required. This value specifies the naming convention for the core dump file. To view the current core_pattern setting, execute the following command:
To modify the core_pattern value, execute the following command:
In this case, we have requested that both the executable name (%e) and the process id (%p) be present in the generated file’s name. The various options available are documented in the core documentation.
If you encounter errors like tee: /proc/sys/kernel/core_pattern: Read-only file system or sysctl: setting key "kernel.core_pattern", ignoring: Read-only file system within a Docker container, it’s advisable to set the kernel.core_pattern parameter on the host system instead of within the container.
As kernel.core_pattern is a system-wide kernel parameter, modifying it on the host should impact all containers. This method, however, necessitates appropriate permissions on the host machine.
Furthermore, when launching a Docker container using docker run, it’s often essential to include the --cap-add=SYS_PTRACE option to enable core dump creation inside the container. Core dump generation typically requires elevated privileges, which are not automatically available to Docker containers.
After the core dump file is generated, you can utilize GDB to debug the core dump file.
Consider a scenario where a segmentation fault is intentionally induced at line 29 in examples/ping_simple/cpp/ping_simple.cpp by adding the line *(int*)0 = 0; to trigger the fault.
Upon running ./examples/ping_simple/cpp/ping_simple, the following output is observed:
It’s apparent that the application has aborted and a core dump file has been generated.
The core dump file can be debugged using GDB by executing gdb <application> <coredump_file>.
It is evident that the application crashed at line 29 of examples/ping_simple/cpp/ping_simple.cpp.
To display the backtrace, the bt command can be executed.
In cases where a distributed application using the UCX library encounters a segmentation fault, you might see stack traces from UCX. This is a default configuration of the UCX library to output stack traces upon a segmentation fault. However, this behavior can be modified by setting the UCX_HANDLE_ERRORS environment variable:
UCX_HANDLE_ERRORS=bt prints a backtrace during a segmentation fault (default setting).UCX_HANDLE_ERRORS=debug attaches a debugger if a segmentation fault occurs.UCX_HANDLE_ERRORS=freeze freezes the application on a segmentation fault.UCX_HANDLE_ERRORS=freeze,bt both freezes the application and prints a backtrace upon a segmentation fault.UCX_HANDLE_ERRORS=none disables backtrace printing during a segmentation fault.While the default action is to print a backtrace on a segmentation fault, it may not always be helpful.
For instance, if a segmentation fault is intentionally caused at line 139 near the start of PingTensorTxOp::compute in /workspace/holoscan-sdk/src/operators/ping_tensor_tx/ping_tensor_tx.cpp (by adding *(int*)0 = 0;), running ./examples/ping_distributed/cpp/ping_distributed will result in the following output:
Although a backtrace is provided, it may not always be helpful as it often lacks source code information. To obtain detailed source code information, using a debugger is necessary.
By setting the UCX_HANDLE_ERRORS environment variable to freeze,bt and running ./examples/ping_distributed/cpp/ping_distributed, we can observe that the thread responsible for the segmentation fault is frozen, allowing us to attach a debugger to it for further investigation.
It is observed that the thread responsible for the segmentation fault is 51 (tid: 51). To attach a debugger to this thread, simply press Enter.
Upon attaching the debugger, a backtrace will be displayed, but it may not be from the thread that triggered the segmentation fault. To handle this, use the info threads command to list all threads, and the thread <thread_id> command to switch to the thread that caused the segmentation fault.
It’s evident that thread ID 14 is responsible for the segmentation fault (LWP 51). To investigate further, we can switch to this thread using the command thread 14 in GDB:
After switching, we can employ the bt command to examine the backtrace of this thread.
Under the backtrace of thread 14, you will find:
This indicates that the segmentation fault occurred at line 139 in /workspace/holoscan-sdk/src/operators/ping_tensor_tx/ping_tensor_tx.cpp.
To view the backtrace of all threads, use the thread apply all bt command.
The Holoscan SDK provides support for tracing and profiling tools, particularly focusing on the compute method of Python operators. Debugging Python operators using Python IDEs can be challenging since this method is invoked from the C++ runtime. This also applies to the initialize, start, and stop methods of Python operators.
Users can leverage IDEs like VSCode/PyCharm (which utilize the PyDev.Debugger) or other similar tools to debug Python operators:
Subsequent sections will detail methods for debugging, profiling, and tracing Python applications using the Holoscan SDK.
The following command initiates a Python application within a pdb debugger session:
For more details, please refer to the pdb_main() method in test_pytracing.py.
It is also possible to launch the pdb session without having to manually add a breakpoint() to the source file before main() as was done in test_pytracing.py. In that case, just launch the existing application using python -m pdb my_app.py. For example, we can launch the existing ping_multi_port.py example included with the SDK like this
We will then be at the pdb prompt, from which we can insert a break point (at the first line of the compute method in this case) and then use c to continue execution until the breakpoint is reached.
Now that we are at the desired breakpoint, we can interactively debug the operator. The following show an example of using s to step by one line then p value1 to print the value of the “value1” variable. The l command is used to show the surrounding context. In the output, the arrow indicates the line where we are currently at in the debugger and the “B” indicates the breakpoint that was previously added.
For profiling, users can employ tools like cProfile or line_profiler for profiling Python applications/operators.
Note that when using a multithreaded scheduler, cProfile or the profile module might not accurately identify worker threads, or errors could occur.
In such cases with multithreaded schedulers, consider using multithread-aware profilers like pyinstrument, pprofile, or yappi.
For further information, refer to the test case at test_pytracing.py.
pyinstrument is a call stack profiler for Python, designed to highlight performance bottlenecks in an easily understandable format directly in your terminal as the code executes.
pprofile is a line-granularity, thread-aware deterministic and statistic pure-python profiler.
yappi is a tracing profiler that is multithreading, asyncio and gevent aware.
profile/cProfile is a deterministic profiling module for Python programs.
line_profiler is a module for doing line-by-line profiling of functions.
The Holoscan SDK provides support for measuring code coverage using Coverage.py.
To record code coverage programmatically, please refer to the coverage_main() method in test_pytracing.py.
You can execute the example application with code coverage enabled by running the following command:
The following command starts a Python application using the trace module:
A test case utilizing the trace module programmatically can be found in the trace_main() method in test_pytracing.py.