Debugging Guide#

nvCOMPDx is distributed in LTO-IR format: the library implementation remains closed within libnvcompdx.a and libnvcompdx.fatbin, while public headers expose device function declarations that allow compression and decompression operations to be fused into user kernels. Breakpoints can be set in user-written device and host code when the recommended build workflow below is followed.

As nvCOMPDx relies on LTO at link time, device debugging differs from conventional PTX/SASS debugging. Combining the -dlto and --device-debug nvcc options in a single compilation step is not supported and produces the following error:

nvcc fatal : Option --link-time-opt cannot be combined with --device-debug

To debug a fused CUDA kernel, compile user device code to PTX (or SASS) with --device-debug first, then link the resulting object against the nvCOMPDx LTO-IR static library. The resulting binary can then be debugged with the CUDA debugger (cuda-gdb), with breakpoints set in user code as usual.

For general build and link instructions, see the Installation Guide.

Note

Through CUDA Toolkit 13.3, the workflow described above is supported only when linking against the static library libnvcompdx.a, and not when using the fatbin library libnvcompdx.fatbin. Furthermore, debugging is only supported on x86_64 Linux platforms.

Command-Line Workflow#

The procedure below assumes a fused kernel in fused_kernel.cu targeting the SM 8.9 (Ada) GPU architecture. Host debugging symbols are enabled with the -g flag.

  1. Compile the fused device code to PTX with relocatable device code and device-debug information:

nvcc -rdc=true --generate-code=arch=compute_89,code=compute_89 fused_kernel.cu \
     -I<mathdx_include_dir> --device-debug -g -c -o fused_kernel.o

  1. Device-link the object against the nvCOMPDx LTO-IR library to generate SASS:

nvcc -dlto -dlink --generate-code=arch=compute_89,code=sm_89 fused_kernel.o \
     -L<mathdx_lib_dir> -lnvcompdx -o fused_kernel_dlinked.o

  1. Perform host linking to produce the final executable:

g++ fused_kernel.o fused_kernel_dlinked.o \
    -L<CUDA_sysroot>/lib64/ -L<mathdx_lib_dir> \
    -lnvcompdx -lcudadevrt -lcudart_static -o fused_kernel

  1. Launch the executable under cuda-gdb:

cuda-gdb --args ./fused_kernel

  1. Set breakpoints in user code and run the program:

break main
run

CMake Configuration#

CMake can automate the debug build workflow described above. The following example configures an executable target fused_kernel from fused_kernel.cu, enables separable device compilation, targets the native GPU architecture, and applies the appropriate compile and link options based on the build type.

add_executable(fused_kernel fused_kernel.cu)
set_target_properties(fused_kernel
     PROPERTIES
         CUDA_SEPARABLE_COMPILATION ON
         CUDA_ARCHITECTURES native
)

if((CMAKE_BUILD_TYPE STREQUAL "DEBUG") OR (CMAKE_BUILD_TYPE STREQUAL "Debug"))
   target_compile_options(fused_kernel PRIVATE --device-debug -g)
else()
   set_property(TARGET fused_kernel PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE)
endif()

target_link_libraries(fused_kernel PRIVATE mathdx::nvcompdx)

When CMAKE_BUILD_TYPE is set to Debug, LTO is disabled and --device-debug is passed to the device compiler. In Release builds, INTERPROCEDURAL_OPTIMIZATION is enabled to allow LTO with the nvCOMPDx library. Both CUDA_SEPARABLE_COMPILATION and INTERPROCEDURAL_OPTIMIZATION are required when linking against mathdx::nvcompdx or mathdx::nvcompdx_fatbin; see Installation Guide for details.

Debugging the Examples#

The nvCOMPDx example build enables user-side debugging automatically when configured with CMAKE_BUILD_TYPE=Debug. Aside from the build type, the configuration and build commands are identical to a Release build.

# Configure and build
mkdir build && cd build
cmake -DNVCOMPDX_CUDA_ARCHITECTURES=89-real \
      -Dmathdx_ROOT=/opt/nvidia/mathdx/XX.Y \
      -DCMAKE_BUILD_TYPE=Debug ..
make -j

# Run the examples with ctest
ctest

# Or run a single example under cuda-gdb
cuda-gdb --args ./nvcompdx/01_introduction/lz4_gpu_compression_introduction