4. Advanced Troubleshooting#

This section provides information about troubleshooting some advanced issues.

4.1. Resolving Hung cuFile* APIs with No Response#

To resolve hung cuFile APIs:

  1. Check whether there are any kernel panics/warnings in dmesg:

    $ dmesg > warnings.txt. less warnings.txt

  2. Check whether the application process is in the D (uninterruptible) state.

  3. If the process is in the D state:

    1. Get the PID of the process by running the following command:

      $ ps axf | grep ' D'

    2. As a root user, get the backtrace of the D state process:

      $ su root
$ cat /proc/<pid>/stack

  4. Verify whether the threads are stuck in the kernel or in user space. For more information, review the backtrace of the D state threads.

  5. Check whether any threads are showing heavy CPU usage.

    1. The htop and mpstat tools should show CPU usage per core.

    2. Get the call graph of where the CPUs are being used. The following code snippet should narrow down whether the threads are hung in user space or in the kernel:

      $ perf top -g

4.2. Sending Relevant Data to Customer Support#

This section describes how to resolve a kernel panic with stack traces using NVSM or the GDS Log Collection tool.

DGX OS:

For DGX BaseOS with the preview network repo enabled and NVSM installed:

$ sudo apt-get install nvsm
$ sudo nvsm dump health

For more details on running NVSM commands, refer to NVIDIA System Management User Guide.

Non DGX:

The GDS Log Collection tool, gds_log_collection.py, may be run by GDS users to collect relevant debugging information from the system when issues with GDS IO are seen.

Some of the important information that this tool captures is highlighted below:

  • dmesg Output and relevant kernel log files.

  • System map files and vmlinux image

  • modinfo output for relevant modules

  • /proc/cmdline output

  • IB devices info like ibdev2net and ibstatus

  • OS distribution information

  • Cpuinfo, meminfo

  • nvidia-fs stats

  • Per process information like cufile.log, cufile.json, gds_stats, stack pointers

  • Any user specified files

To use the log collection tool:

$ sudo /usr/local/cuda/gds//tools/gdstools/gds_log_collection.py -h

This tool is used to collect logs from the system that are relevant for debugging.

It collects logs such as OS and kernel info, nvidia-fs stats, dmesg logs, syslogs, system map files and per-process logs such as cufile.json, cufile.log, gdsstats, process stack, and so on.

Usage:

./gds_log_collection.py [options]

Options:

-h help

-f file1,file2,.. (Note: there should be no spaces between ‘,’)

These files could be any relevant files apart from the one’s being collected (such as crash files).

Usage examples:

sudo ./gds_log_colection.py - Collects all the relevant logs.

sudo ./gds_log_colection.py -f file1,file2 - Collects all the relevant files as well as the user specified files.

4.3. Resolving an IO Failure with EIO and Stack Trace Warning#

You might see an IO failure with EIO and a warning with a stack trace with an nvfs_mgroup_check_and_set function in the trace.

This could mean that the EXAScaler file system did not honor O_DIRECT and fell back to page cache mode. GDS tracks this information in the driver and returns EIO.

Note

The WARNING stack trace is observed only once during the lifetime of the kernel module. You will get an Error: Input/Output (EIO), but the trace message will be printed only once. If you consistently experience this issue, contact support.

4.4. Controlling GPU BAR Memory Usage#

  1. To show how much BAR Memory is available per GPU, run the following command:

    $ /usr/local/cuda-x.y/gds/tools/gdscheck

  2. Review the output, for example:

GPU INFO:
 GPU Index: 0 bar:1 bar size (MB):32768
 GPU Index: 1 bar:1 bar size (MB):32768

GDS uses BAR memory in the following cases:

  • When the process invokes cuFileBufRegister.

  • When GDS uses the cache internally to allocate bounce buffers per GPU.

Note

There is no per-GPU configuration for cache and BAR memory usage.

Each process can control the usage of BAR memory via the configurable property in the /etc/cufile.json file:

"properties": {

// device memory size for reserving bounce buffers for the entire GPU (in KB)
"max_device_cache_size" : 131072,
// limit on maximum memory that can be pinned for a given process (in KB)
"max_device_pinned_mem_size" : 33554432

}

.. note::

   This configuration is per process, and the configuration is set across all GPUs.

4.5. Determining the Amount of Cache to Set Aside#

By default, 128 MB of cache is set in the configurable max_device_cache_size property. However, this does not mean that GDS pre-allocates 128 MB of memory per GPU up front. Memory allocation is done on the fly and is based on need. After the allocation is complete, there is no purging of the cache.

By default, since 128 MB is set, the cache can grow up to 128 MB. Setting the cache is application specific and depends on workload. Refer to the GPUDirect Storage Best Practices Guide to understand the need of cache and how to set the limit based on guidance in the guide.

4.6. Monitoring BAR Memory Usage#

There is no way to monitor the BAR memory usage per process. However, GDS Stats tracks the global BAR usage across all processes. For more information, see the following stat output from /proc/driver/nvidia_fs/stats for the GPU with B:D:F 0000:34:00.0:

GPU 0000:34:00.0  uuid:12a86a5e-3002-108f-ee49-4b51266cdc07 : Registered_MB=32 Cache_MB=10

Registered_MB tracks how much BAR memory is used when applications are explicitly using the cuFileBufRegister API.

Cache_MB tracks GDS usage of BAR memory for internal cache.

4.7. Resolving an ENOMEM Error Code#

-12 ENOMEM error code.

Each GPU has some BAR memory reserved. The cuFileBufRegister function makes the pages that underlie a range of GPU virtual memory accessible to a third-party device. This process is completed by pinning the GPU device memory in BAR space by using the nvidia_p2p_get_pages API. If the application tries to pin memory beyond the available BAR space, the nvidia_p2p_get_pages API returns a -12 (ENOMEM) error code.

To avoid running out of BAR memory, developers should use this output to manage how much memory is pinned by application. Administrators can use this output to investigate how to limit the pinned memory for different applications.

4.8. GDS and Compatibility Mode#

To determine the GDS compatibility mode, complete the following:

  1. In the /etc/cufile.json file, verify that allow_compat_mode is set to true.

  2. gdscheck -p displays whether the allow_compat_mode property is set to true.

  3. Check the cufile.log file for the cufile IO mode: POSIX message.

    This message is in the hot IO path, where logging each instance significantly impacts performance, so the message is only logged when logging:level is explicitly set to the TRACE mode in the /etc/cufile.json file.

4.9. Enabling Compatibility Mode#

Compatibility mode can be used by application developers to test the applications with cuFile-enabled libraries under the following conditions:

  • When there is no support for GDS for a specific file system.

  • The nvidia-fs.ko driver is not enabled in the system by the administrator.

To enable compatibility mode:

  1. Remove the nvidia-fs kernel driver:

    $ rmmod nvidia-fs

  2. In the /etc/cufile.json file, set compat-mode to true.

  3. Set the CUFILE_FORCE_COMPAT_MODE environment variable to true.

The IO through cuFileRead/cuFileWrite will now fall back to the CPU path.

4.10. Tracking the IO After Enabling Compatibility Mode#

When GDS is used in compatibility mode, and cufile_stats is enabled in the /etc/cufile.json file, you can use gds_stats or another standard Linux tools, such as strace, iostat, iotop, SAR, ftrace, and perf. You can also use the BPF compiler collection tools to track and monitor the IO.

When compatibility mode is enabled, internally, cuFileRead and cuFileWrite use POSIX pread and pwrite system calls, respectively.

4.11. Bypassing GPUDirect Storage#

There are some scenarios in which you can bypass GDS.

There are some tunables where GDS IO and POSIX IO can go through simultaneously. The following are cases where GDS can be bypassed without having to remove the GDS driver:

  • On supported file systems and block devices.

    In the /etc/cufile.json file, if the posix_unaligned_writes config property is set to true, the unaligned writes will fall back to the compatibility mode and will not go through GDS. Refer to Before You Install GDS for a list of supported file systems.

  • On EXAScaler and Amazon FSx for Lustre file system:

    In the /etc/cufile.json file, if the posix_gds_min_kb config property is set to a certain value (in KB), the IO for which the size is less than or equal to the set value, will fall back to POSIX mode. For example, if posix_gds_min_kb is set to 8KB, IOs with a size that is less than or equal to 8KB, will fall back to the POSIX mode.

  • On a WekaIO file system:

    In the /etc/cufile.json file, if the allow-compat-mode config property is set to true, cuFileRead automatically falls back to the POSIX mode if:

    • RDMA connections and/or memory registrations cannot be established.

    • cuFileRead fails to allocate an internal bounce buffer for non-4K aligned GPU VA addresses.

Refer to the GPUDirect Storage Best Practices Guide for more information.

4.12. GDS Does Not Work for a Mount#

GDS will not be used for a mount in the following cases:

  • When the necessary GDS drivers are not loaded on the system.

  • The file system associated with that mount is not supported by GDS.

  • The mount point is denylisted in the /etc/cufile.json file.

4.13. Simultaneously Running the GPUDirect Storage IO and POSIX IO on the Same File#

Since a file is opened in O_DIRECT mode for GDS, applications should avoid mixing O_DIRECT and normal I/O to the same file and to overlapping byte regions in the same file.

Even when the file system correctly handles the coherency issues in this situation, overall I/O throughput might be slower than using either mode alone. Similarly, applications should avoid mixing mmap(2) of files with direct I/O to the same files. Refer to the file system-specific documentation for information about additional O_DIRECT limitations.

4.14. Running Data Verification Tests Using GPUDirect Storage#

GDS has an internal data verification utility, gdsio_verify, which is used to test data integrity of reads and writes. Run gdsio_verify -h for detailed usage information.

For example:

$ /usr/local/cuda-11.2/gds/tools/gds_verify -f /mnt/ai200/fio-seq-writes-1 -d 0 -o 0 -s 1G -n 1 -m 1

Sample output:

gpu index :0,file :/mnt/ai200/fio-seq-writes-1, RING buffer size :0,
gpu buffer alignment :0, gpu buffer offset :0, file offset :0,
io_requested :1073741824, bufregister :true, sync :1, nr ios :1,
fsync :0,
address = 0x560d32c17000
Data Verification Success

Note

This test completes data verification of reads and writes through GDS.

On this page