3. Basic Troubleshooting#

3.1. Log Files for the GDS Library#

A cufile.log file is created in the current working directory of the application. Currently the default maximum log file size is 32MB. If the log file size increases to greater than 32MB, the log file is truncated and logging is resumed on the same file. This limit is configurable through the setting in the logging section of cufile.json or the environment variable; setting it to a larger value raises the truncation threshold, and setting it to -1 removes the limit entirely so the log file grows unbounded (recommended only for short debugging runs).

3.2. Enabling a Different cufile.log File for Each Application#

You can enable a different cufile.log file for each application.

There are several relevant cases:

  • If the logging:dir property in the default /etc/cufile.json file is not set, by default, the cufile.log file is generated in the current working directory of the application.

  • If the logging:dir property is set in the default /etc/cufile.json file, the log file is created in the specified directory path.

Note

This is usually not recommended for scenarios where multiple applications use the libcufile.so library.

For example:

"logging": {
    // log directory, if not enabled
    // will create log file under current working
    // directory
      "dir": "/opt/gdslogs/",
}

The cufile.log will be created as a /opt/gdslogs/cufile.log file.

If the application needs to enable a different cufile.log for different applications, the application can override the default JSON path by doing the following steps:

  1. Export CUFILE_ENV_PATH_JSON=”/opt/myapp/cufile.json”.

  2. Edit the /opt/myapp/cufile.json file.

    "logging": {
    // log directory, if not enabled
    // will create log file under current working
    // directory
    "dir": "/opt/myapp",
}

  3. Run the application.

  4. To check for logs, run:

    $ ls -l /opt/myapp/cufile.log

3.3. Enabling Tracing GDS Library API Calls#

There are different logging levels, which can be enabled in the /etc/cufile.json file.

By default, logging level is set to ERROR. Logging will have performance impact as we increase the verbosity levels like INFO, DEBUG, and TRACE, and should be enabled only to debug field issues.

Configure tracing and run the following:

"logging": {
    // log directory, if not enabled
    // will create log file under local directory
    //"dir": "/home/<xxxx>",
    // ERROR|WARN|INFO|DEBUG|TRACE (in decreasing order of priority)
    "level": "ERROR"
},

3.4. cuFileHandleRegister Error#

If you see the cuFileHandleRegister error on the cufile.log file when an IO is issued:

"cuFileHandleRegister error: GPUDirect Storage not supported on current file."

Here are some reasons why this error might occur:

  • The file system is not supported by GDS.

    Refer to CU_FILE_DEVICE_NOT_SUPPORTED for more information.

  • DIRECT_IO functionality is not supported for the mount on which the file resides.

For more information, enable tracing in the /etc/cufile.json file.

3.5. Troubleshooting Applications that Return cuFile Errors#

To troubleshoot cuFile errors:

  1. See the cufile.h file for more information about errors that are returned by the API.

  2. If the IO was submitted to the GDS driver, check whether there are any errors in GDS stats.

    If the IO fails, the error stats should provide information about the type of error.

    See Finding GDS Driver Statistics for more information.

  3. Enable GDS library tracing and monitor the cufile.log file.

  4. Enable GDS Driver debugging:

    $ echo 1 >/sys/module/nvidia_fs/parameters/dbg_enabled

After the driver debug logs are enabled, you might get more information about the error.

3.6. cuFile-* Errors with No Activity in GPUDirect Storage Statistics#

If there are cuFile errors in the GDS statistics, this means that the API failed in the GDS library. You can enable tracing by setting the appropriate logging level in the /etc/cufile.json file to get more information about the failure in cufile.log.

3.7. CUDA Runtime and Driver Mismatch with Error Code 35#

Error code 35 from the CUDA documentation points to cudaErrorInsufficientDriver, which indicates that the installed NVIDIA CUDA driver is older than the CUDA runtime library. This is not a supported configuration. For the application to run, you must update the NVIDIA display driver.

Note

cuFile tools depend on CUDA runtime 10.1 and later. You must ensure that the installed CUDA runtime is compatible with the installed CUDA driver and is at the recommended version.

3.8. CUDA API Errors when Running the cuFile-* APIs#

The GDS library uses the CUDA driver APIs.

If you observe CUDA API errors, you will observe an error code. Refer to the error codes in the CUDA Libraries documentation for more information.

3.9. Finding GDS Driver Statistics#

To find the GDS Driver Statistics, run the following command:

$ cat /proc/driver/nvidia-fs/stats

GDS Driver kernel statistics for READ / WRITE are available for all file systems except for Weka. For Weka file system statistics, refer to WekaFS Troubleshooting for more information about READ / WRITE.

Note

Compatibility mode and P2PDMA I/O statistics are not tracked.

3.10. Tracking IO Activity that Goes Through the GDS Driver#

In GDS Driver statistics, the ops row shows the active IO operation. The Read and Write fields show the current active operation in flight. This information should provide an idea of how many total IOs are in flight across all applications in the kernel. If there is a bottleneck in the userspace, the number of active IOs will be less than the number of threads that are submitting the IO. Additionally, to get more details about the Read and Write bandwidth numbers, look out for counters in the Read/Write rows.

3.11. Read/Write Bandwidth and Latency Numbers in GDS Stats#

Measured latencies begin when the IO is submitted and end when the IO completion is received by the GDS kernel driver. Userspace latencies are not reported. This should provide an idea whether the user space is bottlenecked or whether the IO is bottlenecked on the backend disks/fabric.

Note

The WekaIO file system reads do not go through the nvidia-fs driver, so Read/Write bandwidth stats are not available for WekaIO file system by using this interface.

Refer to the WekaFS Troubleshooting for more information.

3.12. Tracking Registration and Deregistration of GPU Buffers#

In GDS Driver stats, look for the active field in BAR1-map stats row.

The pinning and unpinning of GPU memory through cuFileBufRegister and cuFileBufDeregister is an expensive operation. If you notice a large number of registrations(n) and deregistration(free) in the nvidia-fs stats, it can hurt performance. Refer to the GPUDirect Storage Best Practices Guide for more information about using the cuFileBufRegister API.

3.13. Enabling RDMA-specific Logging for Userspace File Systems#

In order to troubleshoot RDMA related issues for userspace file systems, ensure that the CUFILE_LOGGING_LEVEL environment variable is set to INFO, DEBUG, or TRACE prior to running the application. However, for this to work, cufile.json logging level also should be set to TRACE/DEBUG/INFO level.

For example:

$ export CUFILE_LOGGING_LEVEL=INFO
  This is an example to set log level to INFO via the environment variable.

$ cat /etc/cufile.json
   ....
   "logging": {
        // log directory, if not enabled will create log file
        // under current working directory
        //"dir": "/home/<xxxx>",
        // ERROR|WARN|INFO|DEBUG|TRACE (in decreasing order of priority)
        "level": "DEBUG"
   },
   ....
   This is an example on how to set log level to DEBUG via cufile.json.

3.14. CUDA_ERROR_SYSTEM_NOT_READY After Installation#

On systems with NVSwitch, if you notice the CUDA_ERROR_SYSTEM_NOT_READY error being reported, then make sure that you install the same version of Fabric Manager as the CUDA driver.

For example, if you use:

$ sudo apt install nvidia-driver-460-server -y

then use:

$ apt-get install nvidia-fabricmanager-460

Make sure to restart the Fabric Manager service using:

$ sudo service nvidia-fabricmanager start

3.15. Adding udev Rules for RAID Volumes#

To add udev rules for RAID volumes:

As a sudo user, change the following line in /lib/udev/rules.d/63-md-raid-arrays.rules:

IMPORT{program}="/usr/sbin/mdadm --detail --export $devnode"

Reboot the node or restart the mdadm.

3.16. When You Observe “Incomplete write” on NVME Drives#

During GDS mode writes, you may receive error messages similar to the following:

Tid: 0 incomplete Write, done = 0 issued = 1048576

GPUDirect storage in P2P mode does not support NVMe end to end data protection features. To support GDS in P2P mode, the NVMe must be formatted with Protection Information - Metadata Size is set to zero bytes.

Confirm that the drive has data-integrity mode enabled:

$ sudo nvme id-ns /dev/nvme0n1 -H
-
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded (in use)
LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0x3 Degraded

Note in the preceding example, the metadata size of the drive (nvme0n1) is set to non-zero.

You can set the LBA format to 0 or 2 to disable the protection feature on the drive:

$ sudo nvme format /dev/nvme0n1 -l 2
$ sudo nvme id-ns /dev/nvme0n1 -H
-
LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded
LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best (in use)
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0x3 Degraded

3.17. CUFILE async I/O is failing#

There could be many reasons for which stream based async I/O can fail. This will be logged in cufile.log. One of the common reasons could be that the internal thread pool is not enabled. Refer to cufile.json “execution” section on how to enable it.

3.18. Troubleshooting cuObject Client Configuration#

Configuration changes do not take effect

Check that the application selects its configuration through CUOBJ_ENV_PATH_JSON or /etc/cuobj.json. Verify that supported environment variables are not overriding the JSON settings.

In cuObject client v1.4.0, cuFile and cuObject share a runtime within each process. The first successful initialization selects the runtime configuration, including initialization triggered implicitly by another operation. Subsequent initialization reuses that configuration without loading another JSON file or reapplying environment settings.

This selection remains active until the shared runtime is fully torn down. Restart the application when testing changed initialization settings.

Direct application use of both cuFile and cuObject interfaces in the same process is unsupported in this release. Use separate processes when independent configurations are required. This restriction does not apply to cuObject’s internal use of cuFile functionality.

Mixed initialization may generate a warning. Implicit cuFile initialization does not always generate one.

The selected configuration file is missing

A missing selected JSON file does not, by itself, cause configuration loading to fail. The runtime uses built-in settings and supported environment overrides. If CUOBJ_ENV_PATH_JSON selects a missing custom file, cuObject does not fall back to /etc/cuobj.json. Confirm that the intended file exists and is accessible to the application.

Client initialization fails

Check the runtime logs for file-access errors, malformed JSON, or parameter values rejected by configuration validation. These errors cause configuration initialization to fail. Also verify that the application loads a matching pair of cuObject client and libcufile libraries, and that the RDMA devices configured in properties.rdma_dev_addr_list match the client system.

Use the cuObject platform check described in GDSCHECK Tool to inspect the configuration in a separate diagnostic process.

Logging settings do not take effect

Set logging.level and logging.dir in the selected cuObject JSON file. Use CUOBJ_LOGGING_LEVEL and CUOBJ_LOGFILE_MAX_SIZE_MB for the corresponding environment overrides.

Configure these settings before initialization. If cuFile has already initialized the shared runtime, a later cuObject initialization does not reload the logging configuration.

Enabling execution.parallel_io has no effect

cuObject disables the cuFile parallel I/O thread pool during configuration. Setting execution.parallel_io to true in a cuObject JSON file does not enable that thread pool.

On this page