9. Known Limitations#

This section provides information about the known limitations in this release of GDS.

  • cuFile/cuObject shared-runtime limitation: In GDS v1.20, cuFile and cuObject share a runtime within each process. The first successful initialization, whether explicit or implicit, selects the runtime configuration. Subsequent initialization through either interface reuses the active configuration without loading another JSON file or reapplying environment settings. Configuration selection occurs again only after the shared runtime has been fully torn down and initialized again. Direct application use of both the cuFile and cuObject interfaces in the same process is unsupported in this release. Use separate processes when independent configurations are required. This restriction does not apply to cuObject’s internal use of cuFile functionality.

  • cuObject client library compatibility: Use cuObject client v1.4.0 with the matching libcufile from the same CUDA release. Combining the new cuObject client with an older libcufile is unsupported and can cause application linking or loading failures.

  • GPUDirect Storage with NVME P2PDMA feature is not supported with RAID0 and multipath NVMe support.

  • vGPU supports the latest GPUs with OpenRM; that is, GPUs with Ada Lovelace, Hopper and later are supported. A100 and earlier GPUs are not supported with OpenRM.

  • All OS (BaseOS/RHEL/SLES ) for Grace-Hopper need persistence enabled.

  • CUDA streams based APIs:

    • CUDA graphs are not supported with cuFile Stream APIs.

    • cuFile Stream APIs for GPFS and WekaFS are supported in compatibility mode only.

    • cuFile Stream APIs are not supported when cuFile configuration parameter execution.parallel_io is false or execution.max_io_threads is set to 0.

    • CU_FILE_STREAMS_SUPPORTED bit is not set in Props.fflags when queried with cuFileDriverGetProperties.

  • Available BAR1 memory reported by thenvidia-smi utility is not accurate due to some internal overhead of the CUDA toolkit. Therefore, a huge allocation of BAR1 memory by any GDS application can run into ENOMEM errors, even when nvidia-smi utility shows there is available BAR1 memory.

  • GPUDirect storage in P2P mode does not support NVMe end-to-end data protection features. To support GDS in p2p mode, the NVMe must be formatted with Protection Information where Metadata Size is set to zero bytes.

  • CentOS 7.x is no longer supported.

  • Checksums on the client-side of file systems must be disabled for GDS.

  • cuFile APIs are not supported with applications that use the fork() system call.

  • GDS Compatibility mode is only tested on GDS qualified file systems: ext4, EXAScaler, Amazon FSx for Lustre, XFS, WekaFS, IBM Spectrum Scale, VAST, and BeeGFS.

  • On x86-64 platforms, GDS with “IOMMU=on” or ACS enabled are not guaranteed to work functionally or in a performant way.

  • Refer to the following documentation for IBM Spectrum Scale Limitations with GDS: https://www.ibm.com/docs/en/spectrum-scale/5.1.5?topic=architecture-gpudirect-storage-support-spectrum-scale

  • Upgrading of Linux Kernel version and nv_peer_mem:

    WekaFS does not support newer MLNX_OFED versions 5.3.x and above with GDS. nvidia-peer-memory-dkms=1.1-0-nvidia2 is required for GDS support with WekaFS. Please follow the instructions in section 2.2 of the GDS Troubleshooting and Installation Guide.

  • RHEL 8.3 or later does not have default udev rules for detecting RAID members, which disables GDS on RAID volumes. Refer to the section “Adding udev Rules for RAID Volumes” in GDS Installation and Troubleshooting Guide.

  • Sparse file read performance can be slow with the “NVME P2PDMA” feature.

  • WekaFS is not supported with CUDA 13.1 because of a regression in file handle parsing. This issue causes GPUDirect Storage read and write operations to fail with EBADF (-9). If you use WekaFS, upgrade to CUDA 13.3 or later.

  • On DGX Spark, GPUDirect Storage is supported only in compatibility mode. Do not load nvidia-fs, because doing so can result in errors.

  • On Linux 6.17 and later with nvidia-fs mode enabled, the NVMe patch has a known issue with drives that support scatter-gather lists (SGLs). When the driver uses the SGL path, I/O failures and kernel-level hangs can occur. As a workaround, use the p2pdma path on these kernels.

  • If you plan to use nvidia-fs to access local NVMe drives, DOCA version compatibility depends on your Linux kernel version:

    • On Linux kernel 6.17 or newer, DOCA 3.3.0 or newer is required. This guidance may also apply to older kernels on Linux distributions that have backported the iterative block-layer mapping API; check for it by running grep -w blk_rq_dma_map_iter_start /proc/kallsyms.

    • On Linux kernel 6.16 or older, any DOCA version except 3.3.0 can be used.

    • If you cannot install a DOCA version that is compatible with your kernel as described above, use the p2pdma path as a workaround. Refer to the GDS Installation and Troubleshooting Guide.