8. Known Issues#

  • GDS does not work on OpenShift with NW Operator version 24.

  • When using NW Operator 25.1.0 on OpenShift with ENABLE_NFSRDMA and UNLOAD_STORAGE_DRIVERS set to true, the OpenShift installation might get corrupted. This issue is resolved in NW Operator 25.7 and above.

  • Using nvidia-fs version 2.25.6 may cause cuFile APIs to fail when using GDS P2P mode with the nvidia-fs kernel driver. We recommend using nvidia-fs version 2.25.7 or newer from the GitHub release: nvidia-fs v2.25.7.

  • NVME P2PDMA mode is currently not supported on RHEL 9.5 for x86.

  • NVME P2PDMA can fail on x86_64 platforms when KASLR is enabled in the linux kernel. The feature can be disabled by specifying the nokaslr kernel parameter.

  • Without nvidia-persistenced enabled, NVME P2PDMA can increase the time taken to start the driver initialization for B200 platforms.

  • Sparse file IO performance on Ext4 with NVME P2PDMA is slower on x86_64 platforms compared to GDS P2PDMA with nvidia-fs.ko.

  • When installing via .run file, the installation of GPUDirect Storage components fails if MLNX_OFED is not installed.

  • The cuFile library may fail to load GPU topology information properly when udev information is not accessible to the application. Ensure the correct udev library is installed.

  • nvidia-fs can deadlock in nv_p2p_dma_map_pages and nv_p2p_mem_info_free_callback functions when the user frees the CUDA memory without calling cuFileBufDeregister on registered buffers.

  • On DDN EXAScaler file systems:

    • With stripe count > 1, cuFileRead and cuFileWrite do not work with poll mode enabled for versions older than 2.12.5_ddn10.

    • With 2.12.5_ddn10, any reads beyond EOF causes a BUG_ON inside nvidia-fs.

  • The cuFileRead and cuFileWrite APIs fail when working on cuMemMap allocations with multiple GPUs, when the IO request to a GPU buffer is not 4K aligned and spans across multiple GPUs.

  • When the cufile_stats JSON parameter is set to a non-zero value, the cuFile library may crash during program exit if opened or closed through the dl interface. Workaround: Explicitly call cuFileDriverClose API before closing the library.

  • Prior to CUDA 13.1, enabling stats with verbose logging may cause a race condition during shutdown that can trigger a segmentation fault.

  • The nvidia.ko kernel P2P interface version has changed to 0x00020000 starting from NVIDIA driver version R580.82.05. The nvidia-fs.ko module must be rebuilt against the new symbols after the nvidia.ko driver installation. Remove the existing nvidia-fs-dkms driver manually and reinstall the nvidia-fs.ko driver after the nvidia.ko driver installation by running:

    sudo apt install -y --reinstall nvidia-fs-dkms

  • For GPFS, in the case of asynchronous I/O, only I/Os that fail with -EOPNOTSUPP at submission time are retried via the compat path by libcufile. I/Os that fail with this error during execution time are not retried via the compat path.

  • With libcufile version 1.16.0, when nvidia-fs is loaded but the NVMe/NVMeoF driver is not patched for GDS support, I/O operations on block devices may incorrectly attempt to use the GDS path and fail. This issue only affects file descriptors registered after the first one on the same block device. As a workaround, unload the nvidia-fs module when operating on block devices with an unpatched NVMe/NVMeoF driver, for example:

    modprobe -r nvidia-fs

  • Due to a change in pin_user_pages_fast, upstream kernels 6.11 and later, as well as corresponding recent RHEL kernels, exhibit a performance regression for small I/O sizes up to 128 KB. This regression affects both the NVMe nvidia-fs path and the p2pdma path.

  • For large system deployments on Ubuntu 26.04 and later, it is advisable to use static topology in the cufile configuration to reduce startup time for short-running applications.

  • To retain the previous 16 MiB maximum direct I/O chunk size, set max_direct_io_size_kb to 16384 in cufile.json.

  • In some rare cases, the NVIDIA RM driver can fail during DMA, resulting in a GDS I/O hang on Lustre DDN filesystems. The only way to recover from this hang is to reboot the machine.