1. New Features and Changes#

v1.20

Added separate configuration-file selection starting from cuObject client v1.4.0. cuObject uses /etc/cuobj.json by default. A nonempty CUOBJ_ENV_PATH_JSON selects a custom file in place of /etc/cuobj.json. The custom file is not applied as an overlay.
Added supported CUOBJ_* environment-variable overrides for cuObject client settings. These overrides take precedence over JSON settings. Corresponding CUFILE_* variables are accepted as migration fallbacks only when the matching CUOBJ_* variable is absent. This fallback does not apply to JSON configuration-file selection.
Moved properties.rdma_transport_type from the shipped cufile.json template to cuobj.json for cuObject transport selection.
execution.parallel_io remains libcufile-only and is forced off for cuObject configuration parsing.
Added gdscheck -p --object to display the cuObject client version, environment, effective configuration, and userspace RDMA status.
Added cuObjClient and server APIs to show versioning information.
Added optional static GPU hardware configuration to reduce initialization overhead by caching BAR size and PCI P2PDMA support.
Enabled sparse-file P2P compatibility mode by default.
Added cuObject NVTX profiling visibility.
Added cuObject support for the external RC transport type for Azure Blob Storage.
Added NVTX ranges for client GET, PUT, buffer registration, token operations, and server request and buffer operations.
Improved reliability, validation, error handling, recovery, resource lifecycle management, and diagnostics across libcufile.so, libcufile_rdma.so, nvidia-fs.ko, cuObject, and GDS tools.

Migration note: Applications that previously selected a cuObject configuration file through CUFILE_ENV_PATH_JSON must use CUOBJ_ENV_PATH_JSON. cuObject does not use CUFILE_ENV_PATH_JSON or /etc/cufile.json to select its JSON configuration.

v1.19.1

Added support for VDURA PanFS.
Fixed assorted bugs.

v1.19

Added cuFile APIs to perform scatter-gather I/O across multiple buffers.
Added vanilla_posix_mode, which performs POSIX-mode I/O when any of the libraries required by cuFile are missing.
Improved sparse-file read performance with P2P DMA on EXT4 and XFS by automatically using a more efficient data path when sparse regions are detected. This optimization is enabled by default for larger reads (16 KiB and above) and helps avoid performance degradation when accessing sparse files. To disable it, set the following in /etc/cufile.json:
```
"sparse": {
    "p2p_compat_enable": false
}
```
Added support for RAID1 and RAID10 on NVMe drives in the following configurations:
- x86 platforms: Supported with nvidia-fs.
- Grace-based platforms: Supported with nvidia-fs and C2C mode.
- x86 P2PDMA mode: Supported starting with kernel 7.2 and later.
To enable C2C/P2PDMA mode for RAID1 and RAID10, configure the following settings in /etc/cufile.json:
```
"block": {
    "nvme": {
        "use_pci_p2pdma": true
    },
    "raid1": {
        "use_pci_p2pdma": true
    },
    "raid10": {
        "use_pci_p2pdma": true
    }
}
```

v1.18.1

Improved multithreaded I/O performance for access sizes larger than 8 MiB on distributed file systems

V1.18

Enhanced the accuracy of byte-count reporting during file writes to provide consistent progress information.
Removed the batch I/O size limitation for GDS mode that was caused by the bounce buffer size.
Added support for Linux kernel 7.0 to nvidia-fs.
Improved multithreading performance on Grace-based platforms.
Fixed assorted bugs.

V1.17

Reduced buffer management overhead through DMA-Buf interoperability.
Added tunable bounce buffer sizing for each mode to better match application and topology behavior.
Added deterministic routing and visibility control for multi-GPU and multi-path environments.
Added a structured JSON-based API to simplify integration and automation.
Integrated support for user-defined system topologies.
Updated the compatibility path to support ZFS and BTRFS across all I/O APIs.
Added support for specifying GPU bounce buffer configuration by using gpu_bounce_buffer_slab_config in cufile.json, similar to POSIX slab properties.
Added deprecation warnings for uniform bounce buffer configuration parameters and APIs.
Included assorted bug fixes.

V1.16

Update gdscheck output to specify the modes (p2pdma, nvfs, compat) supported by libcufile.
Added P2P configuration APIs:
- cuFileDriverSetP2PFlags(): Configure P2P transfer flags for specific filesystem or device types.
- cuFileDriverGetP2PFlags(): Query current P2P flag settings.
Removed batch io size limitation due to the bounce buffer size. For any batch I/Os larger than the bounce buffer size, it would now perform a posix I/O.
Added support for Amazon FSx for Lustre with GPUDirect Storage. GPUDirect Storage now interoperates with Amazon FSx for Lustre on Amazon EC2, enabling a direct data path between Lustre and NVIDIA GPUs. This reduces CPU utilization and can increase throughput for data-intensive GPU workloads. See the GPUDirect Storage Overview for supported file systems and configuration notes.
SLES distro supports GDS in P2PDMA mode.
Assorted bug fixes

v1.15.1

Assorted bug fixes.

V1.15

After a reboot, the first time an application launches, cuFileStatsStart may fail when called after cuFileStatsStop. This issue is not observed in subsequent application launches.
Statistics collection API routines for cuFile operations
Removed support for NVIDIA Pascal and Volta architectures
Assorted bug fixes

v1.14

Added support for NEC ScaTeFS.
Updated GDS user level stats to show P2PDMA stats
Updated nvidia-fs.ko to support 6.12 kernel
Introduced Get/Put API interfaces for GDS Parameters, enabling retrieval and modification of parameter values for size_t, bool, and string types.
Assorted bug fixes

Features introduced in previous releases:

v1.13

Improved unregistered buffer IO performance for larger IO size using threadpool.
Improved unregistered buffer IO performance by increasing the internal bounce buffer size. The size can be configured up to a max of 16MiB using the JSON parameter per_buffer_cache_size_kb.
On Grace+Hopper systems, DDN Exascaler GDS p2p mode is supported with 64K kernel PAGE SIZE for transfer sizes that are 4KB aligned but not 64K aligned.
Updated nvidia-fs.ko to support more PCIe Devices, to support Amazon Fsx for Lustre.
Stability fixes during driver close and batch close operations.
Added support for new “NVME P2PDMA” feature for GDS. This mode supports GDS with NVMe linux upstream drivers, minimum kernel version >= 6.2 (Ubuntu 22.04) and 5.14 (RHEL 9.4) on x86_64 based platforms. This feature will eliminate the need for custom MOFED NVMe patches and nvidia-fs.ko to support GDS with Ext4 and XFS with NVMe drives.

v1.11.1

Assorted bug fixes.

v1.11

Added support for RHEL 8.10, RHEL 9.4 and UB 24.04.
Assorted bug fixes.

v1.10.1

Assorted bug fixes.

v1.10

Assorted bug fixes.

v1.9.1

Assorted bug fixes.

v1.9

Added support for RHEL 9.3 and UB 22.04.3.

v1.8.1

Added support for RHEL 9.2 on Grace Hopper platform with 64K host OS page size for EXT4 filesystems with Local NVMe.
Improved the IO throughput performance for applications by adding topology awareness in compatibility mode.

v1.8

Assorted bug fixes.

v1.7.2

Added Grace Hopper platform Support with 64K host OS page size for EXT4 filesystems with Local NVMe on Ubuntu 22.04 with HWE kernels.
Proprietary NVIDIA kernel module is not supported. Only the NVIDIA open kernel module will be supported.
cuFile APIs can be used in Cloud-service-providers environments in compatibility mode.

v1.7

Support for APIs cuFileStreamRegister, cuFileStreamDeregister, cuFileReadAsync, and cuFileWriteAsync is complete. This enables use of CUDA Streams with cuFile APIs.
cuFile APIs can be used with system memory.
cuFile APIs can now be used with non-O_DIRECT file descriptors.
Threadpool support is enabled by default and is required for cuFile APIs supporting CUDA streams.

v1.6.1

Improved batch API performance.
Implemented threadpool in the cuFile library to enable parallelism and improve throughput of a large IO request using a single user thread.

v1.5.1

Assorted bug fixes.

v1.5:

Added support for cuMem* memory allocations with cuFile APIs.

v1.4:

Hopper PCIe support.
RHEL 9.0 and Ubuntu 22.04 support.

v1.3.1:

GDS can be now installed through CUDA .run files.
Support for Ubuntu 22.04 and RHEL9.
Improvements to NIC to GPU affinity for userspace RDMA file systems.

v1.3

Initial support for Linux dma-buf.

v1.2.1

GDS now supports vGPU in VMware context. Refer to https://docs.nvidia.com/grid/latest/grid-vgpu-release-notes-generic-linux-kvm/index.html#gpudirect-technology-support and https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#cuda-open-cl-support-vgpu for more information.

v1.2

Support for BeeGFS.
Support for XFS.
Batch APIs available for use (Alpha level support).

v1.1.1:

Use nvidia_peermem default for userspace RDMA filesystems (GPFS, WekaFS). In order to use nvidia_peermem, load it using:
```
# modprobe nvidia_peermem
```
Added support for BeeGFS (preview).

v1.1:

The XFS file system has been added to the list of supported file systems at a a beta support level.
Improved support for unregistered buffers.
Added options start_offset and io_size to gdsio config file per job options.
Improved performance of 4K and 8K IO sizes for local file systems.
Added user-configurable priority for internal cuFile CUDA streams.

v1.0:

New configuration and environment variables for the cuFile library.
Fixed error handling behavior for Weka retriable and unsupported errors.
Removed hard dependency on librcu-bp.
Added read support for IBM Spectrum Scale.

v0.95:

Compatibility with POSIX IO is enabled by default.
Alpha level support for RHEL 8.3.
GDS is available as Technical preview for DGX OS.
Support for MLNX_OFED 5.3 for NVMe and NVMeOF.
Support for Excelero NVMesh devices.
Support for ScaleFlux computational storage.
Integration with DALI® and PyTorch.
Experimental RAPIDS integration for cuDF, unoptimized, reads only.