O_DIRECT Requirements Guide#
The NVIDIA® GPUDirect Storage® (GDS) O_DIRECT Requirements Guide helps you understand how GDS can provide significant
benefits when it can leverage the O_DIRECT fcntl.h file mode for a direct data path between GPU memory and storage.
This section provides an introduction to the O_DIRECT requirements for the cuFile portion of GDS.
Note
O_DIRECT is the only supported mode before CUDA toolkit 12.2 (GDS version 1.7). CUDA 12.2 (GDS version 1.7) introduces support for non O_DIRECT file descriptors as well. The rest of this guide is still relevant for applications depending upon GDS benefits by expressing intent to use O_DIRECT file mode.
NVIDIA® GPUDirect® Storage (GDS) is the newest addition to the GPUDirect family. GDS enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, which avoids a bounce buffer through the CPU. Using this direct path can relieve effective system bandwidth bottlenecks and decrease the latency and utilization load on the CPU.
GDS can provide significant benefit when it can leverage the O_DIRECT (fcntl.h) file mode for a direct data path between GPU memory and storage. There are many conditions that must be met to achieve the performance benefits of the O_DIRECT mode, and these conditions are not always met by all file systems. The conditions might depend on the transfer size, whether it’s a read or write, whether the write is to new data (past the end of the file or to a hole in the file), and on many other conditions such as whether checksums are required. This document describes the conditions where O_DIRECT, on which GDS relies, can be used.
The target audience for this guide includes:
End users and administrators who:
Understand file systems, so they can carefully consider the implications of features they enable.
Compare support from different file systems to determine the appropriate models and how to use them.
Evaluate the weighted fraction of cases that do or don’t use O_DIRECT effectively.
Middleware developers who:
Consider the design trade-offs that increase the likelihood that the vendor layer can effectively use O_DIRECT.
Filesystems vendors and implementers who:
Accelerate their assessment of the various cases to be handled with O_DIRECT as they enable GDS in new or customized file systems.