3. Deployment#
Effective use of GDS entails having a good understanding of how GDS is deployed, its dependencies, and its limitations and constraints.
3.1. Software Components for Deployment#
Here is some information about the software components that are required to deploy GDS.
cuFile APIs are a supplement to the CUDA driver and runtime APIs and are distributed and installed with the CUDA toolkit.
Applications access cuFile functionality by including cuFile.h and linking against the libcufile.so library.
For the stream subset of the cuFile APIs, a CUDA stream parameter is needed, which takes different forms for runtime
and driver APIs. The cudaFile and cuFile prefixes are used for those two cases, respectively. The conversion from
runtime to driver APIs can be done in header files.
Beyond libcufile.so, there are no linker dependencies that are required to use the cuFile API, but a runtime
dynamic dependency on libcuda.so exists. No link dependency on other CUDA Toolkit libraries, CUDA Runtime
libraries, or any other components of the display driver currently exist. However, an eventual runtime dependency
on the CUDA Runtime library should be anticipated for applications that are using the cudaFile* APIs after they
are added to CUDA Runtime. This step is consistent with the application using any other cuda* API and use of
the CUDA Runtime in deployment is covered in the CUDA deployment documentation at NVIDIA Developer Documentation.
In addition to libcuda.so, cuFile has dependencies on external third-party libraries.
The following table provides information about the third-party libraries and CUDA library levels:
Level |
APIs, Types, and Enum Style |
Dependencies |
Packaged Together |
|---|---|---|---|
cuFile user library |
Matches the following CUDA driver conventions:
|
|
Shipped separately from |
CUDA runtime + toolkit |
Compatibility in |
None |
|
|
|
|
Separately shippable with respect to the NVIDIA driver (until perhaps it gets merged), but it might become co-installed. |
Note
GDS has no internal dependencies on other libraries:
cuFile libraries and drivers do not modify CUDA.
The streams subset of the cuFile APIs use the CUDA user driver (
libcuda.so) and CUDA runtime (libcudart.so).The only APIs used by those drivers are public APIs.
3.2. Using GPUDirect Storage in Containers#
GDS has user-level and kernel-level components. Containers include only user-level code and rely on kernel-level components having been installed on the host machine. Applications can be developed with GDS’s header files and user-level library and distributed in containers. When the appropriate drivers and vendor-enable kernel software are not installed or properly configured, GDS’s compatibility mode enables the cuFile APIs to continue to maintain functional operation with minimized performance impact.
See NVIDIA/MagnumIO for an example of GDS used in a container.