3. Deployment#

Effective use of GDS entails having a good understanding of how GDS is deployed, its dependencies, and its limitations and constraints.

3.1. Software Components for Deployment#

Here is some information about the software components that are required to deploy GDS.

cuFile APIs are a supplement to the CUDA driver and runtime APIs and are distributed and installed with the CUDA toolkit.

Applications access cuFile functionality by including cuFile.h and linking against the libcufile.so library. For the stream subset of the cuFile APIs, a CUDA stream parameter is needed, which takes different forms for runtime and driver APIs. The cudaFile and cuFile prefixes are used for those two cases, respectively. The conversion from runtime to driver APIs can be done in header files.

Beyond libcufile.so, there are no linker dependencies that are required to use the cuFile API, but a runtime dynamic dependency on libcuda.so exists. No link dependency on other CUDA Toolkit libraries, CUDA Runtime libraries, or any other components of the display driver currently exist. However, an eventual runtime dependency on the CUDA Runtime library should be anticipated for applications that are using the cudaFile* APIs after they are added to CUDA Runtime. This step is consistent with the application using any other cuda* API and use of the CUDA Runtime in deployment is covered in the CUDA deployment documentation at NVIDIA Developer Documentation.

In addition to libcuda.so, cuFile has dependencies on external third-party libraries.

The following table provides information about the third-party libraries and CUDA library levels:

Table 3.1 Third-Party Libraries and CUDA Library Levels#

Level

APIs, Types, and Enum Style

Dependencies

Packaged Together

cuFile user library

Matches the following CUDA driver conventions:

  • cuFile APIs

  • cuFile_ enum vals and defines

  • CU_FILE_ errors

  • Provides libcufile.so until perhaps it gets merged into libcuda.so.

  • Provides cuFile.h until perhaps it gets merged into cuda.h.

  • External library dependencies: libmount-dev libnuma-dev

Shipped separately from libcufile.so.

CUDA runtime + toolkit

Compatibility in cufile.h for streams APIs’ usage of cudaStream_t.

None

cufile.h remains distinct from cuda.h and cuda_runtime.h.

nvidia-fs kernel driver

nvfs_ prefix

  • Provides nvidia-fs.ko GPLv2.

  • cuFile uses kernel facilities that are in Linux kernel version 4.15.0.x and later.

  • cuFile has a dependency on MLNX_OFED versions (5.1 and later) for support for RDMA-based filesystems.

  • GPUDirect partners may have dependencies on host-channel adapters that are Mellanox Connect X-5 or later.

Separately shippable with respect to the NVIDIA driver (until perhaps it gets merged), but it might become co-installed.

Note

GDS has no internal dependencies on other libraries:

  • cuFile libraries and drivers do not modify CUDA.

  • The streams subset of the cuFile APIs use the CUDA user driver (libcuda.so) and CUDA runtime (libcudart.so).

    The only APIs used by those drivers are public APIs.

3.2. Using GPUDirect Storage in Containers#

GDS has user-level and kernel-level components. Containers include only user-level code and rely on kernel-level components having been installed on the host machine. Applications can be developed with GDS’s header files and user-level library and distributed in containers. When the appropriate drivers and vendor-enable kernel software are not installed or properly configured, GDS’s compatibility mode enables the cuFile APIs to continue to maintain functional operation with minimized performance impact.

See NVIDIA/MagnumIO for an example of GDS used in a container.