2. Software Architecture#
GDS enables a DMA engine near storage (NVMe or NIC) to push (or pull) data directly into (and out of) GPU memory. cuFile APIs are passed parameters for one file, a file offset, a size to transfer, and a GPU virtual address to or from which data can be read or written. Although the resulting aggregate transfer is one contiguous virtual address range, several smaller transfers may occur in the implementation. The filesystem breaks the contiguous virtual address range into what might become multiple transfers that may span multiple devices. An example is RAID-0 and potentially multiple pages with non-contiguous physical address ranges. The resulting set of physical address ranges is called a scatter-gather list.
Existing operating systems attempting to program DMA engines cannot process GPU virtual addresses without help.
The GDS-enabled kernel drivers use callbacks to the GDS kernel module, nvidia-fs.ko, for all interfaces before
CUDA 12.8 and for non-NVMe mounts for CUDA 12.8 and beyond.. These callbacks provide the GPU virtual addresses needed
in the final scatter-gather list used to program the DMA engine.
2.1. Software Components#
The following layers exist in the GDS software stack:
The application, which includes cufile.h and which makes cuFile API calls from the CPU.
The GDS user-level library,
libcufile.so.The Linux virtual filesystem, VFS.
Linux or vendor kernel storage drivers.
The GDS kernel-level library,
nvidia-fs.ko(see conditions below).
2.2. Primary Components#
The primary components in the GDS software architecture are:
(From NVIDIA)
libcufile.so, which is the user-level cuFile library:Implements the cuFile API, which is the application-facing API for GDS.
cuFileReadis shown in the architecture overview graphic in Software Components for Deployment.There are two alternatives to implement the cuFile API:
Use the
nvidia-fs.kokernel driver.All filesystems that use VFS use this path except for the cases listed below under
nvidia-fs.ko.
The cuFile user library implements an alternative implementation that does the following:
Uses its non-page cache buffering in the CPU system memory.
Uses the standard POSIX call implementations.
Does not need to use the NVFS kernel driver.
This is a compatibility mode that does not enjoy the GDS benefits.
(Not from NVIDIA) Non-block-based or distributed filesystems:
These filesystems might be the standard Linux virtual filesystem (VFS), for example an NFS driver or a third-party proprietary system.
The selection of control paths is based on how filesystems are mounted:
<file path> –> <mount point> –> <filesystem selection>
In some cases, NVIDIA provides patches to these, or alternate, implementations, for example, to kernel modules for NVMe and NVMe-oF.
(From NVIDIA) Kernel-level
nvidia-fs.kodriver:Handles IOCTLs from the cuFile user library.
Implements DMA callbacks to check and translate GPU virtual addresses to physical addresses. These callbacks are called from storage drivers.
Manages the mechanisms and buffering that enable DMA from the device.
As of CUDA 12.8, GDS supports additional peer-to-peer DMA for NVMe devices using upstream kernel PCI P2PDMA infrastructure for x86_64 platforms (local or remote with SNAP). This feature will no longer depend on
nvidia-fs.koor custom patches tonvme.ko. This feature is dependent on OS distributions that support PCI P2PDMA, for example linux kernel version 6.2 and above on Ubuntu distributions. In addition OpenRM driver version 570.x or above must be installed with specific registration keys to allow OpenRM driver to enable the PCI P2PDMA support. RAID0, NVMe Multipathing are currently not enabled with PCI P2PDMA in the upstream kernel and will not work without a specialized patch. See here for configuration details.OS support for the P2PDMA feature can be checked as follows:
$ cat /proc/kallsyms | grep -i p2pdma_pgmap_ops 0000000000000000 d p2pdma_pgmap_ops
Note
The Linux kernel core is completely unmodified.
2.2.1. Workflows for GDS Functionality#
The two workflows that are associated with GDS functionality are illustrated in the following graphic:
Figure 2.1 Workflows for GDS Functionality#
For more information about these workflows, see Workflow 1 and Workflow 2.
2.2.2. Workflow 1#
Use the following steps to complete Workflow 1.
The first workflow pertains to cuFileRead and cuFileWrite usage. The GPU virtual addresses are represented
by proxy CPU system memory addresses. The proxy CPU system memory addresses are passed through the Linux IO stack
and are converted to device-specific DMA bus addresses.
Note
None of the following steps are used on a standard pread or pwrite POSIX call.
``App`` to ``libcufile.so``.
GPU applications or GPU-enabled frameworks link to the cuFile library
The applications or frameworks call the cuFile Driver and IO APIs, such as
cuFileReadandcuFileWrite.
The alignment is handled at this level, and there might be some performance impact, so that buffers do not need to be aligned, such as to 4KB pages or 512KB storage offsets and chunk sizes.
``libcufile``.
libcufilemakes decisions about which mode to use based on the filesystem, the configuration, and the hardware support to select between compatibility mode and GDS, and whether to use internal GPU buffers for efficiency.``libcufile.so`` to ``nvidia-fs.ko``.
The cuFile library,
libcufile.so, services those calls and makes appropriate IOCTL calls to thenvidia-fs.kodriver.The library interacts with the CUDA user-mode driver library,
libcuda.so, as necessary for the stream subset of the cuFile APIs.
``nvidia-fs.ko`` to ``VFS``.
The kernel driver iterates through the set of necessary IO operations and passes in the IO completion callback, in
kiocb->common.ki_completewith the callback function valuenvfs_io_completethat will be used in step 7. Those calls are to the VFS, which calls the appropriate lower layers, such as the standard Linux block system (ext4/XFS and NVMe) or another vendor-distributed filesystem such as EXAScaler.
Storage kernel drivers to ``nvidia-fs.ko``: Callback APIs are registered via the
cuFileDriverOpeninitialization, as described in Filesystems Interoperability in the GDS External Architecture Spec.With this design, drivers only need to handle GPU addresses through the substeps below. GPU memory addresses are available in a separate map, outside the Linux page map, so the
nvidia-fs.koAPIs are used to complete the following tasks:Check whether the DMA target address is on the GPU (
nvfs_is_gpu_page) and needs to be handled differently.Query the list of GPU DMA target addresses by using
nvfs_dma_map_sg*, which are used instead of the CPU system memory address that is passed through the VFS.
Storage kernel drivers to DMA/RDMA engines: After the appropriate GPU memory addresses are obtained, the underlying DMA engines at (for example, NVMe drivers) or near (for example, NIC) storage can be programmed to move data directly between storage (for example, NVMe or storage controller or NIC) and GPU memory. The special proxy addresses in CPU system memory are not accessed by the DMA engines.
DMA/RDMA engines to storage kernel driver: Completion of each block transfer is signaled back to the storage driver layer.
The completion of each iteration is signaled back to the nvidia-fs.ko driver by using the callback that was registered in step 4.
2.2.3. Workflow 2#
This section provides information about the second workflow that relates to reads and writes with user-space RDMA using ib_verbs.
App to ``libcufile.so``: GPU applications or GPU-enabled frameworks link to the cuFile library and call the cuFile Driver and IO APIs. The alignment is handled at this level, though perhaps with some performance impact, so that buffers do not need to be aligned, such as 4KB pages or 512KB storage offsets and chunk sizes.
``libcufile_rdma.so``: obtain RDMA info (keys, GID, LID, and so on) to libcufile.
``libcufile.so``
to vendor library: libcufile calls the appropriate vendor library callback functions to communicate the Rkeys directly in userspace or through thenvidia-fskernel callbacks depending on the vendor driver implementation.
2.3. Aligning with Other Linux Initiatives#
There have been efforts in the Linux community to add native support for DMA among peer devices, which can include NICs and GPUs. As this support is upstreamed, it will take time for all users to adopt the new Linux versions via distributions. In the meantime, NVIDIA has worked with a wide variety of third-party vendors to enable GDS.
Specifically, Linux has added support for PCI P2PDMA in Linux kernel version 6.2 and above kernels, enabling ZONE_DEVICE
address pointers to GPU memory to be passed through the VFS without causing a page fault, in the case of NVMe, nvidia-fs.ko
The cuFile APIs and their implementation are the mechanism by which CUDA adds support for file IO. The cuFile APIs cover explicit transfers between CPU and GPU storage and memory. The APIs also add support for asynchrony and batching, which are not available in POSIX IO. The cuFile APIs will remain relevant after the functionalities mentioned earlier are added to Linux. Only the underlying implementations will change, but not the existing cuFile APIs.
NVIDIA’s initial implementations for cuFile focus on distributed filesystems and systems where appropriate drivers have been installed to enable a direct transfer between storage and GPU memory without using a bounce buffer in the CPU. For compatibility and broader applicability, later implementations may support extensions for local storage and implicit transfers.