cuFile Compatibility Mode

Use Cases

cuFile APIs can be used in different scenarios:


The cuFile library provides a mechanism for cuFile reads and writes to use compatibility mode using POSIX pread and pwrite APIS respectively to system memory and copying to GPU memory. The behavior of compatibility mode with cuFile APIs is determined by the following configuration parameters.
Configuration Option (default) cuFile IO Behavior
"allow_compat_mode": true If true, falls back to using compatibility mode when the library detects that the buffer file descriptor opened cannot use GPUDirect Storage.
"GPUDirect Storage_rdma_write_support": true If false, forces compatibility mode to be used for writes even when the underlying file system is capable of performing GPUDirect Storage writes.
"posix_unaligned_writes" : false

If true, forces compatibility mode to be used for writes where the file offset and/or IO size is not aligned to Page Boundary (4KB).

“lustre:posix_GPUDirect Storage_min_kb" : 0

If greater than 0, compatibility mode is used for IO sizes between [1 - posix_GPUDirect Storage_min_kb] specified in kB.

Note: This option will force posix mode even if “allow_compat_mode” is set to “false”.
"weka:rdma_write_support" : false

If this option is false, WekaFs will use compatibility mode for all writes to the filesystem.

Note: if the option is set to “false”, cuFile library will use the posix path even if the allow_compat_mode option is true or false.

"rdma_dynamic_routing": false,

"rdma_dynamic_routing_order": [ " "SYS_MEM" ]

If rdma_dynamic_routing is set to true and rdma_dynamic_routing_order is set to [“SYS_MEM”] , then all IO for DFS will use compatibility mode.

In addition to the above configuration options, compatibility mode will be used as a fallback option for following use cases.

Use Case cuFile IO Behavior
IBM Spectrum Scale File System writes. Note: All IBM Spectrum Scale in GPUDirect Storage 1.0.0.x release use posix path for writes even if the allow_compat_mode option is set to false.
No BAR1 memory in GPU. Use compatibility mode.

For wekaFS or IBM Spectrum Scale mounts: If there are no rdma_dev_addr_list specified, or failure to register MR with ib device.

Use compatibility mode.
Bounce buffers cannot be allocated in GPU memory. Use compatibility mode.
For WekaFS and IBM Spectrum Scale: If the kernel returns -ENOTSUP for GPUDirect Storage read/write. Retry the IO operation internally using compatibility mode.
The nvidia_fs.ko driver is not loaded. All IO operations will use compatibility mode.