cuFile Compatibility Mode

cuFile APIs can be used in different scenarios:

Developers building GPUDirect Storage applications with cuFile APIs, but don’t have the supported hardware configurations.
Developers building applications running on GPU cards that have CUDA compute capability > 6, but don’t have BAR space exposed.
Deployments where nvidia-fs.ko is not loaded or cannot be loaded.
Deployments where the Linux distribution does not support GPUDirect Storage.
Deployments where the filesystem may be not supported with GPUDirect Storage.
Deployments where the network links are not enabled with RDMA support.
Deployment where the configuration is not optimal for GPUDirect Storage.

The cuFile library provides a mechanism for cuFile reads and writes to use compatibility mode using POSIX pread and pwrite APIS respectively to system memory and copying to GPU memory. The behavior of compatibility mode with cuFile APIs is determined by the following configuration parameters.

Configuration Option (default)	cuFile IO Behavior
`"allow_compat_mode": true`	If `true`, falls back to using compatibility mode when the library detects that the buffer file descriptor opened cannot use GPUDirect Storage.
`"GPUDirect Storage_rdma_write_support": true`	If `false`, forces compatibility mode to be used for writes even when the underlying file system is capable of performing GPUDirect Storage writes.
`"posix_unaligned_writes" : false`	If `true`, forces compatibility mode to be used for writes where the file offset and/or IO size is not aligned to Page Boundary (4KB).
`“lustre:posix_GPUDirect Storage_min_kb" : 0`	If greater than `0`, compatibility mode is used for IO sizes between [1 - `posix_GPUDirect Storage_min_kb`] specified in kB. Note: This option will force posix mode even if “`allow_compat_mode`” is set to “false”.
`"weka:rdma_write_support" : false`	If this option is `false`, WekaFs will use compatibility mode for all writes to the filesystem. Note: if the option is set to “false”, cuFile library will use the posix path even if the `allow_compat_mode` option is `true` or `false`.
`"rdma_dynamic_routing": false,` `"rdma_dynamic_routing_order": [ " "SYS_MEM" ]`	If `rdma_dynamic_routing` is set to `true` and `rdma_dynamic_routing_order` is set to `[“SYS_MEM”]` , then all IO for DFS will use compatibility mode.

Use Case	cuFile IO Behavior
IBM Spectrum Scale File System writes.	Note: All IBM Spectrum Scale in GPUDirect Storage 1.0.0.x release use posix path for writes even if the `allow_compat_mode` option is set to `false`.
No BAR1 memory in GPU.	Use compatibility mode.
For wekaFS or IBM Spectrum Scale mounts: If there are no `rdma_dev_addr_list` specified, or failure to register MR with ib device.	Use compatibility mode.
Bounce buffers cannot be allocated in GPU memory.	Use compatibility mode.
For WekaFS and IBM Spectrum Scale: If the kernel returns `-ENOTSUP` for GPUDirect Storage read/write.	Retry the IO operation internally using compatibility mode.
The `nvidia_fs.ko` driver is not loaded.	All IO operations will use compatibility mode.

Compatible mode does not work in cases where the GPUs have CUDA compute capability less than 6.
Compatible mode is supported only on GPUDirect Storage enabled filesystems.
There is no option to force all IO to use compatibility mode. The user can unload the nvidia_fs.ko or not expose the character devices in the docker container environment.