6. Troubleshooting IO Activity

This section covers issues that are related to IO activity and the interactions with the rest of Linux.

6.1. Managing Coherency of Data in the Page Cache and on Disk

When using GDS, files are often opened with the O_DIRECT mode. When IO is complete, in the context of DIRECT IO, it bypasses the page cache.

Starting with CUDA toolkit 12.2 (GDS version 1.7.x) files can also be opened with non-O_DIRECT mode. Even in such a case, whenever the library software deems fit, it will follow the GDS enabled O_DIRECT path. This conserves coherency by default.

• On EXAScaler file system:

  • For reads, IO bypasses the page cache and fetches the data directly from backend storage.

  • When writes are issued, the nvidia-fs drivers will try to flush the data in the page cache for the range of offset-length before issuing writes to the VFS subsystem.

  • The stats that track this information are:

    • pg_cache

    • pg_cache_fail

    • pg_cache_eio

• On WekaIO file system:

  • For reads, IO bypasses the page cache and fetches the data directly from backend storage.