AIStoreThe tool we have developed is called AIS Loader (aisloader) - a load generator to benchmark and stress-test AIStore, or any S3-compatible backend. AIS Loader generates arbitrary read/write workloads including those (synthetic) ones that mimic training and inference workloads. The latter allows to run benchmarks in isolation and avoid compute-side bottlenecks, if any.
For usage, run: aisloader, or aisloader usage, or aisloader --help.
To get started, go to the aistore root directory and run:
Following in this document are a few easy usage examples and dry-run instructions. For detailed description and more examples, please see aisloader readme.
For the most recently updated command-line options and examples, please run aisloader or aisloader usage.
Destroy existing ais bucket - the first example. Delete all objects in a given Cloud-based bucket - the second example:
Time-based 100% PUT into ais bucket. Upon exit the bucket is emptied (by default):
100% GET from an ais bucket:
Mixed 30%/70% PUT and GET of variable-size objects to/from a Cloud bucket. PUT will generate random object names and is limited by the 10GB total size. Cleanup is not disabled, which means that upon completion all generated objects will be deleted:
PUT 2000 objects named as aisloader/hex({0..2000}{loaderid}):
PUT 30% of random TAR files and 70% GET of transformed TARs to TFRecords with tar2tf transformation.
Full aisloader documentation, including more examples and full options list, is available here.
