CloudAI Benchmark Framework
Overview
CloudAI benchmark framework aims to develop an industry standard benchmark focused on grading Data Center (DC) scale AI systems in the cloud. The primary motivation is to provide automated benchmarking on various systems.
This document contains the following chapters:
Getting Started
CloudAI officially targets Python 3.14 while remaining compatible with Python 3.10 and newer.
Use uv 0.13.4 or newer to install and run CloudAI. For pip-based installation, use pip 26.2.2 or newer. These versions include certificate trust updates for the eMudhra/emSign CA compromise.
Note
For more instructions on how to set up access for enroot, see Installation Requirements.
pip-based Installation
See the preferred Python version in the .python-version file and make sure you have it installed (For installation, see Custom Python Version Installation). Python 3.10 and newer remain supported. Follow these steps:
Note
When upgrading an existing installation, also update the host’s ca-certificates package and the certificate bundles and installers in existing workload environments and container images. Updating CloudAI does not update these environments.
Custom Python Version Installation
If your system Python version is not supported, you can install a custom version using the uv tool:
Key Concepts
CloudAI operates on three main schemas:
-
System Schema: Describes the system, including the scheduler type, node list, and global environment variables
-
Test Schema: An instance of a test template with custom arguments and environment variables
-
Test Scenario Schema: A set of tests with dependencies and additional descriptions about the test scenario
These schemas enable CloudAI to be flexible and compatible with different systems and configurations.
CloudAI Modes Usage Examples
The following are the global options for the cloudai command:
-
--log-file`` ``<path>: specify a file to log output; by defaultdebug.login the current directory is used. Contains log entries of levelDEBUGand higher -
--log-level`` ``<level>: specify logging level for standard output; default isINFO
Available Modes
Usage Examples
run
This mode runs workloads. It automatically installs prerequisites if they are not met.
dry-run
This mode simulates running experiments without actually executing them. This is useful for verifying configurations and testing experiment setups.
generate-report
This mode generates reports under the scenario directory. It automatically runs as part of the run mode after experiments are completed.
install
This mode installs test prerequisites. For more details, refer to the Installation Requirements guide. It automatically runs as part of the run mode if prerequisites are not met.
uninstall
This mode is the opposite of the install mode. This mode removes installed test prerequisites.
list
This mode lists internal components available within CloudAI.
verify-configs
This mode verifies the correctness of system, test, and test scenario configuration files.