> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/dev/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server. # dev ## Docs - [Tutorial](https://docs.nvidia.com/networking/cloudai/dev/tutorial.md) - [User Guide](https://docs.nvidia.com/networking/cloudai/dev/user-guide.md) - [Python API](https://docs.nvidia.com/networking/cloudai/dev/api.md) - [Systems](https://docs.nvidia.com/networking/cloudai/dev/systems.md) - [Development](https://docs.nvidia.com/networking/cloudai/dev/development.md) - [Troubleshooting](https://docs.nvidia.com/networking/cloudai/dev/troubleshooting.md) - [Reporting](https://docs.nvidia.com/networking/cloudai/dev/reporting.md) - [Reports](https://docs.nvidia.com/networking/cloudai/dev/reports.md) - [Pareto Frontiers](https://docs.nvidia.com/networking/cloudai/dev/pareto-frontiers.md) - [Training Report Schema](https://docs.nvidia.com/networking/cloudai/dev/training-report-schema.md) - [Workloads](https://docs.nvidia.com/networking/cloudai/dev/workloads/index.md) - [AI Dynamo](https://docs.nvidia.com/networking/cloudai/dev/workloads/ai-dynamo.md) - [AIConfigurator](https://docs.nvidia.com/networking/cloudai/dev/workloads/aiconfigurator.md) - [Bash Command](https://docs.nvidia.com/networking/cloudai/dev/workloads/bash-cmd.md) - [Chakra Replay](https://docs.nvidia.com/networking/cloudai/dev/workloads/chakra-replay.md) - [DDLB](https://docs.nvidia.com/networking/cloudai/dev/workloads/ddlb.md) - [DeepEP Benchmark](https://docs.nvidia.com/networking/cloudai/dev/workloads/deepep.md) - [DynamoMocker](https://docs.nvidia.com/networking/cloudai/dev/workloads/dynamo-mocker.md) - [fio](https://docs.nvidia.com/networking/cloudai/dev/workloads/fio.md) - [JaxToolbox workloads (DEPRECATED)](https://docs.nvidia.com/networking/cloudai/dev/workloads/jax-toolbox.md) - [MegatronBridge](https://docs.nvidia.com/networking/cloudai/dev/workloads/megatron-bridge.md) - [NCCL](https://docs.nvidia.com/networking/cloudai/dev/workloads/nccl.md) - [NeMo v1.0 aka NemoLauncher (DEPRECATED)](https://docs.nvidia.com/networking/cloudai/dev/workloads/nemo-launcher.md) - [Nemo Run](https://docs.nvidia.com/networking/cloudai/dev/workloads/nemo-run.md) - [NIXL Bench](https://docs.nvidia.com/networking/cloudai/dev/workloads/nixl-bench.md) - [NIXL EP](https://docs.nvidia.com/networking/cloudai/dev/workloads/nixl-ep.md) - [NIXL KVBench](https://docs.nvidia.com/networking/cloudai/dev/workloads/nixl-kvbench.md) - [NIXL CTPerf](https://docs.nvidia.com/networking/cloudai/dev/workloads/nixl-perftest.md) - [OSU](https://docs.nvidia.com/networking/cloudai/dev/workloads/osu.md) - [SGLang](https://docs.nvidia.com/networking/cloudai/dev/workloads/sglang.md) - [Sleep](https://docs.nvidia.com/networking/cloudai/dev/workloads/sleep.md) - [SRT Slurm](https://docs.nvidia.com/networking/cloudai/dev/workloads/srt-slurm.md) - [Slurm Container](https://docs.nvidia.com/networking/cloudai/dev/workloads/slurm-container.md) - [UCC](https://docs.nvidia.com/networking/cloudai/dev/workloads/ucc.md) - [vLLM](https://docs.nvidia.com/networking/cloudai/dev/workloads/vllm.md) - [Installation Requirements](https://docs.nvidia.com/networking/cloudai/dev/workloads/workloads-requirements-installation.md) > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/dev/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server. ## Overview CloudAI benchmark framework aims to develop an industry standard benchmark focused on grading Data Center (DC) scale AI systems in the cloud. The primary motivation is to provide automated benchmarking on various systems. This document contains the following chapters: * [Tutorial](/tutorial) * [User Guide](/user-guide) * [Python API](/api) * [Reporting](/reporting) * [Systems](/systems) * [Workloads](/workloads/index) * [Development](/development) * [Troubleshooting](/troubleshooting) ## Getting Started CloudAI officially targets Python 3.14 while remaining compatible with Python 3.10 and newer. Use uv 0.13.4 or newer to install and run CloudAI. For pip-based installation, use pip 26.2.2 or newer. These versions include certificate trust updates for the eMudhra/emSign CA compromise. ```bash git clone git@github.com:NVIDIA/cloudai.git cd cloudai uv run cloudai --help ``` > **Note** > > For more instructions on how to set up access for `enroot`, see [Installation Requirements](/workloads/workloads-requirements-installation). ### pip-based Installation See the preferred Python version in the `.python-version` file and make sure you have it installed (For installation, see [Custom Python Version Installation](#install-custom-python-version)). Python 3.10 and newer remain supported. Follow these steps: ```bash git clone git@github.com:NVIDIA/cloudai.git cd cloudai python3.14 -m venv venv source venv/bin/activate pip install -e . ``` > **Note** > > When upgrading an existing installation, also update the host's `ca-certificates` package and the certificate bundles and installers in existing workload environments and container images. Updating CloudAI does not update these environments. ### Custom Python Version Installation If your system Python version is not supported, you can install a custom version using the [uv](https://docs.astral.sh/uv/getting-started/installation/) tool: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh source $HOME/.local/bin/env uv venv --seed # picks the python version from .python-version # --seed installs pip and setuptools source .venv/bin/activate ``` ## Key Concepts CloudAI operates on three main schemas: * **System Schema**: Describes the system, including the scheduler type, node list, and global environment variables * **Test Schema**: An instance of a test template with custom arguments and environment variables * **Test Scenario Schema**: A set of tests with dependencies and additional descriptions about the test scenario These schemas enable CloudAI to be flexible and compatible with different systems and configurations. ## CloudAI Modes Usage Examples The following are the global options for the `cloudai` command: * `--log-file `: specify a file to log output; by default `debug.log` in the current directory is used. Contains log entries of level `DEBUG` and higher * `--log-level `: specify logging level for standard output; default is `INFO` ### Available Modes * [run](#run) * [dry-run](#dry-run) * [generate-report](#generate-report) * [install](#install) * [uninstall](#uninstall) * [list](#list) * [verify-configs](#verify-configs) ### Usage Examples #### run This mode runs workloads. It automatically installs prerequisites if they are not met. ```bash cloudai run\ --system-config conf/common/system/example_slurm_cluster.toml\ --tests-dir conf/common/test\ --test-scenario conf/common/test_scenario/sleep.toml ``` #### dry-run This mode simulates running experiments without actually executing them. This is useful for verifying configurations and testing experiment setups. ```bash cloudai dry-run\ --system-config conf/common/system/example_slurm_cluster.toml\ --tests-dir conf/common/test\ --test-scenario conf/common/test_scenario/sleep.toml ``` #### generate-report This mode generates reports under the scenario directory. It automatically runs as part of the `run` mode after experiments are completed. ```bash cloudai generate-report\ --system-config conf/common/system/example_slurm_cluster.toml\ --tests-dir conf/common/test\ --test-scenario conf/common/test_scenario/sleep.toml\ --result-dir /path/to/result_directory ``` #### install This mode installs test prerequisites. For more details, refer to the [Installation Requirements](/workloads/workloads-requirements-installation) guide. It automatically runs as part of the `run` mode if prerequisites are not met. ```bash cloudai install\ --system-config conf/common/system/example_slurm_cluster.toml\ --tests-dir conf/common/test\ --test-scenario conf/common/test_scenario/sleep.toml ``` #### uninstall This mode is the opposite of the install mode. This mode removes installed test prerequisites. ```bash cloudai uninstall\ --system-config conf/common/system/example_slurm_cluster.toml\ --tests-dir conf/common/test\ --test-scenario conf/common/test_scenario/sleep.toml ``` #### list This mode lists internal components available within CloudAI. ```bash cloudai list ``` #### verify-configs This mode verifies the correctness of system, test, and test scenario configuration files. ```bash # verify all at once cloudai verify-configs conf # verify a single file cloudai verify-configs conf/common/system/example_slurm_cluster.toml # verify all scenarios using specific folder with Test TOMLs cloudai verify-configs --tests-dir conf/release/spcx/l40s/test conf/release/spcx/l40s/test_scenario ``` ## Notice and Disclaimer NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses. You are solely responsible for finding, reviewing and complying with all applicable terms, conditions, and licenses, and for verifying the security, integrity and suitability of any retrieved materials for your specific use case. This software is provided "AS IS", without warranty of any kind. The author makes no representations or warranties regarding any retrieved materials, and assumes no liability for any losses, damages, liabilities or legal consequences from your use or inability to use this software or any retrieved materials. Use this software and the retrieved materials at your own risk. ## Documentation Update History | Version | Date | Description | | ------- | -------------- | ---------------------------------------------- | | v1.5.0 | April xx, 2026 | Initial release of this documentation version. |