User Guide
The purpose of this chapter is to help users utilize and understand the different components of the CloudAI framework. The chapter covers topics such as adding new tests and downloading datasets for running NeMo-launcher.
System Schema
In this section, we introduce the concept of the system schema, explain the meaning of each field, and describe how the fields should be used. The system schema is a TOML file that allows users to define a system’s configuration.
Field Descriptions
Field Descriptions
RunAI Scheduler
When using RunAI as the scheduler, additional fields must be specified in the system schema TOML file. The following is a list of required fields and how to set them:
-
After logging into the RunAI web interface, navigate to Access → Applications and create a new application to obtain
app_idandapp_secret -
Use your assigned project and cluster IDs. Contact your administrator if they are not available
-
All other fields follow the same semantics as in the Slurm system schema (e.g.,
install_path,output_path)
Test Scenario Schema
A test scenario is a set of tests with specific dependencies between them. A test scenario is described in a TOML schema file. The following is an example of a test scenario file:
The name field is the test scenario name, which can be any unique identifier for the scenario. Each test has a section name, following the convention Tests.1, Tests.2, etc., with an increasing index. The name of a test should be specified in this section and must correspond to an entry in the test schema. If a test in a test scenario is not present in the test schema, CloudAI will not be able to identify it.
There are two ways to specify nodes:
-
Using the
num_nodesfield as shown in the example -
Specifying nodes explicitly like
nodes`` ``=`` ``["node-001",`` ``"node-002"]
Note: When an explicit node list is provided (e.g., nodes`` ``=`` ``["node-001",`` ``"node-002"]), CloudAI lets Slurm apply the arbitrary distribution policy for task placement.
Alternatively, you can utilize the groups feature in the system schema to specify nodes like nodes`` ``=`` ``['PARTITION_NAME:GROUP_NAME:NUM_NODES'], which allocates num_nodes from the group name in the specified partition. You can also use nodes`` ``=`` ``['PARTITION_NAME:GROUP_NAME:max_avail'], which allocates all the available nodes from the group name in the specified partition.
You can optionally specify a time limit in the Slurm format. Tests can have dependencies. If no dependencies are specified, all tests will run in parallel.
CloudAI supports three types of dependencies:
-
start_post_init -
start_post_comp -
end_post_comp
Dependencies of a test can be described as a subsection of the test. It requires other tests’ id and dependency type.
-
start_post_initmeans the test starts after the prior test begins, with a specified delay. -
start_post_compmeans the test starts after the prior test completes. -
end_post_compmeans the test ends when the prior test completes.
Agents Configuration
For DSE workloads, you can pass agent configuration via agent_config in the scenario.
BaseAgentConfig includes:
-
random_seed: controls deterministic random behavior in agents. -
start_action: controls the very first action strategy ("random"or"first"). -
rewards(optional): nested table mapped toRewardOverrides. When present,CloudAIGymEnvuses it for constraint failures and for substituting failed-metric values in observations.-
constraint_failure: reward returned whenTestDefinition.constraint_checkfails on a step. If omitted or unset, the environment uses-1.0. -
metric_failure: value written into the observation vector when a metric is missing or is the canonical error sentinelMETRIC_ERROR(a distinct object from numeric metrics, so a valid-1.0result is not mistaken for an error). If omitted or unset, observations use-1.0for that slot.
-
Example:
When an agent honors start_action`` ``=`` ``"first", it should start from CloudAIGymEnv.first_sweep (the sweep is built from first values of each sweep parameter). start_action`` ``=`` ``"random" means starting from a random action, typically seeded by random_seed.
Custom agents may extend the BaseAgentConfig and offer more parameters to configure.
DSE parameter exclusions
CloudAI builds the DSE parameter space implicitly from list-valued fields under cmd_args, list-valued extra_env_vars, and list-valued num_nodes. Most lists mean “try each value”, but some workload settings are real list-valued configuration, such as worker port lists or ordered benchmark phases.
Use dse_excluded_args when a list under cmd_args should stay intact instead of becoming a sweep dimension. Entries must be dot-separated paths that start with cmd_args. and may point to either a single field or a parent object. Matching is prefix-based, so excluding cmd_args.foo also excludes nested list-valued fields such as cmd_args.foo.bar from DSE parameter discovery.
In this example, cmd_args.lmcache.chunk_size is still swept, while cmd_args.lmcache.lmcache_worker_ports is passed through as one list value. The exclusion does not remove or mutate the field; it only prevents CloudAI from adding that path to the DSE parameter space.
dse_excluded_args currently applies only to cmd_args paths. It does not exclude list-valued extra_env_vars or num_nodes; those lists are still interpreted as sweep dimensions. To exclude many nested list fields at once, exclude their common parent path. Common examples are cmd_args.aiperf_phases and cmd_args.lmcache.lmcache_worker_ports.
Metric errors and report strategies
Report generation strategies signal a failed or missing metric by returning the singleton METRIC_ERROR from cloudai.core (type MetricErrorSentinel; float(METRIC_ERROR) is -1.0 for numeric use). The declared return type of ReportGenerationStrategy.get_metric is MetricValue (float`` ``|`` ``MetricErrorSentinel).
When branching on success vs failure, use identity (value`` ``is`` ``METRIC_ERROR), not equality to -1.0, so a legitimate metric of -1.0 is not treated as an error. The gym observation path uses that identity check before applying agent_config.rewards.metric_failure.
Configuring HTTP Data Repository
The HTTP Data Repository is currently supported for Slurm systems only. To enable access, you must update your system schema file and create a credential file in your CloudAI project’s root directory.
The following steps are required to configure the HTTP Data Repository:
Updating the System Schema File
Add the following section to your system schema TOML file (e.g., system_schema.toml):
Replace the endpoint with your actual data repository URL.
Creating the Credential File
In the root of your CloudAI project (i.e., the current working directory), create a file named .cloudai.toml with the following content:
Replace <your-api-token-here> with your actual token.
Usage
Both the endpoint and token must be valid for the HTTP Data Repository to function correctly. If either is missing or incorrect, data will not be posted.
Using Test Hooks in CloudAI
A test hook in CloudAI is a specialized test that runs either before or after each main test in a scenario, providing flexibility to prepare the environment or clean up resources. Hooks are defined as pre-test or post-test and referenced in the test scenario’s TOML file using pre_test and post_test fields.
CloudAI organizes hooks in a dedicated directory structure:
-
Hook directory: All hook configurations reside in
conf/hook/ -
Hook test scenarios: Place pre-test hook and post-test hook scenario files in
conf/hook/ -
Hook tests: Place individual tests referenced in hooks within
conf/hook/test/
In the execution flow, pre-test hooks run before the main test, which only executes if the pre-test completes successfully. Post-test hooks follow the main test, provided the prior steps succeed. If a pre-test hook fails, the main test and its post-test hook are skipped.
Downloading DeepSeek Weights
To run DeepSeek R1 tests in CloudAI, you must download the model weights in advance. These weights are distributed via the NVIDIA NGC Registry and must be manually downloaded using the NGC CLI.
The following steps are required to download the DeepSeek weights:
Installing NGC CLI
Download and install the NGC CLI using the following commands:
This will make the ngc command available in your terminal.
Configuring NGC CLI
Authenticate your CLI with your NGC API key by running:
When prompted, paste your API key, which you can obtain from https://org.ngc.nvidia.com/setup.
Downloading the Weights
Navigate to the directory where you want the DeepSeek model weights to be stored, then run:
This command will create a folder named:
inside your current directory.
Verifying the Download
Make sure the full model has been downloaded by checking the folder size:
The expected size is approximately 642 GB. If it’s significantly smaller, remove the folder and re-run the download.
Slurm
The following are subtopics related to Slurm:
Single sbatch vs Per-case sbatch
CloudAI supports two modes for submitting Slurm jobs:
-
(default) Per-case sbatch mode: each case is submitted as a separate sbatch job, allowing flexible scheduling and dependency management.
- Suits best when cases can run in parallel or there are no dependencies between cases.
-
Single sbatch mode: all cases are submitted together in a single sbatch job and share the same nodes. Each next case starts after the previous one completes. To enable it one needs to pass
--single-sbatchflag tocloudai`` ``runcommand (works only for Slurm systems).-
Suits best for jobs when cases need to run on the same nodes for performance reasons.
-
There is no support for dependency management between cases yet; all jobs run one after another.
-
Assuming two cases in a scenario like this:
Regular output directory structure (some files are omitted for clarity):
Single sbatch mode output directory structure:
Most of the files are the same: output files, env vars script, test run metadata. The difference for single sbatch mode is that slurm-job.toml is created for the entire job as well as cloudai_sbatch_script.sh. Extra common.err/common.out files are created for general sbatch outputs.
Extra srun and sbatch Arguments
CloudAI forms sbatch script and srun commands following internal rules. Users can affect the generation by setting special arguments in the system TOML file.
For example (in a system TOML file):
It will result in sbatch file content like this:
Another example (in a system TOML file):
Will result in srun command inside sbatch script like this:
Container Mounts
CloudAI runs all slurm jobs using containers. To simplify file system related tasks, CloudAI mounts the following directories into the container:
-
Test output directory (
<output_path>/<scenario_name_with_timestamp>/<test_name>/<iteration>, likeresults/scenario_2024-06-18_17-40-13/Tests.1/0) is mounted as/cloudai_run_results. -
Test specific mounts can be specified in TOML files:
These mounts are not verified for validity and do not override default mounts.
-
Test specific mounts can be mounted in-code.
Head Node without Shared Storage Available on Compute Nodes
When compute nodes do not share the file system with head node, --enable-cache-without-check for run and dry-run skips the real check for cache existence, but still builds all paths correctly. The flow is like this:
-
[on the head node] run
cloudai`` ``install. -
[on the head node] copy cache to compute nodes.
-
Modify
system.tomlto set compute nodes’ installation root. -
Run
cloudai`` ``run`` ``--enable-cache-without-check`` ``....
Dev Details
SlurmCommandGenStrategy defines abstract method _container_mounts(tr:`` ``TestRun) that must be implemented by every subclass. This method is used in SlurmCommandGenStrategy.container_mounts(tr:`` ``TestRun) (defined as @final) where mounts like /cloudai_run_results (default mount), TestDefinition.extra_container_mounts (from Test TOML) and test specific mounts (defined in-code) are added.
Nsys Tracing
Users can enable Nsys tracing for any workload when running via Slurm. Note that nsys should be available on the compute nodes; CloudAI does not manage it.
The configuration fields are:
Fields with None value are not passed to nsys command.