Reporting
This chapter describes the reporting system in CloudAI. In this chapter, we will cover the following topics:
Overview
CloudAI has two reporting levels:
-
per-test (per each case in a test scenario)
-
per-scenario (per each test scenario)
All reports are generated after the test scenario is completed as part of the main CloudAI process. For Slurm, this means that the login node is used to generate reports.
Per-test reports are linked to a particular workload type (e.g. NcclTest). All per-test reports are implemented as part of the per_test scenario report and can be enabled or disabled via a single configuration option; see Enabling, Disabling and Configuring Reports.
To list all available reports, users can use cloudai`` ``list-reports. Use verbose output to also print report configurations.
Unified Experiment Output
CloudAI writes experiment.json in each scenario’s results directory.
The file contains scenario details, system name, the username of the account running CloudAI in user, and test cases listed under tests. Older files without user are loaded with an empty string. For Slurm runs, the system name is the cluster reported by Slurm. Both Standalone and Slurm execution records are stored in each test case’s runs list. Each record represents an iteration or DSE step and includes its number, process or Slurm job ID, status, timing, result path, and workload metrics.
If username lookup fails, CloudAI logs a warning and records an empty user without interrupting execution.
Timestamps use UTC; durations use seconds. Unknown timestamps are null. A status of unknown indicates the outcome could not be determined. Dry runs include scenario and test-case details without launching workloads.
Live runs status tracking:
-
Standalone jobs are
runningfrom submission. -
Slurm jobs (excluding single-sbatch) start as
pendingwhile waiting for allocation, then transition torunningonce allocated and executing. -
Single-sbatch Slurm run records remain
pendinguntil the allocation finishes and their final statuses are recorded.
Metrics are sourced from TestDefinition.metric_observations(), and are not affected by reporter settings.
MegatronRun exposes iteration_time (milliseconds, lower is better) and tflops_per_gpu (TFLOP/s/GPU, higher is better). Each observation is the mean of the last ten iteration metric lines in stdout.txt, or all available lines when fewer than ten are present, matching the per-test CSV report. No observations are emitted when iteration metric lines are missing.
When a test case is executed successfully, tests[].metrics contains that execution’s metrics. For DSE, it contains metrics from the successful step with the highest valid reward. The search space, selected step, and configuration appear in tests[].dse.
NCCL DSE Example
This example comes from a Slurm NCCL all-reduce run on one node with eight H100 GPUs. DSE tried Ring and Tree in two steps. Ring (step 1) won using inverse latency as the reward, so its measurements appear in tests[].metrics.
Identifiers and paths are anonymized. To keep the example small, each metrics list includes only out-of-place latency and bus bandwidth for 1 MiB messages. Measurement values, timing, and DSE selection are unchanged.
Each file update replaces the previous snapshot atomically. Metric extraction or write errors produce warnings without affecting execution.
General Flow
-
All reports should be registered via
Registry()(.add_report()or.add_scenario_report()) -
Scenario reports are configurable via system config (Slurm-only for now) and scenario config
-
Configuration in a scenario config has the highest priority. Next, system config is checked. Then it defaults to report config from the registry
-
Finally, the report is generated (or not) according to this final config
Enabling, Disabling and Configuring Reports
Note
Only scenario-level reports can be configured.
Enabling or disabling a report needs to be done in the system configuration:
The junit scenario reporter is disabled by default. When enabled, it writes junit.xml in the scenario results directory. It emits one test case for every regular test iteration and every DSE step, including pass/fail status, failure details, scheduler duration when available, and the contents of stdout.txt and stderr.txt. The artifact can be consumed directly by Jenkins, GitLab, GitHub Actions, and other CI systems that support JUnit XML.
Speed-of-Light comparisons
CloudAI can compare NCCL and NIXL Bench measurements with configured Speed-of-Light (SOL) targets. Targets are validated while the system or scenario TOML is parsed. A target without match is the default; a matching target with more dimensions takes precedence:
NCCL targets can distinguish the collective operation and placement:
A test case can replace the targets for a metric inherited from the scenario or system:
The precedence is test case, then scenario, then system. NCCL and NIXL comparison v2 reports include measured, SOL, and percentage-of-SOL columns and draw a shared SOL curve when every compared run resolves the same targets.
Reporting Registration
Report registration is done via Registry class:
Reporting Configuration Implementation
Each report can define its own configuration, which is constructed and passed as an argument to Registry.add_scenario_report. The reports field is parsed during TOML reading and the respective Pydantic model is created for it.
For example, a custom report configuration can be defined as follows:
And it can be used in a test scenario as follows:
Uploading Results to Object Storage
The s3 scenario report publishes the scenario results directory to an S3-compatible bucket. It is disabled by default, because shipping results off-box should be a deliberate choice.
It is registered last, after tarball, so it always observes the complete results directory including every other report’s output.
Files are uploaded under s3://<bucket>/<prefix>/<cluster>/<username>/<results-directory>/, where cluster is the system name and username is the original experiment owner’s user field in experiment.json, including when running generate-report. If the file is missing, unreadable, invalid, or has no recorded owner, uploads use the current account’s username. If that lookup also fails, uploads use unknown. Tarballs use the same path, with <results-directory>.tgz appended.
Install the optional dependency first:
Then enable it in a test scenario:
Or, for Slurm systems, once per cluster in the system config:
Configuration options:
At least one of upload_tree and upload_tarball must be enabled; the config is rejected otherwise.
Environment variables
The destination can be supplied through three environment variables, so a cluster-wide default does not have to be repeated in every scenario:
A value set in TOML takes precedence over the environment variable. The variables are read by the cloudai process when the report configuration is loaded, so export them where cloudai runs:
With these set, enabling the report only needs s3`` ``=`` ``{`` ``enable`` ``=`` ``true`` ``}.
Credentials are never read from CloudAI configuration. They are resolved by boto3’s standard chain: AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, ~/.aws/credentials, or an instance/IAM role.
Because reports run inside a try/except, an upload failure logs a warning and leaves the run’s exit status unchanged.
To upload a results directory from an earlier run, re-run the reports against it: