Skip to navigation

Reporting

View as Markdown

This chapter describes the reporting system in CloudAI. In this chapter, we will cover the following topics:

Overview

CloudAI has two reporting levels:

  • per-test (per each case in a test scenario)
  • per-scenario (per each test scenario)

All reports are generated after the test scenario is completed as part of the main CloudAI process. For Slurm, this means that the login node is used to generate reports.

Per-test reports are linked to a particular workload type (e.g. NcclTest). All per-test reports are implemented as part of the per_test scenario report and can be enabled or disabled via a single configuration option; see Enabling, Disabling and Configuring Reports.

To list all available reports, users can use cloudai list-reports. Use verbose output to also print report configurations.

Unified Experiment Output

CloudAI writes experiment.json in each scenario’s results directory.

The file contains scenario details, system name, the username of the account running CloudAI in user, and test cases listed under tests. Older files without user are loaded with an empty string. For Slurm runs, the system name is the cluster reported by Slurm. Both Standalone and Slurm execution records are stored in each test case’s runs list. Each record represents an iteration or DSE step and includes its number, process or Slurm job ID, status, timing, result path, and workload metrics.

If username lookup fails, CloudAI logs a warning and records an empty user without interrupting execution.

Timestamps use UTC; durations use seconds. Unknown timestamps are null. A status of unknown indicates the outcome could not be determined. Dry runs include scenario and test-case details without launching workloads.

Live runs status tracking:

  • Standalone jobs are running from submission.
  • Slurm jobs (excluding single-sbatch) start as pending while waiting for allocation, then transition to running once allocated and executing.
  • Single-sbatch Slurm run records remain pending until the allocation finishes and their final statuses are recorded.

Metrics are sourced from TestDefinition.metric_observations(), and are not affected by reporter settings.

MegatronRun exposes iteration_time (milliseconds, lower is better) and tflops_per_gpu (TFLOP/s/GPU, higher is better). Each observation is the mean of the last ten iteration metric lines in stdout.txt, or all available lines when fewer than ten are present, matching the per-test CSV report. No observations are emitted when iteration metric lines are missing.

When a test case is executed successfully, tests[].metrics contains that execution’s metrics. For DSE, it contains metrics from the successful step with the highest valid reward. The search space, selected step, and configuration appear in tests[].dse.

NCCL DSE Example

This example comes from a Slurm NCCL all-reduce run on one node with eight H100 GPUs. DSE tried Ring and Tree in two steps. Ring (step 1) won using inverse latency as the reward, so its measurements appear in tests[].metrics.

Identifiers and paths are anonymized. To keep the example small, each metrics list includes only out-of-place latency and bus bandwidth for 1 MiB messages. Measurement values, timing, and DSE selection are unchanged.

{
"id": "nccl-dse_2026-09-21_14-55-45",
"name": "nccl-dse",
"system_name": "example-cluster",
"user": "example-user",
"description": null,
"status": "completed",
"path": "/results/nccl-dse_2026-09-21_14-55-45",
"start": "2026-09-21T14:55:45.275325Z",
"finish": "2026-09-21T15:01:50.432934Z",
"duration": 365.157609,
"tests": [
{
"id": "nccl",
"name": "nccl_base_test",
"description": "NCCL base test configuration",
"status": "completed",
"path": "/results/nccl-dse_2026-09-21_14-55-45/nccl",
"metrics": [
{
"name": "Latency",
"value": 24.76,
"unit": "us",
"dimensions": [
{"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
{"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
{"name": "Size", "value": "1048576", "unit": "", "is_x": true}
]
},
{
"name": "Bandwidth",
"value": 74.12,
"unit": "GB/s",
"dimensions": [
{"name": "Bandwidth basis", "value": "bus", "unit": "", "is_x": false},
{"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
{"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
{"name": "Size", "value": "1048576", "unit": "", "is_x": true}
]
}
],
"runs": [
{
"path": "/results/nccl-dse_2026-09-21_14-55-45/nccl/0/1",
"jobid": "1001",
"status": "completed",
"metrics": [
{
"name": "Latency",
"value": 24.76,
"unit": "us",
"dimensions": [
{"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
{"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
{"name": "Size", "value": "1048576", "unit": "", "is_x": true}
]
},
{
"name": "Bandwidth",
"value": 74.12,
"unit": "GB/s",
"dimensions": [
{"name": "Bandwidth basis", "value": "bus", "unit": "", "is_x": false},
{"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
{"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
{"name": "Size", "value": "1048576", "unit": "", "is_x": true}
]
}
],
"start": "2026-09-21T14:55:46Z",
"finish": "2026-09-21T14:57:28Z",
"duration": 102.0,
"iteration": 0,
"step": 1
},
{
"path": "/results/nccl-dse_2026-09-21_14-55-45/nccl/0/2",
"jobid": "1002",
"status": "completed",
"metrics": [
{
"name": "Latency",
"value": 37.36,
"unit": "us",
"dimensions": [
{"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
{"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
{"name": "Size", "value": "1048576", "unit": "", "is_x": true}
]
},
{
"name": "Bandwidth",
"value": 49.11,
"unit": "GB/s",
"dimensions": [
{"name": "Bandwidth basis", "value": "bus", "unit": "", "is_x": false},
{"name": "Operation", "value": "all_reduce", "unit": "", "is_x": false},
{"name": "Placement", "value": "out_of_place", "unit": "", "is_x": false},
{"name": "Size", "value": "1048576", "unit": "", "is_x": true}
]
}
],
"start": "2026-09-21T14:58:47Z",
"finish": "2026-09-21T15:00:28Z",
"duration": 101.0,
"iteration": 0,
"step": 2
}
],
"dse": {
"space": {
"extra_env_vars.NCCL_ALGO": ["Ring", "Tree"]
},
"best_config": {"extra_env_vars.NCCL_ALGO": "Ring"},
"best_step": 1
}
}
]
}

Each file update replaces the previous snapshot atomically. Metric extraction or write errors produce warnings without affecting execution.

General Flow

  • All reports should be registered via Registry() (.add_report() or .add_scenario_report())
  • Scenario reports are configurable via system config (Slurm-only for now) and scenario config
  • Configuration in a scenario config has the highest priority. Next, system config is checked. Then it defaults to report config from the registry
  • Finally, the report is generated (or not) according to this final config

Enabling, Disabling and Configuring Reports

Only scenario-level reports can be configured.

Enabling or disabling a report needs to be done in the system configuration:

[reports]
per_test = { enable = false }
status = { enable = true }
junit = { enable = true }

The junit scenario reporter is disabled by default. When enabled, it writes junit.xml in the scenario results directory. It emits one test case for every regular test iteration and every DSE step, including pass/fail status, failure details, scheduler duration when available, and the contents of stdout.txt and stderr.txt. The artifact can be consumed directly by Jenkins, GitLab, GitHub Actions, and other CI systems that support JUnit XML.

Speed-of-Light comparisons

CloudAI can compare NCCL and NIXL Bench measurements with configured Speed-of-Light (SOL) targets. Targets are validated while the system or scenario TOML is parsed. A target without match is the default; a matching target with more dimensions takes precedence:

[[sol.bandwidth]]
value = 100.0 # GB/s
[[sol.bandwidth]]
value = 120.0 # GB/s
match = { operation = "write", size_bytes = 1048576 }
[[sol.latency]]
value = 8.0 # us

NCCL targets can distinguish the collective operation and placement:

[[sol.bandwidth]]
value = 250.0 # GB/s
match = { operation = "all_reduce", placement = "out_of_place", bandwidth_basis = "bus" }
[[sol.bandwidth]]
value = 300.0 # GB/s
match = { operation = "all_reduce", placement = "in_place", bandwidth_basis = "bus" }

A test case can replace the targets for a metric inherited from the scenario or system:

[[Tests]]
id = "nixl-case"
test_name = "nixl-bench"
[[Tests.sol.bandwidth]]
value = 120.0

The precedence is test case, then scenario, then system. NCCL and NIXL comparison v2 reports include measured, SOL, and percentage-of-SOL columns and draw a shared SOL curve when every compared run resolves the same targets.

Reporting Registration

Report registration is done via Registry class:

Registry().add_scenario_report("per_test", PerTestReporter, ReportConfig(enable=True))

Inference Report Providers

Test definitions can implement InferenceReportProvider.inference_reports(system, tr) from cloudai.report_generator.inference_report. It returns a list[InferenceReport] with one entry per independent benchmark measurement, or an empty list when no results are available.

Each report contains an InferenceConfig with resolved model, benchmark, load, hardware, and serving-role settings, plus MetricObservation outputs with their units and dimensions. Multiple measurements from the same execution have separate reports, even when their metric names are identical. Unknown optional settings remain None.

vLLM and SGLang return one report per execution; Dynamo returns separate reports for its benchmark phases. The Pareto reporter treats each returned report as one candidate point and applies match to its metric observations. metric_observations() remains available and returns the combined observations from these reports.

Each serving role records its instance count and assigned nodes per instance. An instance is one serving launch, which can span nodes; TP, PP, and DP describe parallelism within it. shared_nodes identifies overlapping prefill/decode placement. Dynamo resolves worker counts from its configured topology and standard launcher defaults. A combined prefill/decode instance is displayed as Serving.

Reporting Configuration Implementation

Each report can define its own configuration, which is constructed and passed as an argument to Registry.add_scenario_report. The reports field is parsed during TOML reading and the respective Pydantic model is created for it.

For example, a custom report configuration can be defined as follows:

class CustomReportConfig(ReportConfig):
greeting: str
Registry().add_scenario_report("custom", CustomReport, CustomReportConfig(greeting="default value"))

And it can be used in a test scenario as follows:

[reports]
custom = { enable = true, greeting = "Hello, world!" }

Uploading Results to Object Storage

The s3 scenario report publishes the scenario results directory to an S3-compatible bucket. It is disabled by default, because shipping results off-box should be a deliberate choice.

It is registered last, after tarball, so it always observes the complete results directory including every other report’s output.

Files are uploaded under s3://<bucket>/<prefix>/<cluster>/<username>/<results-directory>/, where cluster is the system name and username is the original experiment owner’s user field in experiment.json, including when running generate-report. If the file is missing, unreadable, invalid, or has no recorded owner, uploads use the current account’s username. If that lookup also fails, uploads use unknown. Tarballs use the same path, with <results-directory>.tgz appended.

Install the optional dependency first:

pip install 'cloudai[s3]'

Then enable it in a test scenario:

[reports]
s3 = { enable = true, bucket = "my-bucket", prefix = "cloudai/runs", upload_tarball = true }

Or, for Slurm systems, once per cluster in the system config:

[reports]
s3 = { enable = true, bucket = "my-bucket" }

Configuration options:

OptionDefaultDescription
bucket$CLOUDAI_S3_BUCKETDestination bucket. Required; the upload is skipped with a warning if unset.
prefix$CLOUDAI_S3_PREFIXKey prefix. Objects are written under <prefix>/<system_name>/<username>/<results_dir_name>/.
endpoint_url$CLOUDAI_S3_ENDPOINT_URLCustom endpoint, for MinIO or other S3-compatible stores.
regionunsetAWS region. When unset, boto3 resolves it (e.g. AWS_DEFAULT_REGION).
upload_treetrueUpload each file individually, preserving relative paths.
upload_tarballfalseAlso upload a .tgz of the whole directory. It is created if absent and rebuilt if older than the results.
upload_concurrency8Number of files uploaded concurrently when upload_tree is enabled. Must be at least 1.

At least one of upload_tree and upload_tarball must be enabled; the config is rejected otherwise.

Environment variables

The destination can be supplied through three environment variables, so a cluster-wide default does not have to be repeated in every scenario:

VariableSets optionNotes
CLOUDAI_S3_BUCKETbucketDestination bucket.
CLOUDAI_S3_PREFIXprefixKey prefix. Empty by default.
CLOUDAI_S3_ENDPOINT_URLendpoint_urlCustom endpoint, for MinIO or other S3-compatible stores. An empty value is treated as unset.

A value set in TOML takes precedence over the environment variable. The variables are read by the cloudai process when the report configuration is loaded, so export them where cloudai runs:

export CLOUDAI_S3_BUCKET=my-bucket
export CLOUDAI_S3_PREFIX=cloudai/runs
export CLOUDAI_S3_ENDPOINT_URL=http://localhost:9000 # only for MinIO or other S3-compatible stores

With these set, enabling the report only needs s3 = { enable = true }.

Credentials are never read from CloudAI configuration. They are resolved by boto3’s standard chain: AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, ~/.aws/credentials, or an instance/IAM role.

Because reports run inside a try/except, an upload failure logs a warning and leaves the run’s exit status unchanged.

To upload a results directory from an earlier run, re-run the reports against it:

cloudai generate-report --system-config <system.toml> --tests-dir <tests/> \
--test-scenario <scenario.toml> --result-dir results/<scenario>_<timestamp>