NIXL EP
This workload (test_template_name is NixlEP) runs the NIXL Elastic EP benchmark through a Slurm-managed multi-node elastic launcher flow.
Overview
The Slurm launch model is:
- one
elastic.pyprocess per node, started in sequence as the plan progresses - the master node starts first and exposes a TCPStore for rank coordination
- follower nodes connect via
--tcp-server $master_iponce the master is ready - the benchmark runtime comes from the container image
- each run serializes its plan JSON into the output directory
Plan Format
The plan field is a JSON-encoded list of phases. Each phase is a list of rank indices passed directly to the benchmark. CloudAI uses the following convention to drive the elastic launcher:
- Positive rank index — the rank is active. A rank that is new relative to the previous phase causes CloudAI to fire an additional
srunfor that worker. - Negative rank index (e.g.
-6) — signals a contraction: the benchmark sees the absolute value and treats it as temporarily removed. No newsrunis launched for negative indices. - Omitted rank — a rank present in an earlier phase but absent from the current phase list is not relaunched. The benchmark’s own phase logic handles its inactivity.
Example:
Phase completion is detected by polling the primary log for -> end phase N markers.
Usage Examples
Test TOML example:
Test-in-Scenario example:
Reporting
After a run completes, CloudAI prints a single table with one row per (node, rank) measurement. The Phases column shows each phase index colour-coded green (passed) or red (failed). Bandwidth columns report dispatch+combine throughput and timing per rank.
The reported metric (default) is the mean dispatch+combine bandwidth in GB/s across all ranks.
API Documentation
Command Arguments
NixlEPCmdArgs
Command line arguments for the NIXL Elastic EP benchmark.
Test Definition
NixlEPTestDefinition
Test definition for the NIXL Elastic EP benchmark.