NIXL EP
This workload (test_template_name is NixlEP) runs the NIXL Elastic EP benchmark through a Slurm-managed multi-node elastic launcher flow.
Overview
The Slurm launch model is:
-
one
elastic.pyprocess per node, started in sequence as the plan progresses -
the master node starts first and exposes a TCPStore for rank coordination
-
follower nodes connect via
--tcp-server`` ``$master_iponce the master is ready -
the benchmark runtime comes from the container image
-
each run serializes its plan JSON into the output directory
Plan Format
The plan field is a JSON-encoded list of phases. Each phase is a list of rank indices passed directly to the benchmark. CloudAI uses the following convention to drive the elastic launcher:
-
Positive rank index — the rank is active. A rank that is new relative to the previous phase causes CloudAI to fire an additional
srunfor that worker. -
Negative rank index (e.g.
-6) — signals a contraction: the benchmark sees the absolute value and treats it as temporarily removed. No newsrunis launched for negative indices. -
Omitted rank — a rank present in an earlier phase but absent from the current phase list is not relaunched. The benchmark’s own phase logic handles its inactivity.
Example:
Phase completion is detected by polling the primary log for ->`` ``end`` ``phase`` ``N markers.
Usage Examples
Test TOML example:
Test-in-Scenario example:
Reporting
After a run completes, CloudAI prints a single table with one row per (node, rank) measurement. The Phases column shows each phase index colour-coded green (passed) or red (failed). Bandwidth columns report dispatch+combine throughput and timing per rank.
The reported metric (default) is the mean dispatch+combine bandwidth in GB/s across all ranks.
API Documentation
Command Arguments
pydantic model cloudai.workloads.nixl_ep.nixl_ep.NixlEPCmdArgs
Command line arguments for the NIXL Elastic EP benchmark.
field docker_image_url: str [Required]**
URL of the Docker image that contains the NIXL EP benchmark.
field elastic_script: str = ‘/workspace/nixl/examples/device/ep/tests/elastic/elastic.py’**
Path to the benchmark entrypoint, relative to the container’s NIXL runtime root or absolute in the container.
field python_executable: str = ‘python3’**
Python executable to use inside the container.
field plan: str | list[str] [Required]**
Serialized phase plan to write into a per-run JSON file. Use a single string such as “[[0, 1], [0, 1, 2, 3]]” for a single run, or a list of such strings to enable DSE mode (one run per plan).
field num_processes_per_node: int | list[int] [Required]**
Number of local worker processes to spawn on each allocated node.
field service_startup_timeout_seconds: int = 60**
Seconds to wait for the master node’s TCPStore to accept connections.
Constraints:
- ge = 1
field store_port: int = 9999**
TCPStore port used by the benchmark.
Constraints:
-
ge = 1
-
le = 65535
parse_plan() → list[list[int]]
Test Definition
class cloudai.workloads.nixl_ep.nixl_ep.NixlEPTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.nixl_ep.nixl_ep.NixlEPCmdArgs, dse_excluded_args: list[str] = <factory>, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.git_repo.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, training_report: ~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = ‘grid_search’, agent_steps: int = 1, agent_metrics: list[str] = [‘default’], agent_reward_function: str = ‘inverse’, agent_config: dict[str, ~typing.Any] | None = None, env_params: dict[str, ~cloudai.configurator.env_params.EnvParamSpec] = <factory>)
Bases: TestDefinition
Test definition for the NIXL Elastic EP benchmark.
property is_domain_randomization_enabled: bool**
at least one env_params annotation.
Type:
Whether the config declares domain randomization
is_dse_excluded_arg(path: str) → bool
Return whether a dot-separated cmd_args path should be ignored by DSE.
is_env_sampled(cmd_args_path: str) → bool
Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).
metric_observations(
system: System,
tr: TestRun,
) → list[MetricObservation]
Return canonical metric observations produced by this test run, if supported.
validator validate_env_params » * all fields*
Validate env_params annotations against cmd_args.
env_params is an annotation: each key names a cmd_args field whose value is the candidate set (the single source of truth), and the entry carries only how to sample. So each key must name a real cmd_args field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When weights are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in CloudAIGymEnv; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.