> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

This workload (`test_template_name` is `NIXLBench`) runs NIXL benchmarking suite for network and interconnect performance testing.

## Usage Examples

Test TOML example:

```toml
name = "my_nixl_bench_test"
description = "Example NIXL Bench test"
test_template_name = "NIXLBench"

[cmd_args]
docker_image_url = "<docker container url here>"
path_to_benchmark = "/workspace/nixlbench/build/nixlbench"
backend = "UCX"
initiator_seg_type = "VRAM"
target_seg_type = "VRAM"
op_type = "READ"
filepath = "/data"
device_list = "11:F:/store0.bin"
# one could also use <num>kb, <num>mb, <num>gb shortcuts
total_buffer_size = 8000000000
```

Test Scenario example:

```toml
name = "nixl-bench-test"

[[Tests]]
id = "bench.1"
num_nodes = 1
time_limit = "00:10:00"

test_name = "my_nixl_bench_test"
```

Test-in-Scenario example:

```toml
name = "nixl-bench-test"

[[Tests]]
id = "bench.1"
num_nodes = 1
time_limit = "00:10:00"

name = "my_nixl_bench_test"
description = "Example NIXL Bench test"
test_template_name = "NIXLBench"

  [Tests.cmd_args]
  docker_image_url = "<docker container url here>"
  path_to_benchmark = "/workspace/nixlbench/build/nixlbench"
  backend = "UCX"
  initiator_seg_type = "DRAM"
  target_seg_type = "DRAM"
  op_type = "WRITE"
```

## Runtime Coordination

NIXLBench uses ETCD by default. CloudAI starts ETCD from the benchmark image when `etcd_image_url` is omitted, or from the configured image.

To use NIXLBench's direct two-process ASIO runtime instead, set:

```toml
runtime_type = "ASIO"
```

CloudAI resolves `asio_address` to the first allocated node by default and does not install or start ETCD. `asio_address` and `asio_port` can be overridden explicitly. ASIO requires exactly two NIXLBench processes. For UCX, a one-node test runs both processes locally, while a two-node test runs one process on each node.

Storage backends can run without either runtime by using an empty ETCD endpoint:

```toml
backend = "POSIX"
etcd_endpoints = ""
```

This null-runtime mode is limited to storage backends and launches one NIXLBench process by default.

## Independent Storage Processes

To run independent storage benchmarks in one allocation, set `launch_mode = "independent"` and `etcd_endpoints = ""`. CloudAI launches one process per node by default; use `num_nodes` to choose the number of nodes:

```toml
name = "storage-scaling"

[[Tests]]
id = "storage.4nodes"
test_name = "my_storage_test"
num_nodes = 4

  [Tests.cmd_args]
  launch_mode = "independent"
  etcd_endpoints = ""
  num_threads = [4, 8]
```

Here, `my_storage_test` is a NIXLBench test defining the storage backend, container image, benchmark path and workload arguments. CloudAI runs the 4-thread and 8-thread configurations as separate tests.

`launch_mode` selects CloudAI's launch strategy and is not passed to NIXLBench. Its default value, `"default"`, preserves existing behavior, including when task placement options are present. Placement options alone do not enable independent runs.

Independent mode requires `etcd_endpoints = ""` and the default `runtime_type = "ETCD"` setting, which together select NIXLBench's null runtime. Managed/external ETCD and ASIO cannot be used with this mode. CloudAI launches one Slurm step across the requested nodes. Use `extra_srun_args` only when overriding the default task placement. System-level options precede test-level options.

Each task executes in its own shell. Node-specific environment values and command expansions are evaluated there. CloudAI does not discover backend devices.

### Results and Reporting

Independent runs retain `nixlbench/<task-id>.stdout`, `.stderr`, `.status` and `.hostname` files. The `nixlbench/ntasks` file records the actual Slurm task count. A failed process, missing task output or inconsistent block-size/batch-size coverage makes the run unsuccessful; incomplete results are not aggregated.

`nixlbench_per_task.csv` preserves individual measurements with task IDs and hostnames. `nixlbench.csv` summarizes matching block sizes and batch sizes: `avg_lat` and `bw_gb_sec` are arithmetic means across tasks, `bw_min_gb_sec` is the minimum task bandwidth, `bw_sum_gb_sec` is the sum, and `task_count` records the contributors. The native HTML report shows mean latency and minimum/mean/summed bandwidth, with separate plots for each batch size. Comparison reports and metric observations use the mean per-task bandwidth and latency. Existing single-process reports are unchanged.

## API Documentation

### Command Arguments

### `NIXLBenchCmdArgs`

Command line arguments for a NIXL Bench test.

| Field               | Type                                | Default                         | Description                                                                                                                                                                                                                                                          |
| ------------------- | ----------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filepath`          | `str \| None`                       | `None`                          | Directory path (in container) for storage operations. Example: /data                                                                                                                                                                                                 |
| `total_buffer_size` | `str \| list[str] \| None`          | `None`                          | Total buffer size in bytes. Examples: 1024, 1kb, 1mb, 1gb. Use with device_list. The size will be passed into NIXL as integer (bytes)                                                                                                                                |
| `device_list`       | `str \| list[str] \| None`          | `None`                          | Device specs in format 'id:type:path' (e.g., '11:F:/store0.bin,27:K:/dev/nvme0n1')                                                                                                                                                                                   |
| `docker_image_url`  | `str`                               | Required                        | URL of the Docker image to use for the benchmark.                                                                                                                                                                                                                    |
| `etcd_path`         | `str`                               | `'etcd'`                        | Path to the etcd executable.                                                                                                                                                                                                                                         |
| `wait_etcd_for`     | `int`                               | `60`                            | Number of seconds to wait for etcd to become healthy.                                                                                                                                                                                                                |
| `etcd_image_url`    | `str \| None`                       | `None`                          | Optional URL of the Docker image to use for etcd, by default etcd will be run from the same image as the benchmark.                                                                                                                                                  |
| `path_to_benchmark` | `str`                               | Required                        |                                                                                                                                                                                                                                                                      |
| `launch_mode`       | `Literal['default', 'independent']` | `'default'`                     | CloudAI launch mode: 'default' preserves existing orchestration; 'independent' launches one null-runtime task per node in one Slurm step by default. Use num_nodes to select the node count and extra_srun_args to override task placement. Not passed to NIXLBench. |
| `etcd_endpoints`    | `str`                               | `'http://$NIXL_ETCD_ENDPOINTS'` |                                                                                                                                                                                                                                                                      |
| `runtime_type`      | `Literal['ETCD', 'ASIO']`           | `'ETCD'`                        |                                                                                                                                                                                                                                                                      |
| `asio_address`      | `str`                               | `'$NIXL_ASIO_ADDRESS'`          |                                                                                                                                                                                                                                                                      |
| `asio_port`         | `int`                               | `12345`                         | Constraints: `Ge(ge=1)`, `Le(le=65535)`.                                                                                                                                                                                                                             |

### Test Definition

### `NIXLBenchTestDefinition`

Test definition for a NIXL Bench test.

| Field                    | Type                                                       | Default         | Description                                                                                                                                                                                                                   |
| ------------------------ | ---------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                      | Required        |                                                                                                                                                                                                                               |
| `description`            | `str`                                                      | Required        |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                      | Required        |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.nixl_bench.nixl_bench.NIXLBenchCmdArgs` | Required        |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                                | Factory: `list` |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                       | `{}`            |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                           | `{}`            |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                                | `[]`            |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`        | `[]`            |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`        | `None`          |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`          | `None`          |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`     | `None`          |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                      | `'grid_search'` |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                      | `1`             |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                                | `['default']`   |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                      | `'inverse'`     |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                   | `None`          | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]`  | Factory: `dict` | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |