> This page is for version 1.8.1 (default).
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

AI Dynamo workload (test_template_name is `AIDynamo`) runs AI inference benchmarks using the Dynamo framework with distributed prefill and decode workers.

## Run Using Kubernetes

### Prepare Cluster

Before running the AI Dynamo workload on a Kubernetes cluster, ensure that the cluster is set up according to the instructions in the [official documentation](https://docs.nvidia.com/dynamo/dev/getting-started/kubernetes-deployment). Below is a short summary of the required steps:

```bash
export NAMESPACE=dynamo-system
export RELEASE_VERSION=0.7.0  # replace with the desired release version

helm upgrade -n default -i dynamo-crds https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
helm upgrade -n default -i dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz

# The following components are required for multi node only.
# Versions should be aligned with Dynamo version.
helm upgrade -n default -i grove oci://ghcr.io/ai-dynamo/grove/grove-charts:v0.0.0-gd462e65
helm upgrade -n default -i kai-scheduler oci://ghcr.io/nvidia/kai-scheduler/kai-scheduler:0.0.0-4c29820
```

### Launch and Monitor the Job

Note

Both CloudAI and Dynamo will try to access HuggingFace Hub. To avoid `429`` ``Too`` ``Many`` ``Requests` errors and access models under auth, it is recommended to define `HF_TOKEN` environment variable before invoking CloudAI. Once set, run `uv`` ``run`` ``hf`` ``auth`` ``login` to authenticate.

```bash
uv run cloudai run --system-config <k8s system toml> \
   --tests-dir conf/experimental/ai_dynamo/test \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_k8s.toml
```

## Run Using Slurm

### Node Configuration for AI Dynamo

AI Dynamo jobs use three distinct types of nodes:

* **Frontend node**: Hosts the coordination services (etcd, nats), the **frontend server**, the **request generator** (aiperf by default, configurable via `workloads` in the test TOML), and the first decode worker

* **Prefill node(s)**: Handle the prefill stage of inference

* **Decode node(s)**: Handle the decode stage of inference (optional, depending on model and setup)

By default, when `num_nodes` is omitted, CloudAI allocates separate nodes for prefill and decode workers:

```default
num_prefill_nodes + num_decode_nodes
```

Set top-level `num_nodes` explicitly to control the Slurm allocation. A value lower than `num_prefill_nodes`` ``+`` ``num_decode_nodes` enables shared-node disaggregated inference, where prefill and decode roles run on the same allocated node(s) with separate GPU slices.

All node role assignments and orchestration are automatically managed by CloudAI.

### Launch and Monitor the Job

To run the job:

```bash
uv run cloudai run --system-config <slurm system toml> \
   --tests-dir conf/experimental/ai_dynamo/test \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_slurm.toml
```

The job progress monitoring can be done using either of the following options:

```bash
watch squeue --me
```

```bash
watch tail -n 4 ./results/<scenario name>/*.txt
```

The frontend node will initially wait to allow weight loading on all nodes. Once ready, it will launch the configured benchmark tool (`aiperf` by default), which begins generating requests to the frontend server. All servers cooperate to complete inference, and the output will appear in `stdout.txt`.

### Choosing a Benchmark Tool

The benchmark tool is controlled by the `workloads` field in the test TOML. Set `aiperf.sh` to use AIPerf:

```toml
[cmd_args]
workloads = "aiperf.sh"   # uses aiperf, writes aiperf_report.csv
```

To use genai-perf, set:

```toml
[cmd_args]
workloads = "genai_perf.sh"   # uses genai-perf, writes genai_perf_report.csv

[cmd_args.genai_perf]
cmd = "genai-perf profile"
extra-args = "--streaming --verbose -- -v --async"

  [cmd_args.genai_perf.args]
  endpoint-type = "chat"
  output-tokens-mean = 500
  request-count = 50
```

### AIPerf Multi-Phase Runs

`cmd_args.aiperf` is the base AIPerf config. `cmd_args.aiperf_phases` can run several AIPerf rounds against the same live Dynamo stack. By default, CloudAI does not restart prefill, decode, or router processes between phases:

```toml
dse_excluded_args = ["cmd_args.aiperf_phases"]

[cmd_args.aiperf]
health-check-between-phases = true
between-phase-cmd = "true"  # default no-op

  [cmd_args.aiperf.args]
  request-count = 50
  server-metrics = "auto"

[[cmd_args.aiperf_phases]]
name = "round_1"
  [cmd_args.aiperf_phases.args]
  concurrency = 2

[[cmd_args.aiperf_phases]]
name = "round_2"
  [cmd_args.aiperf_phases.args]
  concurrency = 4
```

Single-phase runs keep the old artifact layout: `aiperf_artifacts/`, `aiperf.log`, and `aiperf_report.csv`. Multi-phase runs write per-phase artifacts/logs/reports and copy the last phase report to `aiperf_report.csv` for existing report generation.

`between-phase-cmd` is a bash command run after each non-final phase. The default is a no-op. Set it explicitly for backend-specific cache cleanup, for example `/cloudai_run_results/routerctl.sh`` ``restart` if a test needs to restart the Dynamo router between phases. `health-check-between-phases` probes the frontend after the command.

## Comparison Reports

CloudAI provides the scenario-level `ai_dynamo_comparison` report. It compares AI Dynamo runs using the standard LLM serving tables and charts for TTFT, TPOT, successful prompts, output-token throughput, TPS per user, and TPS per GPU. When `aiperf_accuracy` is configured and its result is available, the report also compares model accuracy. Both AIPerf and GenAI-Perf CSV output are supported.

AIPerf args are rendered as normal CLI flags. Multi-value AIPerf options should be passed with AIPerf CLI syntax, such as `server-metrics-formats`` ``=`` ``"csv,json,jsonl"` or `gpu-telemetry`` ``=`` ``"node1:9401,node2:9401"`. `server-metrics`` ``=`` ``"auto"` expands to the frontend metrics endpoint, Dynamo worker metrics endpoints, and any CloudAI-started DCGM exporters.

### Propagating LMCache Configuration

AIDynamo can pass an LMCache YAML config to the worker processes by setting `LMCACHE_CONFIG_FILE` inside the container. This only propagates the LMCache configuration; the vLLM/SGLang runtime still needs to be launched with the appropriate LMCache or KV-transfer connector for that image/version.

The preferred form is structured TOML under `[cmd_args.lmcache]`. CloudAI converts that object to YAML in the run output directory, mounts that directory as `/cloudai_run_results`, and exports the generated file path as `LMCACHE_CONFIG_FILE`:

```toml
[cmd_args]
  [cmd_args.lmcache]
  chunk_size = 256
  local_cpu = true
  controller_pull_url = "{frontend_node}:8300"
  controller_reply_url = "{frontend_node}:8400"
  lmcache_worker_ports = [8788, 8789, 8790, 8791]
  max_local_cpu_size = 6.0
  nixl_buffer_size = 2079377920
  nixl_buffer_device = "cpu"

    [cmd_args.lmcache.extra_config]
    enable_nixl_storage = false
    nixl_backend = "POSIX"
    nixl_path = "{storage_cache_dir}"
    nixl_pool_size = 2048
```

For an example that uses test-in-scenario mode, see `conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml`. Because the test is fully defined inside the scenario, `--tests-dir` is not required when running that example:

```bash
uv run cloudai run --system-config <slurm system toml> \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml
```

The example sets `dse_excluded_args`` ``=`` ``["cmd_args.lmcache.lmcache_worker_ports"]` because `lmcache_worker_ports` is a list-valued LMCache setting, not a DSE sweep dimension. Other list-valued LMCache fields can still be swept unless their `cmd_args.` path is also excluded.

Alternatively, mount your own LMCache YAML file with `extra_container_mounts` and set `LMCACHE_CONFIG_FILE` through `extra_env_vars`:

```toml
extra_container_mounts = ["/host/lmcache:/lmcache"]
extra_env_vars = { LMCACHE_CONFIG_FILE = "/lmcache/config.yaml" }
```

For multi-node LMCache storage tests, any path referenced by the LMCache YAML, such as `nixl_path` for POSIX-backed storage, must be visible and writable from every node that is expected to share cached data. A node-local path such as `/tmp` is suitable only for single-node smoke tests or configuration propagation checks.

LMCache YAML values can use runtime placeholders. CloudAI renders them inside the Slurm job before launching workers: `{frontend_node}`, `{frontend_ip}`, `{results_dir}`, and `{storage_cache_dir}`. Unknown placeholders fail the run before worker processes start.

### Propagating HiCache Configuration

Define a HiCache storage-backend configuration under `[cmd_args.hicache]`. CloudAI serializes the object as `hicache-config.toml` in the run output directory, exposes its container path through `HICACHE_CONFIG_FILE`, and passes the resolved path to both workers as `--hicache-storage-backend-extra-config`:

```toml
[cmd_args]
  [cmd_args.hicache.plugin.posix]
  active = true
  use_uring = "false"

  [cmd_args.dynamo.prefill_worker]
  extra-args = "--enable-hierarchical-cache"

  [cmd_args.dynamo.prefill_worker.args]
  hicache-size = 8
  hicache-ratio = 0
  hicache-write-policy = "write_through"
  hicache-mem-layout = "page_first"
  hicache-storage-prefetch-policy = "wait_complete"
  hicache-storage-backend = "nixl"

  [cmd_args.dynamo.decode_worker]
  extra-args = "--enable-hierarchical-cache"

  [cmd_args.dynamo.decode_worker.args]
  hicache-size = 8
  hicache-ratio = 0
  hicache-write-policy = "write_through"
  hicache-mem-layout = "page_first"
  hicache-storage-prefetch-policy = "wait_complete"
  hicache-storage-backend = "nixl"
```

The remaining HiCache engine options stay in the worker `args` sections so they can be configured or swept through DSE normally.

The runtime replacement mechanism below supports per-node HiCache values. For example, change the POSIX setting above to `use_uring`` ``=`` ``"${use_uring}"` and have the startup manifest replace `${use_uring}` with `false` or `true`. The worker receives the resulting per-node TOML path automatically.

For a self-contained single-node smoke test using NIXL’s POSIX plugin, run the following on a node with at least two GPUs (one each for the prefill and decode workers):

```bash
uv run cloudai run --system-config <slurm system toml> \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/sglang_hicache.toml
```

### Per-node Startup Commands and Runtime Replacements

On Slurm, `startup_cmd` runs once per allocated node before the workload. It runs on the host by default. Set `startup_cmd_docker_image` to run it in a container using the workload’s mounts:

```toml
[cmd_args]
startup_cmd = "/mnt/discovery/discover-device.sh"
startup_cmd_docker_image = "nvcr.io/example/discovery-tools:latest"
```

CloudAI exports `CLOUDAI_RUNTIME_FILE` to the command. It points to `runtime/<node>.json` in the test output directory, or `/cloudai_run_results/runtime/<node>.json` inside a startup container. Write replacements there:

```json
{
  "LMCACHE_CONFIG_FILE": {
    "${device}": "/dev/ng4n1",
    "${namespace_id}": "1"
  },
  "HICACHE_CONFIG_FILE": {
    "${use_uring}": "false"
  }
}
```

Each top-level key names an environment variable containing a config path. AIDynamo creates a per-node copy, applies literal replacements, then updates the variable to point to that copy.

For `lmcache-config.yaml` on `node-01`, the result is `lmcache-config.node-01.yaml`. The shared source remains unchanged. LMCache placeholders are rendered afterward in the per-node copy. Missing manifest means no replacements.

Startup runs once per test case in normal and `--single-sbatch` modes.

If the selected LMCache mode needs a controller, CloudAI can start one on the frontend node:

```toml
[cmd_args.lmcache_controller]
cmd = "lmcache_controller --host 0.0.0.0 --port 9000 --monitor-ports {\"pull\":8300,\"reply\":8400}"
```

This only launches the process. For disaggregated or multi-node runs, the LMCache YAML still needs controller addresses that resolve to the frontend node from every worker. With the default controller monitor ports, use `controller_pull_url`` ``=`` ``"{frontend_node}:8300"` and `controller_reply_url`` ``=`` ``"{frontend_node}:8400"`. The `lmcache_worker_ports` list must match the number of worker ranks.

### Semantic Degradation With AIPerf Accuracy

AIDynamo uses AIPerf accuracy mode as its semantic degradation signal. Enable it with `[cmd_args.aiperf_accuracy]`. This runs after the configured performance workload, so it can be used with either `aiperf.sh` or `genai_perf.sh`:

```toml
[cmd_args]
workloads = "aiperf.sh"

[cmd_args.aiperf]
  [cmd_args.aiperf.args]
  request-count = 50
  synthetic-input-tokens-mean = 300
  output-tokens-mean = 500
  concurrency = 2

[cmd_args.aiperf_accuracy]
entrypoint = "aiperf profile"
setup-cmd = "python -m pip install --break-system-packages --upgrade aiperf==0.8.0"
cli = '''
--model {model}
--url {url}
--endpoint-type chat
--streaming
--artifact-dir {artifact_dir}
--no-server-metrics
--accuracy-benchmark mmlu
--accuracy-n-shots 5
--accuracy-tasks abstract_algebra
--concurrency 10
--extra-inputs '{"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'
--num-requests 100
'''
```

When `cmd_args.aiperf_accuracy` is configured, CloudAI expects AIPerf to produce `accuracy_results.csv` and exposes the `accuracy` metric from its `OVERALL` row. The metric is reported as a 0.0-1.0 fraction. Keep synthetic prompt and token-length flags out of this mode; the benchmark dataset should come from AIPerf’s accuracy benchmark.

The `entrypoint` and `cli` fields form the accuracy command. CloudAI expands `{model}`, `{url}`, `{endpoint}`, `{result_dir}`, and `{artifact_dir}` in `cli` before launching it. The `setup-cmd` field is optional. It is useful for Dynamo images that include an older system `aiperf` build without the accuracy benchmark plugins. The example upgrades the image-level `aiperf` before launching `aiperf`` ``profile`. MMLU is loaded from `lighteval/mmlu`, so either allow Hugging Face dataset access or pre-cache that dataset before running with `HF_HUB_OFFLINE`/`HF_DATASETS_OFFLINE` enabled. For Qwen3 models, the example disables thinking mode so short MMLU answers can be parsed as choices.

### Custom Accuracy Scripts

`cmd_args.aiperf_accuracy` can also launch a custom mounted script instead of AIPerf. Mount the script or its parent directory with `extra_container_mounts` and set `entrypoint` to the in-container command:

```toml
extra_container_mounts = ["/host/custom_accuracy:/custom_accuracy"]

[cmd_args.aiperf_accuracy]
entrypoint = "python /custom_accuracy/dummy_accuracy.py"
cli = "--model {model} --url {url} --endpoint {endpoint} --artifact-dir {artifact_dir} --prompt ping"
```

CloudAI expands placeholders in `cli` and runs `entrypoint` with that CLI string. The custom command must write `accuracy_results.csv` inside `{artifact_dir}` with an `OVERALL` row. CloudAI copies that file to the run output directory and exposes the same `accuracy` metric as AIPerf accuracy mode.

## Review Benchmark Results

After job completion, CloudAI places output logs and result files in the designated results directory. The result file name depends on the configured `workloads` field:

* `aiperf.sh` → `aiperf_report.csv`

* `genai_perf.sh` → `genai_perf_report.csv`

* `cmd_args.aiperf_accuracy` → `accuracy_results.csv`

If AIPerf accuracy mode is enabled, CloudAI copies `aiperf_accuracy_artifacts/accuracy_results.csv` to `accuracy_results.csv` in the run output directory and marks the run failed if that file is not produced.

Navigate to `./results/<scenario>/<test-id>/0/` and open the CSV to examine performance metrics.

### Shared-Node Disaggregated Runs

For Slurm, set top-level `num_nodes` lower than the sum of `prefill_worker.num-nodes` and `decode_worker.num-nodes` to run both roles on the same allocated node(s). For example, `num_nodes`` ``=`` ``1` with `prefill_worker.num-nodes`` ``=`` ``1` and `decode_worker.num-nodes`` ``=`` ``1` runs one prefill worker and one decode worker on the same node. CloudAI assigns decode GPUs first and prefill GPUs after that based on each role’s `tensor-parallel-size`` ``*`` ``pipeline-parallel-size`. The combined role GPU count must fit on one node.

Example `aiperf_report.csv`:

```default
Metric,avg,min,max,p25,p50,p75,p99,std
Inter Token Latency (ms),2.81,2.66,2.88,2.79,2.83,2.84,2.87,0.04
Time to First Token (ms),49.87,17.15,99.91,49.35,49.87,50.52,92.31,9.20
Time to Second Token (ms),0.50,0.03,4.05,0.03,0.04,0.04,3.47,1.08
Request Latency (ms),1652.30,1203.61,6433.87,1453.19,1462.99,1466.72,6431.16,976.18
Output Sequence Length (tokens),498.06,410.00,501.00,500.00,500.00,500.00,501.00,12.62
Input Sequence Length (tokens),300.00,300.00,300.00,300.00,300.00,300.00,300.00,0.00

Metric,Value
Output Token Throughput (tokens/sec),598.78
Total Token Throughput (tokens/sec),962.32
Request Throughput (requests/sec),1.20
Request Count,50.00
```

## Supported Backends

The following backends are available via the `conf/experimental/ai_dynamo/test/` directory:

* **vLLM** (`vllm.toml`) — use with `test_scenario/vllm_slurm.toml`

* **vLLM with LMCache config propagation** — use self-contained scenario `test_scenario/vllm_lmcache.toml`

* **sglang** (`sglang.toml`) — use with `test_scenario/sglang_slurm.toml`

Both backends use `aiperf` as the default benchmark tool and support disaggregated prefill/decode.

## API Documentation

### Command Arguments

*class* cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs(

*\**,

*dynamo_version: str = 'f7e468c7e8ff0d1426db987564e60572167e8464'*,

*docker_image_url: str*,

*startup_cmd: str | None = None*,

*startup_cmd_docker_image: str | None = None*,

*storage_cache_dir: str | list\[str] | None = '/tmp'*,

*dynamo: \~cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoArgs*,

*hicache: dict | None = None*,

*lmcache: dict | None = None*,

*lmcache_controller: \~cloudai.workloads.ai_dynamo.ai_dynamo.LMCacheController | None = None*,

*genai_perf: \~cloudai.workloads.ai_dynamo.ai_dynamo.GenAIPerf = \<factory>*,

*aiperf: \~cloudai.workloads.ai_dynamo.ai_dynamo.AIPerf = \<factory>*,

*aiperf_phases: list\[\~cloudai.workloads.ai_dynamo.ai_dynamo.AIPerfPhase] | None = None*,

*aiperf_accuracy: \~cloudai.workloads.ai_dynamo.ai_dynamo.AIPerfAccuracy | None = None*,

*workloads: str = 'genai_perf.sh'*,

)

Bases: `CmdArgs`

Arguments for AI Dynamo.

**&#x20;*validator* validate_aiperf_phases*  » \* *all fields**\*

Validate AIPerf phases.

### Test Definition

*class* cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoTestDefinition(*\*, name: str, description: str, test_template_name: str, cmd_args: \~cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs, dse_excluded_args: list\[str] = \<factory>, extra_env_vars: dict\[str, str | \~typing.List\[str]] = \{}, extra_cmd_args: dict\[str, str] = \{}, extra_container_mounts: list\[str] = \[], git_repos: list\[\~cloudai.\_core.installables.git_repo.GitRepo] = \[], nsys: \~cloudai.models.workload.NsysConfiguration | None = None, predictor: \~cloudai.models.workload.PredictorConfig | None = None, training_report: \~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list\[str] = \['default'], agent_reward_function: str = 'inverse', agent_config: dict\[str, \~typing.Any] | None = None, env_params: dict\[str, \~cloudai.configurator.env_params.EnvParamSpec] = \<factory>, script: \~cloudai.\_core.installables.file.File = File(src=PosixPath('src/cloudai/workloads/ai_dynamo/ai_dynamo.sh')), constraints: \~cloudai.workloads.ai_dynamo.ai_dynamo.Constraints = Constraints(prefill_tp_le_decode_tp=True, tp_times_pp_le_gpus_per_node=True), success_marker: str = 'success-marker.txt', failure_marker: str = 'failure-marker.txt'*)

Bases: `TestDefinition`

Test definition for AI Dynamo.

**&#x20;*validator* workload_scripts*  » \* *all fields**\*

Populate prefill/decode args.

**&#x20;get_workload_map() → dict\[str, Workload]**

Get a map of workload scripts to workload objects.

**&#x20;*property* repo*: GitRepo*\*\*

Return the AI Dynamo repository selected by `cmd_args.dynamo_version`.

**&#x20;*property* installables*: list\[Installable]*\*\*

Get all installables for this test definition.

**&#x20;*property* is_domain_randomization_enabled*: bool*\*\*

at least one `env_params` annotation.

**Type:**

Whether the config declares domain randomization

**&#x20;is_dse_excluded_arg(*path: str*) → bool**

Return whether a dot-separated cmd_args path should be ignored by DSE.

**&#x20;is_env_sampled(*cmd_args_path: str*) → bool**

Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

metric_observations(

*system: System*,

*tr: TestRun*,

) → list\[MetricObservation]

Return canonical metric observations produced by this test run, if supported.

**&#x20;*validator* validate_env_params*  » \* *all fields**\*

Validate env_params annotations against cmd_args.

`env_params` is an annotation: each key names a `cmd_args` field whose value is the candidate set (the single source of truth), and the entry carries only *how* to sample. So each key must name a real `cmd_args` field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When `weights` are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in `CloudAIGymEnv`; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.