> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

AI Dynamo workload (`test_template_name` is `AIDynamo`) runs AI inference benchmarks using the Dynamo framework with distributed prefill and decode workers.

## Run Using Kubernetes

### Prepare Cluster

Before running the AI Dynamo workload on a Kubernetes cluster, ensure that the cluster is set up according to the instructions in the [official documentation](https://docs.nvidia.com/dynamo/dev/getting-started/kubernetes-deployment). Below is a short summary of the required steps:

```bash
export NAMESPACE=dynamo-system
export RELEASE_VERSION=0.7.0  # replace with the desired release version

helm upgrade -n default -i dynamo-crds https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
helm upgrade -n default -i dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz

# The following components are required for multi node only.
# Versions should be aligned with Dynamo version.
helm upgrade -n default -i grove oci://ghcr.io/ai-dynamo/grove/grove-charts:v0.0.0-gd462e65
helm upgrade -n default -i kai-scheduler oci://ghcr.io/nvidia/kai-scheduler/kai-scheduler:0.0.0-4c29820
```

### Launch and Monitor the Job

> **Note**
>
> Both CloudAI and Dynamo will try to access HuggingFace Hub. To avoid `429 Too Many Requests` errors and access models under auth, it is recommended to define `HF_TOKEN` environment variable before invoking CloudAI. Once set, run `uv run hf auth login` to authenticate.

```bash
uv run cloudai run --system-config <k8s system toml> \
   --tests-dir conf/experimental/ai_dynamo/test \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_k8s.toml
```

## Run Using Slurm

### Node Configuration for AI Dynamo

AI Dynamo jobs use three distinct types of nodes:

* **Frontend node**: Hosts the coordination services (`etcd`, `nats`), the **frontend server**, the **request generator** (`aiperf` by default, configurable via `workloads` in the test TOML), and the first decode worker
* **Prefill node(s)**: Handle the prefill stage of inference
* **Decode node(s)**: Handle the decode stage of inference (optional, depending on model and setup)

By default, when `num_nodes` is omitted, CloudAI allocates separate nodes for prefill and decode workers:

num_prefill_nodes + num_decode_nodes

Set top-level `num_nodes` explicitly to control the Slurm allocation. A value lower than `num_prefill_nodes + num_decode_nodes` enables shared-node disaggregated inference, where prefill and decode roles run on the same allocated node(s) with separate GPU slices.

All node role assignments and orchestration are automatically managed by CloudAI.

### Multinode Backend Workers on Slurm

`prefill_worker.num-nodes` and `decode_worker.num-nodes` specify the total nodes assigned to each role. Set `nodes-per-worker` to group those nodes into logical backend workers. For example, the following configuration creates one prefill worker spanning two nodes and one decode worker spanning two other nodes:

```toml
[cmd_args.dynamo.prefill_worker]
num-nodes = 2
nodes-per-worker = 2
  [cmd_args.dynamo.prefill_worker.args]
  tensor-parallel-size = 8
  pipeline-parallel-size = 1

[cmd_args.dynamo.decode_worker]
num-nodes = 2
nodes-per-worker = 2
  [cmd_args.dynamo.decode_worker.args]
  tensor-parallel-size = 8
  pipeline-parallel-size = 1
```

When `nodes-per-worker` is omitted, it defaults to `1` and preserves the previous one-worker-per-node behavior. For vLLM TP/PP groups, CloudAI launches rank 0 as the Dynamo-facing process and the remaining ranks with `--headless` using Dynamo's multiprocessing executor. When `data-parallel-size` is greater than one, CloudAI instead launches a full Dynamo vLLM process on every node and derives `data-parallel-size-local`, `data-parallel-start-rank`, the coordinator address, and the RPC port from the worker topology. SGLang receives `--dist-init-addr`, `--nnodes`, and `--node-rank` on every participating node; its configured `data-parallel-size` is forwarded as `dp-size`. The launcher derives prefill and decode disaggregation modes from the configured roles, so worker commands do not need role flags.

For TP/PP-spanning workers, `tensor-parallel-size * pipeline-parallel-size` must be divisible by `nodes-per-worker`. For vLLM multinode data parallelism, `data-parallel-size` must instead be divisible by `nodes-per-worker`, and each DP rank's TP/PP group must fit within one node. SGLang multinode data parallelism requires `--enable-dp-attention`. The resulting per-node GPU footprint must fit on each node. Ray-backed Dynamo vLLM groups are not currently supported. The Slurm launcher explicitly selects etcd discovery and the NATS event plane.

No separate WideEP mode is required. Multinode data-parallel orchestration is inferred from `nodes-per-worker > 1` and `data-parallel-size > 1`. Backend expert-parallel parameters such as vLLM's `--enable-expert-parallel` or SGLang's `ep-size` and `--enable-dp-attention` remain explicit backend configuration. CloudAI uses the allocated Slurm node names for multinode rendezvous. The cluster must make those names consistently resolvable and mutually reachable from every allocated node and workload container. Multi-homed systems should also configure backend network selection, such as `GLOO_SOCKET_IFNAME` and `NCCL_SOCKET_IFNAME`, at the system level. Explicit backend rendezvous addresses take precedence over addresses derived from the Slurm node names. Clusters that cannot provide this hostname contract must supply backend-native per-node address overrides, such as `SGLANG_HOST_IP` or `VLLM_NIXL_SIDE_CHANNEL_HOST`, together with explicit worker rendezvous arguments.

Dedicated examples are available in `test_scenario/vllm_multinode_worker_slurm.toml` and `test_scenario/sglang_multinode_worker_slurm.toml`. The existing `vllm_slurm.toml` and `sglang_slurm.toml` scenarios retain their previous configurations. The vLLM example inherits `tensor-parallel-size = 8` from `test/vllm.toml`; the SGLang example sets it explicitly because `test/sglang.toml` defaults to `1`. WideEP examples are provided in `test_scenario/vllm_wideep_slurm.toml` and `test_scenario/sglang_wideep_slurm.toml`.

### Launch and Monitor the Job

To run the job:

```bash
uv run cloudai run --system-config <slurm system toml> \
   --tests-dir conf/experimental/ai_dynamo/test \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_slurm.toml
```

The job progress monitoring can be done using either of the following options:

```bash
watch squeue --me
```

```bash
watch tail -n 4 ./results/<scenario name>/*.txt
```

The frontend node will initially wait to allow weight loading on all nodes. Once ready, it will launch the configured benchmark tool (`aiperf` by default), which begins generating requests to the frontend server. All servers cooperate to complete inference, and the output will appear in `stdout.txt`.

### Choosing a Benchmark Tool

The benchmark tool is controlled by the `workloads` field in the test TOML. Set `aiperf.sh` to use AIPerf:

```toml
[cmd_args]
workloads = "aiperf.sh"   # uses aiperf, writes aiperf_report.csv
```

To use genai-perf, set:

```toml
[cmd_args]
workloads = "genai_perf.sh"   # uses genai-perf, writes genai_perf_report.csv

[cmd_args.genai_perf]
cmd = "genai-perf profile"
extra-args = "--streaming --verbose -- -v --async"

  [cmd_args.genai_perf.args]
  endpoint-type = "chat"
  output-tokens-mean = 500
  request-count = 50
```

### AIPerf Multi-Phase Runs

`cmd_args.aiperf` is the base AIPerf config. `cmd_args.aiperf_phases` can run several AIPerf rounds against the same live Dynamo stack. By default, CloudAI does not restart prefill, decode, or router processes between phases:

```toml
dse_excluded_args = ["cmd_args.aiperf_phases"]

[cmd_args.aiperf]
health-check-between-phases = true
between-phase-cmd = "true"  # default no-op

  [cmd_args.aiperf.args]
  request-count = 50
  server-metrics = "auto"

[[cmd_args.aiperf_phases]]
name = "round_1"
  [cmd_args.aiperf_phases.args]
  concurrency = 2

[[cmd_args.aiperf_phases]]
name = "round_2"
  [cmd_args.aiperf_phases.args]
  concurrency = 4
```

Single-phase runs keep the old artifact layout: `aiperf_artifacts/`, `aiperf.log`, and `aiperf_report.csv`. Multi-phase runs write per-phase artifacts/logs/reports and copy the last phase report to `aiperf_report.csv` for existing report generation.

`between-phase-cmd` is a bash command run after each non-final phase. The default is a no-op. Set it explicitly for backend-specific cache cleanup, for example `/cloudai_run_results/routerctl.sh restart` if a test needs to restart the Dynamo router between phases. `health-check-between-phases` probes the frontend after the command.

## Comparison Reports

CloudAI provides the scenario-level `ai_dynamo_comparison` report. It compares AI Dynamo runs using the standard LLM serving tables and charts for TTFT, TPOT, successful prompts, output-token throughput, TPS per user, and TPS per GPU. When `aiperf_accuracy` is configured and its result is available, the report also compares model accuracy. Both AIPerf and GenAI-Perf CSV output are supported.

The scenario-level [Pareto report](/networking/cloudai/dev/pareto-frontiers) uses each configured AIPerf phase as a separate point, with that phase's concurrency and benchmark settings. It also supports GenAI-Perf results.

AIPerf args are rendered as normal CLI flags. Multi-value AIPerf options should be passed with AIPerf CLI syntax, such as `server-metrics-formats = "csv,json,jsonl"` or `gpu-telemetry = "node1:9401,node2:9401"`. `server-metrics = "auto"` expands to the frontend metrics endpoint, Dynamo worker metrics endpoints, and any CloudAI-started DCGM exporters.

### Propagating LMCache Configuration

AIDynamo can pass an LMCache YAML config to the worker processes by setting `LMCACHE_CONFIG_FILE` inside the container. This only propagates the LMCache configuration; the vLLM/SGLang runtime still needs to be launched with the appropriate LMCache or KV-transfer connector for that image/version.

The preferred form is structured TOML under `[cmd_args.lmcache]`. CloudAI converts that object to YAML in the run output directory, mounts that directory as `/cloudai_run_results`, and exports the generated file path as `LMCACHE_CONFIG_FILE`:

```toml
[cmd_args]
  [cmd_args.lmcache]
  chunk_size = 256
  local_cpu = true
  controller_pull_url = "{frontend_node}:8300"
  controller_reply_url = "{frontend_node}:8400"
  lmcache_worker_ports = [8788, 8789, 8790, 8791]
  max_local_cpu_size = 6.0
  nixl_buffer_size = 2079377920
  nixl_buffer_device = "cpu"

    [cmd_args.lmcache.extra_config]
    enable_nixl_storage = false
    nixl_backend = "POSIX"
    nixl_path = "{storage_cache_dir}"
    nixl_pool_size = 2048
```

For an example that uses test-in-scenario mode, see `conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml`. Because the test is fully defined inside the scenario, `--tests-dir` is not required when running that example:

```bash
uv run cloudai run --system-config <slurm system toml> \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml
```

The example sets `dse_excluded_args = ["cmd_args.lmcache.lmcache_worker_ports"]` because `lmcache_worker_ports` is a list-valued LMCache setting, not a DSE sweep dimension. Other list-valued LMCache fields can still be swept unless their `cmd_args.` path is also excluded.

Alternatively, mount your own LMCache YAML file with `extra_container_mounts` and set `LMCACHE_CONFIG_FILE` through `extra_env_vars`:

```toml
extra_container_mounts = ["/host/lmcache:/lmcache"]
extra_env_vars = { LMCACHE_CONFIG_FILE = "/lmcache/config.yaml" }
```

For multi-node LMCache storage tests, any path referenced by the LMCache YAML, such as `nixl_path` for POSIX-backed storage, must be visible and writable from every node that is expected to share cached data. A node-local path such as `/tmp` is suitable only for single-node smoke tests or configuration propagation checks.

LMCache YAML values can use runtime placeholders. CloudAI renders them inside the Slurm job before launching workers: `{frontend_node}`, `{frontend_ip}`, `{results_dir}`, and `{storage_cache_dir}`. Unknown placeholders fail the run before worker processes start.

### Propagating HiCache Configuration

Define a HiCache storage-backend configuration under `[cmd_args.hicache]`. CloudAI serializes the object as `hicache-config.toml` in the run output directory, exposes its container path through `HICACHE_CONFIG_FILE`, and passes the resolved path to both workers as `--hicache-storage-backend-extra-config`:

```toml
[cmd_args]
  [cmd_args.hicache.plugin.posix]
  active = true
  use_uring = "false"

  [cmd_args.dynamo.prefill_worker]
  extra-args = "--enable-hierarchical-cache"

  [cmd_args.dynamo.prefill_worker.args]
  hicache-size = 8
  hicache-ratio = 0
  hicache-write-policy = "write_through"
  hicache-mem-layout = "page_first"
  hicache-storage-prefetch-policy = "wait_complete"
  hicache-storage-backend = "nixl"

  [cmd_args.dynamo.decode_worker]
  extra-args = "--enable-hierarchical-cache"

  [cmd_args.dynamo.decode_worker.args]
  hicache-size = 8
  hicache-ratio = 0
  hicache-write-policy = "write_through"
  hicache-mem-layout = "page_first"
  hicache-storage-prefetch-policy = "wait_complete"
  hicache-storage-backend = "nixl"
```

The remaining HiCache engine options stay in the worker `args` sections so they can be configured or swept through DSE normally.

The runtime replacement mechanism below supports per-node HiCache values. For example, change the POSIX setting above to `use_uring = "${use_uring}"` and have the startup manifest replace `${use_uring}` with `false` or `true`. The worker receives the resulting per-node TOML path automatically.

For a self-contained single-node smoke test using NIXL's POSIX plugin, run the following on a node with at least two GPUs (one each for the prefill and decode workers):

```bash
uv run cloudai run --system-config <slurm system toml> \
   --test-scenario conf/experimental/ai_dynamo/test_scenario/sglang_hicache.toml
```

### Per-node Startup Commands and Runtime Replacements

On Slurm, `startup_cmd` runs once per allocated node before the workload. It runs on the host by default. Set `startup_cmd_docker_image` to run it in a container using the workload's mounts:

```toml
[cmd_args]
startup_cmd = "/mnt/discovery/discover-device.sh"
startup_cmd_docker_image = "nvcr.io/example/discovery-tools:latest"
```

CloudAI exports `CLOUDAI_RUNTIME_FILE` to the command. It points to `runtime/<node>.json` in the test output directory, or `/cloudai_run_results/runtime/<node>.json` inside a startup container. Write replacements there:

```json
{
  "LMCACHE_CONFIG_FILE": {
    "${device}": "/dev/ng4n1",
    "${namespace_id}": "1"
  },
  "HICACHE_CONFIG_FILE": {
    "${use_uring}": "false"
  }
}
```

Each top-level key names an environment variable containing a config path. AIDynamo creates a per-node copy, applies literal replacements, then updates the variable to point to that copy.

For `lmcache-config.yaml` on `node-01`, the result is `lmcache-config.node-01.yaml`. The shared source remains unchanged. LMCache placeholders are rendered afterward in the per-node copy. Missing manifest means no replacements.

Startup runs once per test case in normal and `--single-sbatch` modes.

If the selected LMCache mode needs a controller, CloudAI can start one on the frontend node:

```toml
[cmd_args.lmcache_controller]
cmd = "lmcache_controller --host 0.0.0.0 --port 9000 --monitor-ports {\"pull\":8300,\"reply\":8400}"
```

This only launches the process. For disaggregated or multi-node runs, the LMCache YAML still needs controller addresses that resolve to the frontend node from every worker. With the default controller monitor ports, use `controller_pull_url = "{frontend_node}:8300"` and `controller_reply_url = "{frontend_node}:8400"`. The `lmcache_worker_ports` list must match the number of worker ranks.

### Semantic Degradation With AIPerf Accuracy

AIDynamo uses AIPerf accuracy mode as its semantic degradation signal. Enable it with `[cmd_args.aiperf_accuracy]`. This runs after the configured performance workload, so it can be used with either `aiperf.sh` or `genai_perf.sh`:

```toml
[cmd_args]
workloads = "aiperf.sh"

[cmd_args.aiperf]
  [cmd_args.aiperf.args]
  request-count = 50
  synthetic-input-tokens-mean = 300
  output-tokens-mean = 500
  concurrency = 2

[cmd_args.aiperf_accuracy]
entrypoint = "aiperf profile"
setup-cmd = "python -m pip install --break-system-packages --upgrade aiperf==0.8.0"
cli = '''
--model {model}
--url {url}
--endpoint-type chat
--streaming
--artifact-dir {artifact_dir}
--no-server-metrics
--accuracy-benchmark mmlu
--accuracy-n-shots 5
--accuracy-tasks abstract_algebra
--concurrency 10
--extra-inputs '{"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'
--num-requests 100
'''
```

When `cmd_args.aiperf_accuracy` is configured, CloudAI expects AIPerf to produce `accuracy_results.csv` and exposes the `accuracy` metric from its `OVERALL` row. The metric is reported as a 0.0-1.0 fraction. Keep synthetic prompt and token-length flags out of this mode; the benchmark dataset should come from AIPerf's accuracy benchmark.

The `entrypoint` and `cli` fields form the accuracy command. CloudAI expands `{model}`, `{url}`, `{endpoint}`, `{result_dir}`, and `{artifact_dir}` in `cli` before launching it. The `setup-cmd` field is optional. It is useful for Dynamo images that include an older system `aiperf` build without the accuracy benchmark plugins. The example upgrades the image-level `aiperf` before launching `aiperf profile`. MMLU is loaded from `lighteval/mmlu`, so either allow Hugging Face dataset access or pre-cache that dataset before running with `HF_HUB_OFFLINE`/`HF_DATASETS_OFFLINE` enabled. For Qwen3 models, the example disables thinking mode so short MMLU answers can be parsed as choices.

### Custom Accuracy Scripts

`cmd_args.aiperf_accuracy` can also launch a custom mounted script instead of AIPerf. Mount the script or its parent directory with `extra_container_mounts` and set `entrypoint` to the in-container command:

```toml
extra_container_mounts = ["/host/custom_accuracy:/custom_accuracy"]

[cmd_args.aiperf_accuracy]
entrypoint = "python /custom_accuracy/dummy_accuracy.py"
cli = "--model {model} --url {url} --endpoint {endpoint} --artifact-dir {artifact_dir} --prompt ping"
```

CloudAI expands placeholders in `cli` and runs `entrypoint` with that CLI string. The custom command must write `accuracy_results.csv` inside `{artifact_dir}` with an `OVERALL` row. CloudAI copies that file to the run output directory and exposes the same `accuracy` metric as AIPerf accuracy mode.

## Review Benchmark Results

After job completion, CloudAI places output logs and result files in the designated results directory. The result file name depends on the configured `workloads` field:

* `aiperf.sh` → `aiperf_report.csv`
* `genai_perf.sh` → `genai_perf_report.csv`
* `cmd_args.aiperf_accuracy` → `accuracy_results.csv`

If AIPerf accuracy mode is enabled, CloudAI copies `aiperf_accuracy_artifacts/accuracy_results.csv` to `accuracy_results.csv` in the run output directory and marks the run failed if that file is not produced.

Navigate to `./results/<scenario>/<test-id>/0/` and open the CSV to examine performance metrics.

### Shared-Node Disaggregated Runs

For Slurm, set top-level `num_nodes` lower than the sum of `prefill_worker.num-nodes` and `decode_worker.num-nodes` to run both roles on the same allocated node(s). For example, `num_nodes = 1` with `prefill_worker.num-nodes = 1` and `decode_worker.num-nodes = 1` runs one prefill worker and one decode worker on the same node. CloudAI assigns decode GPUs first and prefill GPUs after that based on each role's per-node GPU footprint, including local data-parallel ranks where applicable. The combined role GPU count must fit on one node.

Example `aiperf_report.csv`:

Metric,avg,min,max,p25,p50,p75,p99,std
Inter Token Latency (ms),2.81,2.66,2.88,2.79,2.83,2.84,2.87,0.04
Time to First Token (ms),49.87,17.15,99.91,49.35,49.87,50.52,92.31,9.20
Time to Second Token (ms),0.50,0.03,4.05,0.03,0.04,0.04,3.47,1.08
Request Latency (ms),1652.30,1203.61,6433.87,1453.19,1462.99,1466.72,6431.16,976.18
Output Sequence Length (tokens),498.06,410.00,501.00,500.00,500.00,500.00,501.00,12.62
Input Sequence Length (tokens),300.00,300.00,300.00,300.00,300.00,300.00,300.00,0.00

Metric,Value
Output Token Throughput (tokens/sec),598.78
Total Token Throughput (tokens/sec),962.32
Request Throughput (requests/sec),1.20
Request Count,50.00

## Supported Backends

The following backends are available via the `conf/experimental/ai_dynamo/test/` directory:

* **vLLM** (`vllm.toml`) — use with `test_scenario/vllm_slurm.toml`
* **vLLM multinode worker** — use `test_scenario/vllm_multinode_worker_slurm.toml`
* **vLLM WideEP** — use `test_scenario/vllm_wideep_slurm.toml`
* **vLLM with LMCache config propagation** — use self-contained scenario `test_scenario/vllm_lmcache.toml`
* **sglang** (`sglang.toml`) — use with `test_scenario/sglang_slurm.toml`
* **SGLang multinode worker** — use `test_scenario/sglang_multinode_worker_slurm.toml`
* **SGLang WideEP** — use `test_scenario/sglang_wideep_slurm.toml`

Both backends use `aiperf` as the default benchmark tool and support disaggregated prefill/decode.

## API Documentation

### Command Arguments

### `AIDynamoCmdArgs`

Arguments for AI Dynamo.

| Field                      | Type                                                              | Default                                      | Description                           |
| -------------------------- | ----------------------------------------------------------------- | -------------------------------------------- | ------------------------------------- |
| `dynamo_version`           | `str`                                                             | `'f7e468c7e8ff0d1426db987564e60572167e8464'` | AI Dynamo Git commit, tag, or branch. |
| `docker_image_url`         | `str`                                                             | Required                                     |                                       |
| `startup_cmd`              | `str \| None`                                                     | `None`                                       |                                       |
| `startup_cmd_docker_image` | `str \| None`                                                     | `None`                                       |                                       |
| `storage_cache_dir`        | `str \| list[str] \| None`                                        | `'/tmp'`                                     |                                       |
| `dynamo`                   | `cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoArgs`              | Required                                     |                                       |
| `hicache`                  | `dict \| None`                                                    | `None`                                       |                                       |
| `lmcache`                  | `dict \| None`                                                    | `None`                                       |                                       |
| `lmcache_controller`       | `cloudai.workloads.ai_dynamo.ai_dynamo.LMCacheController \| None` | `None`                                       |                                       |
| `genai_perf`               | `cloudai.workloads.ai_dynamo.ai_dynamo.GenAIPerf`                 | Factory: `GenAIPerf`                         |                                       |
| `aiperf`                   | `cloudai.workloads.ai_dynamo.ai_dynamo.AIPerf`                    | Factory: `AIPerf`                            |                                       |
| `aiperf_phases`            | `list[cloudai.workloads.ai_dynamo.ai_dynamo.AIPerfPhase] \| None` | `None`                                       |                                       |
| `aiperf_accuracy`          | `cloudai.workloads.ai_dynamo.ai_dynamo.AIPerfAccuracy \| None`    | `None`                                       |                                       |
| `workloads`                | `str`                                                             | `'genai_perf.sh'`                            |                                       |

### Test Definition

### `AIDynamoTestDefinition`

Test definition for AI Dynamo.

| Field                    | Type                                                      | Default                                                                        | Description                                                                                                                                                                                                                   |
| ------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                     | Required                                                                       |                                                                                                                                                                                                                               |
| `description`            | `str`                                                     | Required                                                                       |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                     | Required                                                                       |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.ai_dynamo.ai_dynamo.AIDynamoCmdArgs`   | Required                                                                       |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                               | Factory: `list`                                                                |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                      | `{}`                                                                           |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                          | `{}`                                                                           |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                               | `[]`                                                                           |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`       | `[]`                                                                           |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`       | `None`                                                                         |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`         | `None`                                                                         |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`    | `None`                                                                         |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                     | `'grid_search'`                                                                |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                     | `1`                                                                            |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                               | `['default']`                                                                  |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                     | `'inverse'`                                                                    |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                  | `None`                                                                         | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]` | Factory: `dict`                                                                | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |
| `script`                 | `cloudai._core.installables.file.File`                    | `File(src=PosixPath('src/cloudai/workloads/ai_dynamo/ai_dynamo.sh'))`          |                                                                                                                                                                                                                               |
| `constraints`            | `cloudai.workloads.ai_dynamo.ai_dynamo.Constraints`       | `Constraints(prefill_tp_le_decode_tp=True, tp_times_pp_le_gpus_per_node=True)` |                                                                                                                                                                                                                               |
| `success_marker`         | `str`                                                     | `'success-marker.txt'`                                                         |                                                                                                                                                                                                                               |
| `failure_marker`         | `str`                                                     | `'failure-marker.txt'`                                                         |                                                                                                                                                                                                                               |