> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

vLLM workload (`test_template_name` is `vllm`) allows users to execute vLLM benchmarks within the CloudAI framework.

vLLM is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.

## Usage Examples

### Test and Scenario Examples

**test.toml (test definition)**

```toml
name = "vllm_test"
description = "Example vLLM test"
test_template_name = "vllm"

[cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"

[bench_cmd_args]
random_input_len = 16
random_output_len = 128
max_concurrency = 16
num_prompts = 30

[semantic_eval_cmd_args]
entrypoint = "python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py"
cli = "--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json"
```

**scenario.toml (scenario with one test)**

```toml
name = "vllm-benchmark"

[[Tests]]
id = "vllm.1"
num_nodes = 1
time_limit = "00:10:00"
test_name = "vllm_test"
```

### Test-in-Scenario example

**scenario.toml (separate test toml is not needed)**

```toml
name = "vllm-benchmark"

[[Tests]]
id = "vllm.1"
num_nodes = 1
time_limit = "00:10:00"

name = "vllm_test"
description = "Example vLLM test"
test_template_name = "vllm"

[Tests.cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"
```

## Local Models

Set `cmd_args.model` to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use `extra_container_mounts` to mount a host or shared-filesystem directory.

**test.toml (local model)**

```toml
name = "vllm_local_model"
description = "vLLM with a local model"
test_template_name = "vllm"
extra_container_mounts = [
  "/lustre/models/custom:/models/custom:ro",
]

[cmd_args]
docker_image_url = "vllm/vllm-openai:v0.14.0-cu130"
model = "/models/custom"
```

The mounted directory must contain a model, configuration, and tokenizer supported by vLLM.

## Semantic Validation

To run GSM8K semantic validation after the serving benchmark, add `semantic_eval_cmd_args`. CloudAI reports `accuracy` from the eval output, but does not enforce an accuracy threshold.

**test.toml (semantic validation)**

```toml
[semantic_eval_cmd_args]
entrypoint = "python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py"
cli = "--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json"
```

If the runtime image does not contain the eval script, mount a vLLM repository with existing `git_repos` support and point `entrypoint` at the mounted path.

The `cli` string supports `{model}`, `{host}`, `{port}`, `{url}`, `{output_path}`, and `{result_dir}` placeholders.

## Reporting

After a run completes, CloudAI parses `vllm-bench.json` and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If `semantic_eval_cmd_args` is configured, CloudAI also reports semantic validation accuracy.

The reported metric (`default`) is throughput. Additional supported metrics are `throughput`, `tps-per-user`, `tps-per-gpu`, and `accuracy`.

CloudAI also provides the scenario-level `vllm_comparison` report. It compares vLLM test runs in the scenario and uses `bench_cmd_args` values as comparison labels.

### Structured metrics

| Metric                         | Unit        | Preferred | Dimensions                     |
| ------------------------------ | ----------- | --------- | ------------------------------ |
| `request_throughput`           | requests/s  | Higher    | None                           |
| `output_token_throughput`      | tokens/s    | Higher    | None                           |
| `ttft` (time to first token)   | ms          | Lower     | `statistic`: mean, median, p99 |
| `tpot` (time per output token) | ms          | Lower     | `statistic`: mean, median, p99 |
| `accuracy`                     | ratio (0–1) | Higher    | None                           |

## Controlling the Number of GPUs

GPU selection priority, from lowest to highest:

1. `gpus_per_node` system property (scalar value)
2. `decode.gpu_ids` command argument in non-disaggregated mode when `CUDA_VISIBLE_DEVICES` is not set
3. `CUDA_VISIBLE_DEVICES` environment variable (comma-separated list of GPU IDs)
4. `gpu_ids` command argument for both `prefill` and `decode` configurations in disaggregated mode

In disaggregated mode, define both `prefill.gpu_ids` and `decode.gpu_ids`, or omit both.

## Controlling Disaggregation

By default, vLLM will run without disaggregation as a single process. To enable disaggregation, one needs to set `prefill` configuration:

**test.toml (disaggregated prefill/decode)**

```toml
[cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"

[cmd_args.prefill]

[extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

The config above, will automatically split GPUs specified in `CUDA_VISIBLE_DEVICES` into two: - The first half will be used for prefill - The second half will be used for decode

For more control, users can specify the GPU IDs explicitly in `prefill` and `decode` configurations:

**test.toml (disaggregated prefill/decode)**

```toml
[cmd_args.prefill]
gpu_ids = "0,1"

[cmd_args.decode]
gpu_ids = "2,3"
```

In this case `CUDA_VISIBLE_DEVICES` will be ignored and only the GPUs specified in `gpu_ids` will be used.

## Multi-node serving

For non-disaggregated `num_nodes > 1`, CloudAI creates one Ray cluster and starts `vllm serve` on the head node with `--distributed-executor-backend ray`.

For disaggregated serving over more than two nodes, set explicit role sizes:

* `prefill.num_nodes + decode.num_nodes` must equal the test `num_nodes`.
* CloudAI assigns contiguous node slices: prefill first, decode second.
* `tensor_parallel_size` is total per role, not per node.
* `CUDA_VISIBLE_DEVICES` and `gpu_ids` are local GPU IDs on each serving node.

Example: four prefill nodes and four decode nodes, each with four visible GPUs:

**scenario.toml (multi-node disaggregated serving)**

```toml
[[Tests]]
id = "vllm.pd_multi_node"
num_nodes = 8
test_template_name = "vllm"

[Tests.cmd_args.prefill]
num_nodes = 4
tensor_parallel_size = 16

[Tests.cmd_args.decode]
num_nodes = 4
tensor_parallel_size = 16

[Tests.extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

## Readiness health checks

Healthcheck fields:

* `healthcheck`: aggregated server endpoint, default `/healthcheck`.
* `serve_healthcheck`: optional override for serve, prefill, and decode servers.
* `proxy_healthcheck`: disaggregated proxy/router endpoint, default `/healthcheck`.

If `serve_healthcheck` is omitted, disaggregated prefill/decode servers keep the legacy `/health` endpoint. If a disaggregated config sets `healthcheck` but omits `proxy_healthcheck`, the proxy/router uses `healthcheck` for backward compatibility.

## Controlling `proxy_script`

`proxy_script` is used to proxy the requests from the client to the prefill and decode instances. It is ignored for non-disaggregated mode. Default value can be found below.

It can be overridden by setting `proxy_script` by using the latest version of the script from vLLM repository:

**test_scenario.toml (override proxy_script)**

```toml
[[Tests.git_repos]]
url = "https://github.com/vllm-project/vllm.git"
commit = "main"
mount_as = "/vllm_repo"

[Tests.cmd_args]
docker_image_url = "vllm/vllm-openai:v0.14.0-cu130"
proxy_script = "/vllm_repo/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py"
```

In this case the proxy script will be mounted from the vLLM repository (cloned locally) as `/vllm_repo` and used for the test.

## API Documentation

### vLLM Serve Arguments

### `VllmArgs`

Base command arguments for vLLM instances.

| Field          | Type                       | Default | Description                                                                          |
| -------------- | -------------------------- | ------- | ------------------------------------------------------------------------------------ |
| `gpu_ids`      | `str \| list[str] \| None` | `None`  | Comma-separated GPU IDs. If not set, all available GPUs will be used.                |
| `num_nodes`    | `int \| list[int] \| None` | `None`  | Number of Slurm nodes assigned to this role in disaggregated serving mode.           |
| `ray_head`     | `VllmRayStartArgs \| None` | `None`  | Arguments appended to the Ray head startup command for multi-node vLLM roles.        |
| `ray_worker`   | `VllmRayStartArgs \| None` | `None`  | Arguments appended to the Ray worker startup command for multi-node vLLM roles.      |
| `nixl_threads` | `int \| list[int] \| None` | `None`  | Set `kv_connector_extra_config.num_threads` for `--kv-transfer-config` CLI argument. |

### Command Arguments

### `VllmCmdArgs`

vLLM serve command arguments.

| Field                | Type                                           | Default                                                                  | Description                                                                                    |
| -------------------- | ---------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| `docker_image_url`   | `str`                                          | Required                                                                 |                                                                                                |
| `model`              | `str`                                          | `'Qwen/Qwen3-0.6B'`                                                      |                                                                                                |
| `port`               | `int`                                          | `8300`                                                                   | Constraints: `Ge(ge=1)`, `Le(le=65535)`.                                                       |
| `host`               | `str`                                          | `'0.0.0.0'`                                                              | Host/interface for serve or router processes to bind to.                                       |
| `bench_host`         | `str \| None`                                  | `None`                                                                   | Hostname used by the benchmark client. Defaults to the allocated node hostname.                |
| `healthcheck`        | `str`                                          | `'/healthcheck'`                                                         | vLLM server healthcheck endpoint.                                                              |
| `serve_healthcheck`  | `str \| None`                                  | `None`                                                                   | Readiness endpoint for serve, prefill, and decode server processes. Defaults to healthcheck.   |
| `serve_wait_seconds` | `int`                                          | `300`                                                                    |                                                                                                |
| `prefill`            | `cloudai.workloads.vllm.vllm.VllmArgs \| None` | `None`                                                                   | Prefill instance arguments. If not set, a single instance without disaggregation will be used. |
| `decode`             | `cloudai.workloads.vllm.vllm.VllmArgs`         | Factory: `VllmArgs`                                                      | Decode instance arguments.                                                                     |
| `proxy_script`       | `str`                                          | `'/opt/vllm/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py'` |                                                                                                |
| `proxy_healthcheck`  | `str`                                          | `'/healthcheck'`                                                         | vLLM disaggregated proxy/router healthcheck endpoint.                                          |

### Benchmark Command Arguments

### `VllmBenchCmdArgs`

vLLM bench serve command arguments.

| Field               | Type  | Default | Description |
| ------------------- | ----- | ------- | ----------- |
| `random_input_len`  | `int` | `16`    |             |
| `random_output_len` | `int` | `128`   |             |
| `max_concurrency`   | `int` | `16`    |             |
| `num_prompts`       | `int` | `30`    |             |

### Semantic Eval Command Arguments

### `VllmSemanticEvalCmdArgs`

vLLM semantic validation command arguments.

| Field        | Type  | Default                                                                                          | Description |
| ------------ | ----- | ------------------------------------------------------------------------------------------------ | ----------- |
| `entrypoint` | `str` | `'python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py'`                                            |             |
| `cli`        | `str` | `'--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json'` |             |

### Test Definition

### `VllmTestDefinition`

Test object for vLLM.

| Field                    | Type                                                          | Default                                                                                            | Description                                                                                                                                                                                                                   |
| ------------------------ | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                         | Required                                                                                           |                                                                                                                                                                                                                               |
| `description`            | `str`                                                         | Required                                                                                           |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                         | Required                                                                                           |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.vllm.vllm.VllmCmdArgs`                     | Required                                                                                           |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                                   | Factory: `list`                                                                                    |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                          | `{}`                                                                                               |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                              | `{}`                                                                                               |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                                   | `[]`                                                                                               |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`           | `[]`                                                                                               |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`           | `None`                                                                                             |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`             | `None`                                                                                             |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`        | `None`                                                                                             |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                         | `'grid_search'`                                                                                    |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                         | `1`                                                                                                |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                                   | `['default']`                                                                                      |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                         | `'inverse'`                                                                                        |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                      | `None`                                                                                             | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]`     | Factory: `dict`                                                                                    | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |
| `bench_cmd_args`         | `cloudai.workloads.vllm.vllm.VllmBenchCmdArgs`                | `VllmBenchCmdArgs(random_input_len=16, random_output_len=128, max_concurrency=16, num_prompts=30)` |                                                                                                                                                                                                                               |
| `semantic_eval_cmd_args` | `cloudai.workloads.vllm.vllm.VllmSemanticEvalCmdArgs \| None` | `None`                                                                                             |                                                                                                                                                                                                                               |
| `proxy_script_repo`      | `cloudai._core.installables.git_repo.GitRepo \| None`         | `None`                                                                                             |                                                                                                                                                                                                                               |
| `custom_bash`            | `str \| dict[str, str] \| None`                               | `None`                                                                                             |                                                                                                                                                                                                                               |