> This page is for version 1.7.1.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

vLLM workload (`test_template_name` is `vllm`) allows users to execute vLLM benchmarks within the CloudAI framework.

vLLM is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.

## Usage Examples

### Test and Scenario Examples

test.toml (test definition)

```toml
name = "vllm_test"
description = "Example vLLM test"
test_template_name = "vllm"

[cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"

[bench_cmd_args]
random_input_len = 16
random_output_len = 128
max_concurrency = 16
num_prompts = 30

[semantic_eval_cmd_args]
entrypoint = "python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py"
cli = "--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json"
```

scenario.toml (scenario with one test)

```toml
name = "vllm-benchmark"

[[Tests]]
id = "vllm.1"
num_nodes = 1
time_limit = "00:10:00"
test_name = "vllm_test"
```

### Test-in-Scenario example

scenario.toml (separate test toml is not needed)

```toml
name = "vllm-benchmark"

[[Tests]]
id = "vllm.1"
num_nodes = 1
time_limit = "00:10:00"

name = "vllm_test"
description = "Example vLLM test"
test_template_name = "vllm"

[Tests.cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"
```

Workload-specific test definition sections, such as `bench_cmd_args` and `semantic_eval_cmd_args`, are not supported under `[[Tests]]` in a test scenario. Define them in a test definition TOML and reference that test with `test_name` when custom benchmark or semantic-evaluation arguments are needed.

## Semantic Validation

To run GSM8K semantic validation after the serving benchmark, add `semantic_eval_cmd_args`. CloudAI reports `accuracy` from the eval output, but does not enforce an accuracy threshold.

test.toml (semantic validation)

```toml
[semantic_eval_cmd_args]
entrypoint = "python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py"
cli = "--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json"
```

If the runtime image does not contain the eval script, mount a vLLM repository with existing `git_repos` support and point `entrypoint` at the mounted path.

The `cli` string supports `{model}`, `{host}`, `{port}`, `{url}`, `{output_path}`, and `{result_dir}` placeholders.

## Reporting

After a run completes, CloudAI parses `vllm-bench.json` and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If `semantic_eval_cmd_args` is configured, CloudAI also reports semantic validation accuracy.

The reported metric (`default`) is throughput. Additional supported metrics are `throughput`, `tps-per-user`, `tps-per-gpu`, and `accuracy`.

CloudAI also provides the scenario-level `vllm_comparison` report. It compares vLLM test runs in the scenario and uses `bench_cmd_args` values as comparison labels.

## Controlling the Number of GPUs

GPU selection priority, from lowest to highest:

1. `gpus_per_node` system property (scalar value)

2. `decode.gpu_ids` command argument in non-disaggregated mode when `CUDA_VISIBLE_DEVICES` is not set

3. `CUDA_VISIBLE_DEVICES` environment variable (comma-separated list of GPU IDs)

4. `gpu_ids` command argument for both `prefill` and `decode` configurations in disaggregated mode

In disaggregated mode, define both `prefill.gpu_ids` and `decode.gpu_ids`, or omit both.

## Controlling Disaggregation

By default, vLLM will run without disaggregation as a single process. To enable disaggregation, one needs to set `prefill` configuration:

test.toml (disaggregated prefill/decode)

```toml
[cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"

[cmd_args.prefill]

[extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

The config above, will automatically split GPUs specified in `CUDA_VISIBLE_DEVICES` into two: - The first half will be used for prefill - The second half will be used for decode

For more control, users can specify the GPU IDs explicitly in `prefill` and `decode` configurations:

test.toml (disaggregated prefill/decode)

```toml
[cmd_args.prefill]
gpu_ids = "0,1"

[cmd_args.decode]
gpu_ids = "2,3"
```

In this case `CUDA_VISIBLE_DEVICES` will be ignored and only the GPUs specified in `gpu_ids` will be used.

## Multi-node serving

For non-disaggregated `num_nodes`` ``>`` ``1`, CloudAI creates one Ray cluster and starts `vllm`` ``serve` on the head node with `--distributed-executor-backend`` ``ray`.

For disaggregated serving over more than two nodes, set explicit role sizes:

* `prefill.num_nodes`` ``+`` ``decode.num_nodes` must equal the test `num_nodes`.

* CloudAI assigns contiguous node slices: prefill first, decode second.

* `tensor_parallel_size` is total per role, not per node.

* `CUDA_VISIBLE_DEVICES` and `gpu_ids` are local GPU IDs on each serving node.

Example: four prefill nodes and four decode nodes, each with four visible GPUs:

scenario.toml (multi-node disaggregated serving)

```toml
[[Tests]]
id = "vllm.pd_multi_node"
num_nodes = 8
test_template_name = "vllm"

[Tests.cmd_args.prefill]
num_nodes = 4
tensor_parallel_size = 16

[Tests.cmd_args.decode]
num_nodes = 4
tensor_parallel_size = 16

[Tests.extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

## Readiness health checks

Healthcheck fields:

* `healthcheck`: aggregated server endpoint, default `/healthcheck`.

* `serve_healthcheck`: optional override for serve, prefill, and decode servers.

* `proxy_healthcheck`: disaggregated proxy/router endpoint, default `/healthcheck`.

If `serve_healthcheck` is omitted, disaggregated prefill/decode servers keep the legacy `/health` endpoint. If a disaggregated config sets `healthcheck` but omits `proxy_healthcheck`, the proxy/router uses `healthcheck` for backward compatibility.

## Controlling `proxy_script`

`proxy_script` is used to proxy the requests from the client to the prefill and decode instances. It is ignored for non-disaggregated mode. Default value can be found below.

It can be overridden by setting `proxy_script` by using the latest version of the script from vLLM repository:

test_scenario.toml (override proxy_script)

```toml
[[Tests.git_repos]]
url = "https://github.com/vllm-project/vllm.git"
commit = "main"
mount_as = "/vllm_repo"

[Tests.cmd_args]
docker_image_url = "vllm/vllm-openai:v0.14.0-cu130"
proxy_script = "/vllm_repo/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py"
```

In this case the proxy script will be mounted from the vLLM repository (cloned locally) as `/vllm_repo` and used for the test.

## API Documentation

### vLLM Serve Arguments

**&#x20;*pydantic model* cloudai.workloads.vllm.vllm.VllmArgs**

Base command arguments for vLLM instances.

**&#x20;*field* ray_head*: VllmRayStartArgs | None = None*\*\*

Arguments appended to the Ray head startup command for multi-node vLLM roles.

**&#x20;*field* ray_worker*: VllmRayStartArgs | None = None*\*\*

Arguments appended to the Ray worker startup command for multi-node vLLM roles.

**&#x20;*field* nixl_threads*: int | list\[int] | None = None*\*\*

Set `kv_connector_extra_config.num_threads` for `--kv-transfer-config` CLI argument.

**&#x20;*property* serve_args_exclude*: set\[str]*\*\*

Fields consumed internally and excluded from generic serve args.

**&#x20;serialize_serve_arg(*key: str*, *value: object*) → list\[str]**

Serialize a single serve argument to CLI tokens.

**&#x20;*property* serve_args*: list\[str]*\*\*

**&#x20;*field* gpu_ids*: str | list\[str] | None = None*\*\*

Comma-separated GPU IDs. If not set, all available GPUs will be used.

**&#x20;*field* num_nodes*: int | list\[int] | None = None*\*\*

Number of Slurm nodes assigned to this role in disaggregated serving mode.

### Command Arguments

*class* cloudai.workloads.vllm.vllm.VllmCmdArgs(

*\**,

*docker_image_url: str*,

*model: str = 'Qwen/Qwen3-0.6B'*,

*port: \~typing.Annotated\[int*,

*\~annotated_types.Ge(ge=1)*,

*\~annotated_types.Le(le=65535)] = 8300*,

*host: str = '0.0.0.0'*,

*bench_host: str | None = None*,

*healthcheck: str = '/healthcheck'*,

*serve_healthcheck: str | None = None*,

*serve_wait_seconds: int = 300*,

*prefill: \~cloudai.workloads.vllm.vllm.VllmArgs | None = None*,

*decode: \~cloudai.workloads.vllm.vllm.VllmArgs = \<factory>*,

*proxy_script: str = '/opt/vllm/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py'*,

*proxy_healthcheck: str = '/healthcheck'*,

)

Bases: `LLMServingCmdArgs[VllmArgs]`

vLLM serve command arguments.

### Benchmark Command Arguments

*class* cloudai.workloads.vllm.vllm.VllmBenchCmdArgs(

*\**,

*random_input_len: int = 16*,

*random_output_len: int = 128*,

*max_concurrency: int = 16*,

*num_prompts: int = 30*,

*\*\*extra_data: Any*,

)

Bases: `CmdArgs`

vLLM bench serve command arguments.

### Semantic Eval Command Arguments

*class* cloudai.workloads.vllm.vllm.VllmSemanticEvalCmdArgs(

*\**,

*entrypoint: str = 'python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py'*,

*cli: str = '--host \{host} --port \{port} --num-questions 200 --save-results \{output_path}/vllm-gsm8k.json'*,

)

Bases: `CmdArgs`

vLLM semantic validation command arguments.

### Test Definition

**&#x20;*class* cloudai.workloads.vllm.vllm.VllmTestDefinition(*\*, name: str, description: str, test_template_name: str, cmd_args: \~cloudai.workloads.vllm.vllm.VllmCmdArgs, dse_excluded_args: list\[str] = \<factory>, extra_env_vars: dict\[str, str | \~typing.List\[str]] = \{}, extra_cmd_args: dict\[str, str] = \{}, extra_container_mounts: list\[str] = \[], git_repos: list\[\~cloudai.\_core.installables.git_repo.GitRepo] = \[], nsys: \~cloudai.models.workload.NsysConfiguration | None = None, predictor: \~cloudai.models.workload.PredictorConfig | None = None, training_report: \~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = 'grid_search', agent_steps: int = 1, agent_metrics: list\[str] = \['default'], agent_reward_function: str = 'inverse', agent_config: dict\[str, \~typing.Any] | None = None, env_params: dict\[str, \~cloudai.configurator.env_params.EnvParamSpec] = \<factory>, bench_cmd_args: \~cloudai.workloads.vllm.vllm.VllmBenchCmdArgs = VllmBenchCmdArgs(random_input_len=16, random_output_len=128, max_concurrency=16, num_prompts=30), semantic_eval_cmd_args: \~cloudai.workloads.vllm.vllm.VllmSemanticEvalCmdArgs | None = None, proxy_script_repo: \~cloudai.\_core.installables.git_repo.GitRepo | None = None, custom_bash: str | dict\[str, str] | None = None*)**

Bases: `LLMServingTestDefinition[VllmCmdArgs]`

Test object for vLLM.

**&#x20;*property* is_domain_randomization_enabled*: bool*\*\*

at least one `env_params` annotation.

**Type:**

Whether the config declares domain randomization

**&#x20;is_dse_excluded_arg(*path: str*) → bool**

Return whether a dot-separated cmd_args path should be ignored by DSE.

**&#x20;is_env_sampled(*cmd_args_path: str*) → bool**

Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

**&#x20;*validator* validate_env_params*  » \* *all fields**\*

Validate env_params annotations against cmd_args.

`env_params` is an annotation: each key names a `cmd_args` field whose value is the candidate set (the single source of truth), and the entry carries only *how* to sample. So each key must name a real `cmd_args` field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When `weights` are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in `CloudAIGymEnv`; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.