Skip to navigation

vLLM

View as Markdown

vLLM workload (test_template_name is vllm) allows users to execute vLLM benchmarks within the CloudAI framework.

vLLM is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.

Usage Examples

Test and Scenario Examples

test.toml (test definition)

name = "vllm_test"
description = "Example vLLM test"
test_template_name = "vllm"
[cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"
[bench_cmd_args]
random_input_len = 16
random_output_len = 128
max_concurrency = 16
num_prompts = 30
[semantic_eval_cmd_args]
entrypoint = "python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py"
cli = "--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json"

scenario.toml (scenario with one test)

name = "vllm-benchmark"
[[Tests]]
id = "vllm.1"
num_nodes = 1
time_limit = "00:10:00"
test_name = "vllm_test"

Test-in-Scenario example

scenario.toml (separate test toml is not needed)

name = "vllm-benchmark"
[[Tests]]
id = "vllm.1"
num_nodes = 1
time_limit = "00:10:00"
name = "vllm_test"
description = "Example vLLM test"
test_template_name = "vllm"
[Tests.cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"

Local Models

Set cmd_args.model to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use extra_container_mounts to mount a host or shared-filesystem directory.

test.toml (local model)

name = "vllm_local_model"
description = "vLLM with a local model"
test_template_name = "vllm"
extra_container_mounts = [
"/lustre/models/custom:/models/custom:ro",
]
[cmd_args]
docker_image_url = "vllm/vllm-openai:v0.14.0-cu130"
model = "/models/custom"

The mounted directory must contain a model, configuration, and tokenizer supported by vLLM.

Semantic Validation

To run GSM8K semantic validation after the serving benchmark, add semantic_eval_cmd_args. CloudAI reports accuracy from the eval output, but does not enforce an accuracy threshold.

test.toml (semantic validation)

[semantic_eval_cmd_args]
entrypoint = "python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py"
cli = "--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json"

If the runtime image does not contain the eval script, mount a vLLM repository with existing git_repos support and point entrypoint at the mounted path.

The cli string supports {model}, {host}, {port}, {url}, {output_path}, and {result_dir} placeholders.

Reporting

After a run completes, CloudAI parses vllm-bench.json and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If semantic_eval_cmd_args is configured, CloudAI also reports semantic validation accuracy.

The reported metric (default) is throughput. Additional supported metrics are throughput, tps-per-user, tps-per-gpu, and accuracy.

CloudAI also provides the scenario-level vllm_comparison report. It compares vLLM test runs in the scenario and uses bench_cmd_args values as comparison labels.

Structured metrics

MetricUnitPreferredDimensions
request_throughputrequests/sHigherNone
output_token_throughputtokens/sHigherNone
ttft (time to first token)msLowerstatistic: mean, median, p99
tpot (time per output token)msLowerstatistic: mean, median, p99
accuracyratio (0–1)HigherNone

Controlling the Number of GPUs

GPU selection priority, from lowest to highest:

  1. gpus_per_node system property (scalar value)
  2. decode.gpu_ids command argument in non-disaggregated mode when CUDA_VISIBLE_DEVICES is not set
  3. CUDA_VISIBLE_DEVICES environment variable (comma-separated list of GPU IDs)
  4. gpu_ids command argument for both prefill and decode configurations in disaggregated mode

In disaggregated mode, define both prefill.gpu_ids and decode.gpu_ids, or omit both.

Controlling Disaggregation

By default, vLLM will run without disaggregation as a single process. To enable disaggregation, one needs to set prefill configuration:

test.toml (disaggregated prefill/decode)

[cmd_args]
docker_image_url = "nvcr.io#nvidia/ai-dynamo/vllm-runtime:0.7.0"
model = "Qwen/Qwen3-0.6B"
[cmd_args.prefill]
[extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"

The config above, will automatically split GPUs specified in CUDA_VISIBLE_DEVICES into two: - The first half will be used for prefill - The second half will be used for decode

For more control, users can specify the GPU IDs explicitly in prefill and decode configurations:

test.toml (disaggregated prefill/decode)

[cmd_args.prefill]
gpu_ids = "0,1"
[cmd_args.decode]
gpu_ids = "2,3"

In this case CUDA_VISIBLE_DEVICES will be ignored and only the GPUs specified in gpu_ids will be used.

Multi-node serving

For non-disaggregated num_nodes > 1, CloudAI creates one Ray cluster and starts vllm serve on the head node with --distributed-executor-backend ray.

For disaggregated serving over more than two nodes, set explicit role sizes:

  • prefill.num_nodes + decode.num_nodes must equal the test num_nodes.
  • CloudAI assigns contiguous node slices: prefill first, decode second.
  • tensor_parallel_size is total per role, not per node.
  • CUDA_VISIBLE_DEVICES and gpu_ids are local GPU IDs on each serving node.

Example: four prefill nodes and four decode nodes, each with four visible GPUs:

scenario.toml (multi-node disaggregated serving)

[[Tests]]
id = "vllm.pd_multi_node"
num_nodes = 8
test_template_name = "vllm"
[Tests.cmd_args.prefill]
num_nodes = 4
tensor_parallel_size = 16
[Tests.cmd_args.decode]
num_nodes = 4
tensor_parallel_size = 16
[Tests.extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"

Readiness health checks

Healthcheck fields:

  • healthcheck: aggregated server endpoint, default /healthcheck.
  • serve_healthcheck: optional override for serve, prefill, and decode servers.
  • proxy_healthcheck: disaggregated proxy/router endpoint, default /healthcheck.

If serve_healthcheck is omitted, disaggregated prefill/decode servers keep the legacy /health endpoint. If a disaggregated config sets healthcheck but omits proxy_healthcheck, the proxy/router uses healthcheck for backward compatibility.

Controlling proxy_script

proxy_script is used to proxy the requests from the client to the prefill and decode instances. It is ignored for non-disaggregated mode. Default value can be found below.

It can be overridden by setting proxy_script by using the latest version of the script from vLLM repository:

test_scenario.toml (override proxy_script)

[[Tests.git_repos]]
url = "https://github.com/vllm-project/vllm.git"
commit = "main"
mount_as = "/vllm_repo"
[Tests.cmd_args]
docker_image_url = "vllm/vllm-openai:v0.14.0-cu130"
proxy_script = "/vllm_repo/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py"

In this case the proxy script will be mounted from the vLLM repository (cloned locally) as /vllm_repo and used for the test.

API Documentation

vLLM Serve Arguments

VllmArgs

Base command arguments for vLLM instances.

FieldTypeDefaultDescription
gpu_idsstr | list[str] | NoneNoneComma-separated GPU IDs. If not set, all available GPUs will be used.
num_nodesint | list[int] | NoneNoneNumber of Slurm nodes assigned to this role in disaggregated serving mode.
ray_headVllmRayStartArgs | NoneNoneArguments appended to the Ray head startup command for multi-node vLLM roles.
ray_workerVllmRayStartArgs | NoneNoneArguments appended to the Ray worker startup command for multi-node vLLM roles.
nixl_threadsint | list[int] | NoneNoneSet kv_connector_extra_config.num_threads for --kv-transfer-config CLI argument.

Command Arguments

VllmCmdArgs

vLLM serve command arguments.

FieldTypeDefaultDescription
docker_image_urlstrRequired
modelstr'Qwen/Qwen3-0.6B'
portint8300Constraints: Ge(ge=1), Le(le=65535).
hoststr'0.0.0.0'Host/interface for serve or router processes to bind to.
bench_hoststr | NoneNoneHostname used by the benchmark client. Defaults to the allocated node hostname.
healthcheckstr'/healthcheck'vLLM server healthcheck endpoint.
serve_healthcheckstr | NoneNoneReadiness endpoint for serve, prefill, and decode server processes. Defaults to healthcheck.
serve_wait_secondsint300
prefillcloudai.workloads.vllm.vllm.VllmArgs | NoneNonePrefill instance arguments. If not set, a single instance without disaggregation will be used.
decodecloudai.workloads.vllm.vllm.VllmArgsFactory: VllmArgsDecode instance arguments.
proxy_scriptstr'/opt/vllm/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py'
proxy_healthcheckstr'/healthcheck'vLLM disaggregated proxy/router healthcheck endpoint.

Benchmark Command Arguments

VllmBenchCmdArgs

vLLM bench serve command arguments.

FieldTypeDefaultDescription
random_input_lenint16
random_output_lenint128
max_concurrencyint16
num_promptsint30

Semantic Eval Command Arguments

VllmSemanticEvalCmdArgs

vLLM semantic validation command arguments.

FieldTypeDefaultDescription
entrypointstr'python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py'
clistr'--host {host} --port {port} --num-questions 200 --save-results {output_path}/vllm-gsm8k.json'

Test Definition

VllmTestDefinition

Test object for vLLM.

FieldTypeDefaultDescription
namestrRequired
descriptionstrRequired
test_template_namestrRequired
cmd_argscloudai.workloads.vllm.vllm.VllmCmdArgsRequired
dse_excluded_argslist[str]Factory: list
extra_env_varsdict[str, str | typing.List[str]]{}
extra_cmd_argsdict[str, str]{}
extra_container_mountslist[str][]
git_reposlist[cloudai._core.installables.git_repo.GitRepo][]
nsyscloudai.models.workload.NsysConfiguration | NoneNone
predictorcloudai.models.workload.PredictorConfig | NoneNone
training_reportcloudai.models.workload.TrainingReportConfig | NoneNone
agentstr'grid_search'
agent_stepsint1
agent_metricslist[str]['default']
agent_reward_functionstr'inverse'
agent_configdict[str, Any] | NoneNoneAgent configuration.
env_paramsdict[str, cloudai.configurator.env_params.EnvParamSpec]Factory: dictEnvironment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key.
bench_cmd_argscloudai.workloads.vllm.vllm.VllmBenchCmdArgsVllmBenchCmdArgs(random_input_len=16, random_output_len=128, max_concurrency=16, num_prompts=30)
semantic_eval_cmd_argscloudai.workloads.vllm.vllm.VllmSemanticEvalCmdArgs | NoneNone
proxy_script_repocloudai._core.installables.git_repo.GitRepo | NoneNone
custom_bashstr | dict[str, str] | NoneNone