vLLM
vLLM workload (test_template_name is vllm) allows users to execute vLLM benchmarks within the CloudAI framework.
vLLM is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.
Usage Examples
Test and Scenario Examples
test.toml (test definition)
scenario.toml (scenario with one test)
Test-in-Scenario example
scenario.toml (separate test toml is not needed)
Local Models
Set cmd_args.model to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use extra_container_mounts to mount a host or shared-filesystem directory.
test.toml (local model)
The mounted directory must contain a model, configuration, and tokenizer supported by vLLM.
Semantic Validation
To run GSM8K semantic validation after the serving benchmark, add semantic_eval_cmd_args. CloudAI reports accuracy from the eval output, but does not enforce an accuracy threshold.
test.toml (semantic validation)
If the runtime image does not contain the eval script, mount a vLLM repository with existing git_repos support and point entrypoint at the mounted path.
The cli string supports {model}, {host}, {port}, {url}, {output_path}, and {result_dir} placeholders.
Reporting
After a run completes, CloudAI parses vllm-bench.json and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If semantic_eval_cmd_args is configured, CloudAI also reports semantic validation accuracy.
The reported metric (default) is throughput. Additional supported metrics are throughput, tps-per-user, tps-per-gpu, and accuracy.
CloudAI also provides the scenario-level vllm_comparison report. It compares vLLM test runs in the scenario and uses bench_cmd_args values as comparison labels.
Structured metrics
Controlling the Number of GPUs
GPU selection priority, from lowest to highest:
-
gpus_per_nodesystem property (scalar value) -
decode.gpu_idscommand argument in non-disaggregated mode whenCUDA_VISIBLE_DEVICESis not set -
CUDA_VISIBLE_DEVICESenvironment variable (comma-separated list of GPU IDs) -
gpu_idscommand argument for bothprefillanddecodeconfigurations in disaggregated mode
In disaggregated mode, define both prefill.gpu_ids and decode.gpu_ids, or omit both.
Controlling Disaggregation
By default, vLLM will run without disaggregation as a single process. To enable disaggregation, one needs to set prefill configuration:
test.toml (disaggregated prefill/decode)
The config above, will automatically split GPUs specified in CUDA_VISIBLE_DEVICES into two: - The first half will be used for prefill - The second half will be used for decode
For more control, users can specify the GPU IDs explicitly in prefill and decode configurations:
test.toml (disaggregated prefill/decode)
In this case CUDA_VISIBLE_DEVICES will be ignored and only the GPUs specified in gpu_ids will be used.
Multi-node serving
For non-disaggregated num_nodes`` ``>`` ``1, CloudAI creates one Ray cluster and starts vllm`` ``serve on the head node with --distributed-executor-backend`` ``ray.
For disaggregated serving over more than two nodes, set explicit role sizes:
-
prefill.num_nodes`` ``+`` ``decode.num_nodesmust equal the testnum_nodes. -
CloudAI assigns contiguous node slices: prefill first, decode second.
-
tensor_parallel_sizeis total per role, not per node. -
CUDA_VISIBLE_DEVICESandgpu_idsare local GPU IDs on each serving node.
Example: four prefill nodes and four decode nodes, each with four visible GPUs:
scenario.toml (multi-node disaggregated serving)
Readiness health checks
Healthcheck fields:
-
healthcheck: aggregated server endpoint, default/healthcheck. -
serve_healthcheck: optional override for serve, prefill, and decode servers. -
proxy_healthcheck: disaggregated proxy/router endpoint, default/healthcheck.
If serve_healthcheck is omitted, disaggregated prefill/decode servers keep the legacy /health endpoint. If a disaggregated config sets healthcheck but omits proxy_healthcheck, the proxy/router uses healthcheck for backward compatibility.
Controlling proxy_script
proxy_script is used to proxy the requests from the client to the prefill and decode instances. It is ignored for non-disaggregated mode. Default value can be found below.
It can be overridden by setting proxy_script by using the latest version of the script from vLLM repository:
test_scenario.toml (override proxy_script)
In this case the proxy script will be mounted from the vLLM repository (cloned locally) as /vllm_repo and used for the test.
API Documentation
vLLM Serve Arguments
pydantic model cloudai.workloads.vllm.vllm.VllmArgs
Base command arguments for vLLM instances.
field ray_head: VllmRayStartArgs | None = None**
Arguments appended to the Ray head startup command for multi-node vLLM roles.
field ray_worker: VllmRayStartArgs | None = None**
Arguments appended to the Ray worker startup command for multi-node vLLM roles.
field nixl_threads: int | list[int] | None = None**
Set kv_connector_extra_config.num_threads for --kv-transfer-config CLI argument.
property serve_args_exclude: set[str]**
Fields consumed internally and excluded from generic serve args.
serialize_serve_arg(key: str, value: object) → list[str]
Serialize a single serve argument to CLI tokens.
property serve_args: list[str]**
field gpu_ids: str | list[str] | None = None**
Comma-separated GPU IDs. If not set, all available GPUs will be used.
field num_nodes: int | list[int] | None = None**
Number of Slurm nodes assigned to this role in disaggregated serving mode.
Command Arguments
class cloudai.workloads.vllm.vllm.VllmCmdArgs(
*,
docker_image_url: str,
model: str = ‘Qwen/Qwen3-0.6B’,
port: ~typing.Annotated[int,
~annotated_types.Ge(ge=1),
~annotated_types.Le(le=65535)] = 8300,
host: str = ‘0.0.0.0’,
bench_host: str | None = None,
healthcheck: str = ‘/healthcheck’,
serve_healthcheck: str | None = None,
serve_wait_seconds: int = 300,
prefill: ~cloudai.workloads.vllm.vllm.VllmArgs | None = None,
decode: ~cloudai.workloads.vllm.vllm.VllmArgs = <factory>,
proxy_script: str = ‘/opt/vllm/tests/v1/kv_connector/nixl_integration/toy_proxy_server.py’,
proxy_healthcheck: str = ‘/healthcheck’,
)
Bases: LLMServingCmdArgs[VllmArgs]
vLLM serve command arguments.
Benchmark Command Arguments
class cloudai.workloads.vllm.vllm.VllmBenchCmdArgs(
*,
random_input_len: int = 16,
random_output_len: int = 128,
max_concurrency: int = 16,
num_prompts: int = 30,
**extra_data: Any,
)
Bases: CmdArgs
vLLM bench serve command arguments.
Semantic Eval Command Arguments
class cloudai.workloads.vllm.vllm.VllmSemanticEvalCmdArgs(
*,
entrypoint: str = ‘python3 /opt/vllm/tests/evals/gsm8k/gsm8k_eval.py’,
cli: str = ‘—host {host} —port {port} —num-questions 200 —save-results {output_path}/vllm-gsm8k.json’,
)
Bases: CmdArgs
vLLM semantic validation command arguments.
Test Definition
class cloudai.workloads.vllm.vllm.VllmTestDefinition(*, name: str, description: str, test_template_name: str, cmd_args: ~cloudai.workloads.vllm.vllm.VllmCmdArgs, dse_excluded_args: list[str] = <factory>, extra_env_vars: dict[str, str | ~typing.List[str]] = {}, extra_cmd_args: dict[str, str] = {}, extra_container_mounts: list[str] = [], git_repos: list[~cloudai._core.installables.git_repo.GitRepo] = [], nsys: ~cloudai.models.workload.NsysConfiguration | None = None, predictor: ~cloudai.models.workload.PredictorConfig | None = None, training_report: ~cloudai.models.workload.TrainingReportConfig | None = None, agent: str = ‘grid_search’, agent_steps: int = 1, agent_metrics: list[str] = [‘default’], agent_reward_function: str = ‘inverse’, agent_config: dict[str, ~typing.Any] | None = None, env_params: dict[str, ~cloudai.configurator.env_params.EnvParamSpec] = <factory>, bench_cmd_args: ~cloudai.workloads.vllm.vllm.VllmBenchCmdArgs = VllmBenchCmdArgs(random_input_len=16, random_output_len=128, max_concurrency=16, num_prompts=30), semantic_eval_cmd_args: ~cloudai.workloads.vllm.vllm.VllmSemanticEvalCmdArgs | None = None, proxy_script_repo: ~cloudai._core.installables.git_repo.GitRepo | None = None, custom_bash: str | dict[str, str] | None = None)
Bases: LLMServingTestDefinition[VllmCmdArgs]
Test object for vLLM.
metric_observations(
system: System,
tr: TestRun,
) → list[MetricObservation]
Return canonical metric observations produced by this test run, if supported.
property is_domain_randomization_enabled: bool**
at least one env_params annotation.
Type:
Whether the config declares domain randomization
is_dse_excluded_arg(path: str) → bool
Return whether a dot-separated cmd_args path should be ignored by DSE.
is_env_sampled(cmd_args_path: str) → bool
Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).
validator validate_env_params » * all fields*
Validate env_params annotations against cmd_args.
env_params is an annotation: each key names a cmd_args field whose value is the candidate set (the single source of truth), and the entry carries only how to sample. So each key must name a real cmd_args field whose value is a candidate list; a scalar is already fixed, so annotating it is a meaningless label and is rejected here. When weights are declared, the list needs >= 2 values and the weights must align 1:1 with it. Sampling, persistence, the per-trial cmd_args overlay, and the cache key all live in CloudAIGymEnv; keeping this shape check in core lets the overlay stay agent- and workload-agnostic rather than re-implemented per workload.