vLLM
vLLM workload (test_template_name is vllm) allows users to execute vLLM benchmarks within the CloudAI framework.
vLLM is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.
Usage Examples
Test and Scenario Examples
test.toml (test definition)
scenario.toml (scenario with one test)
Test-in-Scenario example
scenario.toml (separate test toml is not needed)
Local Models
Set cmd_args.model to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use extra_container_mounts to mount a host or shared-filesystem directory.
test.toml (local model)
The mounted directory must contain a model, configuration, and tokenizer supported by vLLM.
Semantic Validation
To run GSM8K semantic validation after the serving benchmark, add semantic_eval_cmd_args. CloudAI reports accuracy from the eval output, but does not enforce an accuracy threshold.
test.toml (semantic validation)
If the runtime image does not contain the eval script, mount a vLLM repository with existing git_repos support and point entrypoint at the mounted path.
The cli string supports {model}, {host}, {port}, {url}, {output_path}, and {result_dir} placeholders.
Reporting
After a run completes, CloudAI parses vllm-bench.json and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If semantic_eval_cmd_args is configured, CloudAI also reports semantic validation accuracy.
The reported metric (default) is throughput. Additional supported metrics are throughput, tps-per-user, tps-per-gpu, and accuracy.
CloudAI also provides the scenario-level vllm_comparison report. It compares vLLM test runs in the scenario and uses bench_cmd_args values as comparison labels.
Structured metrics
Controlling the Number of GPUs
GPU selection priority, from lowest to highest:
gpus_per_nodesystem property (scalar value)decode.gpu_idscommand argument in non-disaggregated mode whenCUDA_VISIBLE_DEVICESis not setCUDA_VISIBLE_DEVICESenvironment variable (comma-separated list of GPU IDs)gpu_idscommand argument for bothprefillanddecodeconfigurations in disaggregated mode
In disaggregated mode, define both prefill.gpu_ids and decode.gpu_ids, or omit both.
Controlling Disaggregation
By default, vLLM will run without disaggregation as a single process. To enable disaggregation, one needs to set prefill configuration:
test.toml (disaggregated prefill/decode)
The config above, will automatically split GPUs specified in CUDA_VISIBLE_DEVICES into two: - The first half will be used for prefill - The second half will be used for decode
For more control, users can specify the GPU IDs explicitly in prefill and decode configurations:
test.toml (disaggregated prefill/decode)
In this case CUDA_VISIBLE_DEVICES will be ignored and only the GPUs specified in gpu_ids will be used.
Multi-node serving
For non-disaggregated num_nodes > 1, CloudAI creates one Ray cluster and starts vllm serve on the head node with --distributed-executor-backend ray.
For disaggregated serving over more than two nodes, set explicit role sizes:
prefill.num_nodes + decode.num_nodesmust equal the testnum_nodes.- CloudAI assigns contiguous node slices: prefill first, decode second.
tensor_parallel_sizeis total per role, not per node.CUDA_VISIBLE_DEVICESandgpu_idsare local GPU IDs on each serving node.
Example: four prefill nodes and four decode nodes, each with four visible GPUs:
scenario.toml (multi-node disaggregated serving)
Readiness health checks
Healthcheck fields:
healthcheck: aggregated server endpoint, default/healthcheck.serve_healthcheck: optional override for serve, prefill, and decode servers.proxy_healthcheck: disaggregated proxy/router endpoint, default/healthcheck.
If serve_healthcheck is omitted, disaggregated prefill/decode servers keep the legacy /health endpoint. If a disaggregated config sets healthcheck but omits proxy_healthcheck, the proxy/router uses healthcheck for backward compatibility.
Controlling proxy_script
proxy_script is used to proxy the requests from the client to the prefill and decode instances. It is ignored for non-disaggregated mode. Default value can be found below.
It can be overridden by setting proxy_script by using the latest version of the script from vLLM repository:
test_scenario.toml (override proxy_script)
In this case the proxy script will be mounted from the vLLM repository (cloned locally) as /vllm_repo and used for the test.
API Documentation
vLLM Serve Arguments
VllmArgs
Base command arguments for vLLM instances.
Command Arguments
VllmCmdArgs
vLLM serve command arguments.
Benchmark Command Arguments
VllmBenchCmdArgs
vLLM bench serve command arguments.
Semantic Eval Command Arguments
VllmSemanticEvalCmdArgs
vLLM semantic validation command arguments.
Test Definition
VllmTestDefinition
Test object for vLLM.