SGLang
This workload (test_template_name is sglang) allows users to execute SGLang benchmarks within the CloudAI framework.
SGLang is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.
Usage Examples
Test and Scenario Examples
test.toml (test definition)
scenario.toml (scenario with one test)
Test-in-Scenario example
scenario.toml (separate test toml is not needed)
Local Models
Set cmd_args.model to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use extra_container_mounts to mount a host or shared-filesystem directory.
test.toml (local model)
The mounted directory must contain a model, configuration, and tokenizer supported by SGLang.
Semantic Validation
To run GSM8K semantic validation after the serving benchmark, add semantic_eval_cmd_args. CloudAI reports accuracy from the eval output, but does not enforce an accuracy threshold.
test.toml (semantic validation)
For images that still use the legacy SGLang GSM8K runner, override the entrypoint and raw CLI:
The cli string supports {model}, {host}, {port}, {url}, {output_path}, and {result_dir} placeholders.
Reporting
After a run completes, CloudAI parses sglang-bench.jsonl and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If semantic_eval_cmd_args is configured, CloudAI also reports semantic validation accuracy.
The reported metric (default) is throughput. Additional supported metrics are throughput, tps-per-user, tps-per-gpu, and accuracy.
CloudAI also provides the scenario-level sglang_comparison report. It compares SGLang test runs in the scenario and uses bench_cmd_args values as comparison labels.
Structured metrics
Readiness health checks
Healthcheck fields:
healthcheck: aggregated server and disaggregated router endpoint, default/v1/models.serve_healthcheck: optional override for serve, prefill, and decode servers.
If serve_healthcheck is omitted, disaggregated prefill/decode servers keep the legacy /health endpoint.
Control number of GPUs
GPU selection priority, from lowest to highest:
gpus_per_nodesystem property (scalar value)decode.gpu_idscommand argument in non-disaggregated mode whenCUDA_VISIBLE_DEVICESis not setCUDA_VISIBLE_DEVICESenvironment variable (comma-separated list of GPU IDs)gpu_idscommand argument for bothprefillanddecodeconfigurations in disaggregated mode
In disaggregated mode, define both prefill.gpu_ids and decode.gpu_ids, or omit both.
Control disaggregation
By default, SGLang will run without disaggregation as a single process. To enable disaggregation, one needs to set prefill configuration:
test.toml (disaggregated prefill/decode)
The config above will automatically split GPUs specified in CUDA_VISIBLE_DEVICES into two halves, first half will be used for prefill and second half will be used for decode.
For more control, one can specify the GPU IDs explicitly in prefill and decode configurations:
test.toml (disaggregated prefill/decode)
In this case CUDA_VISIBLE_DEVICES will be ignored and only the GPUs specified in gpu_ids will be used.
Multi-node serving
For non-disaggregated num_nodes > 1, CloudAI starts one sglang.launch_server task per serving node with shared --dist-init-addr, --nnodes, and --node-rank "$SLURM_PROCID".
For disaggregated serving over more than two nodes, set explicit role sizes:
prefill.num_nodes + decode.num_nodesmust equal the testnum_nodes.- CloudAI assigns contiguous node slices: prefill first, decode second.
tpis total per role, not per node.CUDA_VISIBLE_DEVICESandgpu_idsare local GPU IDs on each serving node.
Example: four prefill nodes and four decode nodes, each with four visible GPUs:
scenario.toml (multi-node disaggregated serving)
API Documentation
SGLang Serve Arguments
SglangArgs
Base command arguments for SGLang instances.
Command Arguments
SglangCmdArgs
SGLang serve command arguments.
Benchmark Command Arguments
SglangBenchCmdArgs
SGLang bench_serving command arguments.
Semantic Eval Command Arguments
SglangSemanticEvalCmdArgs
SGLang semantic validation command arguments.
Test Definition
SglangTestDefinition
Test object for SGLang.