SGLang
This workload (test_template_name is sglang) allows users to execute SGLang benchmarks within the CloudAI framework.
SGLang is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.
Usage Examples
Test and Scenario Examples
test.toml (test definition)
scenario.toml (scenario with one test)
Test-in-Scenario example
scenario.toml (separate test toml is not needed)
Workload-specific test definition sections, such as bench_cmd_args and semantic_eval_cmd_args, are not supported under [[Tests]] in a test scenario. Define them in a test definition TOML and reference that test with test_name when custom benchmark or semantic-evaluation arguments are needed.
Semantic Validation
To run GSM8K semantic validation after the serving benchmark, add semantic_eval_cmd_args. CloudAI reports accuracy from the eval output, but does not enforce an accuracy threshold.
test.toml (semantic validation)
For images that still use the legacy SGLang GSM8K runner, override the entrypoint and raw CLI:
The cli string supports {model}, {host}, {port}, {url}, {output_path}, and {result_dir} placeholders.
Reporting
After a run completes, CloudAI parses sglang-bench.jsonl and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If semantic_eval_cmd_args is configured, CloudAI also reports semantic validation accuracy.
The reported metric (default) is throughput. Additional supported metrics are throughput, tps-per-user, tps-per-gpu, and accuracy.
CloudAI also provides the scenario-level sglang_comparison report. It compares SGLang test runs in the scenario and uses bench_cmd_args values as comparison labels.
Readiness health checks
Healthcheck fields:
-
healthcheck: aggregated server and disaggregated router endpoint, default/v1/models. -
serve_healthcheck: optional override for serve, prefill, and decode servers.
If serve_healthcheck is omitted, disaggregated prefill/decode servers keep the legacy /health endpoint.
Control number of GPUs
GPU selection priority, from lowest to highest:
-
gpus_per_nodesystem property (scalar value) -
decode.gpu_idscommand argument in non-disaggregated mode whenCUDA_VISIBLE_DEVICESis not set -
CUDA_VISIBLE_DEVICESenvironment variable (comma-separated list of GPU IDs) -
gpu_idscommand argument for bothprefillanddecodeconfigurations in disaggregated mode
In disaggregated mode, define both prefill.gpu_ids and decode.gpu_ids, or omit both.
Control disaggregation
By default, SGLang will run without disaggregation as a single process. To enable disaggregation, one needs to set prefill configuration:
test.toml (disaggregated prefill/decode)
The config above will automatically split GPUs specified in CUDA_VISIBLE_DEVICES into two halves, first half will be used for prefill and second half will be used for decode.
For more control, one can specify the GPU IDs explicitly in prefill and decode configurations:
test.toml (disaggregated prefill/decode)
In this case CUDA_VISIBLE_DEVICES will be ignored and only the GPUs specified in gpu_ids will be used.
Multi-node serving
For non-disaggregated num_nodes`` ``>`` ``1, CloudAI starts one sglang.launch_server task per serving node with shared --dist-init-addr, --nnodes, and --node-rank`` ``"$SLURM_PROCID".
For disaggregated serving over more than two nodes, set explicit role sizes:
-
prefill.num_nodes`` ``+`` ``decode.num_nodesmust equal the testnum_nodes. -
CloudAI assigns contiguous node slices: prefill first, decode second.
-
tpis total per role, not per node. -
CUDA_VISIBLE_DEVICESandgpu_idsare local GPU IDs on each serving node.
Example: four prefill nodes and four decode nodes, each with four visible GPUs:
scenario.toml (multi-node disaggregated serving)
API Documentation
SGLang Serve Arguments
pydantic model cloudai.workloads.sglang.sglang.SglangArgs
Base command arguments for SGLang instances.
field disaggregation_transfer_backend: str | list[str] | None = ‘nixl’**
Transfer backend used in disaggregated mode. It is consumed by command generation and not emitted as a generic serve argument.
property serve_args_exclude: set[str]**
Fields consumed internally and excluded from generic serve args.
serialize_serve_arg(
key: str,
value: Any,
) → list[str]
Serialize a single serve argument to CLI tokens.
property serve_args: list[str]**
field gpu_ids: str | list[str] | None = None**
Comma-separated GPU IDs. If not set, all available GPUs will be used.
field num_nodes: int | list[int] | None = None**
Number of Slurm nodes assigned to this role in disaggregated serving mode.
Command Arguments
pydantic model cloudai.workloads.sglang.sglang.SglangCmdArgs
Bases: LLMServingCmdArgs[SglangArgs]
SGLang serve command arguments.
field model: str = ‘Qwen/Qwen3-8B’**
field serve_module: str = ‘sglang.launch_server’**
field router_module: str = ‘sglang_router.launch_router’**
field bench_module: str = ‘sglang.bench_serving’**
field healthcheck: str = ‘/v1/models’**
Health check router endpoint.
field prefill: SglangArgs | None = None**
Prefill instance arguments. If not set, a single instance without disaggregation is used.
field decode: SglangArgs [Optional]**
Decode instance arguments.
field docker_image_url: str [Required]**
field port: int = 8300**
Constraints:
-
ge = 1
-
le = 65535
field host: str = ‘0.0.0.0’**
Host/interface for serve or router processes to bind to.
field bench_host: str | None = None**
Hostname used by the benchmark client. Defaults to the allocated node hostname.
field serve_healthcheck: str | None = None**
Readiness endpoint for serve, prefill, and decode server processes. Defaults to healthcheck.
field serve_wait_seconds: int = 300**
Benchmark Command Arguments
pydantic model cloudai.workloads.sglang.sglang.SglangBenchCmdArgs
Bases: CmdArgs
SGLang bench_serving command arguments.
field backend: str = ‘sglang’**
field dataset_name: str = ‘random’**
field num_prompts: int = 30**
field max_concurrency: int = 16**
field random_input: int = 16**
field random_output: int = 128**
field warmup_requests: int = 2**
field random_range_ratio: float = 1.0**
field output_details: bool = True**
Semantic Eval Command Arguments
pydantic model cloudai.workloads.sglang.sglang.SglangSemanticEvalCmdArgs
Bases: CmdArgs
SGLang semantic validation command arguments.
field entrypoint: str = ‘python3 -m sglang.test.run_eval’**
field cli: str = ‘—host {host} —port {port} —eval-name gsm8k —num-examples 200 —num-threads 128 —model {model}‘**
Test Definition
pydantic model cloudai.workloads.sglang.sglang.SglangTestDefinition
Bases: LLMServingTestDefinition[SglangCmdArgs]
Test object for SGLang.
field bench_cmd_args: SglangBenchCmdArgs = SglangBenchCmdArgs(backend=‘sglang’, dataset_name=‘random’, num_prompts=30, max_concurrency=16, random_input=16, random_output=128, warmup_requests=2, random_range_ratio=1.0, output_details=True)**
field custom_bash: CustomBash | None = None**
field semantic_eval_cmd_args: SglangSemanticEvalCmdArgs | None = None**
was_run_successful(
tr: TestRun,
) → JobStatusResult
property cmd_args_dict: Dict[str, str | List[str]]**
constraint_check(
tr: TestRun,
system: System | None,
) → bool
property docker_image: DockerImage**
property extra_args_str: str**
property extra_installables: list[Installable]**
property hf_model: HFModel**
property installables: list[Installable]**
property is_domain_randomization_enabled: bool**
at least one env_params annotation.
Type:
Whether the config declares domain randomization
is_dse_excluded_arg(path: str) → bool
Return whether a dot-separated cmd_args path should be ignored by DSE.
property is_dse_job: bool**
is_env_sampled(cmd_args_path: str) → bool
Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).
field cmd_args: LLMServingCmdArgsT [Required]**
field name: str [Required]**
field description: str [Required]**
field test_template_name: str [Required]**
field dse_excluded_args: list[str] [Optional]**
field extra_env_vars: dict[str, str | List[str]] = {}**
field extra_cmd_args: dict[str, str] = {}**
field extra_container_mounts: list[str] = []**
field git_repos: list[GitRepo] = []**
field nsys: NsysConfiguration | None = None**
field predictor: PredictorConfig | None = None**
field training_report: TrainingReportConfig | None = None**
field agent: str = ‘grid_search’**
field agent_steps: int = 1**
field agent_metrics: list[str] = [‘default’]**
field agent_reward_function: str = ‘inverse’**
field agent_config: dict[str, Any] | None = None**
Agent configuration.
field env_params: dict[str, EnvParamSpec] [Optional]**
Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists to env.csv, and includes them in the trajectory cache key.