Skip to navigation

SGLang

View as Markdown

This workload (test_template_name is sglang) allows users to execute SGLang benchmarks within the CloudAI framework.

SGLang is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.

Usage Examples

Test and Scenario Examples

test.toml (test definition)

name = "sglang_test"
description = "Example SGLang benchmark"
test_template_name = "sglang"
[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"
[bench_cmd_args]
random_input = 16
random_output = 128
max_concurrency = 16
num_prompts = 30
[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.run_eval"
cli = "--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}"

scenario.toml (scenario with one test)

name = "sglang-benchmark"
[[Tests]]
id = "sglang.1"
num_nodes = 1
time_limit = "00:10:00"
test_name = "sglang_test"

Test-in-Scenario example

scenario.toml (separate test toml is not needed)

name = "sglang-benchmark"
[[Tests]]
id = "sglang.1"
num_nodes = 1
time_limit = "00:10:00"
name = "sglang_test"
description = "Example SGLang benchmark"
test_template_name = "sglang"
[Tests.cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"

Local Models

Set cmd_args.model to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use extra_container_mounts to mount a host or shared-filesystem directory.

test.toml (local model)

name = "sglang_local_model"
description = "SGLang with a local model"
test_template_name = "sglang"
extra_container_mounts = [
"/lustre/models/custom:/models/custom:ro",
]
[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "/models/custom"

The mounted directory must contain a model, configuration, and tokenizer supported by SGLang.

Semantic Validation

To run GSM8K semantic validation after the serving benchmark, add semantic_eval_cmd_args. CloudAI reports accuracy from the eval output, but does not enforce an accuracy threshold.

test.toml (semantic validation)

[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.run_eval"
cli = "--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}"

For images that still use the legacy SGLang GSM8K runner, override the entrypoint and raw CLI:

[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.few_shot_gsm8k"
cli = "--host {host} --port {port} --num-questions 200"

The cli string supports {model}, {host}, {port}, {url}, {output_path}, and {result_dir} placeholders.

Reporting

After a run completes, CloudAI parses sglang-bench.jsonl and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If semantic_eval_cmd_args is configured, CloudAI also reports semantic validation accuracy.

The reported metric (default) is throughput. Additional supported metrics are throughput, tps-per-user, tps-per-gpu, and accuracy.

CloudAI also provides the scenario-level sglang_comparison report. It compares SGLang test runs in the scenario and uses bench_cmd_args values as comparison labels.

Structured metrics

MetricUnitPreferredDimensions
request_throughputrequests/sHigherNone
output_token_throughputtokens/sHigherNone
ttft (time to first token)msLowerstatistic: mean, median, p99
tpot (time per output token)msLowerstatistic: mean, median, p99
accuracyratio (0–1)HigherNone

Readiness health checks

Healthcheck fields:

  • healthcheck: aggregated server and disaggregated router endpoint, default /v1/models.
  • serve_healthcheck: optional override for serve, prefill, and decode servers.

If serve_healthcheck is omitted, disaggregated prefill/decode servers keep the legacy /health endpoint.

Control number of GPUs

GPU selection priority, from lowest to highest:

  1. gpus_per_node system property (scalar value)
  2. decode.gpu_ids command argument in non-disaggregated mode when CUDA_VISIBLE_DEVICES is not set
  3. CUDA_VISIBLE_DEVICES environment variable (comma-separated list of GPU IDs)
  4. gpu_ids command argument for both prefill and decode configurations in disaggregated mode

In disaggregated mode, define both prefill.gpu_ids and decode.gpu_ids, or omit both.

Control disaggregation

By default, SGLang will run without disaggregation as a single process. To enable disaggregation, one needs to set prefill configuration:

test.toml (disaggregated prefill/decode)

[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"
[cmd_args.prefill]
[extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"

The config above will automatically split GPUs specified in CUDA_VISIBLE_DEVICES into two halves, first half will be used for prefill and second half will be used for decode.

For more control, one can specify the GPU IDs explicitly in prefill and decode configurations:

test.toml (disaggregated prefill/decode)

[cmd_args.prefill]
gpu_ids = "0,1"
[cmd_args.decode]
gpu_ids = "2,3"

In this case CUDA_VISIBLE_DEVICES will be ignored and only the GPUs specified in gpu_ids will be used.

Multi-node serving

For non-disaggregated num_nodes > 1, CloudAI starts one sglang.launch_server task per serving node with shared --dist-init-addr, --nnodes, and --node-rank "$SLURM_PROCID".

For disaggregated serving over more than two nodes, set explicit role sizes:

  • prefill.num_nodes + decode.num_nodes must equal the test num_nodes.
  • CloudAI assigns contiguous node slices: prefill first, decode second.
  • tp is total per role, not per node.
  • CUDA_VISIBLE_DEVICES and gpu_ids are local GPU IDs on each serving node.

Example: four prefill nodes and four decode nodes, each with four visible GPUs:

scenario.toml (multi-node disaggregated serving)

[[Tests]]
id = "sglang.pd_multi_node"
num_nodes = 8
test_template_name = "sglang"
[Tests.cmd_args.prefill]
num_nodes = 4
tp = 16
[Tests.cmd_args.decode]
num_nodes = 4
tp = 16
[Tests.extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"

API Documentation

SGLang Serve Arguments

SglangArgs

Base command arguments for SGLang instances.

FieldTypeDefaultDescription
gpu_idsstr | list[str] | NoneNoneComma-separated GPU IDs. If not set, all available GPUs will be used.
num_nodesint | list[int] | NoneNoneNumber of Slurm nodes assigned to this role in disaggregated serving mode.
disaggregation_transfer_backendstr | list[str] | None'nixl'Transfer backend used in disaggregated mode. It is consumed by command generation and not emitted as a generic serve argument.

Command Arguments

SglangCmdArgs

SGLang serve command arguments.

FieldTypeDefaultDescription
docker_image_urlstrRequired
modelstr'Qwen/Qwen3-8B'
portint8300Constraints: Ge(ge=1), Le(le=65535).
hoststr'0.0.0.0'Host/interface for serve or router processes to bind to.
bench_hoststr | NoneNoneHostname used by the benchmark client. Defaults to the allocated node hostname.
healthcheckstr'/v1/models'Health check router endpoint.
serve_healthcheckstr | NoneNoneReadiness endpoint for serve, prefill, and decode server processes. Defaults to healthcheck.
serve_wait_secondsint300
prefillcloudai.workloads.sglang.sglang.SglangArgs | NoneNonePrefill instance arguments. If not set, a single instance without disaggregation is used.
decodecloudai.workloads.sglang.sglang.SglangArgsFactory: SglangArgsDecode instance arguments.
serve_modulestr'sglang.launch_server'
router_modulestr'sglang_router.launch_router'
bench_modulestr'sglang.bench_serving'

Benchmark Command Arguments

SglangBenchCmdArgs

SGLang bench_serving command arguments.

FieldTypeDefaultDescription
backendstr'sglang'
dataset_namestr'random'
num_promptsint30
max_concurrencyint16
random_inputint16
random_outputint128
warmup_requestsint2
random_range_ratiofloat1.0
output_detailsboolTrue

Semantic Eval Command Arguments

SglangSemanticEvalCmdArgs

SGLang semantic validation command arguments.

FieldTypeDefaultDescription
entrypointstr'python3 -m sglang.test.run_eval'
clistr'--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}'

Test Definition

SglangTestDefinition

Test object for SGLang.

FieldTypeDefaultDescription
namestrRequired
descriptionstrRequired
test_template_namestrRequired
cmd_argscloudai.workloads.sglang.sglang.SglangCmdArgsRequired
dse_excluded_argslist[str]Factory: list
extra_env_varsdict[str, str | typing.List[str]]{}
extra_cmd_argsdict[str, str]{}
extra_container_mountslist[str][]
git_reposlist[cloudai._core.installables.git_repo.GitRepo][]
nsyscloudai.models.workload.NsysConfiguration | NoneNone
predictorcloudai.models.workload.PredictorConfig | NoneNone
training_reportcloudai.models.workload.TrainingReportConfig | NoneNone
agentstr'grid_search'
agent_stepsint1
agent_metricslist[str]['default']
agent_reward_functionstr'inverse'
agent_configdict[str, Any] | NoneNoneAgent configuration.
env_paramsdict[str, cloudai.configurator.env_params.EnvParamSpec]Factory: dictEnvironment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key.
bench_cmd_argscloudai.workloads.sglang.sglang.SglangBenchCmdArgsSglangBenchCmdArgs(backend='sglang', dataset_name='random', num_prompts=30, max_concurrency=16, random_input=16, random_output=128, warmup_requests=2, random_range_ratio=1.0, output_details=True)
custom_bashstr | dict[str, str] | NoneNone
semantic_eval_cmd_argscloudai.workloads.sglang.sglang.SglangSemanticEvalCmdArgs | NoneNone