> This page is for version 1.7.1.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

This workload (`test_template_name` is `sglang`) allows users to execute SGLang benchmarks within the CloudAI framework.

SGLang is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.

## Usage Examples

### Test and Scenario Examples

test.toml (test definition)

```toml
name = "sglang_test"
description = "Example SGLang benchmark"
test_template_name = "sglang"

[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"

[bench_cmd_args]
random_input = 16
random_output = 128
max_concurrency = 16
num_prompts = 30

[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.run_eval"
cli = "--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}"
```

scenario.toml (scenario with one test)

```toml
name = "sglang-benchmark"

[[Tests]]
id = "sglang.1"
num_nodes = 1
time_limit = "00:10:00"
test_name = "sglang_test"
```

### Test-in-Scenario example

scenario.toml (separate test toml is not needed)

```toml
name = "sglang-benchmark"

[[Tests]]
id = "sglang.1"
num_nodes = 1
time_limit = "00:10:00"

name = "sglang_test"
description = "Example SGLang benchmark"
test_template_name = "sglang"

[Tests.cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"
```

Workload-specific test definition sections, such as `bench_cmd_args` and `semantic_eval_cmd_args`, are not supported under `[[Tests]]` in a test scenario. Define them in a test definition TOML and reference that test with `test_name` when custom benchmark or semantic-evaluation arguments are needed.

## Semantic Validation

To run GSM8K semantic validation after the serving benchmark, add `semantic_eval_cmd_args`. CloudAI reports `accuracy` from the eval output, but does not enforce an accuracy threshold.

test.toml (semantic validation)

```toml
[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.run_eval"
cli = "--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}"
```

For images that still use the legacy SGLang GSM8K runner, override the entrypoint and raw CLI:

```toml
[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.few_shot_gsm8k"
cli = "--host {host} --port {port} --num-questions 200"
```

The `cli` string supports `{model}`, `{host}`, `{port}`, `{url}`, `{output_path}`, and `{result_dir}` placeholders.

## Reporting

After a run completes, CloudAI parses `sglang-bench.jsonl` and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If `semantic_eval_cmd_args` is configured, CloudAI also reports semantic validation accuracy.

The reported metric (`default`) is throughput. Additional supported metrics are `throughput`, `tps-per-user`, `tps-per-gpu`, and `accuracy`.

CloudAI also provides the scenario-level `sglang_comparison` report. It compares SGLang test runs in the scenario and uses `bench_cmd_args` values as comparison labels.

## Readiness health checks

Healthcheck fields:

* `healthcheck`: aggregated server and disaggregated router endpoint, default `/v1/models`.

* `serve_healthcheck`: optional override for serve, prefill, and decode servers.

If `serve_healthcheck` is omitted, disaggregated prefill/decode servers keep the legacy `/health` endpoint.

## Control number of GPUs

GPU selection priority, from lowest to highest:

1. `gpus_per_node` system property (scalar value)

2. `decode.gpu_ids` command argument in non-disaggregated mode when `CUDA_VISIBLE_DEVICES` is not set

3. `CUDA_VISIBLE_DEVICES` environment variable (comma-separated list of GPU IDs)

4. `gpu_ids` command argument for both `prefill` and `decode` configurations in disaggregated mode

In disaggregated mode, define both `prefill.gpu_ids` and `decode.gpu_ids`, or omit both.

## Control disaggregation

By default, SGLang will run without disaggregation as a single process. To enable disaggregation, one needs to set `prefill` configuration:

test.toml (disaggregated prefill/decode)

```toml
[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"

[cmd_args.prefill]

[extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

The config above will automatically split GPUs specified in `CUDA_VISIBLE_DEVICES` into two halves, first half will be used for prefill and second half will be used for decode.

For more control, one can specify the GPU IDs explicitly in `prefill` and `decode` configurations:

test.toml (disaggregated prefill/decode)

```toml
[cmd_args.prefill]
gpu_ids = "0,1"

[cmd_args.decode]
gpu_ids = "2,3"
```

In this case `CUDA_VISIBLE_DEVICES` will be ignored and only the GPUs specified in `gpu_ids` will be used.

## Multi-node serving

For non-disaggregated `num_nodes`` ``>`` ``1`, CloudAI starts one `sglang.launch_server` task per serving node with shared `--dist-init-addr`, `--nnodes`, and `--node-rank`` ``"$SLURM_PROCID"`.

For disaggregated serving over more than two nodes, set explicit role sizes:

* `prefill.num_nodes`` ``+`` ``decode.num_nodes` must equal the test `num_nodes`.

* CloudAI assigns contiguous node slices: prefill first, decode second.

* `tp` is total per role, not per node.

* `CUDA_VISIBLE_DEVICES` and `gpu_ids` are local GPU IDs on each serving node.

Example: four prefill nodes and four decode nodes, each with four visible GPUs:

scenario.toml (multi-node disaggregated serving)

```toml
[[Tests]]
id = "sglang.pd_multi_node"
num_nodes = 8
test_template_name = "sglang"

[Tests.cmd_args.prefill]
num_nodes = 4
tp = 16

[Tests.cmd_args.decode]
num_nodes = 4
tp = 16

[Tests.extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

## API Documentation

### SGLang Serve Arguments

*pydantic model* cloudai.workloads.sglang.sglang.SglangArgs

Base command arguments for SGLang instances.

**&#x20;*field* disaggregation_transfer_backend*: str | list\[str] | None = 'nixl'*\*\*

Transfer backend used in disaggregated mode. It is consumed by command generation and not emitted as a generic serve argument.

**&#x20;*property* serve_args_exclude*: set\[str]*\*\*

Fields consumed internally and excluded from generic serve args.

serialize_serve_arg(

*key: str*,

*value: Any*,

) → list\[str]

Serialize a single serve argument to CLI tokens.

**&#x20;*property* serve_args*: list\[str]*\*\*

**&#x20;*field* gpu_ids*: str | list\[str] | None = None*\*\*

Comma-separated GPU IDs. If not set, all available GPUs will be used.

**&#x20;*field* num_nodes*: int | list\[int] | None = None*\*\*

Number of Slurm nodes assigned to this role in disaggregated serving mode.

### Command Arguments

**&#x20;*pydantic model* cloudai.workloads.sglang.sglang.SglangCmdArgs**

Bases: `LLMServingCmdArgs[SglangArgs]`

SGLang serve command arguments.

**&#x20;*field* model*: str = 'Qwen/Qwen3-8B'*\*\*

**&#x20;*field* serve_module*: str = 'sglang.launch_server'*\*\*

**&#x20;*field* router_module*: str = 'sglang_router.launch_router'*\*\*

**&#x20;*field* bench_module*: str = 'sglang.bench_serving'*\*\*

**&#x20;*field* healthcheck*: str = '/v1/models'*\*\*

Health check router endpoint.

**&#x20;*field* prefill*: [SglangArgs](/networking/cloudai/1.7.1/workloads/sglang#cloudai.workloads.sglang.sglang.SglangArgs "cloudai.workloads.sglang.sglang.SglangArgs") | None = None*\*\*

Prefill instance arguments. If not set, a single instance without disaggregation is used.

**&#x20;*field* decode*: [SglangArgs](/networking/cloudai/1.7.1/workloads/sglang#cloudai.workloads.sglang.sglang.SglangArgs "cloudai.workloads.sglang.sglang.SglangArgs")* *\[Optional]*\*\*

Decode instance arguments.

**&#x20;*field* docker_image_url*: str* *\[Required]*\*\*

**&#x20;*field* port*: int = 8300*\*\*

**Constraints:**

* **ge** = 1

* **le** = 65535

**&#x20;*field* host*: str = '0.0.0.0'*\*\*

Host/interface for serve or router processes to bind to.

**&#x20;*field* bench_host*: str | None = None*\*\*

Hostname used by the benchmark client. Defaults to the allocated node hostname.

**&#x20;*field* serve_healthcheck*: str | None = None*\*\*

Readiness endpoint for serve, prefill, and decode server processes. Defaults to healthcheck.

**&#x20;*field* serve_wait_seconds*: int = 300*\*\*

### Benchmark Command Arguments

**&#x20;*pydantic model* cloudai.workloads.sglang.sglang.SglangBenchCmdArgs**

Bases: `CmdArgs`

SGLang bench_serving command arguments.

**&#x20;*field* backend*: str = 'sglang'*\*\*

**&#x20;*field* dataset_name*: str = 'random'*\*\*

**&#x20;*field* num_prompts*: int = 30*\*\*

**&#x20;*field* max_concurrency*: int = 16*\*\*

**&#x20;*field* random_input*: int = 16*\*\*

**&#x20;*field* random_output*: int = 128*\*\*

**&#x20;*field* warmup_requests*: int = 2*\*\*

**&#x20;*field* random_range_ratio*: float = 1.0*\*\*

**&#x20;*field* output_details*: bool = True*\*\*

### Semantic Eval Command Arguments

**&#x20;*pydantic model* cloudai.workloads.sglang.sglang.SglangSemanticEvalCmdArgs**

Bases: `CmdArgs`

SGLang semantic validation command arguments.

**&#x20;*field* entrypoint*: str = 'python3 -m sglang.test.run_eval'*\*\*

**&#x20;*field* cli*: str = '--host \{host} --port \{port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model \{model}'*\*\*

### Test Definition

*pydantic model* cloudai.workloads.sglang.sglang.SglangTestDefinition

Bases: `LLMServingTestDefinition[SglangCmdArgs]`

Test object for SGLang.

**&#x20;*field* bench_cmd_args*: [SglangBenchCmdArgs](/networking/cloudai/1.7.1/workloads/sglang#cloudai.workloads.sglang.sglang.SglangBenchCmdArgs "cloudai.workloads.sglang.sglang.SglangBenchCmdArgs") = SglangBenchCmdArgs(backend='sglang', dataset_name='random', num_prompts=30, max_concurrency=16, random_input=16, random_output=128, warmup_requests=2, random_range_ratio=1.0, output_details=True)*\*\*

**&#x20;*field* custom_bash*: CustomBash | None = None*\*\*

**&#x20;*field* semantic_eval_cmd_args*: [SglangSemanticEvalCmdArgs](/networking/cloudai/1.7.1/workloads/sglang#cloudai.workloads.sglang.sglang.SglangSemanticEvalCmdArgs "cloudai.workloads.sglang.sglang.SglangSemanticEvalCmdArgs") | None = None*\*\*

was_run_successful(

*tr: TestRun*,

) → JobStatusResult

**&#x20;*property* cmd_args_dict*: Dict\[str, str | List\[str]]*\*\*

constraint_check(

*tr: TestRun*,

*system: System | None*,

) → bool

**&#x20;*property* docker_image*: DockerImage*\*\*

**&#x20;*property* extra_args_str*: str*\*\*

**&#x20;*property* extra_installables*: list\[Installable]*\*\*

**&#x20;*property* hf_model*: HFModel*\*\*

**&#x20;*property* installables*: list\[Installable]*\*\*

**&#x20;*property* is_domain_randomization_enabled*: bool*\*\*

at least one `env_params` annotation.

**Type:**

Whether the config declares domain randomization

**&#x20;is_dse_excluded_arg(*path: str*) → bool**

Return whether a dot-separated cmd_args path should be ignored by DSE.

**&#x20;*property* is_dse_job*: bool*\*\*

**&#x20;is_env_sampled(*cmd_args_path: str*) → bool**

Whether a cmd_args field is env-sampled (env draws it per trial, not the agent).

**&#x20;*field* cmd_args*: LLMServingCmdArgsT* *\[Required]*\*\*

**&#x20;*field* name*: str* *\[Required]*\*\*

**&#x20;*field* description*: str* *\[Required]*\*\*

**&#x20;*field* test_template_name*: str* *\[Required]*\*\*

**&#x20;*field* dse_excluded_args*: list\[str]* *\[Optional]*\*\*

**&#x20;*field* extra_env_vars*: dict\[str, str | List\[str]] = \{}*\*\*

**&#x20;*field* extra_cmd_args*: dict\[str, str] = \{}*\*\*

**&#x20;*field* extra_container_mounts*: list\[str] = \[]*\*\*

**&#x20;*field* git_repos*: list\[GitRepo] = \[]*\*\*

**&#x20;*field* nsys*: NsysConfiguration | None = None*\*\*

**&#x20;*field* predictor*: PredictorConfig | None = None*\*\*

**&#x20;*field* training_report*: TrainingReportConfig | None = None*\*\*

**&#x20;*field* agent*: str = 'grid_search'*\*\*

**&#x20;*field* agent_steps*: int = 1*\*\*

**&#x20;*field* agent_metrics*: list\[str] = \['default']*\*\*

**&#x20;*field* agent_reward_function*: str = 'inverse'*\*\*

**&#x20;*field* agent_config*: dict\[str, Any] | None = None*\*\*

Agent configuration.

**&#x20;*field* env_params*: dict\[str, EnvParamSpec]* *\[Optional]*\*\*

Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists to env.csv, and includes them in the trajectory cache key.