> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

This workload (`test_template_name` is `sglang`) allows users to execute SGLang benchmarks within the CloudAI framework.

SGLang is a high-throughput and memory-efficient inference engine for LLMs. This workload supports both aggregated and disaggregated prefill/decode modes.

## Usage Examples

### Test and Scenario Examples

**test.toml (test definition)**

```toml
name = "sglang_test"
description = "Example SGLang benchmark"
test_template_name = "sglang"

[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"

[bench_cmd_args]
random_input = 16
random_output = 128
max_concurrency = 16
num_prompts = 30

[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.run_eval"
cli = "--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}"
```

**scenario.toml (scenario with one test)**

```toml
name = "sglang-benchmark"

[[Tests]]
id = "sglang.1"
num_nodes = 1
time_limit = "00:10:00"
test_name = "sglang_test"
```

### Test-in-Scenario example

**scenario.toml (separate test toml is not needed)**

```toml
name = "sglang-benchmark"

[[Tests]]
id = "sglang.1"
num_nodes = 1
time_limit = "00:10:00"

name = "sglang_test"
description = "Example SGLang benchmark"
test_template_name = "sglang"

[Tests.cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"
```

## Local Models

Set `cmd_args.model` to an absolute, container-visible path to load an existing model directory instead of downloading a model from Hugging Face. Use `extra_container_mounts` to mount a host or shared-filesystem directory.

**test.toml (local model)**

```toml
name = "sglang_local_model"
description = "SGLang with a local model"
test_template_name = "sglang"
extra_container_mounts = [
  "/lustre/models/custom:/models/custom:ro",
]

[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "/models/custom"
```

The mounted directory must contain a model, configuration, and tokenizer supported by SGLang.

## Semantic Validation

To run GSM8K semantic validation after the serving benchmark, add `semantic_eval_cmd_args`. CloudAI reports `accuracy` from the eval output, but does not enforce an accuracy threshold.

**test.toml (semantic validation)**

```toml
[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.run_eval"
cli = "--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}"
```

For images that still use the legacy SGLang GSM8K runner, override the entrypoint and raw CLI:

```toml
[semantic_eval_cmd_args]
entrypoint = "python3 -m sglang.test.few_shot_gsm8k"
cli = "--host {host} --port {port} --num-questions 200"
```

The `cli` string supports `{model}`, `{host}`, `{port}`, `{url}`, `{output_path}`, and `{result_dir}` placeholders.

## Reporting

After a run completes, CloudAI parses `sglang-bench.jsonl` and prints serving latency, successful prompt count, completion rate, throughput, TPS per user, and TPS per GPU. If `semantic_eval_cmd_args` is configured, CloudAI also reports semantic validation accuracy.

The reported metric (`default`) is throughput. Additional supported metrics are `throughput`, `tps-per-user`, `tps-per-gpu`, and `accuracy`.

CloudAI also provides the scenario-level `sglang_comparison` report. It compares SGLang test runs in the scenario and uses `bench_cmd_args` values as comparison labels.

### Structured metrics

| Metric                         | Unit        | Preferred | Dimensions                     |
| ------------------------------ | ----------- | --------- | ------------------------------ |
| `request_throughput`           | requests/s  | Higher    | None                           |
| `output_token_throughput`      | tokens/s    | Higher    | None                           |
| `ttft` (time to first token)   | ms          | Lower     | `statistic`: mean, median, p99 |
| `tpot` (time per output token) | ms          | Lower     | `statistic`: mean, median, p99 |
| `accuracy`                     | ratio (0–1) | Higher    | None                           |

## Readiness health checks

Healthcheck fields:

* `healthcheck`: aggregated server and disaggregated router endpoint, default `/v1/models`.
* `serve_healthcheck`: optional override for serve, prefill, and decode servers.

If `serve_healthcheck` is omitted, disaggregated prefill/decode servers keep the legacy `/health` endpoint.

## Control number of GPUs

GPU selection priority, from lowest to highest:

1. `gpus_per_node` system property (scalar value)
2. `decode.gpu_ids` command argument in non-disaggregated mode when `CUDA_VISIBLE_DEVICES` is not set
3. `CUDA_VISIBLE_DEVICES` environment variable (comma-separated list of GPU IDs)
4. `gpu_ids` command argument for both `prefill` and `decode` configurations in disaggregated mode

In disaggregated mode, define both `prefill.gpu_ids` and `decode.gpu_ids`, or omit both.

## Control disaggregation

By default, SGLang will run without disaggregation as a single process. To enable disaggregation, one needs to set `prefill` configuration:

**test.toml (disaggregated prefill/decode)**

```toml
[cmd_args]
docker_image_url = "lmsysorg/sglang:dev-cu13"
model = "Qwen/Qwen3-8B"

[cmd_args.prefill]

[extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

The config above will automatically split GPUs specified in `CUDA_VISIBLE_DEVICES` into two halves, first half will be used for prefill and second half will be used for decode.

For more control, one can specify the GPU IDs explicitly in `prefill` and `decode` configurations:

**test.toml (disaggregated prefill/decode)**

```toml
[cmd_args.prefill]
gpu_ids = "0,1"

[cmd_args.decode]
gpu_ids = "2,3"
```

In this case `CUDA_VISIBLE_DEVICES` will be ignored and only the GPUs specified in `gpu_ids` will be used.

## Multi-node serving

For non-disaggregated `num_nodes > 1`, CloudAI starts one `sglang.launch_server` task per serving node with shared `--dist-init-addr`, `--nnodes`, and `--node-rank "$SLURM_PROCID"`.

For disaggregated serving over more than two nodes, set explicit role sizes:

* `prefill.num_nodes + decode.num_nodes` must equal the test `num_nodes`.
* CloudAI assigns contiguous node slices: prefill first, decode second.
* `tp` is total per role, not per node.
* `CUDA_VISIBLE_DEVICES` and `gpu_ids` are local GPU IDs on each serving node.

Example: four prefill nodes and four decode nodes, each with four visible GPUs:

**scenario.toml (multi-node disaggregated serving)**

```toml
[[Tests]]
id = "sglang.pd_multi_node"
num_nodes = 8
test_template_name = "sglang"

[Tests.cmd_args.prefill]
num_nodes = 4
tp = 16

[Tests.cmd_args.decode]
num_nodes = 4
tp = 16

[Tests.extra_env_vars]
CUDA_VISIBLE_DEVICES = "0,1,2,3"
```

## API Documentation

### SGLang Serve Arguments

### `SglangArgs`

Base command arguments for SGLang instances.

| Field                             | Type                       | Default  | Description                                                                                                                    |
| --------------------------------- | -------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `gpu_ids`                         | `str \| list[str] \| None` | `None`   | Comma-separated GPU IDs. If not set, all available GPUs will be used.                                                          |
| `num_nodes`                       | `int \| list[int] \| None` | `None`   | Number of Slurm nodes assigned to this role in disaggregated serving mode.                                                     |
| `disaggregation_transfer_backend` | `str \| list[str] \| None` | `'nixl'` | Transfer backend used in disaggregated mode. It is consumed by command generation and not emitted as a generic serve argument. |

### Command Arguments

### `SglangCmdArgs`

SGLang serve command arguments.

| Field                | Type                                                 | Default                         | Description                                                                                  |
| -------------------- | ---------------------------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------- |
| `docker_image_url`   | `str`                                                | Required                        |                                                                                              |
| `model`              | `str`                                                | `'Qwen/Qwen3-8B'`               |                                                                                              |
| `port`               | `int`                                                | `8300`                          | Constraints: `Ge(ge=1)`, `Le(le=65535)`.                                                     |
| `host`               | `str`                                                | `'0.0.0.0'`                     | Host/interface for serve or router processes to bind to.                                     |
| `bench_host`         | `str \| None`                                        | `None`                          | Hostname used by the benchmark client. Defaults to the allocated node hostname.              |
| `healthcheck`        | `str`                                                | `'/v1/models'`                  | Health check router endpoint.                                                                |
| `serve_healthcheck`  | `str \| None`                                        | `None`                          | Readiness endpoint for serve, prefill, and decode server processes. Defaults to healthcheck. |
| `serve_wait_seconds` | `int`                                                | `300`                           |                                                                                              |
| `prefill`            | `cloudai.workloads.sglang.sglang.SglangArgs \| None` | `None`                          | Prefill instance arguments. If not set, a single instance without disaggregation is used.    |
| `decode`             | `cloudai.workloads.sglang.sglang.SglangArgs`         | Factory: `SglangArgs`           | Decode instance arguments.                                                                   |
| `serve_module`       | `str`                                                | `'sglang.launch_server'`        |                                                                                              |
| `router_module`      | `str`                                                | `'sglang_router.launch_router'` |                                                                                              |
| `bench_module`       | `str`                                                | `'sglang.bench_serving'`        |                                                                                              |

### Benchmark Command Arguments

### `SglangBenchCmdArgs`

SGLang bench_serving command arguments.

| Field                | Type    | Default    | Description |
| -------------------- | ------- | ---------- | ----------- |
| `backend`            | `str`   | `'sglang'` |             |
| `dataset_name`       | `str`   | `'random'` |             |
| `num_prompts`        | `int`   | `30`       |             |
| `max_concurrency`    | `int`   | `16`       |             |
| `random_input`       | `int`   | `16`       |             |
| `random_output`      | `int`   | `128`      |             |
| `warmup_requests`    | `int`   | `2`        |             |
| `random_range_ratio` | `float` | `1.0`      |             |
| `output_details`     | `bool`  | `True`     |             |

### Semantic Eval Command Arguments

### `SglangSemanticEvalCmdArgs`

SGLang semantic validation command arguments.

| Field        | Type  | Default                                                                                                | Description |
| ------------ | ----- | ------------------------------------------------------------------------------------------------------ | ----------- |
| `entrypoint` | `str` | `'python3 -m sglang.test.run_eval'`                                                                    |             |
| `cli`        | `str` | `'--host {host} --port {port} --eval-name gsm8k --num-examples 200 --num-threads 128 --model {model}'` |             |

### Test Definition

### `SglangTestDefinition`

Test object for SGLang.

| Field                    | Type                                                                | Default                                                                                                                                                                                               | Description                                                                                                                                                                                                                   |
| ------------------------ | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                               | Required                                                                                                                                                                                              |                                                                                                                                                                                                                               |
| `description`            | `str`                                                               | Required                                                                                                                                                                                              |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                               | Required                                                                                                                                                                                              |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.sglang.sglang.SglangCmdArgs`                     | Required                                                                                                                                                                                              |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                                         | Factory: `list`                                                                                                                                                                                       |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                                | `{}`                                                                                                                                                                                                  |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                                    | `{}`                                                                                                                                                                                                  |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                                         | `[]`                                                                                                                                                                                                  |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`                 | `[]`                                                                                                                                                                                                  |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`                 | `None`                                                                                                                                                                                                |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`                   | `None`                                                                                                                                                                                                |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`              | `None`                                                                                                                                                                                                |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                               | `'grid_search'`                                                                                                                                                                                       |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                               | `1`                                                                                                                                                                                                   |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                                         | `['default']`                                                                                                                                                                                         |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                               | `'inverse'`                                                                                                                                                                                           |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                            | `None`                                                                                                                                                                                                | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]`           | Factory: `dict`                                                                                                                                                                                       | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |
| `bench_cmd_args`         | `cloudai.workloads.sglang.sglang.SglangBenchCmdArgs`                | `SglangBenchCmdArgs(backend='sglang', dataset_name='random', num_prompts=30, max_concurrency=16, random_input=16, random_output=128, warmup_requests=2, random_range_ratio=1.0, output_details=True)` |                                                                                                                                                                                                                               |
| `custom_bash`            | `str \| dict[str, str] \| None`                                     | `None`                                                                                                                                                                                                |                                                                                                                                                                                                                               |
| `semantic_eval_cmd_args` | `cloudai.workloads.sglang.sglang.SglangSemanticEvalCmdArgs \| None` | `None`                                                                                                                                                                                                |                                                                                                                                                                                                                               |