AI Dynamo
AI Dynamo workload (test_template_name is AIDynamo) runs AI inference benchmarks using the Dynamo framework with distributed prefill and decode workers.
Run Using Kubernetes
Prepare Cluster
Before running the AI Dynamo workload on a Kubernetes cluster, ensure that the cluster is set up according to the instructions in the official documentation. Below is a short summary of the required steps:
Launch and Monitor the Job
Both CloudAI and Dynamo will try to access HuggingFace Hub. To avoid 429 Too Many Requests errors and access models under auth, it is recommended to define HF_TOKEN environment variable before invoking CloudAI. Once set, run uv run hf auth login to authenticate.
Run Using Slurm
Node Configuration for AI Dynamo
AI Dynamo jobs use three distinct types of nodes:
- Frontend node: Hosts the coordination services (
etcd,nats), the frontend server, the request generator (aiperfby default, configurable viaworkloadsin the test TOML), and the first decode worker - Prefill node(s): Handle the prefill stage of inference
- Decode node(s): Handle the decode stage of inference (optional, depending on model and setup)
By default, when num_nodes is omitted, CloudAI allocates separate nodes for prefill and decode workers:
num_prefill_nodes + num_decode_nodes
Set top-level num_nodes explicitly to control the Slurm allocation. A value lower than num_prefill_nodes + num_decode_nodes enables shared-node disaggregated inference, where prefill and decode roles run on the same allocated node(s) with separate GPU slices.
All node role assignments and orchestration are automatically managed by CloudAI.
Multinode Backend Workers on Slurm
prefill_worker.num-nodes and decode_worker.num-nodes specify the total nodes assigned to each role. Set nodes-per-worker to group those nodes into logical backend workers. For example, the following configuration creates one prefill worker spanning two nodes and one decode worker spanning two other nodes:
When nodes-per-worker is omitted, it defaults to 1 and preserves the previous one-worker-per-node behavior. For vLLM TP/PP groups, CloudAI launches rank 0 as the Dynamo-facing process and the remaining ranks with --headless using Dynamo’s multiprocessing executor. When data-parallel-size is greater than one, CloudAI instead launches a full Dynamo vLLM process on every node and derives data-parallel-size-local, data-parallel-start-rank, the coordinator address, and the RPC port from the worker topology. SGLang receives --dist-init-addr, --nnodes, and --node-rank on every participating node; its configured data-parallel-size is forwarded as dp-size. The launcher derives prefill and decode disaggregation modes from the configured roles, so worker commands do not need role flags.
For TP/PP-spanning workers, tensor-parallel-size * pipeline-parallel-size must be divisible by nodes-per-worker. For vLLM multinode data parallelism, data-parallel-size must instead be divisible by nodes-per-worker, and each DP rank’s TP/PP group must fit within one node. SGLang multinode data parallelism requires --enable-dp-attention. The resulting per-node GPU footprint must fit on each node. Ray-backed Dynamo vLLM groups are not currently supported. The Slurm launcher explicitly selects etcd discovery and the NATS event plane.
No separate WideEP mode is required. Multinode data-parallel orchestration is inferred from nodes-per-worker > 1 and data-parallel-size > 1. Backend expert-parallel parameters such as vLLM’s --enable-expert-parallel or SGLang’s ep-size and --enable-dp-attention remain explicit backend configuration. CloudAI uses the allocated Slurm node names for multinode rendezvous. The cluster must make those names consistently resolvable and mutually reachable from every allocated node and workload container. Multi-homed systems should also configure backend network selection, such as GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME, at the system level. Explicit backend rendezvous addresses take precedence over addresses derived from the Slurm node names. Clusters that cannot provide this hostname contract must supply backend-native per-node address overrides, such as SGLANG_HOST_IP or VLLM_NIXL_SIDE_CHANNEL_HOST, together with explicit worker rendezvous arguments.
Dedicated examples are available in test_scenario/vllm_multinode_worker_slurm.toml and test_scenario/sglang_multinode_worker_slurm.toml. The existing vllm_slurm.toml and sglang_slurm.toml scenarios retain their previous configurations. The vLLM example inherits tensor-parallel-size = 8 from test/vllm.toml; the SGLang example sets it explicitly because test/sglang.toml defaults to 1. WideEP examples are provided in test_scenario/vllm_wideep_slurm.toml and test_scenario/sglang_wideep_slurm.toml.
Launch and Monitor the Job
To run the job:
The job progress monitoring can be done using either of the following options:
The frontend node will initially wait to allow weight loading on all nodes. Once ready, it will launch the configured benchmark tool (aiperf by default), which begins generating requests to the frontend server. All servers cooperate to complete inference, and the output will appear in stdout.txt.
Choosing a Benchmark Tool
The benchmark tool is controlled by the workloads field in the test TOML. Set aiperf.sh to use AIPerf:
To use genai-perf, set:
AIPerf Multi-Phase Runs
cmd_args.aiperf is the base AIPerf config. cmd_args.aiperf_phases can run several AIPerf rounds against the same live Dynamo stack. By default, CloudAI does not restart prefill, decode, or router processes between phases:
Single-phase runs keep the old artifact layout: aiperf_artifacts/, aiperf.log, and aiperf_report.csv. Multi-phase runs write per-phase artifacts/logs/reports and copy the last phase report to aiperf_report.csv for existing report generation.
between-phase-cmd is a bash command run after each non-final phase. The default is a no-op. Set it explicitly for backend-specific cache cleanup, for example /cloudai_run_results/routerctl.sh restart if a test needs to restart the Dynamo router between phases. health-check-between-phases probes the frontend after the command.
Comparison Reports
CloudAI provides the scenario-level ai_dynamo_comparison report. It compares AI Dynamo runs using the standard LLM serving tables and charts for TTFT, TPOT, successful prompts, output-token throughput, TPS per user, and TPS per GPU. When aiperf_accuracy is configured and its result is available, the report also compares model accuracy. Both AIPerf and GenAI-Perf CSV output are supported.
The scenario-level Pareto report uses each configured AIPerf phase as a separate point, with that phase’s concurrency and benchmark settings. It also supports GenAI-Perf results.
AIPerf args are rendered as normal CLI flags. Multi-value AIPerf options should be passed with AIPerf CLI syntax, such as server-metrics-formats = "csv,json,jsonl" or gpu-telemetry = "node1:9401,node2:9401". server-metrics = "auto" expands to the frontend metrics endpoint, Dynamo worker metrics endpoints, and any CloudAI-started DCGM exporters.
Propagating LMCache Configuration
AIDynamo can pass an LMCache YAML config to the worker processes by setting LMCACHE_CONFIG_FILE inside the container. This only propagates the LMCache configuration; the vLLM/SGLang runtime still needs to be launched with the appropriate LMCache or KV-transfer connector for that image/version.
The preferred form is structured TOML under [cmd_args.lmcache]. CloudAI converts that object to YAML in the run output directory, mounts that directory as /cloudai_run_results, and exports the generated file path as LMCACHE_CONFIG_FILE:
For an example that uses test-in-scenario mode, see conf/experimental/ai_dynamo/test_scenario/vllm_lmcache.toml. Because the test is fully defined inside the scenario, --tests-dir is not required when running that example:
The example sets dse_excluded_args = ["cmd_args.lmcache.lmcache_worker_ports"] because lmcache_worker_ports is a list-valued LMCache setting, not a DSE sweep dimension. Other list-valued LMCache fields can still be swept unless their cmd_args. path is also excluded.
Alternatively, mount your own LMCache YAML file with extra_container_mounts and set LMCACHE_CONFIG_FILE through extra_env_vars:
For multi-node LMCache storage tests, any path referenced by the LMCache YAML, such as nixl_path for POSIX-backed storage, must be visible and writable from every node that is expected to share cached data. A node-local path such as /tmp is suitable only for single-node smoke tests or configuration propagation checks.
LMCache YAML values can use runtime placeholders. CloudAI renders them inside the Slurm job before launching workers: {frontend_node}, {frontend_ip}, {results_dir}, and {storage_cache_dir}. Unknown placeholders fail the run before worker processes start.
Propagating HiCache Configuration
Define a HiCache storage-backend configuration under [cmd_args.hicache]. CloudAI serializes the object as hicache-config.toml in the run output directory, exposes its container path through HICACHE_CONFIG_FILE, and passes the resolved path to both workers as --hicache-storage-backend-extra-config:
The remaining HiCache engine options stay in the worker args sections so they can be configured or swept through DSE normally.
The runtime replacement mechanism below supports per-node HiCache values. For example, change the POSIX setting above to use_uring = "${use_uring}" and have the startup manifest replace ${use_uring} with false or true. The worker receives the resulting per-node TOML path automatically.
For a self-contained single-node smoke test using NIXL’s POSIX plugin, run the following on a node with at least two GPUs (one each for the prefill and decode workers):
Per-node Startup Commands and Runtime Replacements
On Slurm, startup_cmd runs once per allocated node before the workload. It runs on the host by default. Set startup_cmd_docker_image to run it in a container using the workload’s mounts:
CloudAI exports CLOUDAI_RUNTIME_FILE to the command. It points to runtime/<node>.json in the test output directory, or /cloudai_run_results/runtime/<node>.json inside a startup container. Write replacements there:
Each top-level key names an environment variable containing a config path. AIDynamo creates a per-node copy, applies literal replacements, then updates the variable to point to that copy.
For lmcache-config.yaml on node-01, the result is lmcache-config.node-01.yaml. The shared source remains unchanged. LMCache placeholders are rendered afterward in the per-node copy. Missing manifest means no replacements.
Startup runs once per test case in normal and --single-sbatch modes.
If the selected LMCache mode needs a controller, CloudAI can start one on the frontend node:
This only launches the process. For disaggregated or multi-node runs, the LMCache YAML still needs controller addresses that resolve to the frontend node from every worker. With the default controller monitor ports, use controller_pull_url = "{frontend_node}:8300" and controller_reply_url = "{frontend_node}:8400". The lmcache_worker_ports list must match the number of worker ranks.
Semantic Degradation With AIPerf Accuracy
AIDynamo uses AIPerf accuracy mode as its semantic degradation signal. Enable it with [cmd_args.aiperf_accuracy]. This runs after the configured performance workload, so it can be used with either aiperf.sh or genai_perf.sh:
When cmd_args.aiperf_accuracy is configured, CloudAI expects AIPerf to produce accuracy_results.csv and exposes the accuracy metric from its OVERALL row. The metric is reported as a 0.0-1.0 fraction. Keep synthetic prompt and token-length flags out of this mode; the benchmark dataset should come from AIPerf’s accuracy benchmark.
The entrypoint and cli fields form the accuracy command. CloudAI expands {model}, {url}, {endpoint}, {result_dir}, and {artifact_dir} in cli before launching it. The setup-cmd field is optional. It is useful for Dynamo images that include an older system aiperf build without the accuracy benchmark plugins. The example upgrades the image-level aiperf before launching aiperf profile. MMLU is loaded from lighteval/mmlu, so either allow Hugging Face dataset access or pre-cache that dataset before running with HF_HUB_OFFLINE/HF_DATASETS_OFFLINE enabled. For Qwen3 models, the example disables thinking mode so short MMLU answers can be parsed as choices.
Custom Accuracy Scripts
cmd_args.aiperf_accuracy can also launch a custom mounted script instead of AIPerf. Mount the script or its parent directory with extra_container_mounts and set entrypoint to the in-container command:
CloudAI expands placeholders in cli and runs entrypoint with that CLI string. The custom command must write accuracy_results.csv inside {artifact_dir} with an OVERALL row. CloudAI copies that file to the run output directory and exposes the same accuracy metric as AIPerf accuracy mode.
Review Benchmark Results
After job completion, CloudAI places output logs and result files in the designated results directory. The result file name depends on the configured workloads field:
aiperf.sh→aiperf_report.csvgenai_perf.sh→genai_perf_report.csvcmd_args.aiperf_accuracy→accuracy_results.csv
If AIPerf accuracy mode is enabled, CloudAI copies aiperf_accuracy_artifacts/accuracy_results.csv to accuracy_results.csv in the run output directory and marks the run failed if that file is not produced.
Navigate to ./results/<scenario>/<test-id>/0/ and open the CSV to examine performance metrics.
Shared-Node Disaggregated Runs
For Slurm, set top-level num_nodes lower than the sum of prefill_worker.num-nodes and decode_worker.num-nodes to run both roles on the same allocated node(s). For example, num_nodes = 1 with prefill_worker.num-nodes = 1 and decode_worker.num-nodes = 1 runs one prefill worker and one decode worker on the same node. CloudAI assigns decode GPUs first and prefill GPUs after that based on each role’s per-node GPU footprint, including local data-parallel ranks where applicable. The combined role GPU count must fit on one node.
Example aiperf_report.csv:
Metric,avg,min,max,p25,p50,p75,p99,std Inter Token Latency (ms),2.81,2.66,2.88,2.79,2.83,2.84,2.87,0.04 Time to First Token (ms),49.87,17.15,99.91,49.35,49.87,50.52,92.31,9.20 Time to Second Token (ms),0.50,0.03,4.05,0.03,0.04,0.04,3.47,1.08 Request Latency (ms),1652.30,1203.61,6433.87,1453.19,1462.99,1466.72,6431.16,976.18 Output Sequence Length (tokens),498.06,410.00,501.00,500.00,500.00,500.00,501.00,12.62 Input Sequence Length (tokens),300.00,300.00,300.00,300.00,300.00,300.00,300.00,0.00
Metric,Value Output Token Throughput (tokens/sec),598.78 Total Token Throughput (tokens/sec),962.32 Request Throughput (requests/sec),1.20 Request Count,50.00
Supported Backends
The following backends are available via the conf/experimental/ai_dynamo/test/ directory:
- vLLM (
vllm.toml) — use withtest_scenario/vllm_slurm.toml - vLLM multinode worker — use
test_scenario/vllm_multinode_worker_slurm.toml - vLLM WideEP — use
test_scenario/vllm_wideep_slurm.toml - vLLM with LMCache config propagation — use self-contained scenario
test_scenario/vllm_lmcache.toml - sglang (
sglang.toml) — use withtest_scenario/sglang_slurm.toml - SGLang multinode worker — use
test_scenario/sglang_multinode_worker_slurm.toml - SGLang WideEP — use
test_scenario/sglang_wideep_slurm.toml
Both backends use aiperf as the default benchmark tool and support disaggregated prefill/decode.
API Documentation
Command Arguments
AIDynamoCmdArgs
Arguments for AI Dynamo.
Test Definition
AIDynamoTestDefinition
Test definition for AI Dynamo.