> This page is for version v1.5.1.
> For other versions, use one of these documentation indexes:
> - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt
> - dev: https://docs.nvidia.com/dynamo/dev/llms.txt
> - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt
> - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt
> - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt
> - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt
> - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt
> - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt
> - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt
> - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt
> - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt
> - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt
> - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt
> - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt
> - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt
> - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt
> - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt
> - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt
> - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt
> - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt
> - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Benchmark a Local Deployment with AIPerf

AIPerf sends load to an OpenAI-compatible endpoint and measures latency and throughput. Use this
tutorial after starting a local Dynamo frontend and at least one model worker.

AIPerf measures the live deployment. It does not predict a configuration like AIConfigurator or run
a GPU-free simulation like Mocker and DynoSim.

> **Using an AI coding agent?**
>
> This repository ships agent skills for performance work: clone the repo, point Claude Code,
> Codex, or Cursor at it, and describe your deployment and goal in plain language. The skills
> activate automatically and guide the agent through deploying, benchmarking with AIPerf, and
> iterating on configuration with evidence at every step. See
> [Agent Skills](/dynamo/agent-skills/overview) for the inventory and example prompts.

## Prerequisites

Start a local deployment by following [Model Deployment](/dynamo/cli/model-deployment/introduction). Confirm that the frontend is
available at `http://localhost:8000`:

```bash
curl --fail http://localhost:8000/health
```

Install AIPerf on the machine that will generate load:

```bash
pip install aiperf
```

#### Run a baseline benchmark

Set the model name to the value served by the local worker:

```bash
aiperf profile \
  --model Qwen/Qwen3-32B-FP8 \
  --url http://localhost:8000 \
  --endpoint-type chat \
  --streaming \
  --concurrency 10 \
  --request-count 100 \
  --synthetic-input-tokens-mean 2000 \
  --output-tokens-mean 256
```

AIPerf writes artifacts to `artifacts/` and prints a metrics summary. Record TTFT, Inter-Token
Latency (ITL), end-to-end latency, and output throughput.

#### Compare local configurations

Change one deployment setting at a time, restart the affected workers, and rerun the same AIPerf
command. Common comparisons include:

* one worker versus multiple replicas
* aggregated versus disaggregated serving
* KV-aware routing versus round-robin routing
* different TP or PP sizes recommended by AIConfigurator
* backend or engine-argument changes

Keep the request shape, concurrency, and request count fixed so that the results remain comparable.
Then sweep concurrency to find the saturation point of the selected configuration.

## Next steps

* Use [Sizing with AIConfigurator](/dynamo/cli/disaggregated-serving/sizing-with-ai-configurator) to select additional configurations.
* Use the full [Dynamo Benchmarking guide](/dynamo/recipes/benchmarks/benchmarking) for concurrency sweeps,
  arrival patterns, trace replay, visualization, and GPU telemetry.