Profile OpenAI-Compatible Text APIs Using AIPerf

This guide covers profiling OpenAI-compatible Chat Completions and Completions endpoints with vLLM and AIPerf.

Start a vLLM server

Pull and start a vLLM server using Docker:

$ docker pull vllm/vllm-openai:latest
$ docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
>   --model Qwen/Qwen3-0.6B \
>   --reasoning-parser qwen3

Verify the server is ready:

$ timeout 900 bash -c 'while [ "$(curl -s -o /dev/null -w "%{http_code}" localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"Qwen/Qwen3-0.6B\",\"messages\":[{\"role\":\"user\",\"content\":\"test\"}],\"max_tokens\":1}")" != "200" ]; do sleep 2; done' || { echo "vLLM not ready after 15min"; exit 1; }

Profile Chat Completions API

The Chat Completions API uses the /v1/chat/completions endpoint.

Profile with synthetic inputs

Run AIPerf against the Chat Completions endpoint using synthetic inputs:

$ aiperf profile \
>     --model Qwen/Qwen3-0.6B \
>     --endpoint-type chat \
>     --endpoint /v1/chat/completions \
>     --streaming \
>     --synthetic-input-tokens-mean 100 \
>     --synthetic-input-tokens-stddev 0 \
>     --output-tokens-mean 200 \
>     --output-tokens-stddev 0 \
>     --url localhost:8000 \
>     --request-count 20

Sample Output (Successful Run):

INFO     Starting AIPerf System
INFO     AIPerf System is PROFILING
Profiling: 20/20 |████████████████████████| 100% [00:35<00:00]
INFO     Benchmark completed successfully
INFO     Results saved to: artifacts/Qwen_Qwen3-0.6B-chat-concurrency1/
            NVIDIA AIPerf | LLM Metrics
┃                      Metric ┃     avg ┃     min ┃     max ┃     p99 ┃     p50 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│        Request Latency (ms) │ 1678.90 │ 1456.34 │ 1923.45 │ 1923.45 │ 1667.23 │
│    Time to First Token (ms) │  234.56 │  198.34 │  289.12 │  289.12 │  231.45 │
│    Inter Token Latency (ms) │   13.89 │   11.23 │   17.45 │   17.45 │   13.67 │
│ Output Token Count (tokens) │  200.00 │  200.00 │  200.00 │  200.00 │  200.00 │
│  Request Throughput (req/s) │    5.67 │       - │       - │       - │       - │
└─────────────────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
JSON Export: artifacts/Qwen_Qwen3-0.6B-chat-concurrency1/profile_export_aiperf.json

Profile with custom input file

Create a JSONL input file:

$ cat <<EOF > inputs.jsonl
$ {"texts": ["Hello!"]}
$ {"texts": ["Tell me a joke."]}
$ EOF

Run AIPerf against the Chat Completions endpoint using the custom input file:

$ aiperf profile \
>     --model Qwen/Qwen3-0.6B \
>     --endpoint-type chat \
>     --endpoint /v1/chat/completions \
>     --streaming \
>     --input-file inputs.jsonl \
>     --custom-dataset-type single_turn \
>     --url localhost:8000 \
>     --request-count 10

Profile Completions API

The Completions API uses the /v1/completions endpoint.

Profile with synthetic inputs

Run AIPerf against the Completions endpoint using synthetic inputs:

$ aiperf profile \
>     --model Qwen/Qwen3-0.6B \
>     --endpoint-type completions \
>     --endpoint /v1/completions \
>     --synthetic-input-tokens-mean 64 \
>     --synthetic-input-tokens-stddev 4 \
>     --output-tokens-mean 128 \
>     --output-tokens-stddev 4 \
>     --url localhost:8000 \
>     --request-count 32

Sample Output (Successful Run):

INFO     Starting AIPerf System
INFO     Using /v1/completions endpoint
INFO     AIPerf System is PROFILING
Profiling: 32/32 |████████████████████████| 100% [00:28<00:00]
INFO     Benchmark completed successfully
INFO     Results saved to: artifacts/Qwen_Qwen3-0.6B-completions-concurrency1/
            NVIDIA AIPerf | LLM Metrics
┃                      Metric ┃    avg ┃    min ┃    max ┃    p99 ┃    p50 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│        Request Latency (ms) │ 876.45 │ 789.34 │ 987.12 │ 978.90 │ 871.23 │
│    Time to First Token (ms) │ 156.78 │ 134.56 │ 189.23 │ 185.67 │ 155.12 │
│    Inter Token Latency (ms) │  12.34 │  10.23 │  15.67 │  15.34 │  12.12 │
│ Output Token Count (tokens) │ 128.00 │ 120.00 │ 136.00 │ 135.00 │ 128.00 │
│  Request Throughput (req/s) │  10.89 │      - │      - │      - │      - │
└─────────────────────────────┴────────┴────────┴────────┴────────┴────────┘
JSON Export: artifacts/Qwen_Qwen3-0.6B-completions-concurrency1/profile_export_aiperf.json

Profile with custom input file

Create a JSONL input file:

$ cat <<EOF > inputs.jsonl
$ {"texts": ["How are you?"]}
$ {"texts": ["Give me a poem."]}
$ EOF

Run AIPerf against the Completions endpoint using the custom input file:

$ aiperf profile \
>     --model Qwen/Qwen3-0.6B \
>     --endpoint-type completions \
>     --endpoint /v1/completions \
>     --input-file inputs.jsonl \
>     --custom-dataset-type single_turn \
>     --url localhost:8000 \
>     --request-count 10

$	docker pull vllm/vllm-openai:latest
$	docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
>	--model Qwen/Qwen3-0.6B \
>	--reasoning-parser qwen3

$	aiperf profile \
>	--model Qwen/Qwen3-0.6B \
>	--endpoint-type chat \
>	--endpoint /v1/chat/completions \
>	--streaming \
>	--synthetic-input-tokens-mean 100 \
>	--synthetic-input-tokens-stddev 0 \
>	--output-tokens-mean 200 \
>	--output-tokens-stddev 0 \
>	--url localhost:8000 \
>	--request-count 20

$	cat <<EOF > inputs.jsonl
$	{"texts": ["Hello!"]}
$	{"texts": ["Tell me a joke."]}
$	EOF

$	aiperf profile \
>	--model Qwen/Qwen3-0.6B \
>	--endpoint-type completions \
>	--endpoint /v1/completions \
>	--synthetic-input-tokens-mean 64 \
>	--synthetic-input-tokens-stddev 4 \
>	--output-tokens-mean 128 \
>	--output-tokens-stddev 4 \
>	--url localhost:8000 \
>	--request-count 32

$	cat <<EOF > inputs.jsonl
$	{"texts": ["How are you?"]}
$	{"texts": ["Give me a poem."]}
$	EOF