Is this page helpful?

Benchmarking#

The past few years have witnessed the rise in popularity of generative AI and Large Language Models (LLMs), as part of a broader AI revolution. As LLM-based applications are increasingly rolled out across enterprises, there is a strong and urgent need to benchmark and ensure the cost efficiency of different serving solutions. The cost of an LLM application varies depending on how many queries it can process while being responsive and engaging for the end users. Note that all the cost measurement should be based on reaching an acceptable accuracy measurement, as defined by the application’s use case. This guide focuses on cost measurement and accuracy measurement is not covered.

Standardized benchmarking of LLM performance can be done with many tools, including long-standing tools such as Locust and K6, along with new open-source tools that are specialized for LLMs such as NVIDIA AIPerf. These client-side tools offer specific metrics for LLM-based applications but are not consistent in how they define, measure and calculate different metrics. This guide tries to clarify the common metrics and their differences and limitations. We also give a step-by-step guide on using our preferred tool (AIPerf) to benchmark your LLM applications.

It is worth noting that performance benchmarking and load testing are two distinct approaches to evaluating the deployment of a large language model. Load testing, as exemplified by tools like K6, focuses on simulating a large number of concurrent requests to a model to assess its ability to simulate real-world traffic and scale. This type of testing helps identify issues related to server capacity, auto scaling tactics, network latency, and resource utilization. In contrast, performance benchmarking, as demonstrated by NVIDIA’s AIPerf tool, is concerned with measuring the actual performance of the model itself, such as its throughput, latency, and token-level metrics. This document focuses on this type of testing and helps identify issues related to model efficiency, optimization, and configuration. While load testing is essential for ensuring the model can handle a large volume of requests, performance testing is crucial for understanding the model’s ability to process requests efficiently. By combining both approaches, developers can gain a comprehensive understanding of their large language model deployment’s capabilities and identify areas for improvement.

Important

To learn more about benchmarking LLMs, refer to the NIM LLM and VLM Benchmarking Guide.

The rest of this page is a worked example of that workflow against a NIM LLM and VLM deployment: bring the NIM up with a configuration you can point to, drive it with AIPerf across a concurrency sweep, prove the sweep measured the workload you intended, and read the resulting metrics. The commands were verified with AIPerf 0.10.0.

Prepare the NIM Under Test#

Serve the Model with a Known Configuration#

Start the NIM with the profile you intend to measure. A throughput number means nothing without the configuration that produced it, so record the engine arguments in play before you collect any data. For example, the Blackwell throughput profile for nvidia/nemotron-3-ultra-550b-a55b ships the following engine arguments in its manifest:

{"block_size": 64, "enable_expert_parallel": true, "enable_prefix_caching": true,
 "gpu_memory_utilization": 0.85, "mamba_cache_mode": "align",
 "mamba_ssm_cache_dtype": "bfloat16", "max_num_batched_tokens": 32768,
 "max_num_seqs": 64, "prefix_match_unit": 64}

To benchmark a variation on a shipped profile, pass overrides as direct container arguments. This preserves any NIM_PASSTHROUGH_ARGS value provided by the image. For the environment-only fallback and full precedence order, refer to Advanced Configuration.

Set NIM_LOG_LEVEL=INFO when you start the container:

docker run --gpus all -p 8000:8000 \
  -e NGC_API_KEY \
  -e NIM_LOG_LEVEL=INFO \
  <image>

The default log level suppresses the Resolved configuration: line that reports the arguments the server actually resolved. Without that line, a run where your override never took effect is indistinguishable from one where it did. For more information about log levels, refer to Logging and Observability.

Confirm the Model Name the Server Advertises#

The -m value you pass to AIPerf must match the served model ID exactly. Query the server rather than assuming the name:

curl -s http://localhost:8000/v1/models | python3 -m json.tool

Deployments differ here. A NIM profile might serve nvidia/nemotron-3-ultra-550b-a55b, while a published run of the same weights uses a name such as nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4. Use whichever name /v1/models reports.

Choose Between a Synthetic and a Traced Workload#

AIPerf can generate a synthetic workload from a fixed input and output length, or replay a captured trace of real requests. The two are not interchangeable, and the difference is larger than it looks.

A synthetic workload holds output length constant. A real workload usually does not: in the agentic trace used in the following example, half the requests finish at or under 400 output tokens, but the mean is 2,455 and the tail reaches 26,600. That heavy tail makes the trace roughly six times heavier on decode than a synthetic workload with the same nominal input length and a fixed 400-token output. Both are legitimate measurements, but they answer different questions — so when you plot them, label which is which, and never compare a synthetic point against a traced one.

The remainder of this page walks through a traced workload, which is the harder of the two to get right.

Replay a Trace with AIPerf#

Get the Trace#

Published traces are often stored with Git LFS, which changes how you have to download them. The following example fetches an agentic 64K-context trace from the NVIDIA Dynamo recipes:

mkdir -p traces
TRACE_FILE=nim_turbo_64k_400_90kv_agent_new_noschedule_short_15perc.jsonl
TRACE=traces/$TRACE_FILE

curl -sSL -o "$TRACE" \
  "https://media.githubusercontent.com/media/ai-dynamo/dynamo/main/recipes/nemotron-3-ultra/perf/traces/$TRACE_FILE"

Note the media.githubusercontent.com/media/ host. The usual raw.githubusercontent.com URL returns a 132-byte LFS pointer rather than the trace:

version https://git-lfs.github.com/spec/v1
oid sha256:f20d3f2bc83dd1306cda659fbe34e7c4d85ca5497626c98bc0b1c4d2211379d0
size 2722326

Warning

AIPerf reads that pointer as a valid three-line dataset and produces a complete, successful-looking benchmark of nothing. Validate the file before you spend GPU time on it.

head -c 40 "$TRACE" | grep -q 'git-lfs' && { echo "LFS pointer, not data"; exit 1; }
test "$(grep -c '' "$TRACE")" -eq 3541 || { echo "expected 3541 requests"; exit 1; }
sha256sum "$TRACE"
# f20d3f2bc83dd1306cda659fbe34e7c4d85ca5497626c98bc0b1c4d2211379d0

Inspect the Trace#

Know the shape of the workload before you run it — the trace’s properties determine several of the flags in the next step. The example trace has the following characteristics:

Property	Value
Requests	3,541
Input length	mean 68,435 · p50 67,585 · p90 101,392 · max 1,351,695
Output length	mean 2,455 · p50 399 · p90 6,943 · max 26,600
Block reuse	94% of referenced blocks are shared across requests
Tokens per `hash_id`	502.7 average

The last two rows matter for configuration: the trace’s blocks are about 512 tokens, which is how AIPerf expands the trace’s hash_ids into prompts of the right length with the right degree of sharing between them. AIPerf 0.10.0 and 0.11.0 read this block size from the mooncake_trace loader metadata and reject an explicit --prompt-input-tokens-block-size when --input-file is present, so do not pass that flag with this trace.

Run the Concurrency Sweep#

Sweeping concurrency produces the throughput and latency pairs that form a Pareto curve. Each concurrency level writes its own artifact directory:

MODEL=$(curl -s http://localhost:8000/v1/models | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"][0]["id"])')
TOKENIZER=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
TRACE=traces/nim_turbo_64k_400_90kv_agent_new_noschedule_short_15perc.jsonl

for C in 1 2 4 8 16 32 64; do
  aiperf profile \
    --ui none \
    -m "$MODEL" \
    --tokenizer "$TOKENIZER" \
    --tokenizer-trust-remote-code \
    --url http://localhost:8000 \
    --endpoint-type chat \
    --streaming \
    --concurrency "$C" \
    --workers-max "$C" \
    --input-file "$TRACE" \
    --custom-dataset-type mooncake_trace \
    --no-fixed-schedule \
    --dataset-sampling-strategy sequential \
    --use-server-token-count \
    --extra-inputs ignore_eos:true \
    --extra-inputs "cache_salt:salt_$(openssl rand -hex 6)" \
    --num-requests "$(grep -c '' "$TRACE")" \
    --artifact-dir "artifacts/c$C"
done

Flags That Are Easy to Get Wrong#

Most of these fail silently — they produce a clean run of the wrong workload rather than an error.

Flag	Why it matters
`--endpoint-type chat`	`completions` has no system role. A workload built around a shared system prefix loses that prefix entirely and measures a much shorter input.
`--tokenizer`	The served weights may be quantized (NVFP4 here) while the tokenizer ships with the BF16 repository. Without the matching tokenizer, AIPerf counts tokens differently than the server does, which makes `--use-server-token-count` compare two incompatible counts.
`--no-fixed-schedule`	This trace carries no timestamps, so requests are driven by the concurrency level instead of being replayed against a clock. A trace with timestamps should keep the fixed schedule.
`--dataset-sampling-strategy sequential`	Iterates the trace in order and wraps, so every run at a given concurrency sees the same request order.
`cache_salt`	A fresh salt per run keeps the server’s prefix cache from carrying results across runs and inflating later points in the sweep. Use a hex salt — base64 can emit `/` and `+`.
`--num-requests`	An alias for `--request-count`. Passing the full trace length replays the trace exactly once at each concurrency point.

Validate the Workload Before Trusting the Numbers#

The failure mode with traced workloads is not a crash — it is a clean run of the wrong workload. Before reading any performance number, check the measured sequence lengths against the trace you inspected earlier:

import json, glob
for f in sorted(glob.glob("artifacts/c*/profile_export_aiperf.json")):
    d = json.load(open(f))
    print(f"{f}")
    print(f"   ISL avg {d['input_sequence_length']['avg']:>10,.0f}  (expect ~68,435)")
    print(f"   OSL avg {d['output_sequence_length']['avg']:>10,.0f}  (expect ~2,455)")
    print(f"   OSL mismatches: {d.get('osl_mismatch_count', {}).get('avg', 0)}")

Expect a measured input length within about 10% of the trace’s mean. Two common deviations point at specific mistakes:

A value near a tenth of the expected length (about 6,400 for this trace) means the shared prefix was dropped. Check --endpoint-type.
A very small value means the LFS pointer was replayed instead of the trace. Re-check the download.

Read the Results#

Each artifacts/c<N>/profile_export_aiperf.json file holds the metrics for one point on the curve:

Metric	Field
Throughput	`output_token_throughput.avg`
Per-user throughput	`output_token_throughput_per_user.avg`
Inter-token latency	`inter_token_latency.avg` (also `.p50`, `.p95`, `.p99`)
Time to first token	`time_to_first_token.avg`
Request latency	`request_latency.avg`

Plot output_token_throughput against inter_token_latency across the sweep to get the Pareto curve, and pick the concurrency that meets your latency budget at the highest throughput.

The input_config object in the same file records the configuration AIPerf actually accepted. Together with the Resolved configuration: line from the server log, it is the authoritative record of what was measured — keep both alongside the numbers.