Effective vs Active Metrics in AIPerf

A short technical brief on time-weighted throughput, concurrency, and coordinated-omission-aware latency.

TL;DR

AIPerf reports two complementary families of time-weighted metrics:

• Effective metrics are time-weighted averages of a step function over the full benchmark window. An “average concurrency of 14.6” means that, integrating across every nanosecond from the first credit to the last response, the in-flight request count averaged 14.6.
• Active metrics are the same time-weighted averages restricted to segments where the relevant phase has at least one request in flight. An “active prefill throughput of 28k tok/s” means: while any request was actually in prefill, tokens were being produced at that rate on average.
• effective_latency is a separate per-record metric grouped under EFFECTIVE. It is end_ns - credit_issued_ns — the latency a saturating user actually perceives, including time waiting in the credit queue. It is the only AIPerf latency metric that is coordinated-omission-aware.

Rule of thumb: cite Effective when capacity-planning at the workload mix you measured; cite Active when characterizing peak phase intensity; cite effective_latency when reporting user-perceived latency under load that could be saturating.

Why classical record-averages mislead

LLM inference has three measurement traps that simple arithmetic means hide:

1. Equal-weight averaging over records is biased toward fast requests. A run with one 10-second request and ninety-nine 1-second requests has the same record-arithmetic-mean as a run with one 100-second request and ninety-nine 1-second requests, even though the first run is far healthier. Time-weighted averages weight by duration, so the heavy request contributes proportionally to how long it actually occupied the system.

2. Whole-run averages dilute by idle gaps. LLM inference has two distinct phases per request: prefill (compute-bound, brief, processes the input) and decode (memory-bound, long, generates one token at a time). At any instant most in-flight requests are in decode; prefill windows are brief and sparse. A whole-window average of “prefill throughput” reports a number diluted by all the decode-only time and is much smaller than the per-prefill-burst intensity the hardware actually delivers.

3. Coordinated omission. Under a saturating load generator, requests pile up in the AIPerf credit queue before being dispatched. The server-side timing (request_start_ns to end_ns) excludes that queue wait, so a naive latency understates what an actual user — who issued the request at credit_issued_ns — would have observed. AIPerf addresses this with effective_latency, which charges the queue wait to the request.

The Effective and Active metric groups, together with effective_latency, are AIPerf’s responses to these three traps.

Effective metrics: full-window time-weighted views

AIPerf’s analyzer builds a step function over each quantity of interest (concurrency, decode throughput, prefill throughput, total throughput, tokens-in-flight) using a vectorized sweep-line algorithm on the per-request timestamp columns. The step function holds value v_i from event t_i to event t_{i+1}. An Effective metric is then the time-weighted average

avg = Σ (v_i × Δt_i) / (run_end − run_start)

Percentiles (p50, p90, p95, p99) are also duration-weighted: AIPerf sorts the (v_i, Δt_i) pairs by value, takes a cumulative duration fraction, and reads off the quantile. A “p99 of 920 tok/s” therefore means “99% of the run-window time, decode throughput was at or below 920 tok/s” — not “99% of the records had throughput at or below 920 tok/s”.

The full set of Effective metrics emitted today:

Active metrics: phase-restricted views

Active variants use the same sweep-line rate curve, but the integration window is restricted to segments where the relevant phase mask is strictly positive. For active_prefill_throughput the mask is prefill_concurrency > 0; for active_decode_throughput it is decode_concurrency > 0. Time when no request is in that phase contributes zero duration to the denominator, so the average reflects intensity during the phase rather than diluted by gaps.

In the bar diagram, only the non-zero spikes contribute to both numerator and denominator — the zero-valued bands are excluded.

The Active metrics emitted today:

Worked example (real AIPerf run)

Run: aiperf profile -m mock-model --streaming --concurrency 16 --request-count 200 --synthetic-input-tokens-mean 200 --output-tokens-mean 100 against the in-repo mock server with TTFT=100 ms, ITL=20 ms. Benchmark duration: 27.06 s.

Selected rows from the end-of-run console tables:

Two observations:

• Decode is almost always active, so Effective Decode and Active Decode track each other (727 vs 755 tok/s). Decode dominates the run window — the mean decode concurrency of 14.63 is ≈ 91% of the 16 offered slots (14.63/16.0), so at almost every instant nearly all in-flight requests are in decode.
• Prefill is sparse, so Effective and Active disagree by ~19×. effective_prefill_concurrency averages 0.75 across the whole window — prefill is in flight only a small fraction of the time. When you ask “what is the prefill throughput of this system?”, Active (28k tok/s) is the answer about hardware capability; Effective (1.5k tok/s) is the answer about how much prefill work the workload demanded on average. Both are correct; they answer different questions.

The Effective row’s p50 = 0 for prefill is not a bug — it correctly reports that for more than half of the run window, no request was in prefill, so the time-weighted median of the prefill-throughput step function is exactly zero.

effective_latency: the coordinated-omission-aware latency

effective_latency is grouped under MetricConsoleGroup.EFFECTIVE even though, unlike the sweep-line metrics, it is a per-record metric. The definition is:

effective_latency = end_ns − credit_issued_ns

Compare to the classical request_latency = end_ns − start_ns. The difference, start_ns − credit_issued_ns, is the time the request spent waiting in AIPerf’s credit queue — invisible to the server but real to the user.

This metric is only emitted when at least one record carries a credit_issued_ns timestamp. Every AIPerf load mode dispatches requests through the credit issuer, which stamps each credit with issued_at_ns — the worker then records it as credit_issued_ns — so effective_latency is available for closed-loop (--concurrency) and open-loop (--request-rate, trace replay) runs alike. It is suppressed only when no record in the run carries a credit_issued_ns timestamp. Comparing effective_latency against request_latency tells you how much of perceived latency is queue-induced (load-generator backpressure) versus server-induced (the model itself):

• If they are essentially equal — as in the worked example above, where both averaged ~2,081 ms — your load is not saturating; queue wait is negligible.
• If effective_latency is materially larger than request_latency, you have crossed into a saturating regime. The “real” tail latency users observe is the effective_latency distribution, not the server-side one.

This is AIPerf’s answer to the coordinated-omission problem made famous by Gil Tene: a naïve benchmark that omits queue wait under-reports user-perceived latency precisely when the system is most stressed.

Choosing a metric

Reading the console output

In the standard end-of-run output, AIPerf renders one table per non-empty MetricConsoleGroup in the order EFFECTIVE, ACTIVE, USAGE, CACHE, PREDICTION, AUDIO, REASONING, DEFAULT. A vanilla LLM run typically shows three: NVIDIA AIPerf | LLM Metrics: Effective, NVIDIA AIPerf | LLM Metrics: Active, and the legacy/default NVIDIA AIPerf | LLM Metrics table containing record-level distributions (TTFT, ITL, request latency, OSL, ISL). Endpoint types that emit usage, cache, prediction, audio, or reasoning tokens add intermediate tables. The grouping is driven by the console_group class attribute on each metric.

References

• Definitions and formulas for the record-level metrics referenced here (TTFT, ITL, request latency, ISL/OSL): docs/metrics-reference.md. The Effective/Active families themselves are defined only in this brief.
• Coordinated omission background: Gil Tene, “How NOT to Measure Latency” (Strange Loop 2015)