> This page is for AIPerf, version v0.13.0. > For other versions, use one of these documentation indexes: > - Latest (v0.13.0) (default): https://docs.nvidia.com/aiperf/latest/llms.txt > - dev: https://docs.nvidia.com/aiperf/dev/llms.txt > - v0.13.0: https://docs.nvidia.com/aiperf/v0.13.0/llms.txt > - v0.12.0: https://docs.nvidia.com/aiperf/v0.12.0/llms.txt > - v0.11.0: https://docs.nvidia.com/aiperf/v0.11.0/llms.txt > - v0.9.0: https://docs.nvidia.com/aiperf/v0.9.0/llms.txt > - v0.8.0: https://docs.nvidia.com/aiperf/v0.8.0/llms.txt > - v0.7.0: https://docs.nvidia.com/aiperf/v0.7.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aiperf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aiperf/_mcp/server. # AIPerf Metrics Reference This document provides a comprehensive reference of all metrics available in AIPerf for benchmarking LLM inference performance. Metrics are organized by computation type to help you understand when and how each metric is calculated. ## Table of Contents - [Quick Reference](#quick-reference) - [Understanding Metric Types](#understanding-metric-types) - [Record Metrics](#record-metrics) - [Aggregate Metrics](#aggregate-metrics) - [Derived Metrics](#derived-metrics) - [Detailed Metric Descriptions](#detailed-metric-descriptions) - [Streaming Metrics](#streaming-metrics) - [Time to First Token (TTFT)](#time-to-first-token-ttft) - [Time to Second Token (TTST)](#time-to-second-token-ttst) - [Time to First Output Token (TTFO)](#time-to-first-output-token-ttfo) - [Decode Duration](#decode-duration) - [Inter Token Latency (ITL)](#inter-token-latency-itl) - [Inter Chunk Latency (ICL)](#inter-chunk-latency-icl) - [Output Token Throughput Per User](#output-token-throughput-per-user) - [Prefill Throughput Per User](#prefill-throughput-per-user) - [Token Based Metrics](#token-based-metrics) - [Output Token Count](#output-token-count) - [Output Sequence Length (OSL)](#output-sequence-length-osl) - [Input Sequence Length (ISL)](#input-sequence-length-isl) - [Total Output Tokens](#total-output-tokens) - [Total Output Sequence Length](#total-output-sequence-length) - [Total Input Sequence Length](#total-input-sequence-length) - [E2E Output Token Throughput](#e2e-output-token-throughput) - [Output Token Throughput](#output-token-throughput) - [Total Token Throughput](#total-token-throughput) - [Image Metrics](#image-metrics) - [Number of Images](#number-of-images) - [Image Throughput](#image-throughput) - [Image Latency](#image-latency) - [Video Metrics](#video-metrics) - [Video Inference Time](#video-inference-time) - [Video Peak Memory](#video-peak-memory) - [Audio Metrics](#audio-metrics) - [Audio Duration](#audio-duration) - [Inverse Real-Time Factor (RTFx)](#inverse-real-time-factor-rtfx) - [Reasoning Metrics](#reasoning-metrics) - [Reasoning Token Count](#reasoning-token-count) - [Total Reasoning Tokens](#total-reasoning-tokens) - [Speculative Decoding Metrics](#speculative-decoding-metrics) - [Acceptance Length](#acceptance-length) - [Token-Weighted Acceptance Length](#token-weighted-acceptance-length) - [Draft Acceptance Rate](#draft-acceptance-rate) - [Overall Draft Acceptance Rate](#overall-draft-acceptance-rate) - [Accepted per Verified](#accepted-per-verified) - [Spec Decode Steps](#spec-decode-steps) - [Accepted / Draft Token Counts](#accepted--draft-token-counts) - [Pooled Acceptance Histogram](#pooled-acceptance-histogram) - [Usage Field Metrics](#usage-field-metrics) - [Usage Prompt Tokens](#usage-prompt-tokens) - [Usage Completion Tokens](#usage-completion-tokens) - [Usage Total Tokens](#usage-total-tokens) - [Usage Reasoning Tokens](#usage-reasoning-tokens) - [Usage Prompt Cache Read Tokens](#usage-prompt-cache-read-tokens) - [Usage Prompt Cache Write Tokens](#usage-prompt-cache-write-tokens) - [Usage Prompt Cache Miss Tokens](#usage-prompt-cache-miss-tokens) - [Usage Prompt Audio Tokens](#usage-prompt-audio-tokens) - [Usage Completion Audio Tokens](#usage-completion-audio-tokens) - [Usage Prompt Audio Seconds](#usage-prompt-audio-seconds) - [Usage Tool Use Prompt Tokens](#usage-tool-use-prompt-tokens) - [Usage Accepted Prediction Tokens](#usage-accepted-prediction-tokens) - [Usage Rejected Prediction Tokens](#usage-rejected-prediction-tokens) - [Total Usage Prompt Tokens](#total-usage-prompt-tokens) - [Total Usage Completion Tokens](#total-usage-completion-tokens) - [Total Usage Total Tokens](#total-usage-total-tokens) - [Total Usage Reasoning Tokens](#total-usage-reasoning-tokens) - [Total Usage Prompt Cache Read Tokens](#total-usage-prompt-cache-read-tokens) - [Overall Usage Prompt Cache Read %](#overall-usage-prompt-cache-read-) - [Total Usage Prompt Cache Write Tokens](#total-usage-prompt-cache-write-tokens) - [Total Usage Prompt Cache Miss Tokens](#total-usage-prompt-cache-miss-tokens) - [Total Usage Prompt Audio Tokens](#total-usage-prompt-audio-tokens) - [Total Usage Completion Audio Tokens](#total-usage-completion-audio-tokens) - [Total Usage Prompt Audio Seconds](#total-usage-prompt-audio-seconds) - [Total Usage Tool Use Prompt Tokens](#total-usage-tool-use-prompt-tokens) - [Total Usage Accepted Prediction Tokens](#total-usage-accepted-prediction-tokens) - [Total Usage Rejected Prediction Tokens](#total-usage-rejected-prediction-tokens) - [Usage Discrepancy Metrics](#usage-discrepancy-metrics) - [Usage Prompt Diff %](#usage-prompt-diff-) - [Usage Completion Diff %](#usage-completion-diff-) - [Usage Reasoning Diff %](#usage-reasoning-diff-) - [Usage Discrepancy Count](#usage-discrepancy-count) - [OSL Mismatch Metrics](#osl-mismatch-metrics) - [OSL Mismatch Diff %](#osl-mismatch-diff-) - [OSL Mismatch Count](#osl-mismatch-count) - [Replay Schedule Lag Metrics](#replay-schedule-lag-metrics) - [Replay Send Schedule Offset](#replay-send-schedule-offset) - [Replay Schedule Lag p50 / p90 / p99](#replay-schedule-lag-p50--p90--p99) - [Replay Schedule Degraded](#replay-schedule-degraded) - [Agentic / Trace Metrics](#agentic--trace-metrics) - [Theoretical Prefix Cache Hit](#theoretical-prefix-cache-hit) - [Context Overflow Count](#context-overflow-count) - [Goodput Metrics](#goodput-metrics) - [Good Request Count](#good-request-count) - [Good Request Fraction](#good-request-fraction) - [Goodput](#goodput) - [Error Metrics](#error-metrics) - [Error Input Sequence Length](#error-input-sequence-length) - [Total Error Input Sequence Length](#total-error-input-sequence-length) - [General Metrics](#general-metrics) - [Request Latency](#request-latency) - [Request Throughput](#request-throughput) - [Request Count](#request-count) - [Error Request Count](#error-request-count) - [Request Error Rate](#request-error-rate) - [Minimum Request Timestamp](#minimum-request-timestamp) - [Maximum Response Timestamp](#maximum-response-timestamp) - [Benchmark Duration](#benchmark-duration) - [Network Latency Calibration Metrics](#network-latency-calibration-metrics) - [Network RTT](#network-rtt) - [Network-Adjusted Latency Metrics](#network-adjusted-latency-metrics) - [HTTP Trace Metrics](#http-trace-metrics) - [HTTP Blocked](#http-blocked) - [HTTP DNS Lookup](#http-dns-lookup) - [HTTP Connecting](#http-connecting) - [HTTP Sending](#http-sending) - [HTTP Waiting (TTFB)](#http-waiting-ttfb) - [HTTP Receiving](#http-receiving) - [HTTP Duration (excl. conn)](#http-duration-excl-conn) - [HTTP Connection Overhead](#http-connection-overhead) - [HTTP Total Time](#http-total-time) - [HTTP Data Sent](#http-data-sent) - [HTTP Data Received](#http-data-received) - [HTTP Connection Reused](#http-connection-reused) - [HTTP Chunks Sent](#http-chunks-sent) - [HTTP Chunks Received](#http-chunks-received) - [GPU Power Efficiency Metrics](#gpu-power-efficiency-metrics) - [Total GPU Power](#total-gpu-power) - [Total GPU Energy](#total-gpu-energy) - [Output Tokens per Joule](#output-tokens-per-joule) - [Energy per User](#energy-per-user) - [Average GPU Power](#average-gpu-power) - [Energy per Output Token](#energy-per-output-token) - [Energy per Total Token](#energy-per-total-token) - [Energy per Request](#energy-per-request) - [Performance per Watt](#performance-per-watt) - [Output Tokens per Second per Watt](#output-tokens-per-second-per-watt) - [Goodput per Watt](#goodput-per-watt) - [Energy Delay Product](#energy-delay-product) - [Metric Flags Reference](#metric-flags-reference) --- ## Quick Reference The sections below provide detailed descriptions, requirements, and notes for each metric. --- ## Understanding Metric Types AIPerf computes metrics in three distinct phases during benchmark execution: **Record Metrics**, **Aggregate Metrics**, and **Derived Metrics**. > The metric type also determines which stat fields appear in `profile_export_aiperf.json` per metric — see [JSON Export Schema](/aiperf/reference/json-export-schema) for the per-field presence rules and version history. ## Record Metrics Record Metrics are computed **individually** for **each request** and its **response(s)** during the benchmark run. A single request may have one response (non-streaming) or multiple responses (streaming). These metrics capture **per-request characteristics** such as latency, token counts, and streaming behavior. Record metrics produce **statistical distributions** (min, max, mean, median, p90, p99, etc.) that reveal performance variability across requests. ### Example Metrics `request_latency`, `time_to_first_token`, `inter_token_latency`, `output_token_count`, `input_sequence_length` ### Dependencies Record Metrics can depend on raw request/response data and other Record Metrics from the same request. ### Example Scenario `request_latency` measures the time for each individual request from start to final response. If you send 100 requests, you get 100 latency values that form a distribution showing how latency varies across requests. ## Aggregate Metrics Aggregate Metrics are computed by **tracking** or **accumulating** values across **all requests** in **real-time** during the benchmark. These include counters, min/max timestamps, and other global statistics. Aggregate metrics produce a **single value** representing the entire benchmark run. ### Example Metrics `request_count`, `error_request_count`, `min_request_timestamp`, `max_response_timestamp` ### Dependencies Aggregate Metrics can depend on raw request/response data, Record Metrics and other Aggregate Metrics. ### Example Scenario `request_count` increments by 1 for each successful request. At the end of a benchmark with 100 successful requests, this metric equals 100 (a single value, not a distribution). ## Derived Metrics Derived Metrics are computed by applying **mathematical formulas** to other metric results, but are **not** computed per-record like Record Metrics. Instead, these metrics depend on one or more **prerequisite metrics** being available first and are calculated either **after the benchmark completes** for final results or in **real-time** across **all current data** for live metrics display. Derived metrics can produce either single values or distributions depending on their dependencies. ### Example Metrics `request_throughput`, `output_token_throughput`, `benchmark_duration` ### Dependencies Derived Metrics can depend on Record Metrics, Aggregate Metrics, and other Derived Metrics, but do not have any knowledge of the individual request/response data. ### Example Scenario `request_throughput` is computed from `request_count / benchmark_duration_seconds`. This requires both `request_count` and `benchmark_duration` to be available first, then applies a formula to produce a single throughput value (e.g., 10.5 requests/sec). --- # Detailed Metric Descriptions ## Streaming Metrics All metrics in this section require the `--streaming` flag with a token-producing endpoint and at least one non-empty response chunk. ### Time to First Token (TTFT) **Type:** [Record Metric](#record-metrics) Measures how long it takes to receive the first token (or chunk of tokens) after sending a request. This is critical for user-perceived responsiveness in streaming scenarios, as it represents how quickly the model begins generating output. **Formula:** ```python # nanoseconds ttft_ns = request.content_responses[0].perf_ns - request.start_perf_ns # Convert to milliseconds for display ttft_ms = ttft_ns / 1e6 # Convert to seconds for throughput calculations ttft_seconds = ttft_ns / 1e9 ``` **Notes:** - Includes network latency, queuing time, prompt processing, and generation of the first token (or chunk of tokens). - Raw timestamps are in nanoseconds; converted to milliseconds for display and seconds for rate calculations. - Response chunks refer to individual messages with non-empty content received during streaming. --- ### Time to Second Token (TTST) **Type:** [Record Metric](#record-metrics) Measures the time gap between the first and second chunk of tokens. This metric helps identify generation startup overhead separate from steady-state streaming throughput. **Formula:** ```python # nanoseconds ttst_ns = request.content_responses[1].perf_ns - request.content_responses[0].perf_ns # Convert to milliseconds for display ttst_ms = ttst_ns / 1e6 ``` **Notes:** - Requires at least 2 non-empty response chunks to compute the time between first and second tokens. - Raw timestamps are in nanoseconds; converted to milliseconds for display. --- ### Time to First Output Token (TTFO) **Type:** [Record Metric](#record-metrics) Calculates the time elapsed from request start to the first non-reasoning output token. This metric measures the latency from when a request is initiated to when the first actual output token (non-reasoning content) is received. It is particularly relevant for models that perform extended reasoning before generating output. **Formula:** ```python # nanoseconds # First non-reasoning token: TextResponseData with non-empty text, or # ReasoningResponseData with non-empty content field ttfo_ns = first_non_reasoning_token_perf_ns - request.start_perf_ns # Convert to milliseconds for display ttfo_ms = ttfo_ns / 1e6 ``` **Notes:** - TTFO vs TTFT: Time to First Output (TTFO) measures time to the first non-reasoning token, while Time to First Token (TTFT) measures time to any first token including reasoning tokens. For models without reasoning, TTFO and TTFT are equivalent. - Non-reasoning tokens include TextResponseData with non-empty text, or ReasoningResponseData with non-empty content field (regardless of reasoning field). - Requires at least one non-empty non-reasoning response chunk. --- ### Decode Duration **Type:** [Record Metric](#record-metrics) Measures the client-observed wall-clock interval between the first and final non-empty streamed content responses. Unlike ITL, this metric is not normalized by the number of generated tokens. **Formula:** ```python decode_duration_ns = request_latency_ns - time_to_first_token_ns decode_duration_ms = decode_duration_ns / 1e6 ``` **Notes:** - This is an end-to-end client observation. It includes response delivery and client-observed gaps after TTFT; it does not isolate server kernel execution time. - Compare Decode Duration with Output Sequence Length when early stopping or a generation-length change is possible. - A run can have similar Decode Duration but a much higher ITL if it produces fewer output tokens. - Requires valid `time_to_first_token` and `request_latency` metrics. --- ### Inter Token Latency (ITL) **Type:** [Record Metric](#record-metrics) Measures the average time between consecutive tokens during generation, excluding the initial TTFT overhead. This represents the steady-state token generation rate. **Formula:** ```python # Default (all runs): assume one token in the first chunk. # Calculate in nanoseconds, then convert to seconds inter_token_latency_ns = (request_latency_ns - time_to_first_token_ns) / (output_sequence_length - 1) # Convert to seconds for throughput calculations inter_token_latency_seconds = inter_token_latency_ns / 1e9 # Convert to milliseconds for display inter_token_latency_ms = inter_token_latency_ns / 1e6 ``` **Notes:** - By default the divisor is `output_sequence_length - 1`, which is exact for servers that stream one token per chunk. This is the formula every run uses unless `--per-chunk-usage` is enabled. - **Opt-in first-chunk correction (`--per-chunk-usage`).** The decode window (`request_latency - time_to_first_token`) covers only the tokens that arrive *after* the first content chunk. When a server bundles several tokens into the first streamed chunk (for example TRT-LLM's `stream-interval`), assuming a single token over-counts the decode tokens and inflates Output Token Throughput Per User (`1 / ITL`). Passing `--per-chunk-usage` (which also requires `--use-server-token-count`, a `chat` endpoint, and `--streaming`) makes the server report cumulative per-chunk usage via `stream_options.continuous_usage_stats`; the divisor then becomes `output_sequence_length - tokens_in_first_content_chunk`, where that count is the server-reported cumulative `completion_tokens` (reasoning included, matching OSL which is `output + reasoning = completion`) through the first content chunk. Requires a server that honors `continuous_usage_stats` (e.g. vLLM, TRT-LLM); otherwise the divisor stays `output_sequence_length - 1`. - Requires an output sequence length of at least 2, plus valid `time_to_first_token`, `request_latency`, and `output_sequence_length` metrics. If the first-chunk count is inconsistent with OSL (>= OSL), the divisor degrades to `output_sequence_length - 1` and a one-time warning is logged rather than dropping the metric. - ITL is generated-token-normalized decode duration. It is not the total wall-clock decode duration. - Compare runs at equivalent output lengths, or inspect Decode Duration and Output Sequence Length alongside ITL. - Streaming chunks can contain multiple tokens. ITL uses token count, while ICL uses chunk arrival timestamps. - Result is in seconds when used for throughput calculations (Output Token Throughput Per User). --- ### Inter Chunk Latency (ICL) **Type:** [Record Metric](#record-metrics) Captures the time gaps between all consecutive response chunks in a streaming response, providing a distribution of chunk arrival times rather than a single average. Note that this is different from the ITL metric, which measures the time between consecutive tokens regardless of chunk size. **Formula:** ```python inter_chunk_latency = [request.content_responses[i].perf_ns - request.content_responses[i-1].perf_ns for i in range(1, len(request.content_responses))] ``` **Notes:** - Requires at least 2 response chunks. - Unlike ITL (which produces a single average), ICL provides the full distribution of inter-chunk times. - Useful for detecting variability, jitter, or issues in streaming delivery. - Analyzing ICL distributions can reveal batching behavior, scheduling issues, or network variability. --- ### Output Token Throughput Per User **Type:** [Record Metric](#record-metrics) This metric is computed per-request, and it excludes the TTFT from the equation, so it is **not** directly comparable to the [Output Token Throughput](#output-token-throughput) metric. The token generation rate experienced by an individual user/request, measured as the inverse of inter-token latency. This represents single-request streaming performance. **Formula:** ```python output_token_throughput_per_user = 1.0 / inter_token_latency_seconds ``` **Notes:** - Computes the inverse of ITL to show tokens per second from an individual user's perspective. - Differs from Output Token Throughput (aggregate across all concurrent requests) by focusing on single-request experience. - Useful for understanding the user experience independent of concurrency effects. --- ### Prefill Throughput Per User **Type:** [Record Metric](#record-metrics) Measures the rate at which input tokens are processed during the prefill phase, calculated as input tokens per second based on TTFT. This is only applicable to streaming responses. **Formula:** ```python prefill_throughput_per_user = input_sequence_length / time_to_first_token_seconds ``` **Notes:** - Higher values indicate faster prompt processing. - Useful for understanding input processing capacity and bottlenecks. - Depends on Input Sequence Length and TTFT metrics. --- ## Token Based Metrics All metrics in this section require token-producing endpoints that return text content (chat, completion, etc.). These metrics are not available for embeddings or other non-generative endpoints. ### Output Token Count **Type:** [Record Metric](#record-metrics) The number of output tokens generated for a single request, _excluding reasoning tokens_. This represents the output tokens returned to the user across all responses for the request. **Formula:** ```python output_token_count = len(tokenizer.encode(content, add_special_tokens=False)) ``` **Notes:** - Tokenization uses `add_special_tokens=False` to count only content tokens, excluding special tokens added by the tokenizer. - For streaming requests with multiple responses, the responses are joined together and then tokens are counted. - For models that expose reasoning in a separate `reasoning_content` field, this metric counts only non-reasoning output tokens. - If reasoning appears inside the regular `content` (e.g., `` blocks), those tokens will be counted unless explicitly filtered. --- ### Output Sequence Length (OSL) **Type:** [Record Metric](#record-metrics) The total number of completion tokens (output + reasoning) generated for a single request across all its responses. This represents the complete token generation workload for the request. **Formula:** ```python output_sequence_length = (output_token_count or 0) + (reasoning_token_count or 0) ``` **Notes:** - For models that do not support/separate reasoning tokens, OSL equals the output token count. --- ### Input Sequence Length (ISL) **Type:** [Record Metric](#record-metrics) The number of input/prompt tokens for a single request. This represents the size of the input sent to the model. **Formula:** ```python input_sequence_length = len(tokenizer.encode(prompt, add_special_tokens=False)) ``` **Notes:** - Tokenization uses `add_special_tokens=False` to count only content tokens, excluding special tokens added by the tokenizer. - Useful for understanding the relationship between input size and latency/throughput. --- ### Total Output Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all output tokens (excluding reasoning tokens) generated across all requests. This represents the total output token workload. **Formula:** ```python total_output_tokens = sum(r.output_token_count for r in records if r.valid) ``` **Notes:** - Aggregates output tokens across all successful requests. - Useful for capacity planning and cost estimation. --- ### Total Output Sequence Length **Type:** [Derived Metric](#derived-metrics) The sum of all completion tokens (output + reasoning) generated across all requests. This represents the complete token generation workload. **Formula:** ```python total_osl = sum(r.output_sequence_length for r in records if r.valid) ``` **Notes:** - Aggregates the complete token generation workload including both output and reasoning tokens. - For models without reasoning tokens, this equals Total Output Tokens. --- ### Total Input Sequence Length **Type:** [Derived Metric](#derived-metrics) The sum of all input/prompt tokens processed across all requests. This represents the total input workload sent to the model. **Formula:** ```python total_isl = sum(r.input_sequence_length for r in records if r.valid) ``` **Notes:** - Useful for understanding the input workload, capacity planning, and analyzing the relationship between input size and system performance. --- ### E2E Output Token Throughput **Type:** [Record Metric](#record-metrics) Per-request output token throughput based on end-to-end request latency. Unlike [Output Token Throughput Per User](#output-token-throughput-per-user) (which uses 1/ITL and excludes TTFT), this metric includes TTFT, queuing, and all other overhead in the denominator. Available for both streaming and non-streaming responses. **Formula:** ```python e2e_output_token_throughput = output_sequence_length / request_latency_seconds ``` **Notes:** - Uses total request latency (not ITL), so values will be slightly lower than Output Token Throughput Per User for streaming responses. - Available for non-streaming responses (unlike Output Token Throughput Per User which requires streaming). - Flags: `PRODUCES_TOKENS_ONLY | LARGER_IS_BETTER` - Depends on Output Sequence Length and Request Latency metrics. --- ### Output Token Throughput **Type:** [Derived Metric](#derived-metrics) This metric is computed as a single value across all requests and includes TTFT in the equation, so it is **not** directly comparable to the [Output Token Throughput Per User](#output-token-throughput-per-user) metric. The aggregate token generation rate across all concurrent requests, measured as total tokens per second. This represents the system's overall token generation capacity. **Formula:** ```python output_token_throughput = total_osl / benchmark_duration_seconds ``` **Notes:** - Measures aggregate throughput across all concurrent requests; represents the overall system token generation rate. - Higher values indicate better system utilization and capacity. --- ### Total Token Throughput **Type:** [Derived Metric](#derived-metrics) Calculates the total token throughput metric, combining both input and output token processing across all concurrent requests. **Formula:** ```python total_token_throughput = (total_isl + total_osl) / benchmark_duration_seconds ``` **Notes:** - Measures the combined input and output token processing rate. - Includes reasoning tokens in the output count (via total_osl). - Useful for understanding total system token processing capacity. --- ## Image Metrics All metrics in this section require image-capable endpoints (e.g., image generation APIs). These metrics are not available for text-only or other non-image endpoints. ### Number of Images **Type:** [Record Metric](#record-metrics) The number of logical image references in the wire request. This is the foundation metric used by Image Throughput and Image Latency. **Formula:** ```python num_images = count_image_content_parts(wire_payload) ``` **Notes:** - Requires at least one image in at least one turn. - Counts UUID cache-only references as logical images even when their URL is empty. - Does not measure uploaded image bytes or cache misses. - Not displayed in console output (`console_group = MetricConsoleGroup.NONE`). --- ### Image Throughput **Type:** [Record Metric](#record-metrics) Calculates the image throughput from the record by dividing the number of images by the request latency. **Formula:** ```python image_throughput = num_images / request_latency_seconds ``` **Notes:** - Higher values indicate faster image generation. --- ### Image Latency **Type:** [Record Metric](#record-metrics) Calculates the image latency from the record by dividing the request latency by the number of images. **Formula:** ```python image_latency = request_latency_ms / num_images ``` **Notes:** - Lower values indicate faster per-image generation. --- ## Video Metrics All metrics in this section require video-producing endpoints (e.g., SGLang video generation). These metrics rely on server-reported fields in the response and are not available for non-video endpoints. ### Video Inference Time **Type:** [Record Metric](#record-metrics) Server-reported GPU generation time for video inference, extracted from the `inference_time_s` field in video generation responses (e.g., SGLang). **Formula:** ```python video_inference_time = response.data.inference_time_s ``` **Notes:** - Value comes from the server, not computed by AIPerf. - Displayed in milliseconds. --- ### Video Peak Memory **Type:** [Record Metric](#record-metrics) Server-reported peak GPU memory usage during video generation, extracted from the `peak_memory_mb` field in video generation responses. **Formula:** ```python video_peak_memory = response.data.peak_memory_mb ``` **Notes:** - Value comes from the server, not computed by AIPerf. - Unit is megabytes. --- ## Audio Metrics Metrics in this section require an audio input on the request (e.g., ASR datasets such as LibriSpeech, GigaSpeech, AMI, VoxPopuli). They are not computed for text-only or non-audio requests. ### Audio Duration **Type:** [Record Metric](#record-metrics) Per-request input audio duration in seconds. Hidden from the console summary; available in JSON / CSV record exports for characterizing dataset shape and verifying RTFx calculations. **Notes:** - Only computed when the request carries `audio_duration_seconds` (e.g., ASR datasets such as LibriSpeech). - Aggregate stats (avg, p50, p99) are computed automatically. ### Inverse Real-Time Factor (RTFx) **Type:** [Record Metric](#record-metrics) The ratio of input audio duration to request latency. The standard ASR throughput metric, used by the HuggingFace Open ASR Leaderboard, NVIDIA Riva, and NVIDIA NeMo. **Formula:** ```python rtfx = audio_duration_seconds / request_latency_seconds ``` **Notes:** - Higher is better. A value of 10 means the server transcribed audio 10× faster than real-time playback. - RTFx < 1 means the server is slower than real-time and not suitable for live transcription. - Requires `audio_duration` and `request_latency` metrics to be computed first. --- ## Reasoning Metrics All metrics in this section require models and backends that expose reasoning content in a separate `reasoning_content` field, distinct from the regular `content` field. ### Reasoning Token Count **Type:** [Record Metric](#record-metrics) The number of reasoning tokens generated for a single request. These are tokens used for "thinking" or chain-of-thought reasoning before generating the final output. **Formula:** ```python reasoning_token_count = len(tokenizer.encode(reasoning_content, add_special_tokens=False)) ``` **Notes:** - Tokenization uses `add_special_tokens=False` to count only content tokens, excluding special tokens added by the tokenizer. - Does **not** differentiate `` tags or extract reasoning from within the regular `content` field. --- ### Total Reasoning Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all reasoning tokens generated across all requests. This represents the total reasoning/thinking workload. **Formula:** ```python total_reasoning_tokens = sum(r.reasoning_token_count for r in records if r.valid) ``` **Notes:** - Useful for understanding the reasoning overhead and cost for reasoning-enabled models. --- ## Speculative Decoding Metrics All metrics in this section read only the engine-neutral per-request [`SpecDecodeAcceptanceRecord`](/aiperf/reference/per-request-speculative-decoding-acceptance) attached to each record. They are fully engine-agnostic: any engine whose adapter fills the record lights them up unchanged, and none of them branch on the engine. When spec decode is off (or a request had no verify steps) the record is absent and every metric here drops out cleanly -- nothing is shown or exported. Acceptance length can be **concurrency / batch-size dependent for adaptive drafters** (DSpark, vLLM's dynamic speculative decoding), which tune the number of draft tokens per step -- or disable speculation -- as load rises. For fixed-length drafters (a fixed-`k` draft model, Medusa, classic or dynamic-tree EAGLE) the per-token acceptance rate is a property of the draft/target model pair and the context, so acceptance length is essentially load-invariant. Either way these metrics report at the run's offered load; when the drafter is adaptive, cross-run comparisons are only meaningful with concurrency held fixed. These metrics render in a dedicated `Spec Decode` console section (`console_group = MetricConsoleGroup.SPEC_DECODE`), followed by a one-line pooled acceptance histogram. Notation: `j` = accepted draft tokens in a step, `l` = proposed draft tokens in a step; the always-accepted bonus token adds the `+1`. ### Acceptance Length **Type:** [Record Metric](#record-metrics) · **Unit:** ratio · larger is better Per-request mean tokens emitted per verify step, including the always-accepted bonus token (the `j + 1` acceptance length). **Formula:** ```python spec_decode_acceptance_length = 1 + num_accepted_draft_tokens / num_spec_steps ``` **Notes:** - Averages per-request means equally. Compare against [Token-Weighted Acceptance Length](#token-weighted-acceptance-length), which weights by verify steps. The two can diverge when requests differ in how many verify steps they run and that step count correlates with per-request acceptance -- e.g. a few long, high-acceptance requests pull the token-weighted value above the equal-per-request mean. They coincide when every request runs a similar number of steps. --- ### Token-Weighted Acceptance Length **Type:** [Derived Metric](#derived-metrics) · **Unit:** ratio · larger is better Run-level acceptance length weighting every verify step equally rather than every request. **Formula:** ```python spec_decode_token_weighted_acceptance_length = 1 + sum(num_accepted_draft_tokens) / sum(num_spec_steps) ``` **Notes:** - Reported **alongside** the per-request-mean acceptance length and clearly labeled, because the two answer different questions (per-request vs per-step weighting). --- ### Draft Acceptance Rate **Type:** [Record Metric](#record-metrics) · **Unit:** % · larger is better Per-request fraction of proposed draft tokens that were accepted, as a percentage. Draft-only -- excludes the bonus token. **Formula:** ```python spec_decode_draft_acceptance_rate = 100 * num_accepted_draft_tokens / num_draft_tokens ``` --- ### Overall Draft Acceptance Rate **Type:** [Derived Metric](#derived-metrics) · **Unit:** % · larger is better Run-level (draft-volume-weighted) draft acceptance rate: summed accepted drafts over summed proposed drafts across the whole run, so large and small requests contribute in proportion to their draft volume. **Formula:** ```python spec_decode_overall_draft_acceptance_rate = 100 * sum(num_accepted_draft_tokens) / sum(num_draft_tokens) ``` --- ### Accepted per Verified **Type:** [Record Metric](#record-metrics) · **Unit:** ratio · larger is better Per-request emitted tokens over verified tokens: accepted drafts plus one bonus per step, over proposed drafts plus one bonus per step. **Formula:** ```python spec_decode_accepted_per_verified = (num_accepted_draft_tokens + num_spec_steps) / (num_draft_tokens + num_spec_steps) # = (j + 1) / (l + 1) ``` --- ### Spec Decode Steps **Type:** [Record Metric](#record-metrics) · **Unit:** count Per-request number of speculative verification steps (`num_spec_steps`). Equals the sum of the request's acceptance-histogram counts. The run-level `total_spec_decode_steps` is its sum across requests. --- ### Accepted / Draft Token Counts **Type:** [Record Metrics](#record-metrics) · **Unit:** tokens · `console_group = NONE` `spec_decode_accepted_draft_tokens` (`num_accepted_draft_tokens`) and `spec_decode_draft_tokens` (`num_draft_tokens`) travel per request and sum into `total_accepted_draft_tokens` / `total_draft_tokens`. They are exported to JSON/CSV/JSONL but hidden from the console to keep the section compact. --- ### Pooled Acceptance Histogram **Type:** dedicated reducer (outside the metric registry) The per-request `acceptance_histogram {j: steps}` pooled elementwise across the run into a single `{j: total_steps}` -- verify steps bucketed by accepted-draft count over the whole run. AIPerf's metric accumulator handles scalars and lists but not dicts, so this one aggregate is pooled by a dedicated per-phase reducer in the accumulator and rides on `ProfileResults.pooled_spec_decode_acceptance_histogram`. - **Console:** a one-line row beneath the Spec Decode table showing the % of steps per bucket, capped to buckets `0 .. 7` with any `j >= 8` folded into a trailing `>=8` bucket. - **Aggregate JSON** (`profile_export_aiperf.json`): the full `{j: count}` object under `pooled_spec_decode_acceptance_histogram` (no cap). - **CSV:** the scalar metrics carry the CSV; the structured histogram is JSON-only. **Reconciliation:** the pooled bucket counts sum to `total_spec_decode_steps`, which equals the sum of every request's `num_spec_steps`. --- ## Usage Field Metrics All metrics in this section track API-reported token counts from the `usage` field in API responses. These are **not displayed in console output** but are available in exports. These metrics are useful for comparing client-side token counts with server-reported counts to detect discrepancies. ### Usage Prompt Tokens **Type:** [Record Metric](#record-metrics) The number of input/prompt tokens as reported by the API's `usage.prompt_tokens` field for a single request. **Formula:** ```python usage_prompt_tokens = response.usage.prompt_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - May differ from client-side Input Sequence Length due to different tokenizers or special tokens. - For streaming responses, uses the last non-None value reported. --- ### Usage Completion Tokens **Type:** [Record Metric](#record-metrics) The number of completion tokens as reported by the API's `usage.completion_tokens` field for a single request. **Formula:** ```python usage_completion_tokens = response.usage.completion_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - May differ from client-side Output Sequence Length due to different tokenizers or counting methods. - For streaming responses, uses the last non-None value reported. --- ### Usage Total Tokens **Type:** [Record Metric](#record-metrics) The total number of tokens (prompt + completion) as reported by the API's `usage.total_tokens` field for a single request. **Formula:** ```python usage_total_tokens = response.usage.total_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - Should generally equal `usage_prompt_tokens + usage_completion_tokens`. - For streaming responses, uses the last non-None value reported. --- ### Usage Reasoning Tokens **Type:** [Record Metric](#record-metrics) The number of reasoning tokens as reported by the API's `usage.completion_tokens_details.reasoning_tokens` field for a single request. Only available for reasoning-enabled models. **Formula:** ```python usage_reasoning_tokens = response.usage.completion_tokens_details.reasoning_tokens ``` **Notes:** - Taken from the API response for reasoning-enabled models. - May differ from client-side Reasoning Token Count due to different tokenizers. - For streaming responses, uses the last non-None value reported. --- ### Usage Prompt Cache Read Tokens **Type:** [Record Metric](#record-metrics) The number of prompt tokens that were served from cache (cache hits) as reported by the API's `usage` field for a single request. **Formula:** ```python # OpenAI shape: nested under prompt_tokens_details usage_prompt_cache_read_tokens = response.usage.prompt_tokens_details.cached_tokens # from last non-None response # Anthropic shape: top-level usage_prompt_cache_read_tokens = response.usage.cache_read_input_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - OpenAI surfaces cache reads as `prompt_tokens_details.cached_tokens` (or `input_tokens_details.cached_tokens`); writes are transparent and not reported. - Anthropic surfaces cache reads at the top level as `cache_read_input_tokens`; writes are reported separately as [Usage Prompt Cache Write Tokens](#usage-prompt-cache-write-tokens). - **Self-hosted servers gate this behind a flag:** vLLM needs `--enable-prompt-tokens-details` and SGLang needs `--enable-cache-report` (both default off); TRT-LLM emits `cached_tokens` by default. Without the flag these metrics read all-None even when the server is caching — see [Vendor Usage Field Reference](/aiperf/reference/vendor-usage-field-reference). - For streaming responses, uses the last non-None value reported. --- ### Usage Prompt Cache Write Tokens **Type:** [Record Metric](#record-metrics) The number of prompt tokens written to cache (cache creations) as reported by the API's `usage.cache_creation_input_tokens` field for a single request. Anthropic-specific. **Formula:** ```python usage_prompt_cache_write_tokens = response.usage.cache_creation_input_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - Reported only by APIs that bill cache writes separately (Anthropic). OpenAI does not surface cache writes — they happen transparently and are not billed separately, so this metric is empty for OpenAI workloads. - Cache writes are typically billed at a premium relative to ordinary input tokens but enable cheap reads on subsequent requests, so the metric is intentionally not flagged "larger is better." - For streaming responses, uses the last non-None value reported. --- ### Usage Prompt Audio Tokens **Type:** [Record Metric](#record-metrics) The number of audio tokens from the prompt as reported by the API's `usage.prompt_tokens_details.audio_tokens` field for a single request. **Formula:** ```python usage_prompt_audio_tokens = response.usage.prompt_tokens_details.audio_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - Only available for audio-capable endpoints. - For streaming responses, uses the last non-None value reported. --- ### Usage Completion Audio Tokens **Type:** [Record Metric](#record-metrics) The number of audio tokens in the completion as reported by the API's `usage.completion_tokens_details.audio_tokens` field for a single request. **Formula:** ```python usage_completion_audio_tokens = response.usage.completion_tokens_details.audio_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - Only available for audio-capable endpoints. - For streaming responses, uses the last non-None value reported. --- ### Usage Accepted Prediction Tokens **Type:** [Record Metric](#record-metrics) The number of accepted prediction tokens as reported by the API's `usage.completion_tokens_details.accepted_prediction_tokens` field for a single request. These are tokens from a predicted completion that the model actually used. **Formula:** ```python usage_accepted_prediction_tokens = response.usage.completion_tokens_details.accepted_prediction_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - Only relevant when using predicted outputs (speculative decoding). - For streaming responses, uses the last non-None value reported. --- ### Usage Rejected Prediction Tokens **Type:** [Record Metric](#record-metrics) The number of rejected prediction tokens as reported by the API's `usage.completion_tokens_details.rejected_prediction_tokens` field for a single request. These are tokens from a predicted completion that the model did not use. **Formula:** ```python usage_rejected_prediction_tokens = response.usage.completion_tokens_details.rejected_prediction_tokens # from last non-None response ``` **Notes:** - Taken from the API response `usage` object, not computed by AIPerf. - Only relevant when using predicted outputs (speculative decoding). - For streaming responses, uses the last non-None value reported. --- ### Usage Prompt Cache Miss Tokens **Type:** [Record Metric](#record-metrics) The number of prompt tokens that *missed* cache (and required fresh processing) as reported by the API's `usage.prompt_cache_miss_tokens` field for a single request. **DeepSeek-specific.** **Formula:** ```python usage_prompt_cache_miss_tokens = response.usage.prompt_cache_miss_tokens # from last non-None response ``` **Notes:** - DeepSeek bills cache hits and misses at different rates and surfaces both as their own fields. Other vendors don't report a separate miss count (you can derive it from `prompt_tokens - prompt_cache_read_tokens`, but it's not its own first-class field). - Not flagged "larger is better" — misses are unhelpful (they're the part you didn't cache). - For streaming responses, uses the last non-None value reported. --- ### Usage Tool Use Prompt Tokens **Type:** [Record Metric](#record-metrics) The number of prompt tokens consumed by tool / function-call declarations sent in the request, separate from user-content prompt tokens. **Gemini-specific.** **Formula:** ```python # Gemini wraps usage in usageMetadata; the property reads through the envelope. usage_tool_use_prompt_tokens = response.usage.toolUsePromptTokenCount # from last non-None response ``` **Notes:** - Surfaces what fraction of input tokens are spent on function/tool definitions vs user content. Useful for tool-heavy agentic workloads. - Other vendors fold tool definitions into the regular `prompt_tokens` count, so this metric will raise `NoMetricValue` for OpenAI / Anthropic / etc. - For streaming responses, uses the last non-None value reported. ### Usage Prompt Audio Seconds **Type:** [Record Metric](#record-metrics) The audio duration of the input prompt in **seconds (not tokens)** as reported by the API's `usage.prompt_audio_seconds` field for a single request. **Mistral-specific.** **Formula:** ```python usage_prompt_audio_seconds = response.usage.prompt_audio_seconds # from last non-None response ``` **Notes:** - Distinct from [Usage Prompt Audio Tokens](#usage-prompt-audio-tokens) — this is a duration in seconds, not a token count. Both can coexist for frameworks that report both. - Returned as `float` (so `12.5s` is preserved exactly even when the API reports an integer). - For streaming responses, uses the last non-None value reported. --- ### Total Usage Prompt Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported prompt tokens across all requests. **Formula:** ```python total_usage_prompt_tokens = sum(r.usage_prompt_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported input tokens across all requests. --- ### Total Usage Completion Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported completion tokens across all requests. **Formula:** ```python total_usage_completion_tokens = sum(r.usage_completion_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported completion tokens across all requests. --- ### Total Usage Total Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported total tokens across all requests. **Formula:** ```python total_usage_total_tokens = sum(r.usage_total_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported total tokens across all requests. --- ### Total Usage Reasoning Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported reasoning tokens across all requests. **Formula:** ```python total_usage_reasoning_tokens = sum(r.usage_reasoning_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported reasoning tokens across all requests. --- ### Total Usage Prompt Cache Read Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported prompt cache-read tokens across all requests. **Formula:** ```python total_usage_prompt_cache_read_tokens = sum(r.usage_prompt_cache_read_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported cache-read prompt tokens across all requests (OpenAI `prompt_tokens_details.cached_tokens` or Anthropic top-level `cache_read_input_tokens`). --- ### Overall Usage Prompt Cache Read % **Type:** [Derived Metric](#derived-metrics) Run-aggregate share of input tokens served from prompt cache, weighted by token volume. Computed from the run totals so a request with 10k prompt tokens contributes 100x as much weight as a request with 100 prompt tokens — the resulting number reflects the actual fraction of input tokens the API served from cache across the whole benchmark. **Formula:** ```python overall_usage_prompt_cache_read_pct = ( total_usage_prompt_cache_read_tokens / total_usage_prompt_tokens ) * 100 ``` **Notes:** - No value is produced if `total_usage_prompt_tokens` is zero (e.g. all requests errored before reporting usage). --- ### Total Usage Prompt Cache Write Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported prompt cache-write (cache creation) tokens across all requests. Anthropic-specific. **Formula:** ```python total_usage_prompt_cache_write_tokens = sum(r.usage_prompt_cache_write_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported cache-write prompt tokens across all requests (Anthropic top-level `cache_creation_input_tokens`). Empty for OpenAI workloads. --- ### Total Usage Prompt Audio Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported prompt audio tokens across all requests. **Formula:** ```python total_usage_prompt_audio_tokens = sum(r.usage_prompt_audio_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported prompt audio tokens across all requests. --- ### Total Usage Completion Audio Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported completion audio tokens across all requests. **Formula:** ```python total_usage_completion_audio_tokens = sum(r.usage_completion_audio_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported completion audio tokens across all requests. --- ### Total Usage Accepted Prediction Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported accepted prediction tokens across all requests. **Formula:** ```python total_usage_accepted_prediction_tokens = sum(r.usage_accepted_prediction_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported accepted prediction tokens across all requests. --- ### Total Usage Rejected Prediction Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported rejected prediction tokens across all requests. **Formula:** ```python total_usage_rejected_prediction_tokens = sum(r.usage_rejected_prediction_tokens for r in records if r.valid) ``` **Notes:** - Aggregates server-reported rejected prediction tokens across all requests. --- ### Total Usage Prompt Cache Miss Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported prompt cache-miss tokens across all requests. **DeepSeek-specific.** **Formula:** ```python total_usage_prompt_cache_miss_tokens = sum(r.usage_prompt_cache_miss_tokens for r in records if r.valid) ``` **Notes:** - Aggregates DeepSeek's top-level `prompt_cache_miss_tokens` across all requests. Empty for vendors that don't surface a separate miss field. --- ### Total Usage Tool Use Prompt Tokens **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported tool-use prompt tokens across all requests. **Gemini-specific.** **Formula:** ```python total_usage_tool_use_prompt_tokens = sum(r.usage_tool_use_prompt_tokens for r in records if r.valid) ``` **Notes:** - Aggregates Gemini's `toolUsePromptTokenCount` across all requests. Useful for understanding what fraction of total prompt tokens were spent on tool/function declarations in tool-heavy agentic workloads. --- ### Total Usage Prompt Audio Seconds **Type:** [Derived Metric](#derived-metrics) The sum of all API-reported prompt audio durations across all requests, in **seconds (not tokens)**. **Mistral-specific.** **Formula:** ```python total_usage_prompt_audio_seconds = sum(r.usage_prompt_audio_seconds for r in records if r.valid) ``` **Notes:** - Aggregates Mistral's `prompt_audio_seconds`. Unit is seconds; do not confuse with [Total Usage Prompt Audio Tokens](#total-usage-prompt-audio-tokens). --- ## Usage Discrepancy Metrics These metrics measure the percentage difference between API-reported token counts (`usage` fields) and client-computed token counts. They are **not displayed in console output** but help identify tokenizer mismatches or counting discrepancies. ### Usage Prompt Diff % **Type:** [Record Metric](#record-metrics) The percentage difference between API-reported prompt tokens and client-computed Input Sequence Length. **Formula:** ```python usage_prompt_tokens_diff_pct = abs((usage_prompt_tokens - input_sequence_length) / input_sequence_length) * 100 ``` **Notes:** - Values close to 0% indicate good agreement between client and server token counts. - Large differences may indicate tokenizer mismatches or special token handling differences. --- ### Usage Completion Diff % **Type:** [Record Metric](#record-metrics) The percentage difference between API-reported completion tokens and client-computed Output Sequence Length. **Formula:** ```python usage_completion_tokens_diff_pct = abs((usage_completion_tokens - output_sequence_length) / output_sequence_length) * 100 ``` **Notes:** - Values close to 0% indicate good agreement between client and server token counts. - Large differences may indicate tokenizer mismatches or different counting methods. --- ### Usage Reasoning Diff % **Type:** [Record Metric](#record-metrics) The percentage difference between API-reported reasoning tokens and client-computed Reasoning Token Count. **Formula:** ```python usage_reasoning_tokens_diff_pct = abs((usage_reasoning_tokens - reasoning_token_count) / reasoning_token_count) * 100 ``` **Notes:** - Only available for reasoning-enabled models. - Values close to 0% indicate good agreement between client and server reasoning token counts. --- ### Usage Discrepancy Count **Type:** [Aggregate Metric](#aggregate-metrics) The number of requests where token count differences exceed a threshold (default 10%). **Formula:** ```python usage_discrepancy_count = sum(1 for r in records if r.any_diff > threshold) ``` **Notes:** - Default threshold is 10% difference. - Counts requests where prompt, completion, or reasoning token differences are significant. - Useful for monitoring overall token count agreement quality. --- ## OSL Mismatch Metrics These metrics measure the difference between requested output sequence length (`--osl`/`max_tokens`) and actual output tokens generated. They help identify when the server is not honoring the requested output length, typically because EOS tokens stop generation early. These metrics are **not displayed in console output** but are available in exports and used by the end-of-benchmark warning. ### OSL Mismatch Diff % **Type:** [Record Metric](#record-metrics) The signed percentage difference between actual output sequence length and requested OSL. Negative values mean the server stopped early (actual < requested), positive values mean it generated more than requested. **Formula:** ```python osl_mismatch_diff_pct = ((output_sequence_length - requested_osl) / requested_osl) * 100 ``` **Notes:** - Negative = stopped early (hit EOS before max_tokens) - Positive = generated more than requested - 0% = exact match between requested and actual - Example: Requested 100 tokens, got 50 → Diff = -50% - Example: Requested 100 tokens, got 120 → Diff = 20% --- ### OSL Mismatch Count **Type:** [Aggregate Metric](#aggregate-metrics) The count of requests where the absolute token difference exceeds the effective threshold. Used to trigger the end-of-benchmark warning panel. **Formula:** ```python # Effective threshold is capped to be tighter for large OSL values threshold_tokens = min(requested_osl * (pct_threshold / 100), max_token_threshold) diff_tokens = abs(actual_osl - requested_osl) osl_mismatch_count = sum(1 for r in records if diff_tokens > threshold_tokens) ``` **Notes:** - Default percentage threshold is 5% (`AIPERF_METRICS_OSL_MISMATCH_PCT_THRESHOLD`). - Default max token threshold is 50 (`AIPERF_METRICS_OSL_MISMATCH_MAX_TOKEN_THRESHOLD`). - The `min()` makes threshold tighter for large OSL: requesting 2000 tokens caps at 50 token diff instead of 100 (5%). - Counts both early stops (negative diff) and over-generation (positive diff). - When this count is non-zero, a warning panel is displayed at the end of the benchmark. - To ensure servers honor `--osl`, use `--extra-inputs ignore_eos:true` or `--extra-inputs min_tokens:`. - If discrepancy is due to tokenizer mismatch between client and server, use `--use-server-token-count`. **Server support for `min_tokens`:** | Server | Parameter | Notes | |--------|-----------|-------| | [vLLM](https://docs.vllm.ai/en/latest/api/vllm/sampling_params/) | `min_tokens` | Default: 0 | | [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html) | `min_tokens` | Default: 1 | | [SGLang](https://github.com/sgl-project/sglang) | `min_new_tokens` | Default: 0 | | [TGI](https://github.com/huggingface/text-generation-inference) | `min_new_tokens` | Unclear API support; TGI in maintenance mode | --- ## Replay Schedule Lag Metrics These metrics report how faithfully a fixed-schedule trace replay delivered its offered (recorded) request schedule. The whole family is `FIXED_SCHEDULE_ONLY`: it is computed only when the profiling phase type is `fixed_schedule`, and stays inactive under any other timing mode even when the dataset carries turn timestamps (e.g. a timestamped trace swept with `--no-fixed-schedule`). The metrics are **not displayed in console output** but are available in exports; a warning is logged when the run is flagged degraded. Note that an explicit `--fixed-schedule` flag is not required for the family to activate: a timestamped `--custom-dataset-type` trace (a top-level `timestamp` field in the first record of JSONL traces such as `mooncake_trace` and `bailian_trace`, or a `timestamp_start_unix_ms` column in `baseten_trace` Parquet) auto-promotes the profiling phase to `fixed_schedule`, so these metrics then appear in profile exports. Pass `--no-fixed-schedule` to keep the user-selected timing mode, which also deactivates the family. Every absolutely-scheduled turn carries its intended send time (`Turn.timestamp`, ms relative to the schedule zero), and the worker stamps `RequestRecord.timestamp_ns` (wall clock) when the request is dispatched to the transport. The wall-clock instant of the schedule zero is not recorded, so lag is reported **relative to the least-late request** of the run (`lag = offset - min(offset)`). The derived family is computed once over the whole run and is **excluded from `--slice-duration` timeslice exports**: re-deriving it per slice would re-anchor each slice at its own least-late request and erase the cumulative schedule drift these metrics exist to expose. **What this cannot measure:** - A constant delay applied uniformly to every request (the least-late request defines zero). - Lag of delay-scheduled continuation turns (back-pressure replay fires them relative to the prior turn's completion; they carry no absolute intended time and are excluded). - True wire time (`timestamp_ns` is stamped worker-side at transport dispatch, before the TCP write). ### Replay Send Schedule Offset **Type:** [Record Metric](#record-metrics) Internal building block: the raw per-request send offset. Values are epoch-scale nanoseconds; only differences between them are meaningful. **Formula:** ```python replay_send_schedule_offset = record.timestamp_ns - turn.timestamp * 1e6 ``` **Notes:** - `INTERNAL`: hidden from output unless `AIPERF_DEV_SHOW_INTERNAL_METRICS` is enabled. - Skipped for records whose dispatched turn has no absolute schedule timestamp. --- ### Replay Schedule Lag p50 / p90 / p99 **Type:** [Derived Metric](#derived-metrics) Percentiles of the anchored send lag across the run, in milliseconds. Schedule degradation (event-loop stalls, worker saturation, queue buildup) shows up as growing lag percentiles. **Formula:** ```python lag_ms = (offsets - offsets.min()) / 1e6 replay_sched_lag_p50 = percentile(lag_ms, 50) replay_sched_lag_p90 = percentile(lag_ms, 90) replay_sched_lag_p99 = percentile(lag_ms, 99) ``` --- ### Replay Schedule Degraded **Type:** [Derived Metric](#derived-metrics) Boolean (0/1) signal that the replay could not keep up with the offered schedule: anchored send-lag p99 exceeded 500 ms. When degraded, a warning with the lag percentiles is logged at most once per run (the first time the run is flagged); the metric value itself is re-derived on every summarize tick. **Formula:** ```python replay_sched_degraded = 1 if percentile(lag_ms, 99) > 500.0 else 0 ``` --- ## Agentic / Trace Metrics These metrics appear for agentic/WEKA trace replay workloads. Theoretical prefix-cache accounting requires a loader that stamps per-turn block counts (e.g. `weka_trace`). Context-overflow counting applies whenever the runtime classifier tags an error as context overflow. ### Theoretical Prefix Cache Hit **Type:** Accumulator summary (not a record/aggregate metric plugin) Ideal infinite-cache prefix hit rate implied by the trace's hash_id block overlap, as stamped by the WEKA loader. Emitted by `TheoreticalPrefixCacheAccumulator` on the `metric_records` channel. **Formula:** ```python theoretical_prefix_cache_hit = 100.0 * sum(hit_blocks) / sum(total_blocks) ``` **Export fields:** - `avg` / `current`: hit rate percent - `sum`: cumulative hit blocks (numerator) - `count`: cumulative total blocks (denominator) **Notes:** - Phase-scoped via `export_results(ctx)` so warmup blocks do not leak into profiling (and vice versa). - Only appears when at least one conversation turn carries both `theoretical_prefix_cache_hit_blocks` and `theoretical_prefix_cache_total_blocks`. --- ### Context Overflow Count **Type:** [Aggregate Metric](#aggregate-metrics) Number of requests whose error body was classified as a context-length / context-overflow rejection. Used by the InferenceX AgentX scenario to flip `submission_valid=false` when the overflow rate exceeds 1%. **Formula:** ```python context_overflow_count = sum(1 if request.context_overflow else 0) ``` **Notes:** - Marked `ERROR_ONLY` / `NO_INDIVIDUAL_RECORDS` (aggregate signal only). - Under `--scenario inferencex-agentx-mvp`, matching overflows may take the records-side skip path (`context_overflow_skip`) and land in the `skipped_context_overflow_count` side channel instead of normal error metrics; the aggregate exporter still merges both into the overflow rate denominator (`request_count + error_request_count + skipped_context_overflow_count`). --- ## Goodput Metrics Goodput metrics measure the throughput of requests that meet user-defined Service Level Objectives (SLOs). See the [Goodput tutorial](/aiperf/tutorials/metrics-analysis/benchmark-goodput-with-ai-perf) for configuration details. ### Good Request Count **Type:** [Aggregate Metric](#aggregate-metrics) The number of requests that meet all user-defined SLO thresholds during the benchmark. **Formula:** ```python good_request_count = sum(1 for r in records if r.all_slos_met) ``` **Notes:** - Requires SLO thresholds to be configured (e.g., `--goodput`). - Only counts requests where ALL SLO constraints are satisfied. - Used to calculate Goodput metric. --- ### Good Request Fraction **Type:** [Derived Metric](#derived-metrics) **Tag:** `good_request_fraction` The fraction of all attempted requests that satisfied every per-request SLO. Returns a ratio in `[0.0, 1.0]`. Errored requests count toward the denominator so a backend that drops traffic under load cannot look "good" simply because the surviving requests stayed under the latency budget. **Formula:** ```python attempted = request_count + error_request_count good_request_fraction = good_request_count / attempted if attempted > 0 else 0.0 ``` **Flags:** `GOODPUT | LARGER_IS_BETTER | NO_CONSOLE` **Unit:** `RATIO` (0.0–1.0) **Required upstream metrics:** None. Each counter may legitimately be absent: valid-request counters on an all-error run, and `error_request_count` on a clean run. **Notes:** - Requires SLO thresholds to be configured (e.g., `--goodput`); without SLOs, `good_request_count` is always 0 and this metric is 0. - Returns `0.0` when no requests were attempted (`request_count + error_request_count == 0`). - Hidden from console output (`NO_CONSOLE`); appears in JSON, CSV, and Parquet exports. - Powers the SLA-feasibility gate of the [`max-goodput-under-slo`](/aiperf/sweeping-adaptive-search/search-recipes) search recipe (`good_request_fraction:avg:ge:`); without it, the recipe filter dereferences a missing tag and Bayesian optimization treats every iteration as infeasible. --- ### Goodput **Type:** [Derived Metric](#derived-metrics) The rate of SLO-compliant requests per second. This represents the effective throughput of requests meeting quality requirements. **Formula:** ```python goodput = good_request_count / benchmark_duration_seconds ``` **Notes:** - Requires SLO thresholds to be configured. - Always less than or equal to Request Throughput. - Useful for capacity planning and comparing systems based on quality-adjusted throughput. --- ## Error Metrics These metrics are computed only for failed/error requests and are **not displayed in console output**. ### Error Input Sequence Length **Type:** [Record Metric](#record-metrics) The number of input tokens for requests that resulted in errors. This helps analyze whether input size correlates with errors. **Formula:** ```python error_isl = input_sequence_length # for error requests only ``` **Notes:** - Only computed for requests that failed. - Useful for identifying if certain input sizes trigger errors. --- ### Total Error Input Sequence Length **Type:** [Derived Metric](#derived-metrics) The sum of all input tokens from requests that resulted in errors. **Formula:** ```python total_error_isl = sum(r.error_isl for r in records if not r.valid) ``` **Notes:** - Aggregates input tokens across all failed requests. --- ## General Metrics Metrics in this section are available for all benchmark runs with no special requirements. ### Request Latency **Type:** [Record Metric](#record-metrics) Measures the total end-to-end time from sending a request until receiving the final response. For streaming requests with multiple responses, this measures until the last response is received. This is the complete time experienced by the client for a single request. **Formula:** ```python request_latency_ns = request.content_responses[-1].perf_ns - request.start_perf_ns ``` **Notes:** - Includes all components: network time, queuing, prompt processing, token generation, and response transmission. - For streaming requests, measures from request start to the final chunk received. --- ### Request Throughput **Type:** [Derived Metric](#derived-metrics) The overall rate of completed requests per second across the entire benchmark. This represents the system's ability to process requests under the given concurrency and load. **Formula:** ```python request_throughput = request_count / benchmark_duration_seconds ``` **Notes:** - Captures the aggregate request processing rate; higher values indicate better system throughput. - Affected by concurrency level, request complexity, output sequence length, and system capacity. --- ### Request Count **Type:** [Aggregate Metric](#aggregate-metrics) The total number of **successfully completed** requests in the benchmark. This includes all requests that received valid responses, regardless of streaming mode. **Formula:** ```python request_count = sum(1 for r in records if r.valid) ``` --- ### Error Request Count **Type:** [Aggregate Metric](#aggregate-metrics) The total number of failed/error requests encountered during the benchmark. This includes network errors, HTTP errors, timeout errors, and other failures. **Formula:** ```python error_request_count = sum(1 for r in records if not r.valid) ``` **Notes:** - Error rate can be computed as `error_request_count / (request_count + error_request_count)`. --- ### Request Error Rate **Type:** [Derived Metric](#derived-metrics) The percentage of completed requests that ended in error. **Formula:** ```python request_error_rate = 100.0 * error_request_count / (request_count + error_request_count) ``` **Notes:** - Omitted only when both successful and error counts are zero. --- ### Minimum Request Timestamp **Type:** [Aggregate Metric](#aggregate-metrics) The wall-clock timestamp of the first request sent in the benchmark. This is used to calculate the benchmark duration and represents the start of the benchmark run. **Formula:** ```python min_request_timestamp = min(r.timestamp_ns for r in records) ``` --- ### Maximum Response Timestamp **Type:** [Aggregate Metric](#aggregate-metrics) The wall-clock timestamp of the last response received in the benchmark. This is used to calculate the benchmark duration and represents the end of the benchmark run. **Formula:** ```python max_response_timestamp = max(r.timestamp_ns + r.request_latency for r in records) ``` --- ### Benchmark Duration **Type:** [Derived Metric](#derived-metrics) The total elapsed time from the first request sent to the last response received. This represents the complete wall-clock duration of the benchmark run. **Formula:** ```python benchmark_duration = max_response_timestamp - min_request_timestamp ``` **Notes:** - Uses wall-clock timestamps representing real calendar time. - Used as the denominator for throughput calculations; represents the effective measurement window. --- ## Network Latency Calibration Metrics These metrics appear only when network latency calibration is enabled (with `--network-latency-automatic` or `--network-latency-mean`). They make benchmark runs taken from different client network locations comparable by removing the client-to-endpoint network round-trip from the reported latencies. With `--network-latency-automatic`, AIPerf opens a fresh TCP connection to the endpoint host:port on an interval throughout profiling (`--network-latency-ping-interval`, default 1s) and records the handshake RTT. The handshake is one network round-trip with no server compute, so it isolates the network leg and is immune to server load. Each probe is written to `profile_export_network_latency.jsonl`. The **mean** RTT over successful probes is then subtracted from the request-start-anchored latency metrics. Alternatively, a fixed mean RTT can be supplied with `--network-latency-mean ` to skip probing (the two flags are mutually exclusive). The chosen RTT is logged at run completion. Subtracting a constant from a distribution shifts the mean and every percentile by that constant and leaves the standard deviation unchanged, so the adjustment is applied to the aggregated results (clamped at 0). The raw metrics are always preserved. ### Network RTT **Type:** [Derived Metric](#derived-metrics) The mean network RTT that was subtracted from the adjusted latency metrics (single value). **Notes:** - Measured via TCP handshakes throughout the run, or supplied directly via `--network-latency-mean`. - Full per-probe samples and per-target statistics are written to `profile_export_network_latency.jsonl`. --- ### Network-Adjusted Latency Metrics **Type:** [Derived Metric](#derived-metrics) Non-destructive, RTT-corrected variants of the request-start-anchored latency metrics: | Metric | Tag | Source metric | |---|---|---| | Network-Adjusted Request Latency | `network_adjusted_request_latency` | Request Latency | | Network-Adjusted Time to First Token | `network_adjusted_time_to_first_token` | Time to First Token (TTFT) | | Network-Adjusted Time to First Output Token | `network_adjusted_time_to_first_output_token` | Time to First Output Token (TTFO) | **Formula:** ```python # nanoseconds, applied to every aggregated statistic (avg, min, max, p1..p99) network_adjusted_ns = max(0, source_metric_ns - mean_network_rtt_ns) ``` **Notes:** - Only metrics whose interval starts at the client request-send timestamp are adjusted. Time to Second Token (second_response - first_response), Inter Token Latency (ITL), and Inter Chunk Latency (ICL) are intentionally **not** adjusted: they are intra-stream gaps that do not include the connection RTT (and for ITL the RTT cancels in `(request_latency - ttft)`), so they are already network-invariant. - Values are clamped at 0; if the subtracted RTT exceeds some measured latencies a warning is logged (the RTT may be larger than the true network leg). --- ## HTTP Trace Metrics All metrics in this section require HTTP trace data to be collected during requests. These metrics provide detailed HTTP request lifecycle timing following k6 naming conventions. See the [HTTP Trace Metrics tutorial](/aiperf/tutorials/metrics-analysis/http-trace-metrics-guide) for configuration details. ### HTTP Blocked **Type:** [Record Metric](#record-metrics) Time spent blocked waiting for a free TCP connection slot from the pool. This metric measures the time a request spent waiting in the connection pool queue before a connection became available. High values indicate connection pool saturation. **Formula:** ```python http_req_blocked = connection_pool_wait_end_perf_ns - connection_pool_wait_start_perf_ns ``` **Notes:** - k6 equivalent: `http_req_blocked` - HAR equivalent: `blocked` - Returns 0 if no pool wait occurred (connection immediately available). - Only available for AioHttpTraceData. --- ### HTTP DNS Lookup **Type:** [Record Metric](#record-metrics) Time spent on DNS resolution. This metric measures the time spent resolving the hostname to an IP address. **Formula:** ```python http_req_dns_lookup = dns_lookup_end_perf_ns - dns_lookup_start_perf_ns ``` **Notes:** - k6 equivalent: `http_req_looking_up` - HAR equivalent: `dns` - Returns 0 if DNS cache hit or connection was reused. - Only available for AioHttpTraceData. --- ### HTTP Connecting **Type:** [Record Metric](#record-metrics) Time spent establishing TCP connection to the remote host. For HTTPS requests, this includes both TCP connection establishment and TLS handshake time (combined measurement from aiohttp). **Formula:** ```python http_req_connecting = tcp_connect_end_perf_ns - tcp_connect_start_perf_ns ``` **Notes:** - k6 equivalent: `http_req_connecting` - HAR equivalent: `connect` - Returns 0 if connection was reused. - Only available for AioHttpTraceData. --- ### HTTP Sending **Type:** [Record Metric](#record-metrics) Time spent sending data to the remote host. This metric measures the time from when the request started being sent to when the full request (headers + body) was transmitted. **Formula:** ```python http_req_sending = request_send_end_perf_ns - request_send_start_perf_ns ``` **Notes:** - k6 equivalent: `http_req_sending` - HAR equivalent: `send` --- ### HTTP Waiting (TTFB) **Type:** [Record Metric](#record-metrics) Time to First Byte (TTFB) - time waiting for the server to respond. This metric measures the time from when the request was fully sent to when the first byte of the response body was received. This represents server processing time plus network latency. **Formula:** ```python http_req_waiting = response_receive_start_perf_ns - request_send_end_perf_ns ``` **Notes:** - k6 equivalent: `http_req_waiting` (also known as TTFB) - HAR equivalent: `wait` - Note that this is not the same as the time to first token (TTFT), which is the time from request start to the first valid token received. The server may send non-token data first. --- ### HTTP Receiving **Type:** [Record Metric](#record-metrics) Time spent receiving response data from the remote host. This metric measures the time from when the first byte of the response was received to when the last byte was received. **Formula:** ```python http_req_receiving = response_receive_end_perf_ns - response_receive_start_perf_ns ``` **Notes:** - k6 equivalent: `http_req_receiving` - HAR equivalent: `receive` - Returns 0 if response was a single chunk. --- ### HTTP Duration (excl. conn) **Type:** [Record Metric](#record-metrics) Time for HTTP request/response exchange, excluding connection overhead. This measures only the request/response exchange time: `sending + waiting + receiving`. **Formula:** ```python http_req_duration = response_receive_end_perf_ns - request_send_start_perf_ns ``` **Notes:** - k6 equivalent: `http_req_duration` - HAR equivalent: `time` - EXCLUDES connection overhead (blocked, dns_lookup, connecting). - For full end-to-end time including connection setup, use `http_req_total`. - Note: This uses trace-level timestamps for more accurate measurement than application-level request latency. --- ### HTTP Connection Overhead **Type:** [Record Metric](#record-metrics) Total connection overhead time (blocked + dns_lookup + connecting). This metric combines all pre-request overhead. **Formula:** ```python http_req_connection_overhead = http_req_blocked + http_req_dns_lookup + http_req_connecting ``` **Notes:** - Useful for identifying total connection establishment costs. - Returns 0 if connection was reused with no pool wait. - Only available for AioHttpTraceData. --- ### HTTP Total Time **Type:** [Record Metric](#record-metrics) Sum of all HTTP timing phases from connection pool to last chunk received. This is the sum of all 6 timing components: `blocked + dns_lookup + connecting + sending + waiting + receiving`. **Formula:** ```python http_req_total = http_req_blocked + http_req_dns_lookup + http_req_connecting + http_req_sending + http_req_waiting + http_req_receiving ``` **Notes:** - This ensures the math adds up: individual timing metrics sum exactly to this total. - Only available for AioHttpTraceData (requires connection overhead metrics). --- ### HTTP Data Sent **Type:** [Record Metric](#record-metrics) Total bytes sent in the HTTP request (headers + body). **Formula:** ```python http_req_data_sent = trace.request_bytes_total ``` **Notes:** - k6 equivalent: `data_sent` (per request) - Measures total bytes written to the transport layer. --- ### HTTP Data Received **Type:** [Record Metric](#record-metrics) Total bytes received in the HTTP response (headers + body). **Formula:** ```python http_req_data_received = trace.response_bytes_total ``` **Notes:** - k6 equivalent: `data_received` (per request) - Measures total bytes read from the transport layer. --- ### HTTP Connection Reused **Type:** [Record Metric](#record-metrics) Whether the HTTP connection was reused from the connection pool. Returns 1 if reused, 0 if new connection was established. **Formula:** ```python http_req_connection_reused = 1 if connection_reused_perf_ns is not None else 0 ``` **Notes:** - Helps identify connection reuse patterns and keep-alive effectiveness. - Only available for AioHttpTraceData. --- ### HTTP Chunks Sent **Type:** [Record Metric](#record-metrics) Number of transport-level write operations during the request. Useful for debugging chunked transfers. **Formula:** ```python http_req_chunks_sent = trace.request_chunks_count ``` **Notes:** - Not displayed in console output (`console_group = MetricConsoleGroup.NONE`). --- ### HTTP Chunks Received **Type:** [Record Metric](#record-metrics) Number of transport-level read operations during the response. Useful for debugging chunked/streaming responses. **Formula:** ```python http_req_chunks_received = trace.response_chunks_count ``` **Notes:** - Not displayed in console output (`console_group = MetricConsoleGroup.NONE`). --- ## GPU Power Efficiency Metrics All metrics in this section require `--gpu-telemetry` to be enabled and the underlying collector to expose the relevant signal. They are computed **per vendor**, once per profiling phase, by the `EnergyEfficiencyAnalyzer` — an `analyzer` plugin that joins the GPU-telemetry accumulator (energy/power) to the metrics-accumulator summary (tokens/throughput/latency) via the `SummaryContext` — not by the standard derivation walk. See the [Externally-Injected Derived Metric pattern](/aiperf/architecture-internals/ai-perf-code-patterns#externally-injected-derived-metric-pattern) and the `analyzer` plugin category. Each metric exists in an NVIDIA variant (tag prefix `nvidia_`, from `nvidia_power_usage` / `nvidia_energy_consumption`, populated by the DCGM and pynvml collectors) and an AMD variant (tag prefix `amd_`, from `amd_power` / `amd_energy_consumption`, populated by the amdsmi collector). A mixed NVIDIA+AMD run reports both, each summing only its own vendor's GPUs. Each vendor's metrics render in their own console section — `GPU Power Efficiency (NVIDIA)` (`console_group = MetricConsoleGroup.GPU_POWER_EFFICIENCY_NVIDIA`) and `GPU Power Efficiency (AMD)` (`MetricConsoleGroup.GPU_POWER_EFFICIENCY_AMD`) — separate from the main metrics table. A section is omitted entirely when no GPU of that vendor reported. **Energy source.** Per-vendor total GPU energy is taken from the vendor's energy counter delta when available (`source = dcgm_counter` for NVIDIA; `source = power_integration` fallback for amdsmi when only the power gauge is exposed), and the time-averaged fleet power is derived from it as `total_gpu_energy / profiling_duration_s`. If neither signal is present the family is skipped for that vendor. The family is skipped entirely when GPU telemetry is not collected. Each tag is independently omitted when its underlying signal is unavailable (e.g. `nvidia_energy_per_user` only appears for concurrency runs; `nvidia_goodput_per_watt` only when goodput SLOs are configured). NVIDIA tags: `nvidia_energy_delay_product`, `nvidia_performance_per_watt`, `nvidia_output_tps_per_watt`, `nvidia_goodput_per_watt`, `nvidia_average_gpu_power`, `nvidia_total_gpu_energy`, `nvidia_total_gpu_power`, `nvidia_energy_per_total_token`, `nvidia_energy_per_output_token`, `nvidia_energy_per_request`, `nvidia_output_tokens_per_joule`, `nvidia_energy_per_user`. AMD tags follow the same pattern with the `amd_` prefix. ### Total GPU Power **Type:** [Derived Metric](#derived-metrics) (externally injected) Sum of average GPU power across all reporting GPUs during the profiling phase, in watts. Useful as a baseline for cross-run power comparisons. **Formula:** ```python # Vendor source fields: # NVIDIA: nvidia_power_usage (pynvml / DCGM collectors) # AMD: amd_power (amdsmi collector) # # Per GPU: average of the vendor power gauge samples in the profiling window # (warmup excluded). Summed across all GPUs that reported valid samples. total_gpu_power_w = sum( avg(vendor_power_field[start_ns:end_ns]) for gpu in reporting_gpus ) ``` **Notes:** - Unit: watts (`W`). - Time-filtered to the profiling-phase window; warmup samples are excluded. - Power is a gauge, so the window stays bounded — post-bench idle samples don't drag the average down. - Omitted when no GPU reports `nvidia_power_usage` (NVIDIA) or `amd_power` (AMD) in the window. --- ### Total GPU Energy **Type:** [Derived Metric](#derived-metrics) (externally injected) Sum of energy consumed across all reporting GPUs during the profiling phase, in joules. Computed as a counter delta (`final − baseline`) per GPU and summed. **Formula:** ```python # Vendor source fields: # NVIDIA: nvidia_energy_consumption (pynvml / DCGM collectors, in megajoules) # AMD: amd_energy_consumption (amdsmi collector, in megajoules) # # Per GPU: delta of the vendor energy monotonic counter over the profiling # window, widened on the end by FINAL_SCRAPE_GRACE_NS so the trailing scrape # that lands just after requests_end_ns is captured. grace_ns = Environment.GPU.FINAL_SCRAPE_GRACE_NS # default 666_000_000 (~666 ms) total_gpu_energy_j = sum( delta(vendor_energy_field[start_ns : end_ns + grace_ns]) for gpu in reporting_gpus ) # Negative deltas are clamped to 0 to handle counter resets (DCGM restart). ``` **Notes:** - Unit: joules (`J`). Source samples are reported in megajoules and converted via `EnergyMetricUnit.MEGAJOULE.joules`. - The end-of-window grace is bounded (not open-ended) so cooldown samples and any subsequent-phase samples cannot leak into the delta. Tune via `AIPERF_GPU_FINAL_SCRAPE_GRACE_NS` if you also tune `AIPERF_GPU_COLLECTION_INTERVAL` — keep grace at roughly `2x` the collection cadence. - Per-GPU deltas use the nearest non-NaN baseline and the nearest non-NaN final sample; arrays containing transient NaN sensor failures still yield a meaningful delta. - Omitted when no GPU reports `nvidia_energy_consumption` (NVIDIA) or `amd_energy_consumption` (AMD) in the window. --- ### Output Tokens per Joule **Type:** [Derived Metric](#derived-metrics) (externally injected) Inference energy efficiency: number of output tokens produced per joule of GPU energy consumed during the profiling phase. Higher is better. **Formula:** ```python output_tokens_per_joule = total_osl / total_gpu_energy ``` **Notes:** - Unit: `tokens/J`. - Flagged `LARGER_IS_BETTER | PRODUCES_TOKENS_ONLY`. - Numerator is total output tokens (`total_osl` from the metrics summary); denominator comes from the GPU telemetry counter delta (or integrated power) above. - Omitted when `total_osl` is absent from the summary or aggregate `total_gpu_energy` is zero. --- ### Energy per User **Type:** [Derived Metric](#derived-metrics) (externally injected) Per-user energy footprint during the profiling phase: total GPU energy consumed divided by the configured concurrency. Lower is better — a more efficient deployment serves the same load for less energy per concurrent user. **Formula:** ```python # concurrency from the resolved profiling phase config # (run.cfg.get_profiling_phases()[0].concurrency). energy_per_user_j = total_gpu_energy / concurrency ``` **Notes:** - Unit: `joules/user`. - Flagged `MetricFlags.NONE` — smaller-is-better is the default for unflagged metrics. - Denominator is the profiling phase's configured `concurrency` (`run.cfg.get_profiling_phases()[0].concurrency`). The resolver defaults this to `1` when `--concurrency` isn't specified in concurrency-mode runs, so the metric is emitted in the common case. - Omitted when concurrency is unset (e.g. pure `--request-rate` mode) or aggregate GPU energy is unavailable. --- ### Average GPU Power **Type:** [Derived Metric](#derived-metrics) (externally injected) Time-averaged total GPU power over the profiling window, in watts (`W`) — the fleet power that actually produced `total_gpu_energy`. Under the DCGM counter path this is derived from energy; under power integration it equals `total_gpu_power`. **Formula:** ```python # dcgm_counter source: average_gpu_power_w = total_gpu_energy / profiling_duration_s # power_integration source: average_gpu_power_w = total_gpu_power ``` --- ### Energy per Output Token **Type:** [Derived Metric](#derived-metrics) (externally injected) The primary energy-efficiency metric: GPU energy spent per output token, in millijoules per token (`mJ/token`). Lower is better. Normalizes across model sizes, hardware, and batch sizes. **Formula:** ```python energy_per_output_token_mj = total_gpu_energy * 1000 / total_osl ``` --- ### Energy per Total Token **Type:** [Derived Metric](#derived-metrics) (externally injected) GPU energy per total (input + output) token, in `mJ/token`. Captures combined prefill + decode cost; useful when comparing systems with different input/output ratios. **Formula:** ```python energy_per_total_token_mj = total_gpu_energy * 1000 / (total_isl + total_osl) ``` --- ### Energy per Request **Type:** [Derived Metric](#derived-metrics) (externally injected) GPU energy consumed per request, in joules per request (`joules/request`). **Formula:** ```python energy_per_request_j = total_gpu_energy / request_count ``` --- ### Performance per Watt **Type:** [Derived Metric](#derived-metrics) (externally injected) Request throughput per watt of GPU power draw, in `requests/sec/W`. Higher is better — the standard HPC/MLPerf efficiency metric, enabling apples-to-apples comparison across GPU SKUs and power envelopes. **Formula:** ```python performance_per_watt = request_throughput / average_gpu_power ``` --- ### Output Tokens per Second per Watt **Type:** [Derived Metric](#derived-metrics) (externally injected) Output-token throughput per watt of GPU power draw, in `tokens/sec/W`. Higher is better — the token-level companion to `performance_per_watt`, insensitive to request size. **Formula:** ```python output_tps_per_watt = output_token_throughput / average_gpu_power ``` **Notes:** - Flagged `LARGER_IS_BETTER | PRODUCES_TOKENS_ONLY`. - Omitted when `output_token_throughput` is absent or average power is zero. --- ### Goodput per Watt **Type:** [Derived Metric](#derived-metrics) (externally injected) SLO-passing request throughput (goodput) per watt of GPU power draw, in `good-req/s/W`. Higher is better — measures efficient work that actually met its latency SLOs, not raw throughput. **Formula:** ```python goodput_per_watt = goodput / average_gpu_power ``` **Notes:** - Flagged `LARGER_IS_BETTER`. - Only meaningful when goodput SLOs are configured (`--goodput`); omitted otherwise or when average power is zero. --- ### Energy Delay Product **Type:** [Derived Metric](#derived-metrics) (externally injected) Joint energy-and-latency figure of merit, in joule-seconds (`J*s`). Lower is better — penalizes both slow-but-efficient and fast-but-wasteful configurations, useful for finding the Pareto-optimal operating point. **Formula:** ```python energy_delay_product = energy_per_request * mean_request_latency_s ``` --- ## Multi-Run Aggregate Metrics These metrics are only available when using `--num-profile-runs > 1` for confidence reporting. When running multiple profile iterations with `--num-profile-runs`, AIPerf computes aggregate statistics across all runs to quantify measurement variance and repeatability. These statistics are written to `aggregate/profile_export_aiperf_aggregate.json` and `aggregate/profile_export_aiperf_aggregate.csv`. For detailed information about aggregate statistics, their mathematical definitions, and interpretation guidelines, see the [Multi-Run Confidence Tutorial](/aiperf/tutorials/metrics-analysis/multi-run-confidence-reporting). ### Quick Reference The following aggregate statistics are computed for each metric: - **mean**: Average value across all runs - **std**: Standard deviation (measure of spread) - **min**: Minimum value observed - **max**: Maximum value observed - **cv**: Coefficient of Variation (normalized variability) - **se**: Standard Error (uncertainty in the mean) - **ci_low, ci_high**: Confidence interval bounds - **t_critical**: t-distribution critical value used ### Aggregate Metadata The aggregate output also includes metadata about the multi-run benchmark: - **aggregation_type**: Always "confidence" for multi-run confidence reporting - **num_profile_runs**: Total number of runs requested - **num_successful_runs**: Number of runs that completed successfully - **failed_runs**: List of failed runs with error details - **confidence_level**: Confidence level used for intervals (e.g., 0.95) - **cooldown_seconds**: Cooldown duration between runs - **run_labels**: Labels for each run (e.g., ["trial_0001", "trial_0002", ...]) --- # Metric Flags Reference Metric flags are used to control when and how metrics are computed, displayed, and grouped. Flags can be combined using bitwise operations to create composite behaviors. ## Individual Flags | Flag | Description | Impact | |------|-------------|--------| | `NONE` | No flags set | Metric has default behavior with no special restrictions | | `STREAMING_ONLY` | Only computed for streaming responses | Requires Server-Sent Events (SSE) with multiple response chunks; skipped for non-streaming requests | | `ERROR_ONLY` | Only computed for error requests | Tracks error-specific information; computed only for invalid/failed requests | | `PRODUCES_TOKENS_ONLY` | Only computed for token-producing endpoints | Requires endpoints that return text/token content; skipped for embeddings and non-generative endpoints | | `LARGER_IS_BETTER` | Higher values indicate better performance | Used for throughput and count metrics to indicate optimization direction | | `INTERNAL` | Internal AIPerf metric | Used for AIPerf system diagnostics; not displayed in console or exported without developer mode | | `SUPPORTS_AUDIO_ONLY` | Only computed for audio endpoints | Requires audio-capable endpoints; skipped for other endpoint types | | `SUPPORTS_IMAGE_ONLY` | Only computed for image endpoints | Requires image-capable endpoints; skipped for other endpoint types | | `SUPPORTS_REASONING` | Requires reasoning token support | Only available for models and endpoints that expose reasoning content in separate fields | | `EXPERIMENTAL` | Experimental/unstable metric | May change or be removed in future releases; not displayed in console or exported without developer mode | | `GOODPUT` | Only computed when goodput is enabled | Requires SLO thresholds to be configured (e.g., `--goodput`); skipped otherwise | | `NO_INDIVIDUAL_RECORDS` | Not exported for individual records | Aggregate metrics not relevant to individual records (e.g., request count, min/max timestamps); excluded from per-record exports | | `TOKENIZES_INPUT_ONLY` | Only computed when endpoint tokenizes input | Requires endpoints that process and tokenize input text; skipped for non-text endpoints | | `HTTP_TRACE_ONLY` | Only computed when HTTP trace data is available | Requires HTTP request tracing to be enabled; provides detailed HTTP lifecycle timing metrics | | `SUPPORTS_VIDEO_ONLY` | Only computed for video endpoints | Requires video-capable endpoints; skipped for other endpoint types | | `USAGE_DIFF_ONLY` | Only computed when usage field data is available | Requires API responses to include usage field with token counts for comparison with client-computed values | | `PRODUCES_VIDEO_ONLY` | Only computed for video-producing endpoints | Requires endpoints that produce video output (e.g., SGLang video generation) | ## Composite Flags These flags are combinations of multiple individual flags for convenience: | Flag | Composition | Description | |------|-------------|-------------| | `STREAMING_TOKENS_ONLY` | `STREAMING_ONLY` + `PRODUCES_TOKENS_ONLY` | Requires both streaming support and token-producing endpoints | --- # Metric Console Group Reference The `console_group` class attribute on a metric controls which console table the metric appears in (or hides it entirely). It is independent of [`MetricFlags`](#metric-flags-reference) — flags filter by axis (`ERROR_ONLY`, `INTERNAL`, `EXPERIMENTAL`); `console_group` selects a display bucket. | Group | Description | |-------|-------------| | `MetricConsoleGroup.NONE` | Hidden from console; still exported to JSON/CSV/JSONL. Replaces the legacy `NO_CONSOLE` flag. | | `MetricConsoleGroup.DEFAULT` | Standard `LLM Metrics` table. Default for new metrics. | | `MetricConsoleGroup.USAGE` | API-reported usage token metrics (prompt/completion/total). Rendered as `LLM Metrics: Usage`. | | `MetricConsoleGroup.CACHE` | Cache-related token metrics (e.g. prompt cache hits). | | `MetricConsoleGroup.PREDICTION` | Speculative prediction token metrics (accepted/rejected). | | `MetricConsoleGroup.AUDIO` | Audio token metrics (prompt/completion). | | `MetricConsoleGroup.REASONING` | Reasoning token metrics. | | `MetricConsoleGroup.SPEC_DECODE` | Speculative-decoding acceptance metrics (acceptance length, draft acceptance rate, accepted-per-verified, spec-decode steps). Rendered in a dedicated `Spec Decode` section, followed by a one-line pooled accepted-draft histogram from a separate exporter. | | `MetricConsoleGroup.GPU_POWER_EFFICIENCY_NVIDIA` | NVIDIA cross-GPU power efficiency totals (`nvidia_total_gpu_power`, `nvidia_total_gpu_energy`, `nvidia_output_tokens_per_joule`, `nvidia_energy_per_user`). Rendered in a dedicated `GPU Power Efficiency (NVIDIA)` section instead of the main table. | | `MetricConsoleGroup.GPU_POWER_EFFICIENCY_AMD` | AMD cross-GPU power efficiency totals (`amd_total_gpu_power`, `amd_total_gpu_energy`, `amd_output_tokens_per_joule`, `amd_energy_per_user`). Rendered in a dedicated `GPU Power Efficiency (AMD)` section instead of the main table. | Set as a class attribute on a `BaseMetric` subclass: ```python class MyUsageMetric(BaseRecordMetric[int]): tag = "my_usage_metric" console_group = MetricConsoleGroup.USAGE ``` --- ## Timing Namespace (`aiperf.timing.*`) The `TimingResultsStrategy` emits phase-level timing snapshots as OTel counters and up-down-counters under the `aiperf.timing.*` namespace. These metrics track credit-phase progression in real time and are sourced from `CreditPhaseStats` fields. ### Counters | Metric Name | OTel Instrument | Unit | Description | `CreditPhaseStats` Field | Requirement | |---|---|---|---|---|---| | `aiperf.timing.requests.sent` | Counter | `1` | Total requests dispatched in this phase | `requests_sent` | 13.2 | | `aiperf.timing.requests.completed` | Counter | `1` | Requests that received a complete response | `requests_completed` | 13.2 | | `aiperf.timing.requests.cancelled` | Counter | `1` | Requests cancelled before completion | `requests_cancelled` | 13.2 | | `aiperf.timing.requests.errors` | Counter | `1` | Requests that ended in error | `request_errors` | 13.2 | | `aiperf.timing.sessions.sent` | Counter | `1` | Sessions initiated in this phase | `sent_sessions` | 13.2 | | `aiperf.timing.sessions.completed` | Counter | `1` | Sessions that finished all turns | `completed_sessions` | 13.2 | | `aiperf.timing.sessions.cancelled` | Counter | `1` | Sessions cancelled before completion | `cancelled_sessions` | 13.2 | | `aiperf.timing.sessions.turns_total` | Counter | `1` | Cumulative session turns executed | `total_session_turns` | 13.2 | ### Up-Down-Counters (Gauges) | Metric Name | OTel Instrument | Unit | Description | `CreditPhaseStats` Field | Requirement | |---|---|---|---|---|---| | `aiperf.timing.requests.in_flight` | UpDownCounter | `1` | Requests currently awaiting a response | `in_flight_requests` | 13.2 | | `aiperf.timing.sessions.in_flight` | UpDownCounter | `1` | Sessions with at least one turn in progress | `in_flight_sessions` | 13.2 | | `aiperf.timing.phase.timeout_triggered` | UpDownCounter | `1` | Whether the phase hard-timeout fired (0 or 1) | `timeout_triggered` | 13.2 | | `aiperf.timing.phase.grace_timeout_triggered` | UpDownCounter | `1` | Whether the grace-period timeout fired (0 or 1) | `grace_period_timeout_triggered` | 13.2 | | `aiperf.timing.phase.was_cancelled` | UpDownCounter | `1` | Whether the phase was user-cancelled (0 or 1) | `was_cancelled` | 13.2 | | `aiperf.timing.phase.elapsed_sec` | UpDownCounter | `s` | Wall-clock seconds elapsed in the phase | `requests_elapsed_time` | 13.2 | **Notes:** - All timing metrics carry the three GenAI spec Required attributes (`gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`) so they can be joined with spec-named request metrics in dashboards. - Counter metrics emit deltas (current - previous snapshot) and skip zero-delta updates. - Up-down-counter metrics emit the signed difference from the previous snapshot and skip near-zero (< 1e-9) deltas. --- ## OpenTelemetry GenAI Semantic Convention Mapping AIPerf translates its internal metric names onto the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics/) so that downstream dashboards and alerting can consume spec-standard metric names directly. ### Metric Name Rename Table | AIPerf Source | GenAI Spec Metric | Unit | Instrument | |---|---|---|---| | `request_latency` | `gen_ai.client.operation.duration` | s | Histogram | | `time_to_first_token` | `gen_ai.client.operation.time_to_first_chunk` | s | Histogram | | `inter_token_latency` | `gen_ai.client.operation.time_per_output_chunk` | s | Histogram | | `input_token_count` + `output_token_count` (merged) | `gen_ai.client.token.usage` with `gen_ai.token.type=input\|output` | {token} | Histogram | Duration metrics are converted from nanoseconds to seconds. Token counts use the identity conversion. ### `gen_ai.operation.name` Mapping Derived from the AIPerf `endpoint.type` configuration value: | AIPerf `endpoint.type` | `gen_ai.operation.name` | |---|---| | `chat` | `chat` | | `completions` | `text_completion` | | `embeddings` | `embeddings` | | anything else | `chat` (fallback) | ### `gen_ai.provider.name` Host Auto-Inference The provider attribute is resolved using the following precedence: 1. Explicit `--gen-ai-provider` CLI override (highest priority) 2. Host pattern inference from the endpoint URL (see table below) 3. `_OTHER` fallback | URL Host Pattern | Provider Value | |---|---| | `api.openai.com` | `openai` | | `api.anthropic.com` | `anthropic` | | `api.deepseek.com` | `deepseek` | | `api.mistral.ai` | `mistral_ai` | | `api.cohere.ai` / `api.cohere.com` | `cohere` | | `api.x.ai` | `x_ai` | | `api.groq.com` | `groq` | | `api.perplexity.ai` | `perplexity` | | `generativelanguage.googleapis.com` | `gcp.gemini` | | `*-aiplatform.googleapis.com` | `gcp.vertex_ai` | | `bedrock-runtime.*.amazonaws.com` | `aws.bedrock` | | `*.openai.azure.com` | `azure.ai.openai` | | `*.services.ai.azure.com` | `azure.ai.inference` | | `*.ibm.com` (with Watsonx paths) | `ibm.watsonx.ai` | | anything else | `_OTHER` | ### `error.type` Classification Error conditions on individual requests are classified into spec-standard `error.type` attribute values: | AIPerf Condition | `error.type` Value | |---|---| | asyncio/HTTP timeout | `timeout` | | HTTP 5xx response | `http_5xx` | | HTTP 4xx response | `http_4xx` | | JSON parse error | `parse_error` | | User-initiated cancel | `cancelled` | | anything else | `_OTHER` | The `error.type` attribute is only attached when an error is present; successful requests omit it entirely. ### Timing Namespace and GenAI Spec Interoperability The `aiperf.timing.*` metrics retain AIPerf-specific names because the GenAI semantic convention specification has no equivalent phase-level timing metrics. However, these metrics receive the same Required attributes (`gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`) as the spec-named request metrics so that downstream systems can join across both namespaces for correlation and alerting. ### Metrics NOT Emitted AIPerf is a client-side benchmarking tool and does **not** emit any server-side metrics: - No `gen_ai.server.*` metrics are produced. AIPerf also does **not** emit any opt-in GenAI events: - `gen_ai.input.messages` - `gen_ai.output.messages` - `gen_ai.system_instructions` - `gen_ai.tool.definitions` These events are excluded because AIPerf's purpose is performance measurement, not request/response content logging.