Metrics#
This section defines common LLM inference metrics. Tool implementations vary, so compare results only when definitions align. Refer to Using AIPerf to Benchmark for collecting these metrics with AIPerf.
The following diagram summarizes widely used latency and throughput metrics.
Figure 1. Overview of popular LLM inference performance metrics.#
Time to First Token#
Time to first token (TTFT) measures how long you wait before seeing the model’s output. It is the time from query submission to the first received token, if the response is not empty.
Figure 2: TTFT - Time to First Token including both the tokenization and de-tokenization steps for the first output token.#
Note
NVIDIA AIPerf disregards initial responses with no content or an empty string. TTFT is meaningless when the first response contains no token.
Time to first token generally includes request queuing time, prefill time, and network latency. Longer prompts increase TTFT because the attention mechanism uses the full input sequence to create the KV cache before generation begins. In a production application, several requests can be in progress at the same time, so one request’s prefill phase can overlap with another request’s generation phase.
Note
Traditional web service benchmarking tools such as K6 can also provide TTFT by using timing events in the HTTP request.
End-to-End Request Latency#
End-to-end request latency, or e2e request latency, measures how long it takes from submitting a query to receiving the full response. This includes queueing, batching, and network latency. Figure 3 shows the request latency timeline.
Figure 3. End-to-end Request latency#
Note
In streaming mode, the de-tokenization step can be done multiple times when partial results are returned to the user.
For an individual request, the end-to-end request latency is the time difference between the request sent and the final token received:
Note
Generation_time is the duration from the first token received to the final token received, as depicted in Figure 1. AIPerf also removes the final [done] signal or empty response so it is not included in e2e latency.
Inter-token Latency#
Inter-token latency (ITL) is defined as the average time between consecutive tokens and is also known as time per output token (TPOT).
Figure 4: ITL - latency between successive token generations.#
Although the definition is straightforward, tools differ on whether TTFT is included in the average. AIPerf excludes TTFT.
AIPerf defines ITL as follows:
The equation for this metric does not include the first token. The denominator subtracts one so ITL characterizes only the decoding part of request processing.
With longer output sequences, the KV cache and its memory cost grow. Attention computation cost also grows with each new token because this cost is linear in the length of the input and output sequence so far. Consistent inter-token latencies indicate efficient memory management, memory bandwidth, and attention computation.
Tokens Per Second#
Total tokens per second (TPS) per system represents total output token throughput across all simultaneous requests. As the number of requests increases, total TPS per system increases until it saturates the available GPU compute resources. Beyond that point, TPS can decrease.
Consider the following timeline of the entire benchmark with n total requests:
Figure 5: Timeline of events in a benchmarking run#
where the following variables are defined as:
L<i>: End-to-end latency of request i
T_start: Start of benchmark
Tx: Timestamp of the first request
Ty: Timestamp of the last response of the last request
T_end: End of benchmark
AIPerf defines TPS as total output tokens divided by the end-to-end latency between the first request and the last response of the last request.
This calculation is batch-oriented and is not a live running metric. AIPerf excludes warmup if you configure a warmup phase.
TPS per user represents throughput from a single-client perspective. It is defined as output sequence length divided by e2e_latency for each request, which asymptotically approaches 1/ITL as the output sequence length increases. As the number of concurrent requests increases, total system TPS increases while TPS per user decreases as latency increases.
Requests Per Second#
Requests per second (RPS) is the average number of requests that can be successfully completed by the system in a 1-second period. It is calculated as: