Is this page helpful?

Parameters and Best Practices#

After reviewing Metrics, configure test parameters and sweep ranges that reflect your deployment. The settings below help produce meaningful, comparable benchmark results.

Use Cases#

Your application’s use cases influence input sequence length (ISL) and output sequence length (OSL). Sequence lengths affect how fast a system processes input, builds the KV cache, and generates output tokens. Longer input sequences increase prefill memory requirements and TTFT. Longer output sequences increase generation memory requirements and ITL. Understand the distribution of inputs and outputs in your LLM deployment to optimize hardware utilization. Common use cases and likely ISL/OSL pairs include the following:

  • Translation: Translation between languages and code. Similar ISL and OSL, typically 500 to 2,000 tokens each.

  • Generation: Code, story, email, and search-based content generation, typically with OSL near 1,000 tokens and ISL near 100 tokens.

  • Summarization: Retrieval, chain-of-thought prompting, and multi-turn conversations, typically with ISL near 1,000 tokens and OSL near 100 tokens.

If you have production traffic, you can use real prompts as inputs.

Load Control#

  • Concurrency N is the number of concurrent clients, each with one active request. Equivalently, it is the number of requests that an LLM service handles concurrently. After each request receives a complete response, another request is sent so the system always has N active requests.

    Concurrency is most frequently used to describe and control the load induced on the inference system.

  • Max batch size: Batch is the group of simultaneous requests that the inference engine processes. This can be a subset of the concurrent requests. The maximum batch size parameter defines the maximum number of requests that the inference engine can process simultaneously.

    If the concurrency exceeds the maximum batch size multiplied by the number of active replicas, some requests wait in a queue for later processing. In this case, time to first token can increase due to queueing delay.

  • Request rate controls load by determining the rate at which new requests are sent. Using a constant request rate r means one request is sent every 1/r seconds. Using a Poisson request rate determines the average inter-arrival time.

    AIPerf supports both concurrency and request rate. Prefer concurrency for most benchmarks: with request rate, outstanding requests can grow without bound when the arrival rate exceeds system throughput.

    When you specify concurrencies to test, sweep from one request to a maximum value slightly greater than the max batch size. When concurrency exceeds the engine’s max batch size, some requests wait in a queue. Throughput generally saturates near the max batch size while latency steadily increases.

Other Parameters#

  • ignore_eos: Most LLMs have a special end-of-sequence (EOS) token, which signifies the end of generation. It indicates that the LLM has generated a complete response and should stop. In general usage, LLM inference should respect this signal and stop generating further tokens. For benchmarking, set ignore_eos to true so the model continues generating tokens until it reaches the max_tokens limit. This helps the benchmark reach the intended output length and obtain consistent measurements.

  • Sampling compared to greedy decoding: Different sampling strategies can affect LLM generation speed. Greedy decoding selects the token with the highest logit and avoids normalizing and sorting the probability distribution over tokens. Refer to the Hugging Face generation strategies blog for a detailed explanation of sampling methods. Keep the sampling method consistent within the same benchmarking setup.