Control Hooks by Inference Server

How AIPerf’s endpoint.reset_kv_cache and endpoint.server_profiler map onto vLLM, SGLang, and TensorRT-LLM (trtllm-serve).

For the general feature (ownership, failure policy, CLI flags), see Benchmark Control Hooks.

Quick matrix

AIPerf always sends empty-body POSTs and treats any 2xx as success. Optional JSON bodies that some servers accept on /start_profile are out of scope for the first control-hook pass — use empty POST + manual stop (AIPerf already pairs start/stop to profiling barriers).

vLLM

Source

• Reset: vllm/entrypoints/serve/dev/cache/api_router.py → @router.post("/reset_prefix_cache"). Mounted only via register_vllm_dev_api_routers() when VLLM_SERVER_DEV_MODE=1 (vllm/envs.py).
• Profiler: vllm/entrypoints/serve/profile/api_router.py → /start_profile, /stop_profile. Mounted only when profiler_config.profiler is set (local-dev warning in code).

Caveats

• Reset returns HTTP 200 with {"success": bool}. A failed reset (blocks still held) is still 2xx, so AIPerf will treat it as success. Ensure the engine is idle before the hook, or inspect server logs.
• Enable both gates before relying on AIPerf defaults:

$
export VLLM_SERVER_DEV_MODE=1
$
# plus whatever flag/config enables profiler_config.profiler for your vLLM version

1
endpoint:
2
  type: chat
3
  url: http://127.0.0.1:8000
4
  reset_kv_cache: true
5
  server_profiler: true

$
aiperf profile \
>
  --model <model> \
>
  --url http://127.0.0.1:8000 \
>
  --reset-kv-cache \
>
  --server-profiler \
>
  ...

SGLang

Source

• Flush: python/sglang/srt/entrypoints/http_server.py → @app.api_route("/flush_cache", methods=["GET", "POST"]) with query timeout: float = 0.0. Returns 200 on success, 400 on failure (e.g. not idle).
• Profiler: same file → /start_profile, /stop_profile (GET/POST).
• gRPC deployments: grpc_server.py documents an HTTP sidecar that exposes /start_profile and /stop_profile (flush may also live on the HTTP surface depending on deployment).

Config

1
endpoint:
2
  type: chat
3
  url: http://127.0.0.1:30000
4
  reset_kv_cache:
5
    path: /flush_cache
6
    # Optional: wait for idle (query string is allowed on relative paths)
7
    # path: /flush_cache?timeout=30
8
  server_profiler: true

$
aiperf profile \
>
  --model <model> \
>
  --url http://127.0.0.1:30000 \
>
  --reset-kv-cache \
>
  --reset-kv-cache-path /flush_cache \
>
  --server-profiler \
>
  ...

Notes:

• Prefer ?timeout=30 (and matching timeout_seconds on the hook) when cells may still have in-flight work; timeout=0 fails fast with 400.
• Point --url at the HTTP sidecar origin for gRPC-only servers if that is where admin routes are exposed.

Docs: Native APIs, Benchmark and Profiling.

TensorRT-LLM (trtllm-serve)

Cache reset

Verified on TensorRT-LLM main: no /reset_prefix_cache or /flush_cache on tensorrt_llm/serve/openai_server.py. Related admin routes exist for RL/weight workflows (/release_memory, /resume_memory, /update_weights) — not a prefix-cache flush.

KV block reuse is controlled at startup via kv_cache_config / enable_block_reuse (see KV cache reuse).

For sweep hygiene:

• Disable reuse when cross-cell pollution is unacceptable, or
• Restart / redeploy trtllm-serve between cells (outside AIPerf hooks).

Do not enable endpoint.reset_kv_cache against stock trtllm-serve unless your deployment documents a custom admin route.

Profiler

On current TensorRT-LLM main, /start_profile and /stop_profile are not registered. The in-tree tensorrt_llm/serve/scripts/benchmark_serving.py calls those URLs when --profile is set, but the server handlers live in open PR #13872 (state: OPEN, not merged).

Until that PR (or equivalent) lands in your build:

1
endpoint:
2
  type: chat
3
  url: http://127.0.0.1:8000
4
  # omit reset_kv_cache and server_profiler

After the routes exist on your binary, confirm then enable:

$
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:8000/start_profile
$
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:8000/stop_profile

Expect 2xx. Missing routes make profiler start fatal — leave server_profiler off.

Mock server (local verification)

The in-repo mock server implements:

Example against mock:

$
aiperf-mock-server --port 8000 --fast --access-logs &
$
$
# vLLM-shaped defaults
$
aiperf profile --model mock-model --url http://127.0.0.1:8000 \
>
  --reset-kv-cache --server-profiler --request-count 2 --ui none ...
$
$
# SGLang-shaped reset path
$
aiperf profile --model mock-model --url http://127.0.0.1:8000 \
>
  --reset-kv-cache --reset-kv-cache-path /flush_cache \
>
  --server-profiler --request-count 2 --ui none ...

Access logs should show reset/flush before profiling traffic, and start_profile / stop_profile around the profiling phase.

Choosing overrides

Always keep paths relative (leading /). AIPerf joins them to each unique endpoint origin (scheme://host:port).