Is this page helpful?

Payload Capture#

Payload capture is an opt-in feature that records each inference request as JSONL suitable for replay with NVIDIA AIPerf. It lets you capture your real traffic and replay it later – for example to measure speculative-decoding acceptance and speedup on your own workload, or to reproduce a production load pattern.

Warning

Payload capture records exact request prompts, which may contain PII, secrets, or other sensitive customer data. It is disabled by default. Enable it only when you intend to, write captures to an access-controlled location, and delete them when no longer needed.

Enable it#

Capture is configured with two environment variables (mirroring NIM_SPECDEC_ENABLE / NIM_SPECDEC_ARGS): NIM_CAPTURE_ENABLE is the toggle, and NIM_CAPTURE_ARGS is an optional JSON object of tuning keys. Point the capture path at a writable, mounted directory:

export IMG=nvcr.io/nim/<org>/<nim>:<tag>
docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -e NIM_CAPTURE_ENABLE=1 \
  -e NIM_CAPTURE_ARGS='{"path": "/captures/requests.jsonl", "format": "mooncake_payload"}' \
  -v /data/captures:/captures \
  -p 8000:8000 \
  $IMG

The container runs as a non-root user, so mount a directory it can write to and set the capture path inside it. NIM logs a warning at startup if the destination is not writable. NIM_CAPTURE_ENABLE=1 alone (no NIM_CAPTURE_ARGS) captures to the default path in mooncake_payload format.

Configuration#

NIM_CAPTURE_ENABLE (0/1, default 0) is the master switch. NIM_CAPTURE_ARGS is a JSON object; all keys are optional and validated (a bad value falls back to its default; invalid JSON disables capture so prompts are never written to an unintended location):

`NIM_CAPTURE_ARGS` key	Default	Description
`path`	`/opt/nim/captures/requests.jsonl`	Destination JSONL file (must be writable).
`format`	`mooncake_payload`	`mooncake_payload` or `raw_payload` (see below).
`max_request_bytes`	`1048576`	Requests larger than this are forwarded normally but not recorded (a truncated payload is not replayable); a warning is logged.
`sample_rate`	`1.0`	Probability a matching request is recorded. `1.0` = all; `0.1` = ~10%; `0.0` = none. Use to trim I/O under high QPS.
`endpoint_pattern`	`^/v1/(chat/completions\|completions)$`	Regex; only matching request paths are captured.
`max_queue_bytes`	`1073741824`	In-memory ceiling (bytes) for the background writer queue. Captures are serialized and written off the request path by a writer thread; if the sink cannot keep up, records are dropped with a warning (the request is never blocked). Set >= `max_request_bytes`.

Both formats record the actual request payload (the full request JSON), never hashed or synthetic prompts – so the replay reflects your true workload (e.g. coding vs. chat traffic), not a generic corpus.

Output formats#

`mooncake_payload` (default)#

One timestamped line per request; the timestamp is milliseconds relative to the first captured request:

{"timestamp":0,"payload":{"model":"m","messages":[{"role":"user","content":"hello"}],"max_tokens":128}}
{"timestamp":540,"payload":{"model":"m","prompt":"hi","max_tokens":64}}

Replay (preserves inter-request timing):

aiperf profile \
  --model <served-model-name> --endpoint-type chat --streaming \
  --url localhost:8000 \
  --custom-dataset-type mooncake_trace \
  --input-file ./requests.jsonl \
  --server-metrics http://localhost:8000/v1/metrics

`raw_payload`#

One request JSON object per line:

{"model":"m","messages":[{"role":"user","content":"hello"}],"max_tokens":128}

Replay:

aiperf profile \
  --model <served-model-name> --endpoint-type chat --streaming \
  --url localhost:8000 \
  --custom-dataset-type raw_payload \
  --input-file ./requests.jsonl \
  --server-metrics http://localhost:8000/v1/metrics

Replaying: one endpoint type per run#

AIPerf’s payload modes send each captured payload verbatim to the single endpoint named by --endpoint-type. A trace that mixes chat (messages) and completion (prompt) payloads cannot be replayed in one run – the off-type lines are rejected. The default endpoint_pattern captures both /v1/chat/completions and /v1/completions, so for a single-run replay either:

scope capture to one endpoint, e.g. "endpoint_pattern": "^/v1/chat/completions$", or
split the capture by endpoint and replay each file with its matching --endpoint-type (chat for messages, completions for prompt).

AIPerf also loads a tokenizer for the served model to compute token metrics. If it cannot be fetched (private model name or air-gapped host), pass a local tokenizer directory with --tokenizer <dir> (or --tokenizer builtin).

Handling sensitive captures (PII)#

Captured payloads contain the exact prompts. If you are only testing performance, you can protect any sensitive data at rest by replacing the payloads with block-hashed values.

However, if you are testing speculative-decoding acceptance rate or acceptance length, the original payloads should remain in place. Hashing discards the prompt content, which severely degrades the capture’s value for measuring speculative-decoding AR/AL (acceptance depends on the actual tokens, not just token lengths or prefix structure).

Warning

Either way, treat the capture file as sensitive: write it to an access-controlled, short-lived location and delete it when finished.