> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Request Trace Reference Request tracing emits replay metadata, optional chat payloads, and optional harness tool events to one or more sinks. For the capture and replay workflow, see [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#capture-and-replay-requests). ## Configuration | Variable | Default when enabled | Values | Description | | -------------------------------------------- | --------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `DYN_REQUEST_TRACE` | unset | Truthy value | Master switch. When enabled and `DYN_REQUEST_TRACE_RECORDS` is unset, emits `request_end,tool`. | | `DYN_REQUEST_TRACE_RECORDS` | `request_end,tool` | `request_end`, `request_payload`, `tool` | Comma-separated record types. Setting the variable enables only the listed types. | | `DYN_REQUEST_TRACE_SINKS` | `file` | `file`, `stderr`, `nats`, `otel` | Comma-separated sinks. | | `DYN_REQUEST_TRACE_FILE_PATH` | `/tmp/dynamo-request-trace` | Path or prefix | Literal path for `jsonl`; segment prefix for `jsonl_gz`. | | `DYN_REQUEST_TRACE_FILE_FORMAT` | `jsonl_gz` | `jsonl`, `jsonl_gz` | File encoding and rotation mode. | | `DYN_REQUEST_TRACE_CAPACITY` | `1024` | Positive integer | Best-effort in-process broadcast capacity. | | `DYN_REQUEST_TRACE_NATS_SUBJECT` | `dynamo.request_trace.v1` | NATS subject | Subject used by the `nats` sink. | | `DYN_REQUEST_TRACE_OTEL_MAX_PAYLOAD_BYTES` | `4194304` | Positive integer bytes | Maximum serialized OTLP payload attribute. Oversized payloads emit an incomplete marker. | | `DYN_REQUEST_TRACE_FILE_BUFFER_BYTES` | `1048576` | Integer bytes | File batching threshold. | | `DYN_REQUEST_TRACE_FILE_FLUSH_INTERVAL_MS` | `1000` | Integer milliseconds | Periodic file flush interval. | | `DYN_REQUEST_TRACE_FILE_ROLL_BYTES` | `268435456` | Positive integer bytes | Gzip roll threshold in uncompressed bytes. | | `DYN_REQUEST_TRACE_FILE_ROLL_LINES` | unset | Positive integer records | Optional gzip roll threshold in records. | | `DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_ENDPOINT` | unset | ZMQ bind address | Optional PULL endpoint for harness tool events. Configure it on only one process. | | `DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_TOPIC` | `agent-tool-events` | ZMQ topic | First-frame topic filter. | | `DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST` | unset | Header names | Comma- or whitespace-separated allowlist captured only on `request_payload` rows. Values are unredacted. | Legacy `jsonl` and `jsonl_gz` sink values, `DYN_REQUEST_TRACE_OUTPUT_PATH`, and `DYN_REQUEST_TRACE_JSONL_*` remain migration aliases. `DYN_AUDIT_*` variables are migration shims, not wire-compatibility aliases. Prefer `DYN_REQUEST_TRACE_*` for new deployments. ## Sink Behavior The `otel` sink maps each row to one OTLP `LogRecord` in scope `dynamo.request_trace`. It resolves transport settings in this order: * Protocol: `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL`, then `OTEL_EXPORTER_OTLP_PROTOCOL`, then `grpc`. * Endpoint: `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then the protocol default. Dynamo appends `/v1/logs` only to a generic HTTP/protobuf endpoint. * Service name: `OTEL_SERVICE_NAME`, default `dynamo`. Setting `DYN_REQUEST_TRACE_SINKS=stderr` does not enable OTLP export. Include `otel`, for example `file,otel` or `stderr,otel`. ## Record Types ### `request_end` Contains compact replay metadata. Session headers can enrich the row with session identity, request metrics, finish metadata, and tool-call metadata. Session identity does not enable sticky routing. ### `request_payload` Contains the OpenAI `/v1/chat/completions` client request and, when available, its response. Dynamo emits one row per eligible request. Canceled or failed requests omit `payload.response`. Oversized OTLP bodies use `payload_complete=false` and `payload_drop_reason`. Allowlisted headers appear at `payload.http_request_headers`. Matching is case-insensitive. Empty, non-UTF-8, and unlisted values are omitted. Because values are not redacted, do not capture credential-bearing headers. ### Tool Events Harness events enter through the optional ZMQ endpoint and are normalized into request trace rows before fan-out to the configured sinks. ## Record Schema All rows use `schema: dynamo.request.trace.v1` and an `event_type`. A context-free replay row contains request timing, output length, KV block size, input length, and rolling input-sequence hashes: ```json { "schema": "dynamo.request.trace.v1", "event_type": "request_end", "event_time_unix_ms": 1777312801000, "request": { "request_id": "dynamo-request-id", "request_received_ms": 1777312800000, "output_tokens": 16, "replay": { "trace_block_size": 64, "input_length": 128, "input_sequence_hashes": [14879255164371896291] } } } ``` A payload row stores the client request under `payload.request`, the completed response under `payload.response`, and a `payload_complete` marker. Optional captured headers appear under `payload.http_request_headers`. ```json { "schema": "dynamo.request.trace.v1", "event_type": "request_payload", "event_source": "dynamo", "payload": { "request_id": "dynamo-request-id", "endpoint": "openai.chat_completion", "model": "my-model", "request": {"model": "my-model", "messages": []}, "response": {"id": "chatcmpl-example", "choices": []}, "payload_complete": true } } ``` `input_sequence_hashes` are Dynamo's sequence-aware rolling hashes. DynoSim maps them to compact internal IDs while loading the original trace; it does not create an intermediate Mooncake file. ## Supported Replay Requests Context-free replay rows must represent one model request. Dynamo skips requests with `n > 1`, `best_of > 1`, prompt embeddings, multimodal inputs, or no usable tracker and KV cache block size. Session context does not bypass these checks. DynoSim accepts `dynamo.request.trace.v1` JSONL and JSONL.GZ directly, derives the trace block size, and rejects mixed session-aware and context-free trace sets. ## Related * [Observe a Local Deployment](/dynamo/dev/cli/operations/observability#capture-and-replay-requests) * [Logging Reference](/dynamo/dev/observability/logging#request-payload-export) * [Environment Variables](/dynamo/dev/observability/environment-variables#request-tracing)