Agents

Agent Tracing

Export Dynamo request traces, tool-call metadata, and Perfetto timelines

View as Markdown

Agent tracing captures request timing, token counts, worker placement, finish metadata, and replay hashes for eligible LLM requests. Requests with session identity also carry agent context, which lets analysis tools group LLM turns and tool activity into the same run.

Request traces contain metadata, not payloads. Dynamo does not store prompts, responses, or tool-call arguments in these traces.

Capture Your First Agent Trace

1

Enable request tracing

Set the master switch before starting the Dynamo frontend:

Enable the default gzip trace sink
$export DYN_REQUEST_TRACE=1

Dynamo writes rotating gzip-compressed JSONL segments to /tmp/dynamo-request-trace.NNNNNN.jsonl.gz.

To use another segment prefix, set DYN_REQUEST_TRACE_OUTPUT_PATH:

Choose an output path
$export DYN_REQUEST_TRACE=1
$export DYN_REQUEST_TRACE_OUTPUT_PATH=/mnt/captures/run-42/request-trace
2

Run your agent workload

Send requests through a configured Agent Harness or custom client. Session identity enriches eligible request_end rows automatically; it does not require another trace flag.

3

Convert the capture to Perfetto

Run the converter from the Dynamo repository:

Create a Perfetto trace
$python3 benchmarks/request_trace/convert_to_perfetto.py \
>  "${DYN_REQUEST_TRACE_OUTPUT_PATH}".*.jsonl.gz \
>  --output "${DYN_REQUEST_TRACE_OUTPUT_PATH}.perfetto.json"
4

Open the timeline

Open /tmp/dynamo-request-trace.perfetto.json in the Perfetto UI. The default view shows LLM request slices, prefill and decode stages, and tool slices when the trace contains tool metadata.

For every request-trace environment variable and default, see Request Trace Reference.

Choose the Tool Detail You Need

Capture methodHarness changesWhat it records
Automatic tool metadataNoneTool-call names and IDs from response parsing, plus inferred tool-wait spans in Perfetto
Explicit harness eventsRequiredPer-tool timing, status, output size, and error type

Tool Call Observability

Record Tool Calls Automatically

When request tracing is enabled and the worker uses --dyn-tool-call-parser, Dynamo records the final finish reason and each parsed tool call’s name and ID in request_end.finish_reason_metadata. Tool-call arguments are intentionally omitted.

The Perfetto converter infers a tool span from the end of a tool-call response to the next request in the same trajectory. You can calculate the same interval from the trace:

Approximate tool and agent wait time
tool_wait(turn N) ~= next.request_received_ms - this.event_time_unix_ms

This interval includes tool execution and agent overhead. It cannot separate parallel tools or attribute time to one tool.

Record Precise Tool Timing

For per-tool attribution, configure the frontend to receive events from the harness:

Enable explicit tool-event ingress
$export DYN_REQUEST_TRACE=1
$export DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_ENDPOINT=tcp://127.0.0.1:20390

The endpoint is a ZMQ PULL bind address. The harness publishes agent-tool-events as the first frame unless you override the topic with DYN_REQUEST_TRACE_TOOL_EVENTS_ZMQ_TOPIC.

Tool-event wire format and example

The wire format is [topic, seq_be_u64, msgpack(RequestTraceRecord)]. Use a monotonic sequence and a background PUSH publisher with a bounded queue. Include started_at_unix_ms, ended_at_unix_ms, and duration_ms on terminal tool_end and tool_error events so timing survives a dropped tool_start.

Use the same agent context as the surrounding LLM requests. Make each tool_call_id unique within its trajectory.

Explicit tool_end event
1{
2  "schema": "dynamo.request.trace.v1",
3  "event_type": "tool_end",
4  "event_time_unix_ms": 1777312801500,
5  "event_source": "harness",
6  "agent_context": {
7    "session_id": "research-run-42",
8    "trajectory_id": "research-run-42:researcher"
9  },
10  "tool": {
11    "tool_call_id": "call-abc",
12    "tool_class": "web_search",
13    "status": "succeeded",
14    "started_at_unix_ms": 1777312801080,
15    "ended_at_unix_ms": 1777312801500,
16    "duration_ms": 420.5
17  }
18}

Optional tool fields are output_tokens, output_bytes, tool_name_hash, and error_type. Status values are running, succeeded, error, and cancelled; supported aliases include ok, success, failed, timeout, and canceled.

Dynamo request_end Record

Dynamo emits request_end after an eligible response stream finishes or is dropped. The record can contain agent context, request timing and token metrics, worker information, finish metadata, and replay hashes.

Full agent-enriched request_end example
1{
2  "schema": "dynamo.request.trace.v1",
3  "event_type": "request_end",
4  "event_time_unix_ms": 1777312801000,
5  "event_source": "dynamo",
6  "agent_context": {
7    "session_id": "research-run-42",
8    "trajectory_id": "research-run-42:researcher",
9    "parent_trajectory_id": "research-run-42:planner"
10  },
11  "request": {
12    "request_id": "dynamo-request-id",
13    "model": "my-model",
14    "input_tokens": 128,
15    "output_tokens": 16,
16    "request_received_ms": 1777312800000,
17    "total_time_ms": 1000,
18    "finish_reason_metadata": {
19      "finish_reason": "tool_calls",
20      "tool_calls": [
21        {
22          "choice_index": 0,
23          "tool_call_index": 0,
24          "id": "call-abc",
25          "name": "web_search"
26        }
27      ]
28    },
29    "replay": {
30      "trace_block_size": 64,
31      "input_length": 128,
32      "input_sequence_hashes": [
33        14879255164371896291,
34        274632075616497421
35      ]
36    }
37  }
38}

For chat streams, Dynamo records finish metadata after parser and jail rewrites. Completion streams record the final OpenAI-compatible completion finish reason.

Request tracing currently covers eligible Rust OpenAI chat-completions and completions requests. It skips unsupported replay shapes, including n > 1, best_of > 1, prompt_embeds, multimodal inputs, and requests without a tracker or usable KV cache block size. Sinks use best-effort delivery and can drop records when they lag, so check warnings and validate row counts before treating a capture as complete.