Reading Results

Every run_evaluation.py invocation writes a timestamped session directory under --output-dir (default ./eval_results). The top level contains session aggregates, and each scenario has an evidence subdirectory. Use this page to select the artifact that answers your investigation question. The saved evidence includes output from the large language model (LLM) judge when the judge runs. The command-line interface (CLI) invocation determines the session directory and selected scenarios. Stateful scenarios can also include database (DB) hash evidence.

Directory Layout

Each run directory combines session-level aggregates with one evidence directory per scenario.

eval_results/eval_YYYYMMDD_HHMMSS/
├── evaluation_log.txt            # runner log for the whole session
├── run_args.json                 # CLI invocation history (appended on --resume)
├── all_metrics.json              # JSON array: every scenario's metrics.json, concatenated
├── all_latencies.csv             # one CSV row per latency measurement
├── all_summary.txt               # human-readable per-scenario + run-level summary
└── <scenario_name>/
    ├── metrics.json              # per-scenario scores, turns, latency, token usage
    ├── judge_result.json         # LLM judge output (written whenever the judge ran)
    ├── final_agent_response.json # actions the bots actually performed
    ├── final_scenario_db_hash.txt# post-run DB hash(es)
    ├── conversation_log.txt      # timestamped transcript with latency annotations
    ├── conversation_log.seglst.json  # segLST speaker segments
    ├── conversation_log.wav      # stereo audio, L = user→agent, R = agent→user
    ├── bridge_log.txt            # bridge debug log for this scenario
    ├── bot_logs_agent/llm_context.json  # agent bot's full LLM context
    ├── bot_logs_user/llm_context.json   # user-sim bot's full LLM context
    └── scenario_config/          # snapshot of the scenario as it was run
        ├── metadata.json
        ├── reference_answer.json
        ├── agent_prompt.txt
        ├── agent_tools.json
        ├── user_prompt.txt
        └── user_tools.json

Which File Answers Which Question

Start with the artifact that most directly answers your investigation question.

Session-Level Files

Read all_summary.txt first. It starts with scenario, duration, and turn totals. A per-scenario block then reports Is successful, task success without clean exit, applicable signals, turn count, duration, and latency. The remaining sections report overall latency, stalled-run warnings, composite and per-signal rates, token usage, and per-domain breakdowns when the run spans multiple domains.

Two lines deserve special attention:

• WARNING: N scenario(s) had fewer than 3 agent turn(s) ... and were counted as failures. — the --min-agent-turns filter fired. Those scenarios are hard failures in the composite rate and are skipped in the per-signal rates, so they depress the headline number while shrinking every per-signal denominator. Re-run them with --resume as described in Resuming a Run.
• Overall Success Rate: N/A (no scenarios had any applicable signal) — nothing was scoreable at all.

all_metrics.json is the JSON array of every scenario’s metrics.json, convenient for scripted aggregation. all_latencies.csv has the header Scenario,User_Transcript,Agent_Transcript,Latency_ms and one row per measured user-to-agent response. evaluation_log.txt is the runner’s own log (appended, not truncated, on resume). run_args.json holds an invocations list — each entry records started_at, argv, parsed_args, and the resolved scenario list, with the judge API key redacted.

metrics.json

The per-scenario source of truth. Fields always present:

Conditional fields appear only when the corresponding signal applies: db_state_match plus db_state_expected_hash / db_state_actual_hash, db_state_assertion_pass_rate plus per-predicate db_state_assertion_verdicts, nl_assertion_pass_rate, judge_score and judge_passed, and insufficient_agent_turns (only when the stall filter fired). A trace_metrics block is folded in when a trace_metrics.json file exists in the scenario directory or in bot_logs_agent/.

The names inside success_breakdown are the SuccessSignal string values — is_action_match, db_state_match, db_state_assertion, nl_assertion, judge_passed, clean_exit. excluded lists signals that were computed but are not in the scenario’s whitelist, so they are informational only. Refer to Scoring for how each signal is derived.

judge_result.json

judge_result.json is written whenever the LLM judge runs. --judge-url and --judge-model both carry defaults, so the judge is constructed on every run unless you blank one of them. Always carries score, reason, and judge_input (the verbatim system_prompt and user_content the judge saw — use it to triage a surprising score without re-deriving the prompt). Scenarios with natural-language assertions also include nl_assertion_verdicts (one entry per assertion) and nl_assertion_pass_rate.

Action and DB Artifacts

final_agent_response.json is always list-wrapped as [{"actions": [...]}]. Each action record carries the upstream tool name, its arguments, a result, and a side tag stamped by the bridge according to which bot produced it ("agent" or "user"). For single-side domains, everything is tagged "agent". For the dual-side telecom domain, user-sim tool calls arrive tagged "user". Refer to tau2-telecom.

final_scenario_db_hash.txt records one or two lines, db_hash: <sha256> and, when the scenario has user-side state, user_db_hash: <sha256>. The full database never crosses the WebSocket.

To diagnose a db_state_match failure, compare this file with expected_db_hash or expected_user_db_hash in scenario_config/metadata.json. When the hashes differ, inspect bot_logs_agent/llm_context.json to find the tool call that diverged from the reference.

Transcript and Audio

conversation_log.txt opens with the scenario start time, then one block per turn:

[ 21.624s -  45.425s] (23.801s) AGENT: Here are the pizza options we have: ...
  → Response latency: 7931.2ms

It ends with the end time and Stop reason. conversation_log.seglst.json is the same content in segLST form (session_id, words, speaker, start_time, end_time) for diarization and automatic speech recognition (ASR) tooling. conversation_log.wav is stereo at --output-sample-rate: left channel is the audio sent to the agent, right channel is the audio sent to the user-sim.

LLM Context and Bridge Log

bot_logs_agent/llm_context.json and bot_logs_user/llm_context.json are lists of role / content messages including tool calls and tool results — the ground truth for what each bot actually did, as opposed to what it said. Compare the user-sim’s intended utterance against the agent’s recognized transcript to separate ASR errors from reasoning errors. For dual-side domains, read both files: the user-sim’s phone-tool calls exist only on the user side.

bridge_log.txt covers audio routing, real-time voice interface (RTVI) events, scenario initialization, cross-side sync dispatches, and the end-of-scenario summary pull. Start here when a scenario produced no metrics.json at all — the bridge log records the connection or initialization failure.

Triage Recipes

Use these commands to identify failed or stalled scenarios and inspect their latency and database-state evidence.

$
cd eval_results/eval_20260730_075304
$
$
# Which scenarios failed, and on which signal?
$
jq -r 'select(.is_successful != true)
>
       | "\(.scenario_name)\tfailed=\(.success_breakdown.failed | join(","))"' all_metrics.json
$
$
# Scenarios the stall filter counted as failures.
$
jq -r 'select(.insufficient_agent_turns == true) | .scenario_name' all_metrics.json
$
$
# P95 latency per scenario.
$
jq -r '"\(.scenario_name)\t\(.latency_stats.p95_ms)"' all_metrics.json
$
$
# Expected vs actual DB hash for one scenario.
$
jq -r .expected_db_hash tau2_retail__1/scenario_config/metadata.json
$
cat tau2_retail__1/final_scenario_db_hash.txt

Next Steps

Continue with the scoring definitions or operational guides that match what you found in the artifacts.

• Scoring — how each of the six signals is computed and combined.
• Resuming a Run — re-running stalled or in-flight scenarios into the same session directory.
• Metrics Reference — field-by-field reference.
• Eval CLI Reference — every run_evaluation.py flag.