Evaluation CLI

The NeMo Labs Voice Agent evaluation harness ships two command-line entry points in evaluation/:

The argparse definitions in those two files specify every default below.

Before You Run

Start both bot servers before you run run_evaluation.py. The code resolves SERVER_CONFIG_PATH against the current working directory, so run cd evaluation first. The run_agent.sh and run_user.sh helpers also change to that directory and export the following ports.

# Terminal 1 — simulated user bot
cd evaluation && WEBSOCKET_PORT=8766 FASTAPI_PORT=7861 \
  SERVER_CONFIG_PATH=server_configs/user.yaml python bot_server.py
# Terminal 2 — agent under test
cd evaluation && WEBSOCKET_PORT=8765 FASTAPI_PORT=7860 \
  SERVER_CONFIG_PATH=server_configs/agent.yaml python bot_server.py
# Terminal 3 — driver
cd evaluation && python run_evaluation.py --domain eva_airline

Refer to Environment Variables for the complete bot-server variable list and Evaluation Quickstart for the end-to-end walkthrough.

run_evaluation.py

Use the following option groups to select scenarios, connect the bots, control runs, and configure scoring.

Scenario Selection and Listing

Use these flags to list or select the scenarios included in a run.

Domain filtering is a literal prefix match, so --domain tau2_telecom selects only the tau2_telecom__… scenarios. The parallel workflow-policy registration is a separate domain, --domain tau2_telecom_workflow. The four benchmark domains carry 50 (eva_airline), 50 (tau2_airline), 114 (tau2_retail), and 114 (tau2_telecom, mirrored by tau2_telecom_workflow) scenarios. For domain details, refer to Benchmark Domains.

Connection and Audio

Use these flags to configure bot endpoints, result storage, and audio streaming.

Run Control

Use these flags to set scenario limits, resume runs, and control matching behavior.

--min-agent-turns is a stall filter for runs where the LLM backend hung. Scenarios below the threshold are counted as failures in the composite success rate and skipped in the per-signal rates (action-match, DB-state, NL-assertion) — they are not dropped from the run. Under --resume they are additionally treated as in-flight and re-run. The turn count comes from the live-recorded token_usage.agent.n_calls in metrics.json, falling back to the saved agent LLM context for older runs.

The default is 2 because the floor has to sit below the shortest legitimate conversation. Across 500 measured scenarios, the agent LLM-call distribution bottoms out at exactly 3 calls, where 21 scenarios sit. Every one of those is a clean exit (stop_reason of [EXIT]), and 11 of them succeeded. A floor of 3 therefore lands inside that legitimate cluster and fails scenarios that did the work; 2 sits below it.

The default tool-only conversation-end policy preserves existing behavior: the agent must call EndConversationTool. Select valid-terminal-state when the agent integration cannot reliably emit the tool call but the simulator or final-speaker evidence can establish a valid ending. The runner still records whether the agent called the tool, independently of the selected policy. The default 120-second inactivity timer applies under both policies, but only valid-terminal-state can treat a user-final inactivity timeout as a clean exit.

Automatic Retry

The runner can re-run a scenario immediately, inside the same session, when the finished attempt looks like an infrastructure failure instead of a task failure. This is separate from --resume, which you invoke by hand after a run ends.

The two triggers differ in how certain the failure is:

• Zero conversation turns, enabled by default. The bots never exchanged audio, so the attempt holds no measurement that a retry could bias. Turn off with --no-auto-resume-on-stale.
• Fewer agent turns than --min-agent-turns, disabled by default. That floor is a heuristic, and legitimately short scenarios sit on it. Because an insufficient-turns scenario is forced to is_successful: false, retrying it converts a guaranteed failure into a fresh draw and biases the success rate upward. Run-to-run churn within a single arm of these benchmarks measures 12% to 32%, so the bias is real rather than theoretical. Enable it only when you accept that trade.

The stale trigger reads the bridge’s own conversation record, not token_usage.agent.n_calls. Token usage increments only when the agent bot emits an RTVI token-usage message, so a backend that does not report usage reads as zero calls on a perfectly healthy run. Gating the default-on trigger on that counter would silently re-run every scenario.

A retried attempt is preserved on disk as <scenario>.killed.autoretry<N>.<timestamp>/ with a __KILLED__ marker file inside, so its logs stay available for triage. Every metrics.json records auto_retry_count, and auto_retry_reasons appears when at least one retry fired. Use those fields for downstream analysis rather than matching directory names.

LLM Judge

Use these flags to connect and configure the optional LLM judge.

The runner constructs the judge only when both --judge-url and --judge-model are nonempty. Passing an empty string to either option disables judging for the run. Judge thinking is always enabled and is not exposed as a flag. Changing that behavior would silently change score semantics across runs.

The runner validates numeric options before the run starts. A violation exits through parser.error (status 2):

The runner passes --judge-temperature and --judge-seed through without validation.

Exit Codes

The evaluation driver exits with one of the following status codes.

Invocation Record

Every run writes run_args.json into the session directory with the shape {"invocations": [...]}. Each entry records:

• started_at and the raw argv.
• The parsed arguments, with the judge API key redacted.
• The resolved scenario names and count.

A --resume invocation appends a new entry and soft-checks it against the previous one on the scoring-relevant fields domain, scenarios, duration, judge_url, judge_model, judge_threshold, judge_max_tokens, judge_temperature, judge_top_p, judge_seed, strict_match, conversation_end_policy, and inactivity_timeout. Mismatches log a warning but do not block the run.

For result artifacts (all_metrics.json, all_summary.txt, and per-scenario metrics.json), refer to Reading Results and the Metrics Dictionary. For the six scoring signals, refer to Scoring.

check_resume.py

Use check_resume.py to classify a finished or interrupted session directory without moving or deleting anything. The script applies the same classification as the runner, so its output predicts what --resume would do.

cd evaluation
python check_resume.py ../eval_results/eval_20260618_072325 --min-agent-turns 2

The --min-agent-turns default here is 2, matching run_evaluation.py. Pass the same value you intend to use with --resume whenever you override it, so the preview matches the run. Raising the floor above 2 is rarely what you want. Legitimately short scenarios bottom out at 3 agent responses, so a floor of 3 or more re-runs real results instead of stalls.

Each subdirectory is bucketed into one of three states:

Directories already named *.killed.* or containing __KILLED__, and top-level files such as run_args.json and evaluation_log.txt, are ignored.

The script prints the per-bucket counts first, then up to three lists in this order:

The retry list is informational. It reports which finished results needed an automatic re-run before they succeeded, and it does not change the completed, rerun, or fresh counts. It covers only scenarios in the completed bucket. A scenario that was retried and still classifies as re-runnable stays under Would re-run, where the inline suffix flags it — that combination is the most diagnostic of the two, because it means the automatic retry did not fix the problem. The suffix needs a readable metrics.json, so a scenario that has no metrics file at all is listed without it. When nothing is queued for a re-run and no scenario is fresh, the script prints All scenarios are complete — nothing to resume.

Detect retried scenarios through the auto_retry_count field in metrics.json, which is what this script reads. Do not glob for <scenario>.killed.autoretry*/ directories: that layout is an implementation detail of how the runner preserves failed attempts.

For the resume workflow, refer to Resuming a Run.