ReferenceEvaluation

Evaluation CLI

View as Markdown

The NeMo Labs Voice Agent evaluation harness ships two command-line entry points in evaluation/:

ScriptPurpose
run_evaluation.pyDrives the bridge: runs scenarios against a live agent bot and a simulated-user bot, then scores and aggregates the results.
check_resume.pyDry-run inspection of an existing result directory: reports which scenarios --resume would re-run. Never writes to disk.

The argparse definitions in those two files specify every default below.

Before You Run

Start both bot servers before you run run_evaluation.py. The code resolves SERVER_CONFIG_PATH against the current working directory, so run cd evaluation first. The run_agent.sh and run_user.sh helpers also change to that directory and export the following ports.

$# Terminal 1 — simulated user bot
$cd evaluation && WEBSOCKET_PORT=8766 FASTAPI_PORT=7861 \
>  SERVER_CONFIG_PATH=server_configs/user.yaml python bot_server.py
$
$# Terminal 2 — agent under test
$cd evaluation && WEBSOCKET_PORT=8765 FASTAPI_PORT=7860 \
>  SERVER_CONFIG_PATH=server_configs/agent.yaml python bot_server.py
$
$# Terminal 3 — driver
$cd evaluation && python run_evaluation.py --domain eva_airline

Refer to Environment Variables for the complete bot-server variable list and Evaluation Quickstart for the end-to-end walkthrough.

run_evaluation.py

Use the following option groups to select scenarios, connect the bots, control runs, and configure scoring.

Scenario Selection and Listing

Use these flags to list or select the scenarios included in a run.

FlagDefaultDescription
--listoffPrint every registered scenario, grouped by domain, and exit.
--list-domainsoffPrint every domain with its scenario count, and exit.
--scenarios NAME [NAME ...]all registered scenariosExplicit scenario names. Takes precedence over --domain.
--domain DOMAINNoneRun every scenario whose name starts with the DOMAIN__ prefix. Exits with status 1 if nothing matches.

Domain filtering is a literal prefix match, so --domain tau2_telecom selects only the tau2_telecom__… scenarios. The parallel workflow-policy registration is a separate domain, --domain tau2_telecom_workflow. The four benchmark domains carry 50 (eva_airline), 50 (tau2_airline), 114 (tau2_retail), and 114 (tau2_telecom, mirrored by tau2_telecom_workflow) scenarios. For domain details, refer to Benchmark Domains.

Connection and Audio

Use these flags to configure bot endpoints, result storage, and audio streaming.

FlagDefaultDescription
--user-url URLws://localhost:8766WebSocket URL of the simulated-user bot.
--agent-url URLws://localhost:8765WebSocket URL of the agent under test.
--output-dir DIR./eval_resultsParent directory; each run creates eval_<TIMESTAMP>/ inside it.
--output-sample-rate HZ16000Sample rate of the recorded conversation audio written per scenario.
--audio-chunk-in-seconds SEC0.016Size of each audio chunk the bridge streams between the bots.
--pause SEC0.5Pause between scenario setup and the scenario run.

Run Control

Use these flags to set scenario limits, resume runs, and control matching behavior.

FlagDefaultDescription
--duration SECNoneHard cap per scenario. When unset, each scenario’s own max_duration applies (900 s for the eva and tau2 bases, shorter for the small demo domains).
--min-agent-turns N3Minimum agent large language model (LLM) responses for a scenario to be scored on its own merits. Pass 0 to disable.
--resume TIMESTAMPNoneReuse the existing eval_<TIMESTAMP>/ session directory under --output-dir. Exits with status 1 if that directory does not exist.
--strict-matchoffForce disallow_extra_items=True on every scenario, overriding each scenario’s own setting, so the action-list comparator requires exact-length matches.

--min-agent-turns is a stall filter for runs where the LLM backend hung. Scenarios below the threshold are counted as failures in the composite success rate and skipped in the per-signal rates (action-match, DB-state, NL-assertion) — they are not dropped from the run. Under --resume they are additionally treated as in-flight and re-run. The turn count comes from the live-recorded token_usage.agent.n_calls in metrics.json, falling back to the saved agent LLM context for older runs.

LLM Judge

Use these flags to connect and configure the optional LLM judge.

FlagDefaultDescription
--judge-url URLhttp://localhost:8000/v1/chat/completionsOpenAI-compatible chat-completions endpoint for the judge.
--judge-model NAMEnvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4Judge model name.
--judge-api-key KEYNoneInline API key. Redacted to <redacted> in run_args.json.
--judge-api-key-name VARJUDGE_API_KEYEnvironment variable read for the key when --judge-api-key is not given.
--judge-threshold F0.9Score threshold above which the judge verdict counts as a pass.
--judge-timeout SEC120.0Per-request timeout.
--judge-max-tokens N100000Generation budget. Reasoning judges spend most of it on thinking, so lowering it can truncate the verdict.
--judge-temperature F1.0Sampling temperature.
--judge-top-p F0.95Nucleus sampling top_p.
--judge-seed N42Sampling seed, for run-to-run reproducibility.
--judge-thinking-token-budget NNoneProvider-specific thinking budget; only sent when set.
--judge-include-conversationoffInclude the bridge transcript turns in the judge input.
--judge-compact-contextoffCompact the LLM context histories before sending them to the judge.
--judge-context-message-limit NNoneMax context messages, applied when --judge-compact-context is on.
--judge-context-system-string-limit NNoneMax system-message length, applied when --judge-compact-context is on.
--judge-context-string-limit NNoneMax non-system string length, applied when --judge-compact-context is on.

The runner constructs the judge only when both --judge-url and --judge-model are nonempty. Passing an empty string to either option disables judging for the run. Judge thinking is always enabled and is not exposed as a flag. Changing that behavior would silently change score semantics across runs.

The runner validates numeric options before the run starts. A violation exits through parser.error (status 2):

OptionRule
--judge-threshold, --judge-top-pfinite float between 0 and 1 inclusive
--judge-timeoutfinite float greater than 0
--judge-max-tokens, --judge-thinking-token-budget, --judge-context-message-limit, --judge-context-system-string-limit, --judge-context-string-limitpositive integer

The runner passes --judge-temperature and --judge-seed through without validation.

Exit Codes

The evaluation driver exits with one of the following status codes.

CodeMeaning
0Run completed, or --list / --list-domains printed and exited.
1Unknown scenario name, empty domain, no registered scenarios, missing --resume directory, KeyboardInterrupt, or an unhandled exception during the run.
2argparse usage error, including the judge numeric validation above.

Invocation Record

Every run writes run_args.json into the session directory with the shape {"invocations": [...]}. Each entry records:

  • started_at and the raw argv.
  • The parsed arguments, with the judge API key redacted.
  • The resolved scenario names and count.

A --resume invocation appends a new entry and soft-checks it against the previous one on the scoring-relevant fields domain, scenarios, duration, judge_url, judge_model, judge_threshold, judge_max_tokens, judge_temperature, judge_top_p, judge_seed, and strict_match. Mismatches log a warning but do not block the run.

For result artifacts (all_metrics.json, all_summary.txt, and per-scenario metrics.json), refer to Reading Results and the Metrics Dictionary. For the six scoring signals, refer to Scoring.

check_resume.py

Use check_resume.py to classify a finished or interrupted session directory without moving or deleting anything. The script applies the same classification as the runner, so its output predicts what --resume would do.

$cd evaluation
$python check_resume.py ../eval_results/eval_20260618_072325 --min-agent-turns 3
ArgumentDefaultDescription
eval_dirrequiredPath to the eval_<TIMESTAMP>/ session directory. Exits with status 1 if it is not a directory.
--min-agent-turns N0Flag scenarios with fewer than N agent LLM responses as stalled. 0 disables the check.

The --min-agent-turns default here is 0, unlike the runner’s 3. Pass the same value you intend to use with --resume if you want the preview to match.

Each subdirectory is bucketed into one of three states:

StateMeaning
completedHas a readable metrics.json with at least one turn and enough agent turns. --resume loads its metrics from disk and skips the scenario.
rerunMissing or unreadable metrics.json, zero turns, or fewer agent turns than --min-agent-turns. --resume moves it to <scenario>.killed.<resume_timestamp>/ (dropping a __KILLED__ marker file inside) and runs it again.
freshNo subdirectory yet; it runs normally.

Directories already named *.killed.* or containing __KILLED__, and top-level files such as run_args.json and evaluation_log.txt, are ignored. The script prints per-bucket counts followed by the re-run and fresh lists, with the classification reason next to each re-run entry.

For the resume workflow, refer to Resuming a Run.