Metrics Dictionary | NeMo Labs Voice Agent

This reference defines each metric that the NeMo Labs Voice Agent evaluation harness writes. The producers are nemo_voice_agent/evaluation/runner.py for per-scenario scoring and run aggregation, nemo_voice_agent/evaluation/bridge.py for turns, latency, and token counters, and nemo_voice_agent/evaluation/utils.py for the large language model (LLM) judge. For metric concepts, refer to Scoring. For artifact locations, refer to Reading Results.

Value Conventions

Use these conventions to distinguish an absent signal from an applicable signal without evidence.

Convention	Meaning
Key absent	The signal did not apply to this scenario at all (for example, no `expected_scenario_db`, or no judge configured).
`"N/A"` (string)	The signal applies in principle but had no evidence. `is_action_match` is `"N/A"` when the scenario ships no reference answer; `is_successful` is `"N/A"` when no whitelisted signal was applicable.
Pass rate as a gate	A `_pass_rate` float only counts as a pass* for the composite when it is exactly `1.0` — every assertion must hold.
Rate is `None`	A run-level rate is omitted from `all_summary.txt` entirely when its bucket is empty.

metrics.json — Always Present

The harness writes one metrics.json file in each scenario directory. These keys are present for every scenario that completes a bridge run.

Field	Type	Meaning
`scenario_name`	string	Scenario class `name`. The substring before the first `__` is the domain bucket used for per-domain rollups.
`scenario_directory`	string	Path to this scenario’s output directory. Refreshed on resume if the directory moved.
`total_turns`	int	Number of entries in `turns`.
`turns`	list	Each entry is `timestamp` (ISO 8601), `role` (`"user"` or `"agent"`), `text` (finalized transcript for that speaker segment).
`duration_seconds`	float	Bridge-measured elapsed time between its own start and end timestamps.
`scenario_duration`	float	Runner-measured wall clock around `bridge.run_scenario`. Slightly larger than `duration_seconds`.
`latency_stats`	object	Aggregate over this scenario’s `latencies`, with fields defined after this table.
`latencies`	list	One entry per measurement: `user_transcript`, `agent_transcript`, `latency_ms`.
`stop_reason`	string	Lower-level bridge stop reason, such as `"[EXIT]"`, `"[SIMULATOR_EXIT]"`, `"[INACTIVITY_TIMEOUT]"`, or `"[TIMEOUT]"`.
`inactivity_timeout_seconds`	float	Configured inactivity interval for the bridge. The default is `120.0` seconds.
`clean_exit`	bool	Compatibility success signal evaluated under `conversation_end_policy`. Under the default `tool-only` policy, only an agent `EndConversationTool` exit passes.
`end_conversation_tool_called`	bool	Whether the agent called `EndConversationTool`, independent of the selected policy verdict.
`simulator_end_reported`	bool	Whether the simulated user emitted an `<exit>...</exit>` real-time voice interface (RTVI) server message.
`last_speaker`	string or `null`	Role of the final recorded turn: `"user"`, `"agent"`, or `null` when no turn has a role.
`conversation_end_policy`	string	Policy used to evaluate `clean_exit`: `"tool-only"` or `"valid-terminal-state"`.
`conversation_end_reason`	string	Diagnostic reason that explains why the selected conversation-end policy passed or failed. Values are defined after this table.
`is_action_match`	bool or `"N/A"`	Deterministic reference-vs-prediction action-list comparison. `false` when the prediction file is missing; `"N/A"` when the reference file is missing.
`token_usage`	object	`agent` and `user` sub-objects, each with `n_calls`, `prompt`, `completion`. Accumulated from RTVI `metrics` events during the run.
`success_breakdown`	object	Signal names bucketed into `passed` / `failed` / `not_applicable` / `excluded`.
`is_successful`	bool or `"N/A"`	Composite verdict: strict AND over the applicable signals in the scenario’s `success_signals` whitelist.
`is_task_successful`	bool	Same conjunction with `clean_exit` removed from the failure list. Written whenever `is_successful` is a bool.
`auto_retry_count`	int	Number of automatic in-run re-runs that this scenario needed. `0` when the first attempt was accepted.

The latency_stats object contains count (int), plus mean_ms, p50_ms, p95_ms, min_ms, and max_ms (floats). All values are zero when count is 0. One latency is recorded per user-to-agent handoff: the wall-clock gap from the user bot’s last audio frame to the agent’s first subsequent audio frame. The value therefore includes automatic speech recognition (ASR), LLM, and text-to-speech (TTS) time-to-first-byte, not only LLM latency.

With conversation_end_policy: "valid-terminal-state", clean_exit is also true when the simulator reports a successful end or [INACTIVITY_TIMEOUT] occurs after a final user turn. An inactivity timeout after a final agent turn and the overall [TIMEOUT] both remain false. The --min-agent-turns check remains authoritative and forces stalled scenarios to fail regardless of their conversation-end evidence.

The inactivity timer resets on meaningful activity from either bot, including speaking lifecycle events, text-to-speech events, relevant server messages or actions, and audio frames detected as speech. It does not reset on continuously streamed silent audio. inactivity_timeout_seconds records the configured interval, while duration_seconds records actual elapsed runtime and scenario_duration includes runner overhead.

The conversation_end_reason value identifies the evidence used for the verdict:

Value	Meaning
`agent_end_conversation_tool`	The agent called `EndConversationTool`; both policies pass.
`simulator_reported_end`	The simulator emitted `<exit>...</exit>`; `valid-terminal-state` passes. The bridge records `stop_reason: "[SIMULATOR_EXIT]"` when this policy is enabled.
`inactivity_timeout_after_user_final_turn`	The bridge recorded `[INACTIVITY_TIMEOUT]` and `last_speaker` is `"user"`; `valid-terminal-state` passes. An overall `[TIMEOUT]` does not qualify.
`agent_end_conversation_tool_missing`	The `tool-only` policy failed because the agent did not call `EndConversationTool`.
`no_valid_terminal_evidence`	The `valid-terminal-state` policy found no accepted terminal evidence.

metrics.json — Conditional

The harness writes the following keys only when the corresponding scenario signal or run feature applies.

Field	Type	Written when
`db_state_expected_hash`	string	The scenario exposes `expected_scenario_db`. SHA-256 of the runner’s in-process gold replay.
`db_state_actual_hash`	string or `null`	Same condition. The hash the bot returned from `get_scenario_summary`; `null` if it returned none.
`db_state_match`	bool	Same condition. `true` when and only when the two hashes are equal. A missing bot hash is a `false`, not `"N/A"`.
`db_state_assertion_verdicts`	list	The scenario exposes `db_state_assertions` (tau2-telecom today). One verdict per predicate.
`db_state_assertion_pass_rate`	float	Same condition. Passing predicates divided by total predicates.
`nl_assertion_pass_rate`	float	The scenario carries `nl_assertions` and a judge ran. Passing verdicts divided by total verdicts.
`judge_score`	float	A judge was configured (`--judge-url`). Raw score in the range 0 to 1.
`judge_passed`	bool	A judge was configured and `--judge-threshold` was set. `judge_score >= judge_threshold`.
`insufficient_agent_turns`	bool	`--min-agent-turns` (default 2) fired: the agent produced fewer LLM responses than the floor. Always `true` when present.
`auto_retry_reasons`	list	`auto_retry_count` is at least 1. One short string per automatic re-run, in order, naming why the previous attempt was discarded.
`trace_metrics`	object	A `trace_metrics.json` exists in the scenario directory or in `bot_logs_agent/`. Copied verbatim; the runner never interprets it.

Each db_state_assertion_verdicts entry contains func_name (registered predicate name) and side ("agent" or "user", which selects the pulled database). It also contains passed (bool), expected (the assertion’s assert_value), actual (the predicate result, or null on error), and message (optional upstream label). The error field is null or contains the failure reason. If the predicate is missing, raises an exception, or cannot access the required database, passed is false.

success_breakdown Buckets

Names inside each bucket are the SuccessSignal string values: is_action_match, db_state_match, db_state_assertion, nl_assertion, judge_passed, clean_exit.

Bucket	Contents
`passed`	Whitelisted signals that evaluated to `true`.
`failed`	Whitelisted signals that evaluated to `false`. Non-empty means `is_successful` is `false`.
`not_applicable`	Whitelisted signals with no verdict for this scenario. If this covers the whole whitelist, `is_successful` is `"N/A"`.
`excluded`	Signals computed but not in the whitelist. Informational only; they never affect `is_successful`.

is_task_successful is derived from this object: it is true when failed contains nothing other than clean_exit.

Stalled Scenarios

When --min-agent-turns N is set (default 2), the runner counts the agent’s LLM calls — preferring token_usage.agent.n_calls, falling back to the message count in bot_logs_agent/llm_context.json. Below the floor, the scenario is forced to is_successful: false and is_task_successful: false, gets insufficient_agent_turns: true, and is skipped in every per-signal bucket. It is a counted failure in the composite rate, not an exclusion. Token usage still rolls up, so cost accounting remains accurate. Refer to Resuming a Run to retry them.

Automatic Retry Provenance

auto_retry_count tells you whether the numbers in a metrics.json came from the first attempt or from a re-run that the runner performed automatically during the session. The field is always written, so 0 means the first attempt was accepted rather than that the feature was unavailable. auto_retry_reasons appears only when the count is at least 1 and carries one entry per discarded attempt, such as 0 conversation turns (bots never exchanged audio).

Each discarded attempt keeps its own directory at <scenario>.killed.autoretry<N>.<timestamp>/, holding that attempt’s logs plus a __KILLED__ marker file. Read auto_retry_count when you need retry counts for analysis. Matching directory names is fragile, because the same .killed. naming also covers subdirectories that --resume moved aside. check_resume.py reads the same field: it lists retried scenarios that finished under Completed after an automatic in-run retry, and annotates retried scenarios that would be re-run again with ; already auto-retried <n>x in-run.

Automatic retry is controlled by --no-auto-resume-on-stale, --auto-resume-on-insufficient-turns, --max-auto-retries, and --auto-retry-backoff-secs, documented in Eval CLI Reference.

judge_result.json

The harness writes judge_result.json once per scenario when a judge is configured.

Field	Type	Meaning
`score`	float	0 to 1. Copied into `metrics.json` as `judge_score`. `0.0` on any request or parse error.
`reason`	string	The judge’s explanation, with quoted evidence. On error, `"Error: ..."`.
`judge_input`	object	`system_prompt` and `user_content` — the verbatim text the judge saw, for triage without re-deriving the prompt.
`nl_assertion_verdicts`	list	Present only when the scenario carries `nl_assertions`. One entry per assertion.
`nl_assertion_pass_count`	int	Same condition. Number of verdicts with `passed: true`.
`nl_assertion_total`	int	Same condition. Equals the number of assertions declared by the scenario.
`nl_assertion_pass_rate`	float	Same condition. `nl_assertion_pass_count / nl_assertion_total`.

The user_content inside judge_input is assembled from tagged sections, in this order and only when each input is non-empty: <reference> (the gold action list), <prediction> (the agent’s recorded actions), <conversation> (the transcript turns, included only under --judge-include-conversation), <nl_assertions> (the scenario’s assertions rendered as a 1-based numbered list), <agent_context_history> (the agent bot’s LLM context, including its system prompt and every tool call with arguments and results), and <user_context_history> (the simulated user’s LLM context, which carries the user-side tool calls in dual-side domains). Scanning for those tags is the quickest way to confirm the judge actually saw the evidence a surprising score should have rested on.

Each nl_assertion_verdicts entry has index (1-based, matching the numbered assertion the judge was shown), assertion (the assertion text, inlined so the file is self-describing), passed (bool), and reason (string). The list is normalized to exactly one entry per assertion: missing, out-of-range, or non-boolean verdicts are filled as passed: false with an explanatory reason.

all_summary.txt Run-Level Aggregates

The runner renders these rates as text without serializing them to JSON. The first column contains internal variable names, and the second contains the labels in the file. A rate is omitted when its bucket is empty.

Runner name	Label in `all_summary.txt`	Denominator
`success_rate`	`Overall Success Rate`	Scenarios with a boolean `is_successful` (the `"N/A"` ones are excluded).
`task_success_rate`	`Task Success Rate (excl. clean_exit)`	Same set of scenarios.
`action_match_rate`	`Action-list match`	Scenarios where `is_action_match` was a bool.
`db_state_success_rate`	`DB-State match`	Scenarios that declared `expected_scenario_db`.
`db_state_assertion_success_rate`	`DB-State-Assertion pass`	Predicates emitted in the run, not scenarios.
`nl_assertion_success_rate`	`NL-Assertion pass`	Assertions emitted in the run, not scenarios.
`judge_score_mean`	`Judge score mean`	Mean of `judge_score` over scenarios that were judged.
`judge_pass_rate`	`Judge passed (>= threshold)`	Scenarios with a `judge_passed` bool.
`clean_exit_rate`	`Clean exit`	Scenarios that completed a bridge run, evaluated under each run’s conversation-end policy.

The two assertion rates use assertions rather than scenarios as the denominator so that they remain comparable across domains where scenarios carry different assertion counts.

Also in the file:

Overall Latency Statistics — overall_latency_stats, pooled over every latency measurement in the run: count, mean_ms, p50_ms, p95_ms, min_ms, max_ms. Each is -1 when nothing was measured.
Token Usage — per side, n_calls, prompt, completion, and their sum, plus a Run total across both sides. The entire block is suppressed when no token events were received.
Stall warning — WARNING: N scenario(s) had fewer than N agent turn(s) ... and were counted as failures. followed by the scenario names and the suggested resume command.

Per-Domain Breakdowns

The runner prints per-domain breakdowns only when the run spans more than one domain bucket. Each breakdown uses the scenario-name prefix before __ as its key. Breakdowns are available for success rate, task success rate (excluding clean exit), DB-state match rate, NL-assertion pass rate, and DB-state-assertion pass rate. Judge score, judge pass, action match, and clean exit have no per-domain block.

Aggregation on Resume

Scenarios skipped by --resume do not re-run. Their metrics.json is loaded from disk and folded into the same buckets using RunAggregator.add_scenario. Each signal is appended only when its key is present and well-typed, so an older metrics.json missing a field simply contributes nothing to that bucket. NL-assertion verdicts are re-read from judge_result.json on this path, because only the pass rate lives in metrics.json. The resulting all_summary.txt is identical whether a scenario ran live or was loaded from a prior session.

Use these pages to connect the field definitions to scoring behavior, result inspection, and domain-specific use.

Scoring — how each of the six signals is derived and combined.
Reading Results — the full artifact layout and triage recipes.
Eval CLI Reference — the flags that switch these fields on.
tau2-telecom — the domain that exercises db_state_assertions and dual-side DB pulls.