Scoring Model

The NeMo Labs Voice Agent evaluation runner emits six independent scoring signals per scenario and combines them into the is_successful composite verdict. The canonical list is the SuccessSignal enum in nemo_voice_agent/evaluation/scenarios/classes.py. Scenario.compute_is_successful computes the composite, and nemo_voice_agent/evaluation/runner.py writes it.

Every signal is opt-in per scenario through the success_signals whitelist. Signals that are computed but not whitelisted are still saved — they land in success_breakdown.excluded as diagnostics.

Signal Matrix

The following matrix maps each supported signal to its persisted field, value type, and current domain use.

Note the two float signals: the enum value written to success_breakdown is db_state_assertion / nl_assertion, while the numeric field in metrics.json carries the _pass_rate suffix.

Signal Details

ACTION_MATCH — check_if_task_success (in nemo_voice_agent/evaluation/utils.py) recursively compares scenario_config/reference_answer.json against final_agent_response.json. Every dictionary in the reference must have an order-independent match in the prediction. Extra prediction keys and list items are tolerated unless the scenario sets disallow_extra_items or the run passes --strict-match. String comparison honors the scenario’s ignore_capitalization, ignore_punctuation, and clean_text flags. "N/A" when the scenario has no reference answer. The result is False when the reference exists but the agent produced no prediction file.

DB_STATE_MATCH — the bot computes get_dict_hash(shared_state["db"]) inside the get_scenario_summary real-time voice interface (RTVI) handler and returns only the SHA-256 string. The runner hashes scenario.expected_scenario_db from its in-process gold replay and compares the values. The database (DB) itself never crosses the WebSocket. Path-independent: any action sequence landing on the correct end state passes. If the scenario declares an expected DB but the bot returned no hash, the result is False, not "N/A". Both hashes are also written to final_scenario_db_hash.txt for triage. Only the agent-side hash is scored — the user-side user_db_hash pulled for dual-side domains is recorded but does not gate this signal.

DB_STATE_ASSERTION — per-predicate scoring. Each entry in scenario.db_state_assertions contains func_name, arguments, assert_value, and side. The runner dispatches it through evaluate_db_state_assertion in nemo_voice_agent/evaluation/db_state_predicates.py. The function selects the agent or user DB based on side and calls a pure predicate registered under (domain, func_name). The pass rate is passes over total predicates. Per-predicate verdicts land in metrics.json under db_state_assertion_verdicts. Missing or raising predicates produce passed=False with an error field rather than aborting the run. Used where the solution space is open and several valid end states satisfy the same outcome. Refer to tau2_telecom.

NL_ASSERTION — natural-language claims about the conversation, judged per assertion by the large language model (LLM) judge. Only populated when the scenario declares nl_assertions and the judge ran. Per-assertion verdicts live in judge_result.json under nl_assertion_verdicts. The scenario-level rate is nl_assertion_pass_rate in metrics.json.

JUDGE_PASSED — judge_score >= --judge-threshold (threshold default 0.9). The raw float is saved separately as judge_score. The judge receives the reference and prediction payloads when they exist, both bots’ llm_context.json histories, the numbered NL assertions, and — only with --judge-include-conversation — the bridge transcript turns. Both --judge-url and --judge-model carry defaults pointing at a local OpenAI-compatible endpoint, so the judge is constructed on every run. Override them to target your own judge. Refer to the Evaluation Command-Line Interface (CLI) Reference.

CLEAN_EXIT — True only when bridge.stop_reason is [EXIT], meaning the agent voluntarily called EndConversationTool. [TIMEOUT] always fails. The raw reason is saved as stop_reason.

Composite is_successful

Scenario.compute_is_successful takes the dict of all six verdicts and returns:

• the strict AND over the whitelist entries whose verdict is not None, or
• the literal string "N/A" when no whitelisted signal was applicable (for example, a qa run with no reachable judge).

Two runner-level overrides sit on top of this:

• Stalled scenarios. With --min-agent-turns (default 3), a scenario whose agent produced fewer completed turns is forced to is_successful = False. It counts as a composite-rate failure but is skipped in the per-signal rates. The individual measurements are meaningless for a conversation that never happened. The scenario remains in the denominator.
• is_task_successful. The same conjunction with clean_exit removed from the failed set, so you can read “did the agent do the work” separately from “did the agent hang up properly”. Reported as Task Success Rate (excl. clean_exit) in all_summary.txt.

Per-Domain Whitelists

Whitelists are declared on each domain’s base scenario — a ClassVar tuple when fixed, or a cached_property when it depends on a per-task opt-in such as nl_assertions.

Design rule: prefer DB_STATE_MATCH over ACTION_MATCH wherever a domain ships an expected DB, because it is path-independent. Use JUDGE_PASSED as a gate only when no deterministic alternative exists. NL_ASSERTION runs through the judge but contributes per-claim verdicts, so it is safe to gate on.

Why CLEAN_EXIT Is Universal

Closure discipline gates every domain. An agent that performs the right work but never stops talking is not a successful agent, and a timed-out scenario uses more compute than a clean exit. The gate matters most for policy-refusal scenarios, where the expected and initial states match. Without CLEAN_EXIT, an agent that crashes at the greeting would pass DB_STATE_MATCH by doing nothing. The regression test test_every_concrete_scenario_includes_clean_exit in tests/unit/test_runner_is_successful.py fails the build if a new domain omits it.

Scenario.__init_subclass__ also raises TypeError at class-definition time if a concrete scenario (one that declares name) resolves success_signals to an empty sequence. This check prevents accidentally authoring an unscored domain. Refer to Authoring Domains.

Strict Thresholds for Float Signals

The two pass-rate signals are normalized to booleans with a threshold of exactly 1.0. A 95% pass rate means one assertion failed, and that is a defect to investigate rather than noise to round away. Values of None or "N/A" normalize to “not applicable” and drop out of the conjunction entirely.

success_breakdown

Every scenario’s metrics.json carries a success_breakdown object with four buckets of enum-value strings:

The excluded bucket is how you spot “all gating signals passed, but the agent took an unusual path” — for telecom, db_state_match and is_action_match land there on every scenario.

Where the Numbers Land

Use the session and scenario artifacts according to the level of detail you need.

For the field-by-field schema, refer to Metrics Reference. For the directory layout, refer to Reading Results.