> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# nemo_voice_agent.evaluation.runner

Dynamic Voice Agent Evaluation Runner

Runs evaluation scenarios with dynamic system prompt updates.
Accepts structured Scenario objects instead of raw dicts.

## Module Contents

### Classes

| Name                                                                 | Description                   |
| -------------------------------------------------------------------- | ----------------------------- |
| [`RunAggregator`](#nemo_voice_agent-evaluation-runner-RunAggregator) | Run-level metric accumulator. |

### Functions

| Name                                                                                               | Description                                                             |
| -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [`_load_optional_trace_metrics`](#nemo_voice_agent-evaluation-runner-_load_optional_trace_metrics) | -                                                                       |
| [`run_dynamic_evaluation`](#nemo_voice_agent-evaluation-runner-run_dynamic_evaluation)             | Run evaluation with dynamic scenario switching and latency measurement. |

### API

```python
class nemo_voice_agent.evaluation.runner.RunAggregator(
    success_results: typing.List[bool] = list(),
    task_success_results: typing.List[bool] = list(),
    action_match_results: typing.List[bool] = list(),
    judge_score_results: typing.List[float] = list(),
    judge_pass_results: typing.List[bool] = list(),
    db_state_results: typing.List[bool] = list(),
    nl_assertion_results: typing.List[bool] = list(),
    db_state_assertion_results: typing.List[bool] = list(),
    clean_exit_results: typing.List[bool] = list(),
    insufficient_turns_skipped: typing.List[str] = list(),
    per_domain_success: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_task_success: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_action_match: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_judge_score: typing.Dict[str, typing.List[float]] = dict(),
    per_domain_judge_pass: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_db_state: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_nl_assertion: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_db_state_assertion: typing.Dict[str, typing.List[bool]] = dict(),
    per_domain_clean_exit: typing.Dict[str, typing.List[bool]] = dict(),
    run_token_usage: dict = (lambda: {'agent': {'n_call...
)
```

Dataclass

Run-level metric accumulator.

Holds per-signal pass-rate buckets + per-domain breakdowns + per-side
token totals across all scenarios in a run. `add_scenario(metrics, domain)`
folds a per-scenario metrics dict into all buckets uniformly — used by
both the freshly-run path (after computing metrics) and the resume-skip
path (after loading metrics.json from disk for an already-completed
scenario). Centralizing this logic guarantees the final `all_summary.txt`
aggregate is identical regardless of whether a scenario ran live or was
loaded from a prior session.

Bucket lists are kept as flat `List[bool|float]` (not keyed by
`SuccessSignal`) so the existing downstream consumers in
`run_dynamic_evaluation` (summary writer, per-domain rollups) can
reference them by name without changing 100+ call sites. Dict-key
lookups on incoming `metrics` use `SuccessSignal.*` members directly
(`StrEnum` makes them str-equal to their JSON-key values).

**`action_match_results`**

---

**`clean_exit_results`**

---

**`db_state_assertion_results`**

---

**`db_state_results`**

---

**`insufficient_turns_skipped`**

---

**`judge_pass_results`**

---

**`judge_score_results`**

---

**`nl_assertion_results`**

---

**`per_domain_action_match`**

---

**`per_domain_clean_exit`**

---

**`per_domain_db_state`**

---

**`per_domain_db_state_assertion`**

---

**`per_domain_judge_pass`**

---

**`per_domain_judge_score`**

---

**`per_domain_nl_assertion`**

---

**`per_domain_success`**

---

**`per_domain_task_success`**

---

**`run_token_usage`**

---

**`success_results`**

---

**`task_success_results`**

---

```python
nemo_voice_agent.evaluation.runner.RunAggregator.add_scenario(
    metrics: dict,
    domain: str,
    min_agent_turns: int = 0
) -> None
```

Append one scenario's metrics into all run-level + per-domain buckets.

Idempotent against incomplete metrics dicts — each signal is only
appended when its corresponding key is present and well-typed in
`metrics`. Signals not opted into by the scenario (no
`expected_scenario_db`, no NL assertions, etc.) simply don't
contribute to their bucket.

`SuccessSignal` enum members are used as the dict keys when
looking up signal values in `metrics` — StrEnum members compare
equal to their string values, so this preserves byte-stability of
the on-disk metrics.json format while pinning the lookup to a
typo-resistant symbol.

When `min_agent_turns > 0`, scenarios with fewer agent LLM responses
(stalled agent) are counted as failures in the composite success/task
buckets, skipped in the per-signal buckets (those measurements are
meaningless for a run that stalled), and recorded in
`insufficient_turns_skipped` for the resume warning. Token usage is
still rolled up so cost accounting stays accurate.

```python
nemo_voice_agent.evaluation.runner._load_optional_trace_metrics(
    scenario_dir: str
) -> typing.Optional[dict]
```

```python
nemo_voice_agent.evaluation.runner.run_dynamic_evaluation(
    user_url: str,
    agent_url: str,
    output_dir: str,
    scenarios: typing.List[nemo_voice_agent.evaluation.scenarios.classes.Scenario],
    audio_chunk_in_seconds: float = 0.016,
    duration_per_scenario: typing.Optional[int] = None,
    pause_between_scenarios: float = 0.5,
    user_output_sample_rate: int = 24000,
    agent_output_sample_rate: int = 24000,
    user_input_sample_rate: int = 16000,
    agent_input_sample_rate: int = 16000,
    output_sample_rate: int = 24000,
    global_timestamp: str = None,
    logger: nemo_voice_agent.utils.FileLogger = None,
    judge: typing.Optional[nemo_voice_agent.evaluation.utils.LLMJudge] = None,
    judge_threshold: typing.Optional[float] = None,
    judge_include_conversation: bool = False,
    strict_match: bool = False,
    min_agent_turns: int = 0
)
```

async

Run evaluation with dynamic scenario switching and latency measurement.

**Parameters:**

**`user_url`**

WebSocket URL of user (simulated user)

---

**`agent_url`**

WebSocket URL of agent being tested

---

**`output_dir`**

Output directory for results

---

**`scenarios`**

List of Scenario objects defining each evaluation scenario

---

**`audio_chunk_in_seconds`**

Audio chunk in seconds for the audio stream (default: 0.016)

---

**`duration_per_scenario`**

Maximum duration per scenario in seconds, which overrides the scenario's own max\_duration if set.

---

**`pause_between_scenarios`**

Seconds to pause between scenarios

---

**`user_output_sample_rate`**

User TTS output sample rate (default: 24000)

---

**`agent_output_sample_rate`**

Agent TTS output sample rate (default: 24000)

---

**`user_input_sample_rate`**

User STT input sample rate (default: 16000)

---

**`agent_input_sample_rate`**

Agent STT input sample rate (default: 16000)

---

**`output_sample_rate`**

Output sample rate for recorded audio (default: 24000)

---

**`global_timestamp`**

Timestamp string for output file naming

---

**`logger`**

FileLogger instance for logging

---

**`judge`**

LLMJudge instance for judging the scenario

---

**`judge_threshold`**

Threshold for judging the scenario if binary result is desired, None for score based result

---

**`judge_include_conversation`**

If True, include bridge transcript turns in the judge input.
Disabled by default because interrupted/cross-talk segments can make these turns noisy.

---

**`strict_match`**

If True, force `disallow_extra_items=True` on every scenario for this run,
overriding each scenario's own setting. Default False respects per-scenario flags.

---

**`min_agent_turns`**

scenarios with agent turns less than this number will be treated as incomplete

---