> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# nemo_voice_agent.evaluation.scenarios.classes

## Module Contents

### Classes

| Name                                                                            | Description                                                                     |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`Actions`](#nemo_voice_agent-evaluation-scenarios-classes-Actions)             | Actions configuration for the scenario.                                         |
| [`Persona`](#nemo_voice_agent-evaluation-scenarios-classes-Persona)             | Persona configuration for the scenario.                                         |
| [`Resources`](#nemo_voice_agent-evaluation-scenarios-classes-Resources)         | Resources configuration for the scenario.                                       |
| [`Scenario`](#nemo_voice_agent-evaluation-scenarios-classes-Scenario)           | Base class for all evaluation scenarios.                                        |
| [`SuccessSignal`](#nemo_voice_agent-evaluation-scenarios-classes-SuccessSignal) | Per-scenario scoring signals that participate in the `is_successful` composite. |
| [`Task`](#nemo_voice_agent-evaluation-scenarios-classes-Task)                   | Task configuration for the scenario.                                            |

### API

```python
class nemo_voice_agent.evaluation.scenarios.classes.Actions(
    instructions: typing.List[str] = list(),
    guidelines: typing.List[str] = list()
)
```

Dataclass

Actions configuration for the scenario.

**`guidelines`**

---

**`instructions`**

---

```python
nemo_voice_agent.evaluation.scenarios.classes.Actions.to_prompt_section() -> str
```

Render these actions as a prompt section.

```python
class nemo_voice_agent.evaluation.scenarios.classes.Persona(
    role: str,
    name: typing.Optional[str] = None,
    background: typing.Optional[str] = None,
    personality: typing.Optional[str] = None,
    language: typing.Optional[str] = None,
    accent: typing.Optional[str] = None,
    behavior_config: typing.Optional[typing.Dict[str, typing.Any]] = None,
    voice_config: typing.Optional[typing.Dict[str, typing.Any]] = None
)
```

Dataclass

Persona configuration for the scenario.

**`accent`**

---

**`background`**

---

**`behavior_config`**

---

**`language`**

---

**`name`**

---

**`personality`**

---

**`role`**

---

**`voice_config`**

---

```python
nemo_voice_agent.evaluation.scenarios.classes.Persona.to_prompt_section() -> str
```

Render this persona as a prompt section.

```python
class nemo_voice_agent.evaluation.scenarios.classes.Resources(
    tools: typing.Dict[str, typing.Dict[str, str]] = dict(),
    documents: typing.Dict[str, str] = dict(),
    information: typing.List[str] = list(),
    info_sections: typing.Optional[typing.Dict[str, str]] = None
)
```

Dataclass

Resources configuration for the scenario.

**`documents`**

---

**`info_sections`**

---

**`information`**

---

**`tools`**

---

```python
nemo_voice_agent.evaluation.scenarios.classes.Resources.to_prompt_section() -> str
```

Render this resource set as a prompt section.

```python
nemo_voice_agent.evaluation.scenarios.classes.Resources.to_tools_json_string() -> str
```

Get the tools for the scenario as a JSON string.

```python
class nemo_voice_agent.evaluation.scenarios.classes.Scenario(
    noise_config: typing.Optional[nemo_voice_agent.utils.audio.NoiseConfig] = None,
    name: typing.Optional[str] = None,
    description: typing.Optional[str] = None,
    max_duration: typing.Optional[int] = None,
    reference_answer: typing.Optional[typing.Union[typing.List[typing.Dict[str, typing.Any]], typing.Dict[str, typing.Any]]] = None,
    ignore_capitalization: typing.Optional[bool] = False,
    ignore_punctuation: typing.Optional[bool] = False,
    clean_text: typing.Optional[bool] = False,
    disallow_extra_items: typing.Optional[bool] = False,
    expected_scenario_db: typing.Optional[typing.Dict[str, typing.Any]] = None,
    nl_assertions: typing.Optional[typing.List[str]] = None,
    db_state_assertions: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None,
    initialization_actions: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None,
    expected_user_db: typing.Optional[typing.Dict[str, typing.Any]] = None
)
```

Base class for all evaluation scenarios.

**`agent_actions`**

Instructions and guidelines for the agent. Override in subclasses.

---

**`agent_persona`**

Persona for the agent under test. Override in subclasses.

---

**`agent_resources`**

Resources (tools, documents, information) available to the agent. Override in subclasses.

---

**`agent_task`**

The Task for the agent under test. Override in subclasses.

---

**`domain`**

---

**`reference_file`**

---

**`success_signals`**

---

**`user_actions`**

Instructions and guidelines for the user. Override in subclasses.

---

**`user_persona`**

Persona for the simulated user. Override in subclasses.

---

**`user_resources`**

Resources (tools, documents, information) available to the user. Override in subclasses.

---

**`user_task`**

The Task for the simulated user. Override in subclasses.

---

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.__init_subclass__(
    kwargs = {}
)
```

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.compute_is_successful(
    signals: typing.Dict[nemo_voice_agent.evaluation.scenarios.classes.SuccessSignal, typing.Optional[bool]]
) -> typing.Union[bool, str]
```

Combine per-signal verdicts into the composite `is_successful`.

Default behavior: strict AND over the intersection of
`self.success_signals` and the signals that produced a non-None
value for this scenario.

**Parameters:**

**`signals`**

Mapping of every known `SuccessSignal` to its verdict
for this scenario. `None` means the signal was not
applicable (e.g., the scenario didn't opt in, or the run
config didn't enable it — judge disabled, etc.).

---

**Returns:** `Union[bool, str]`

`True` / `False` when at least one signal in

**Raises:**

* `ValueError`: if `success_signals` references a name that's
  not a valid `SuccessSignal` member.

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_agent_prompt() -> str
```

Get the agent prompt for the scenario.

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_agent_tools() -> str
```

Get the tools for the agent in a json string.

The json string should be in the following format:

```python
{
    "tool_name_1": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    "tool_name_2": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    ...
}
```

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_user_prompt() -> str
```

Get the user prompt for the scenario.

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_user_tools() -> str
```

Get the tools for the user in a json string.
The json string should be in the following format:

```python
{
    "tool_name_1": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    "tool_name_2": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    ...
}
```

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.save(
    output_dir: str
)
```

Save the scenario to a file.

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.setup_shared_state(
    state: dict,
    side: str
) -> None
```

Populate per-side `shared_state` before tools are instantiated.

Called by the runner once per scenario, separately for `side="user"`
and `side="agent"`. The resulting state is JSON-serialized and sent
to the corresponding bot server via the `shared_state_init` arg of
`update_system_prompt`.

Default no-op. Override in subclasses to seed scenario fixtures (e.g.,
a database path that the action handler resolves and loads).

Convention: any `*_path` keys placed in `state` are treated as
relative to `get_eval_data_root()` and resolved/loaded by the action
handler. `state["db_path"]` becomes `state["db"]` after resolution.

```python
nemo_voice_agent.evaluation.scenarios.classes.Scenario.sync_state(
    agent_db: dict,
    user_db: dict
) -> typing.Dict[str, typing.Dict[str, typing.Any]]
```

Reconcile cross-side state after a write action fired on either side.

Mirrors upstream tau2's `Environment.sync_tools()` — both DBs
live in different processes in voice mode (each bot owns one),
so this function runs on the bridge's in-process shadow copies
and returns per-side deltas the bridge pushes back to the bots.

Default no-op: returns `{"agent": {}, "user": {}}`. Single-side
domains (eva, tau2\_airline, tau2\_retail) keep the default —
their scenarios never have user-side LLM tools so there's
nothing to propagate.

**Contract for overrides** (currently only `Tau2TelecomBaseScenario`):

* Inputs `agent_db` and `user_db` are the bridge's shadow
  dicts, already updated by replaying the just-fired action.
* The override MAY mutate both dicts in place AND MUST return
  the per-side delta dicts the bridge dispatches via
  `apply_sync_delta`.
* A scenario that overrides `sync_state` MUST also provide
  `_build_tool_map(state) → {name: tool}` where each tool has
  a sync `invoke(**kwargs)` method. The bridge uses this to
  replay actions onto the shadow DBs before calling
  `sync_state`. Tau2 satisfies this via `_Tau2InvokeMixin`.

Delta shape is **domain-defined** — the bridge transports it
verbatim to the registered `apply_sync_delta` applier on the
receiving bot. See `evaluation/sync_appliers.py` for the
generic default applier (dotted-path field set) and per-domain
ports (e.g. `tau2_telecom_sync.apply_telecom_sync_delta`).

```python
class nemo_voice_agent.evaluation.scenarios.classes.SuccessSignal
```

**Bases:** `enum.Enum`

Per-scenario scoring signals that participate in the `is_successful` composite.

Single source of truth: scenarios reference these members in their
`success_signals` whitelist; the runner uses the same members as the
keys of its per-signal verdict dict. Values are the canonical JSON keys
written to `metrics.json` / `success_breakdown` — StrEnum members
are string-equal to their values so the on-disk format is byte-stable
across renames.

To rename a signal: edit the value here, update the corresponding
`metrics.get(<old>)` lookup in the runner, and every scenario
declaration auto-picks up the new value because they reference the
enum member (not the literal string).

**`ACTION_MATCH`**

---

**`CLEAN_EXIT`**

---

**`DB_STATE_ASSERTION`**

---

**`DB_STATE_MATCH`**

---

**`JUDGE_PASSED`**

---

**`NL_ASSERTION`**

---

```python
class nemo_voice_agent.evaluation.scenarios.classes.Task(
    goal: str,
    background: str = ''
)
```

Dataclass

Task configuration for the scenario.

**`background`**

---

**`goal`**

---

```python
nemo_voice_agent.evaluation.scenarios.classes.Task.to_prompt_section() -> str
```

Render this task as a prompt section.