nemo_voice_agent.evaluation.scenarios.classes

Module Contents

Classes

Name	Description
`Actions`	Actions configuration for the scenario.
`Persona`	Persona configuration for the scenario.
`Resources`	Resources configuration for the scenario.
`Scenario`	Base class for all evaluation scenarios.
`SuccessSignal`	Per-scenario scoring signals that participate in the `is_successful` composite.
`Task`	Task configuration for the scenario.

API

class nemo_voice_agent.evaluation.scenarios.classes.Actions(
    instructions: typing.List[str] = list(),
    guidelines: typing.List[str] = list()
)

Dataclass

Actions configuration for the scenario.

guidelines

List[str] = field(default_factory=list)

instructions

List[str] = field(default_factory=list)

nemo_voice_agent.evaluation.scenarios.classes.Actions.to_prompt_section() -> str

Render these actions as a prompt section.

class nemo_voice_agent.evaluation.scenarios.classes.Persona(
    role: str,
    name: typing.Optional[str] = None,
    background: typing.Optional[str] = None,
    personality: typing.Optional[str] = None,
    language: typing.Optional[str] = None,
    accent: typing.Optional[str] = None,
    behavior_config: typing.Optional[typing.Dict[str, typing.Any]] = None,
    voice_config: typing.Optional[typing.Dict[str, typing.Any]] = None
)

Dataclass

Persona configuration for the scenario.

accent

Optional[str] = None

background

Optional[str] = None

behavior_config

Optional[Dict[str, Any]] = None

language

Optional[str] = None

name

Optional[str] = None

personality

Optional[str] = None

role

str

voice_config

Optional[Dict[str, Any]] = None

nemo_voice_agent.evaluation.scenarios.classes.Persona.to_prompt_section() -> str

Render this persona as a prompt section.

class nemo_voice_agent.evaluation.scenarios.classes.Resources(
    tools: typing.Dict[str, typing.Dict[str, str]] = dict(),
    documents: typing.Dict[str, str] = dict(),
    information: typing.List[str] = list(),
    info_sections: typing.Optional[typing.Dict[str, str]] = None
)

Dataclass

Resources configuration for the scenario.

documents

Dict[str, str] = field(default_factory=dict)

info_sections

Optional[Dict[str, str]] = None

information

List[str] = field(default_factory=list)

tools

Dict[str, Dict[str, str]] = field(default_factory=dict)

nemo_voice_agent.evaluation.scenarios.classes.Resources.to_prompt_section() -> str

Render this resource set as a prompt section.

nemo_voice_agent.evaluation.scenarios.classes.Resources.to_tools_json_string() -> str

Get the tools for the scenario as a JSON string.

class nemo_voice_agent.evaluation.scenarios.classes.Scenario(
    noise_config: typing.Optional[nemo_voice_agent.utils.audio.NoiseConfig] = None,
    name: typing.Optional[str] = None,
    description: typing.Optional[str] = None,
    max_duration: typing.Optional[int] = None,
    reference_answer: typing.Optional[typing.Union[typing.List[typing.Dict[str, typing.Any]], typing.Dict[str, typing.Any]]] = None,
    ignore_capitalization: typing.Optional[bool] = False,
    ignore_punctuation: typing.Optional[bool] = False,
    clean_text: typing.Optional[bool] = False,
    disallow_extra_items: typing.Optional[bool] = False,
    expected_scenario_db: typing.Optional[typing.Dict[str, typing.Any]] = None,
    nl_assertions: typing.Optional[typing.List[str]] = None,
    db_state_assertions: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None,
    initialization_actions: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None,
    expected_user_db: typing.Optional[typing.Dict[str, typing.Any]] = None
)

Base class for all evaluation scenarios.

agent_actions

Actions

Instructions and guidelines for the agent. Override in subclasses.

agent_persona

Persona

Persona for the agent under test. Override in subclasses.

agent_resources

Resources

Resources (tools, documents, information) available to the agent. Override in subclasses.

agent_task

Task

The Task for the agent under test. Override in subclasses.

domain

str = 'default'

reference_file

= 'reference_answer.json'

success_signals

Optional[Sequence[SuccessSignal]] = None

user_actions

Actions

Instructions and guidelines for the user. Override in subclasses.

user_persona

Persona

Persona for the simulated user. Override in subclasses.

user_resources

Resources

Resources (tools, documents, information) available to the user. Override in subclasses.

user_task

Task

The Task for the simulated user. Override in subclasses.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.__init_subclass__(
    kwargs = {}
)

nemo_voice_agent.evaluation.scenarios.classes.Scenario.compute_is_successful(
    signals: typing.Dict[nemo_voice_agent.evaluation.scenarios.classes.SuccessSignal, typing.Optional[bool]]
) -> typing.Union[bool, str]

Combine per-signal verdicts into the composite is_successful.

Default behavior: strict AND over the intersection of self.success_signals and the signals that produced a non-None value for this scenario.

Parameters:

signals

Dict[SuccessSignal, Optional[bool]]

Mapping of every known SuccessSignal to its verdict for this scenario. None means the signal was not applicable (e.g., the scenario didn’t opt in, or the run config didn’t enable it — judge disabled, etc.).

Returns: Union[bool, str]

True / False when at least one signal in

Raises:

ValueError: if success_signals references a name that’s not a valid SuccessSignal member.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_agent_prompt() -> str

Get the agent prompt for the scenario.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_agent_tools() -> str

Get the tools for the agent in a json string.

The json string should be in the following format:

{
    "tool_name_1": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    "tool_name_2": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    ...
}

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_user_prompt() -> str

Get the user prompt for the scenario.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_user_tools() -> str

Get the tools for the user in a json string. The json string should be in the following format:

{
    "tool_name_1": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    "tool_name_2": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    ...
}

nemo_voice_agent.evaluation.scenarios.classes.Scenario.save(
    output_dir: str
)

Save the scenario to a file.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.setup_shared_state(
    state: dict,
    side: str
) -> None

Populate per-side shared_state before tools are instantiated.

Called by the runner once per scenario, separately for side="user" and side="agent". The resulting state is JSON-serialized and sent to the corresponding bot server via the shared_state_init arg of update_system_prompt.

Default no-op. Override in subclasses to seed scenario fixtures (e.g., a database path that the action handler resolves and loads).

Convention: any *_path keys placed in state are treated as relative to get_eval_data_root() and resolved/loaded by the action handler. state["db_path"] becomes state["db"] after resolution.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.sync_state(
    agent_db: dict,
    user_db: dict
) -> typing.Dict[str, typing.Dict[str, typing.Any]]

Reconcile cross-side state after a write action fired on either side.

Mirrors upstream tau2’s Environment.sync_tools() — both DBs live in different processes in voice mode (each bot owns one), so this function runs on the bridge’s in-process shadow copies and returns per-side deltas the bridge pushes back to the bots.

Default no-op: returns {"agent": {}, "user": {}}. Single-side domains (eva, tau2_airline, tau2_retail) keep the default — their scenarios never have user-side LLM tools so there’s nothing to propagate.

Contract for overrides (currently only Tau2TelecomBaseScenario):

Inputs agent_db and user_db are the bridge’s shadow dicts, already updated by replaying the just-fired action.
The override MAY mutate both dicts in place AND MUST return the per-side delta dicts the bridge dispatches via apply_sync_delta.
A scenario that overrides sync_state MUST also provide _build_tool_map(state) → {name: tool} where each tool has a sync invoke(**kwargs) method. The bridge uses this to replay actions onto the shadow DBs before calling sync_state. Tau2 satisfies this via _Tau2InvokeMixin.

Delta shape is domain-defined — the bridge transports it verbatim to the registered apply_sync_delta applier on the receiving bot. See evaluation/sync_appliers.py for the generic default applier (dotted-path field set) and per-domain ports (e.g. tau2_telecom_sync.apply_telecom_sync_delta).

class nemo_voice_agent.evaluation.scenarios.classes.SuccessSignal

Bases: enum.Enum

Per-scenario scoring signals that participate in the is_successful composite.

Single source of truth: scenarios reference these members in their success_signals whitelist; the runner uses the same members as the keys of its per-signal verdict dict. Values are the canonical JSON keys written to metrics.json / success_breakdown — StrEnum members are string-equal to their values so the on-disk format is byte-stable across renames.

To rename a signal: edit the value here, update the corresponding metrics.get(<old>) lookup in the runner, and every scenario declaration auto-picks up the new value because they reference the enum member (not the literal string).

ACTION_MATCH

= 'is_action_match'

CLEAN_EXIT

= 'clean_exit'

DB_STATE_ASSERTION

= 'db_state_assertion'

DB_STATE_MATCH

= 'db_state_match'

JUDGE_PASSED

= 'judge_passed'

NL_ASSERTION

= 'nl_assertion'

class nemo_voice_agent.evaluation.scenarios.classes.Task(
    goal: str,
    background: str = ''
)

Dataclass

Task configuration for the scenario.

background

str = field(default='')

goal

str

nemo_voice_agent.evaluation.scenarios.classes.Task.to_prompt_section() -> str

Render this task as a prompt section.