nemo_voice_agent.evaluation.scenarios.classes

Module Contents

Classes

Name	Description
`Actions`	Actions configuration for the scenario.
`Persona`	Persona configuration for the scenario.
`Resources`	Resources configuration for the scenario.
`Scenario`	Base class for all evaluation scenarios.
`SuccessSignal`	Per-scenario scoring signals that participate in the `is_successful` composite.
`Task`	Task configuration for the scenario.

API

class nemo_voice_agent.evaluation.scenarios.classes.Actions(
    instructions: typing.List[str] = list(),
    guidelines: typing.List[str] = list()
)

Dataclass

Actions configuration for the scenario.

guidelines

List[str] = field(default_factory=list)

An itemized list of guidelines that the user/agent must comply with. For example:

For user: [ "" ]
For agent: [ “Do not make up any items not on the menu”, “Always use the PlaceOrderTool tool to place the order”, “Always confirm with the user if the order is correct before placing the order”, ]

instructions

List[str] = field(default_factory=list)

An itemized list of instructions for the user/agent must follow step by step in order to complete the task. For example, for a task of ordering a pizza:

For user: [ “Ask the agent for the available pizza options”, “Order a pepperoni pizza and ask for the price”, “Ask the agent if extra cheese is available and add it if available”, “Finish the order and ask for the price”, ]
For agent: [ “Greet the user by saying ‘welcome to the pizza restaurant! ” “How can I help you today?’”, “Ask the user for what they would like to order and help them make the order”, “Summarize the order and confirm with the user if the order is correct”, “Ask the user for their name and associate it with the order”, “Place the order using the PlaceOrderTool tool, and confirm with the user ” “if the order is placed successfully”, “Thank the user for their order and say goodbye”, ]

nemo_voice_agent.evaluation.scenarios.classes.Actions.to_prompt_section() -> str

Render these actions as a prompt section.

class nemo_voice_agent.evaluation.scenarios.classes.Persona(
    role: str,
    name: typing.Optional[str] = None,
    background: typing.Optional[str] = None,
    personality: typing.Optional[str] = None,
    language: typing.Optional[str] = None,
    accent: typing.Optional[str] = None,
    behavior_config: typing.Optional[typing.Dict[str, typing.Any]] = None,
    voice_config: typing.Optional[typing.Dict[str, typing.Any]] = None
)

Dataclass

Persona configuration for the scenario.

accent

Optional[str] = None

The accent of the persona if any, e.g. “American”, “British”, “Australian”, etc. Only used for TTS generation. If provided, the prompt will have additional information about the accent.

background

Optional[str] = None

The background of the persona, e.g.,

For user: “You are a student who is studying at the university. You like to play basketball in your free time”
For agent: “You are a helpful AI agent who can help the user with their questions and tasks.”

behavior_config

Optional[Dict[str, Any]] = None

Pipeline-layer persona knobs that the LLM alone cannot realize — e.g. interrupt_tendency, enable_interruptions, backchannel_min_threshold, use_llm_backchannel. Currently parked: not consumed by to_prompt_section and not wired into the turn-taking pipeline. Future work; preserved here as a metric-slicing label and as a forward-compatible carrier for tau2-derived persona configs.

language

Optional[str] = None

The language used by the persona, e.g. “English”, “Chinese”, “Spanish”, etc. Only used for TTS generation. If provided, the prompt will have additional information about the language.

name

Optional[str] = None

The name of the persona, e.g. “Bob”, “Lisa”, “Charlie”, etc. The name and role will be combined to a sentence like “You are a {role} named {name}.” in the system prompt.

personality

Optional[str] = None

Detailed description on the personality of the persona. For example:

For user:
- “You are determined and straightforward, but sometime you make mistakes.”
- “You are passive in communication, unclear in needs, repeatedly seeks confirmation, and slow in decision-making.”
For agent:
- “You have a great sense of humor while being helpful and friendly to the user. Your responses are concise and conversational.”
- “You are friendly and helpful to the user. You can guide the user to finish their task when they show hesitation.”

role

str

The role of the persona, e.g., “human user” or “helpful AI agent”.

voice_config

Optional[Dict[str, Any]] = None

TTS-layer acoustic/voice knobs (persona_name voice binding, channel/source/speech_effects, telephony_enabled, environment, background_noise_file, etc.). Currently parked: not consumed by to_prompt_section and not wired into TTS. Future work.

nemo_voice_agent.evaluation.scenarios.classes.Persona.to_prompt_section() -> str

Render this persona as a prompt section.

class nemo_voice_agent.evaluation.scenarios.classes.Resources(
    tools: typing.Dict[str, typing.Dict[str, str]] = dict(),
    documents: typing.Dict[str, str] = dict(),
    information: typing.List[str] = list(),
    info_sections: typing.Optional[typing.Dict[str, str]] = None
)

Dataclass

Resources configuration for the scenario.

documents

Dict[str, str] = field(default_factory=dict)

A dictionary of available documents, where the key is the document name and the value is a file path. The file can be read by using a read_file tool.

info_sections

Optional[Dict[str, str]] = None

Optional structured info, rendered as ### <key> blocks under ## Additional Information. Used by tau2 to expose known_info / unknown_info as separate Things you know / Things you don't know subsections so the simulator stops fabricating facts it doesn’t have (see eval_20260603_072747 audit: user-sim invented order IDs because unknown_info was being silently dropped). None (default) preserves existing flat-list rendering for eva/customer_service/restaurant/qa/etc.

information

List[str] = field(default_factory=list)

A list of additional information strings. For example, the agent will have some FAQs or other information that is relevant to the scenario.

tools

Dict[str, Dict[str, str]] = field(default_factory=dict)

A dictionary of available tools, where the key is the tool name and the value is a dictionary of tool arguments to be passed to the tool constructor.

nemo_voice_agent.evaluation.scenarios.classes.Resources.to_prompt_section() -> str

Render this resource set as a prompt section.

nemo_voice_agent.evaluation.scenarios.classes.Resources.to_tools_json_string() -> str

Get the tools for the scenario as a JSON string.

class nemo_voice_agent.evaluation.scenarios.classes.Scenario(
    noise_config: typing.Optional[nemo_voice_agent.utils.audio.NoiseConfig] = None,
    name: typing.Optional[str] = None,
    description: typing.Optional[str] = None,
    max_duration: typing.Optional[int] = None,
    reference_answer: typing.Optional[typing.Union[typing.List[typing.Dict[str, typing.Any]], typing.Dict[str, typing.Any]]] = None,
    ignore_capitalization: typing.Optional[bool] = False,
    ignore_punctuation: typing.Optional[bool] = False,
    clean_text: typing.Optional[bool] = False,
    disallow_extra_items: typing.Optional[bool] = False,
    expected_scenario_db: typing.Optional[typing.Dict[str, typing.Any]] = None,
    nl_assertions: typing.Optional[typing.List[str]] = None,
    db_state_assertions: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None,
    initialization_actions: typing.Optional[typing.List[typing.Dict[str, typing.Any]]] = None,
    expected_user_db: typing.Optional[typing.Dict[str, typing.Any]] = None
)

Base class for all evaluation scenarios.

agent_actions

Actions

Instructions and guidelines for the agent. Override in subclasses.

agent_persona

Persona

Persona for the agent under test. Override in subclasses.

agent_resources

Resources

Resources (tools, documents, information) available to the agent. Override in subclasses.

agent_task

Task

The Task for the agent under test. Override in subclasses.

domain

str = 'default'

reference_file

= 'reference_answer.json'

success_signals

Optional[Sequence[SuccessSignal]] = None

user_actions

Actions

Instructions and guidelines for the user. Override in subclasses.

user_persona

Persona

Persona for the simulated user. Override in subclasses.

user_resources

Resources

Resources (tools, documents, information) available to the user. Override in subclasses.

user_task

Task

The Task for the simulated user. Override in subclasses.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.__init_subclass__(
    kwargs = {}
)

nemo_voice_agent.evaluation.scenarios.classes.Scenario.compute_is_successful(
    signals: typing.Dict[nemo_voice_agent.evaluation.scenarios.classes.SuccessSignal, typing.Optional[bool]]
) -> typing.Union[bool, str]

Combine per-signal verdicts into the composite is_successful.

Default behavior: strict AND over the intersection of self.success_signals and the signals that produced a non-None value for this scenario.

Parameters:

signals

Dict[SuccessSignal, Optional[bool]]

Mapping of every known SuccessSignal to its verdict for this scenario. None means the signal was not applicable (e.g., the scenario didn’t opt in, or the run config didn’t enable it — judge disabled, etc.).

Returns: Union[bool, str]

True / False when at least one signal in

Raises:

ValueError: if success_signals references a name that’s not a valid SuccessSignal member.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_agent_prompt() -> str

Get the agent prompt for the scenario.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_agent_tools() -> str

Get the tools for the agent in a json string.

The json string should be in the following format:

{
    "tool_name_1": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    "tool_name_2": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    ...
}

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_user_prompt() -> str

Get the user prompt for the scenario.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.get_user_tools() -> str

Get the tools for the user in a json string. The json string should be in the following format:

{
    "tool_name_1": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    "tool_name_2": {
        "arg1_name": "value1",
        "arg2_name": "value2",
    },
    ...
}

nemo_voice_agent.evaluation.scenarios.classes.Scenario.save(
    output_dir: str
)

Save the scenario to a file.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.setup_shared_state(
    state: dict,
    side: str
) -> None

Populate per-side shared_state before tools are instantiated.

Called by the runner once per scenario, separately for side="user" and side="agent". The resulting state is JSON-serialized and sent to the corresponding bot server via the shared_state_init arg of update_system_prompt.

Default no-op. Override in subclasses to seed scenario fixtures (e.g., a database path that the action handler resolves and loads).

Convention: any *_path keys placed in state are treated as relative to get_eval_data_root() and resolved/loaded by the action handler. state["db_path"] becomes state["db"] after resolution.

nemo_voice_agent.evaluation.scenarios.classes.Scenario.sync_state(
    agent_db: dict,
    user_db: dict
) -> typing.Dict[str, typing.Dict[str, typing.Any]]

Reconcile cross-side state after a write action fired on either side.

Mirrors upstream tau2’s Environment.sync_tools() — both DBs live in different processes in voice mode (each bot owns one), so this function runs on the bridge’s in-process shadow copies and returns per-side deltas the bridge pushes back to the bots.

Default no-op: returns {"agent": {}, "user": {}}. Single-side domains (eva, tau2_airline, tau2_retail) keep the default — their scenarios never have user-side LLM tools so there’s nothing to propagate.

Contract for overrides (currently only Tau2TelecomBaseScenario):

Inputs agent_db and user_db are the bridge’s shadow dicts, already updated by replaying the just-fired action.
The override MAY mutate both dicts in place AND MUST return the per-side delta dicts the bridge dispatches via apply_sync_delta.
A scenario that overrides sync_state MUST also provide _build_tool_map(state) → {name: tool} where each tool has a sync invoke(**kwargs) method. The bridge uses this to replay actions onto the shadow DBs before calling sync_state. Tau2 satisfies this via _Tau2InvokeMixin.

Delta shape is domain-defined — the bridge transports it verbatim to the registered apply_sync_delta applier on the receiving bot. See evaluation/sync_appliers.py for the generic default applier (dotted-path field set) and per-domain ports (e.g. tau2_telecom_sync.apply_telecom_sync_delta).

class nemo_voice_agent.evaluation.scenarios.classes.SuccessSignal

Bases: enum.Enum

Per-scenario scoring signals that participate in the is_successful composite.

Single source of truth: scenarios reference these members in their success_signals whitelist; the runner uses the same members as the keys of its per-signal verdict dict. Values are the canonical JSON keys written to metrics.json / success_breakdown — StrEnum members are string-equal to their values so the on-disk format is byte-stable across renames.

To rename a signal: edit the value here, update the corresponding metrics.get(<old>) lookup in the runner, and every scenario declaration auto-picks up the new value because they reference the enum member (not the literal string).

ACTION_MATCH

= 'is_action_match'

CLEAN_EXIT

= 'clean_exit'

DB_STATE_ASSERTION

= 'db_state_assertion'

DB_STATE_MATCH

= 'db_state_match'

JUDGE_PASSED

= 'judge_passed'

NL_ASSERTION

= 'nl_assertion'

class nemo_voice_agent.evaluation.scenarios.classes.Task(
    goal: str,
    background: str = ''
)

Dataclass

Task configuration for the scenario.

background

str = field(default='')

The background of the task for user/agent. For example:

For user: “You are hungry and just arrived at a pizza restaurant.”
For agent: “You are a restaurant assistant who wants to help the user to order food at the restaurant.”

goal

str

The goal of the task for user/agent. For example:

For user: “Order a chicken sandwich and a side salad”
For agent: “Help the user to order food at the restaurant.”

nemo_voice_agent.evaluation.scenarios.classes.Task.to_prompt_section() -> str

Render this task as a prompt section.