Authoring Scenarios

A scenario is a Python class that specifies one evaluation run: the simulated user’s goal, the agent’s instructions, each side’s tools, and the scoring contract. Scenario classes live under nemo_voice_agent/evaluation/scenarios/data/.

Every domain has a base class that implements domain-level defaults and is not registered. Concrete scenarios subclass the base, override only what differs, and register themselves with @register_eval_scenario. In the tau2 and eva domains most subclasses are under 20 lines because everything derives from a single tau2_id / eva_id class attribute.

Where Scenarios Live

The following table shows the package shape used by each benchmark-derived domain.

New modules must be side-imported from scenarios/data/__init__.py so the decorators run at import time. Refer to Authoring Domains for the full new-domain checklist.

The Eight Per-Side Properties

Each scenario supplies four dataclasses per side, for both user and agent — eight properties in total. They are defined in nemo_voice_agent/evaluation/scenarios/classes.py and rendered into the system prompt by get_user_prompt() / get_agent_prompt().

Persona also carries behavior_config and voice_config. Prompt rendering and the pipeline do not use these fields, so treat them only as metric-slicing labels.

Scenario-Level Fields

Use these fields to define the scenario identity, runtime limits, scoring contract, and fixture state.

The success_signals Contract

success_signals is the whitelist of signals that gate the composite is_successful verdict. It must resolve to a non-empty sequence of SuccessSignal members. Scenario.__init_subclass__ raises TypeError at class-definition time for any class that declares name without one.

Pass-rate signals (db_state_assertion_pass_rate, nl_assertion_pass_rate) are binarized at a threshold of 1.0, so every assertion must pass. The default verdict is a strict AND over whitelisted signals that produced a non-None value. If none apply, the scenario scores "N/A" and is excluded from the run rate. Signals outside the whitelist are still computed and saved under success_breakdown.excluded in metrics.json. Refer to Scoring and Metrics Reference.

Two declaration patterns cover every shipped domain — a ClassVar tuple when the whitelist is uniform, and a cached_property when it depends on per-task opt-ins. A single outlier scenario can also declare its own tuple, which shadows the base.

1
# Uniform across the domain.
2
class Tau2AirlineBaseScenario(Tau2BaseScenario):
3
    success_signals = (SuccessSignal.DB_STATE_MATCH, SuccessSignal.CLEAN_EXIT)
4
5
# Derived from a per-task opt-in, so it cannot drift from the data.
6
class Tau2RetailBaseScenario(Tau2BaseScenario):
7
    @cached_property
8
    def success_signals(self) -> tuple:
9
        if self.nl_assertions:
10
            return (SuccessSignal.DB_STATE_MATCH, SuccessSignal.NL_ASSERTION, SuccessSignal.CLEAN_EXIT)
11
        return (SuccessSignal.DB_STATE_MATCH, SuccessSignal.CLEAN_EXIT)

If strict AND is the wrong combinator for your scenario, override compute_is_successful(self, signals) instead of contorting the whitelist.

Worked Example

A complete scenario for the in-repo restaurant domain. It inherits agent_persona, agent_task, user_resources, max_duration, the text-normalization flags, and success_signals from RestaurantBaseScenario, so only the scenario-specific pieces appear here.

1
from nemo_voice_agent.evaluation.scenarios import register_eval_scenario
2
from nemo_voice_agent.evaluation.scenarios.classes import Actions, Persona, Resources, Task
3
from nemo_voice_agent.evaluation.scenarios.data.restaurant import (
4
    PIZZA_PALACE_MENU,
5
    RestaurantBaseScenario,
6
)
7
8
9
@register_eval_scenario
10
class PizzaPepperoni(RestaurantBaseScenario):
11
    """Order a pepperoni pizza with extra cheese at Pizza Palace."""
12
13
    name = "restaurant__pizza_pepperoni"
14
    description = "Order a pepperoni pizza with extra cheese at Pizza Palace"
15
    reference_answer = {
16
        "items": [
17
            {"name": "Pepperoni Pizza", "unit_price": "9.99", "quantity": "1"},
18
            {"name": "Extra Cheese", "unit_price": "1.50", "quantity": "1"},
19
        ],
20
        "customer_name": "Charlie",
21
        "customer_phone": "314-527-8960",
22
        "total_price": "11.49",
23
    }
24
25
    @property
26
    def user_persona(self) -> Persona:
27
        return Persona(
28
            role="human user",
29
            name="Charlie",
30
            background="You are a graphic designer. Your phone number is 314-527-8960.",
31
            personality="Communicative and positive, with clear needs and prompt decision-making.",
32
        )
33
34
    @property
35
    def user_task(self) -> Task:
36
        return Task(
37
            goal="Order a pepperoni pizza with extra cheese.",
38
            background="You are hungry after work and just walked into Pizza Palace.",
39
        )
40
41
    @property
42
    def user_actions(self) -> Actions:
43
        return Actions(
44
            instructions=[
45
                "Ask the agent what pizza options are available.",
46
                "Order one pepperoni pizza.",
47
                "Ask if you can add extra cheese, and add it to the order.",
48
                "Confirm the order and ask for the total price.",
49
            ],
50
            guidelines=["Provide your name and phone number when asked."],
51
        )
52
53
    @property
54
    def agent_actions(self) -> Actions:
55
        return Actions(
56
            instructions=[
57
                "Greet the user and ask what they would like to order.",
58
                "Summarize the order and confirm it is correct.",
59
                "Ask for the user's name and phone number.",
60
                "Place the order with the `PlaceOrderTool` tool and confirm it succeeded.",
61
                "Say goodbye and call the `EndConversationTool` tool.",
62
            ],
63
            guidelines=["Do not make up any items that are not on the menu."],
64
        )
65
66
    @property
67
    def agent_resources(self) -> Resources:
68
        return Resources(
69
            tools={
70
                "GetMenuTool": {"menu": PIZZA_PALACE_MENU},
71
                "PlaceOrderTool": {"auto_validate": "False"},
72
                "EndConversationTool": {},
73
            },
74
            information=["You can use the `GetMenuTool` tool to retrieve the restaurant menu."],
75
        )

EndConversationTool is mandatory in every domain: it emits the exit signal the bridge waits for, and it is what makes the CLEAN_EXIT signal pass. Without it the bridge idles until max_duration expires. Tool base classes and registration are covered in Authoring Tools.

Seeding Fixture Data

Scenarios that need a database override setup_shared_state(self, state, side). The runner calls it one time per side. The resulting dictionary is JSON-serialized into the shared_state_init argument of the apply_initialization real-time voice interface (RTVI) action. The bot handler merges the data into its own shared_state before tools are instantiated.

1
def setup_shared_state(self, state: dict, side: str) -> None:
2
    if side == "agent":
3
        state["db_path"] = f"{self.domain}/db.json"

Any db_path value is resolved bot-side against get_eval_data_root() and replaced with the loaded db key. Fixtures live in nemo_voice_agent/evaluation/data/, overridable with the EVAL_DATA_ROOT environment variable. Send a path rather than inline content for anything large — the tau2 databases exceed the WebSocket frame limit if inlined. At end of scenario the bridge pulls get_scenario_summary from each bot. The response contains the recorded actions and a db_hash. The inline DB returns only when the bridge opts in with include_db, which it does for scenarios that declare db_state_assertions. Dual-side domains that must propagate state between the two DBs also override sync_state. Refer to tau2-telecom.

Verify

Run from the evaluation/ directory, since SERVER_CONFIG_PATH and the scenario runner resolve paths against the current working directory.

$
cd evaluation
$
$
# The new scenario should appear under its domain heading.
$
python run_evaluation.py --list
$
$
# Run it alone against a live user bot and agent bot.
$
python run_evaluation.py --scenarios restaurant__pizza_pepperoni

Scenarios that produce fewer than --min-agent-turns agent turns (default 3) count as failures in the composite success rate. The per-signal rates skip them because a scenario that does not start is a defect, not an exclusion. Pass --min-agent-turns 0 to disable the filter. To start the two bot servers, refer to the Evaluation Quickstart. To interpret the output, refer to Results and the Eval CLI reference.