nemo_voice_agent.evaluation.scenarios.data.tau2_common

Shared base for tau2-derived evaluation scenarios.

This module is loaded by every tau2 domain (airline / retail / telecom). It is not loaded by eva_airline — keep eva-specific assumptions out of here.

Two pieces of machinery live here:

_load_tau2_voice_task_index(domain, split="base") — module-level @cache’d loader. Joins tasks.json (definitions) with tasks_voice.json (the voice-eligible id list + persona) and intersects with split_tasks.json[split]. Returns id → {"task": <tasks.json entry>, "persona_name": <str>}.
Tau2BaseScenario — superclass for every tau2 domain’s base scenario. Provides:
- tau2_task / persona_name cached_properties reading from the index.
- policy cached_property loading policy.md from disk (shared across all scenarios in the domain — one read per process).
- _gold_replay cached_property that deepcopies the seeded DB, replays evaluation_criteria.actions through the scenario’s toolset, and captures (final_db, recorded_actions). Both expected_scenario_db and reference_answer are derived from the same replay pass (one execution, two ground-truth signals — see plan §7 Q3).

Subclasses must implement _build_tool_map(state) so _gold_replay can dispatch tau2 action records to the corresponding ported tool instances.

Module Contents

Classes

Name	Description
`Tau2BaseScenario`	Base class for scenarios ported from tau2-bench voice-user-sim-v1.0.

Functions

Name	Description
`_load_tau2_voice_task_index`	Build `id → {"task", "persona_name"}` for one tau2 domain + split.
`_normalize_env_record`	Translate upstream tau2 `env_type`-keyed records into our `side` shape.

API

class nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario()

Bases: Scenario

Base class for scenarios ported from tau2-bench voice-user-sim-v1.0.

Subclasses must set:

domain: one of "airline", "retail", "telecom".
tau2_id: the task id within that domain (key of tasks_voice.json).

And must implement:

_build_tool_map(state): return {tool_name: tool_instance} for the domain’s full toolset, each tool instance bound to state as its shared_state. Used by _gold_replay to dispatch reference actions.

Optional class attribute:

has_user_state: when True (telecom), setup_shared_state is called with side="user" during gold replay to seed state["user_db"]. Note: gold replay runs in-process so the same state dict holds both db (agent side) and user_db (user side) at the same time. In a live run each bot’s shared_state holds only its own DB at state["db"]; the agent-vs-user labeling lives at the bridge boundary. See create_get_scenario_summary_action for the live shape; gold replay deliberately diverges to keep the replay single-pass.

Everything else (tau2_task, persona_name, policy, db, expected_scenario_db, reference_answer) is derived via cached properties from the upstream data files.

_gold_replay

Tuple[Dict[str, Any], Optional[Dict[str, Any]], List[Dict[str, Any]]]

Replay evaluation_criteria.actions against a fresh gold environment.

Mirrors tau2-bench’s evaluator_env.py flow: deepcopy the seeded DB, instantiate the full toolset bound to the gold state, dispatch each reference action by name, and capture the resulting (final_db, final_user_db, recorded_actions) tuple. final_user_db is None for single-side domains (airline / retail).

DB seeding bypasses setup_shared_state for the agent side. setup_shared_state writes state["db_path"] (a small string) which the bot server’s rtvi_actions handler resolves into state["db"] by loading from disk — that resolution only happens in-process on the bot server, never in this code path. Gold replay runs entirely in-process so we load self.db directly here.

side tagging: tool code itself records {"type", "args", "result"} with no side field — same WriteScenarioTool instance could be used in either bot. In the live run, the bridge stamps side based on which ws produced the record. Here in the gold replay, we stamp side based on the requestor field of the action entry — telecom-only since airline/retail omit the key (.get() returns None → mapped to "agent", matching their single-side semantics).

Failures during replay are logged but don’t crash — a broken gold action just produces an empty/partial reference set, which the runner will then report as a mismatch.

_index_entry

Dict[str, Any]

_user_scenario

Dict[str, Any]

Raw user_scenario from this task, or an empty stub if absent.

Defensive: some tau2 tasks (or test fixtures) may omit fields. The user_* properties below dig with .get(...) or "" to handle that.

agent_actions

Actions

Stub Actions for the agent — instructions/guidelines live in policy.md.

agent_persona

Persona

Stub Persona for the agent — carries persona_name label only.

Not used to assemble the agent prompt (policy.md is the source of truth — see get_agent_prompt docstring). name carries the tau2 persona label for per-persona metric slicing later.

agent_resources

Resources

Stub Resources for the agent — tools are registered at the bot-server level.

agent_task

Task

Stub Task for the agent — content lives in policy.md.

Dict[str, Any]

Initial DB state — loaded once per domain. Subclasses override for telecom (TOML).

Returns the raw parsed dict. setup_shared_state deep-copies this on every scenario instantiation so per-scenario mutations don’t leak.

domain

str = ''

expected_scenario_db

Dict[str, Any]

Post-replay agent-side DB — primary signal for runner DB-hash matching.

expected_user_db

Optional[Dict[str, Any]]

Post-replay user-side DB (telecom only). None for single-side domains.

has_user_state

bool = False

max_duration

= 900

persona_name

Optional[str]

tau2 persona-name label (metric slicing). May be None for some tasks.

policy

str

The agent’s system prompt — loaded once per domain from policy.md.

Same content for every scenario in the domain. Subclasses that compose multiple policy files (telecom: main_policy + tech_support_workflow + per-issue workflows) should override.

reference_answer

Dict[str, List[Dict[str, Any]]]

Wrapped action list — {"actions": [...]} shape matching eva_airline.

Wrapping makes eva and tau2 produce the same reference file shape so a single comparator path handles both: the runner’s check_if_task_success Situation 2 (dict ref + list-of-dict pred → match the pred’s last dict against the ref) consumes {"actions": [...]} references uniformly, regardless of domain. The bridge’s prediction file final_agent_response.json is shaped [{"actions": [...]}] (a list-of-1 dict carrying the same actions key), so Situation 2 lines them up cleanly.

The underlying flat list (the gold-replay’s recorded actions, including the side field stamped by _gold_replay) is identity-shared with _gold_replay[2]. Same record schema as WriteScenarioTool._record_action emits during the live run, so the runner does apples-to-apples comparison without schema translation. Tasks with evaluation_criteria.actions == [] (e.g. policy-refusal tasks) get {"actions": []} — the agent is expected to make no mutations.

split

str = 'base'

tau2_id

str = ''

tau2_task

Dict[str, Any]

The joined tasks.json entry for this scenario.

user_actions

Actions

Default user-side guidelines — voice readability rule applies to every tau2 domain.

Every tau2 domain involves spoken alphanumeric IDs (confirmation numbers, user IDs, phone numbers, SIM PINs). Subclasses can extend by overriding and concatenating extra guidelines.

user_persona

Persona

Simulated-user persona derived from user_scenario.instructions.

task_instructions (behavioral guidance) → personality.
name is deliberately None — narrative identity (real reservation holder name, user_id, or “you are a frequent flyer” framing) comes entirely from tau2’s hand-authored known_info. Setting name to tau2’s persona_name (e.g. "lisa_brenner") would prepend an inconsistent “Your name is lisa_brenner.” line that contradicts the known_info content (e.g. "Your user id is 'daiki_muller_1116'."). scenario.persona_name is still available on the class for metric-slicing per plan §7 Q7; it just doesn’t flow into the prompt.

known_info and unknown_info are NOT placed in background. They live in user_resources.info_sections as Things you know / Things you don't know subsections. Reason: Persona is identity + style; these are facts. Separating them lets the prompt clearly signal which details the simulator should NOT invent (anything not in known_info, and especially anything explicitly in unknown_info).

user_resources

Resources

User-side resources — known_info + unknown_info as info_sections.

Renders into the user-sim prompt as::

Additional Information

Things you know

<known_info content>

Things you don’t know

<unknown_info content>

Telecom subclasses override to also register user-side tools.

Why both subsections: the user simulator otherwise fabricates identifiers it doesn’t have (e.g. tau2_retail__16 simulator invented PEND456 / WATCH001 instead of saying “I don’t have my order IDs”). unknown_info is tau2’s authored hint about what the user explicitly does not know — for task 16 it’s “You do not remember your email address”. Exposing it tells the simulator both what to share AND what to admit ignorance about. Combined with GENERAL_PROMPT’s anti-fabrication rule, this prevents the invent-plausible-IDs failure mode while preserving the agent’s discovery path (the agent must still call find_user_id_by_name_zip → get_user_details → get_order_details to locate the actual orders).

user_task

Task

Simulated-user task — reason_for_call is the user’s goal.

nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario._build_tool_map(
    state: dict
) -> typing.Dict[str, typing.Any]

Return {tool_name: tool_instance} for this domain’s toolset.

Each tool instance must be bound to state as its shared_state so mutations from gold-replay land in the gold state (not a live bot’s state). Subclasses implement this by instantiating their domain’s full Tool set with shared_state=state.

Used by _gold_replay to dispatch reference-action records to the matching tool implementation.

nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario.get_agent_prompt() -> str

Tau2 policy.md + a minimal voice-realization addendum.

Body (policy.md) is verbatim from tau2. Sierra Research’s published voice-leaderboard numbers assume the policy goes to the agent unchanged, so we don’t splice or paraphrase. The abstract agent_persona / agent_task / agent_actions / agent_resources properties exist as Scenario-contract stubs only — they do NOT participate in prompt assembly (see agent_persona docstring).

However, two voice-specific addenda are appended after policy.md:

GENERAL_PROMPT: “spoken aloud” guidance (concise, plain text, spell numbers as words). Without this the LLM produces written-text style replies that don’t synthesize well.
VOICE_ALPHANUMERIC_RULE: how to spell confirmation numbers / IDs. Tau2’s text-mode agent never needed this — the data round-trips through ASR/TTS in voice mode and benefits significantly.

These additions don’t conflict with policy.md; they’re realization guidance, not policy content. They sit in a clearly-marked ## Additional Notes section so a future reader can identify what’s verbatim-tau2 vs added.

Subclasses can further append (e.g. self.policy + extra) by overriding, but must call super().get_agent_prompt() if they want the addenda.

nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario.setup_shared_state(
    state: dict,
    side: str
) -> None

Seed the agent side with a db_path pointing at the domain’s db.json.

Why path-based, not inline: the runner JSON-serializes state into shared_state_init and the bridge sends it through the WebSocket inside an update_system_prompt action. Tau2’s airline db.json is ~7 MB; serialized + protobuf-wrapped it exceeds pipecat’s default 1 MB WebSocket frame limit, which closes the connection with code 1009 before the action ever reaches the bot. Path-based seeding sends a short string instead; the bot server’s rtvi_actions.create_update_system_prompt_action handler pops db_path and loads the file from disk on its side (relative to EVAL_DATA_ROOT).

Gold replay bypasses this entirely — it runs in-process so it loads self.db directly without going through db_path resolution.

Subclasses with dual-DB needs (telecom) should override to also handle side == "user" and populate state["user_db"].

nemo_voice_agent.evaluation.scenarios.data.tau2_common._load_tau2_voice_task_index(
    domain: str,
    split: str = 'base'
) -> typing.Dict[str, typing.Dict[str, typing.Any]]

Build id → {"task", "persona_name"} for one tau2 domain + split.

domain is the registry namespace string ("tau2_airline", "tau2_retail", "tau2_telecom") — it also serves as the data subdirectory name under nemo_voice_agent/evaluation/data/.

Filtering pipeline (intersection):

ids = tasks_voice.json["configs"].keys() (voice-eligible)
ids &= split_tasks.json[split] (split membership)
Each retained id is joined with its tasks.json entry.

Banking has no split_tasks.json — for that domain, step 2 is skipped automatically when the file is absent.

Other fields under configs.<id>.configs.<preset> (background noise, channel/source/speech effects, interruption flags) are deliberately discarded — see plan §1 non-goal. persona_name is read from configs.<id>.configs.control.persona_name and used as a metric-slicing label only (no voice binding).

Cached via functools.cache so the join runs at most once per (domain, split) per process. The data dir (nemo_voice_agent/evaluation/data/tau2_<domain>/) must exist for the requested domain — until it’s populated, this function raises FileNotFoundError, which is the desired behavior for not-yet-ported domains.

nemo_voice_agent.evaluation.scenarios.data.tau2_common._normalize_env_record(
    rec: typing.Dict[str, typing.Any]
) -> typing.Dict[str, typing.Any]

Translate upstream tau2 env_type-keyed records into our side shape.

Upstream task JSON uses env_type ∈ {"user", "assistant"} on both initial_state.initialization_actions[] and evaluation_criteria.env_assertions[]. Our framework uses side ∈ {"user", "agent"} (matches the bridge’s existing side-tagging on action records). This helper applies the paired rename — key env_type → side, value "assistant" → "agent" — at the scenario translation boundary so the runner, bridge, predicate registry, and init-function registry all see a uniform shape.

Returns a new dict; doesn’t mutate the input. Other fields pass through unchanged.