> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server. # nemo_voice_agent.evaluation.scenarios.data.tau2_common Shared base for tau2-derived evaluation scenarios. This module is loaded by every tau2 domain (airline / retail / telecom). It is **not** loaded by eva\_airline — keep eva-specific assumptions out of here. Two pieces of machinery live here: 1. `_load_tau2_voice_task_index(domain, split="base")` — module-level @cache'd loader. Joins `tasks.json` (definitions) with `tasks_voice.json` (the voice-eligible id list + persona) and intersects with `split_tasks.json[split]`. Returns `id → {"task": <tasks.json entry>, "persona_name": <str>}`. 2. `Tau2BaseScenario` — superclass for every tau2 domain's base scenario. Provides: * `tau2_task` / `persona_name` cached\_properties reading from the index. * `policy` cached\_property loading `policy.md` from disk (shared across all scenarios in the domain — one read per process). * `_gold_replay` cached\_property that deepcopies the seeded DB, replays `evaluation_criteria.actions` through the scenario's toolset, and captures `(final_db, recorded_actions)`. Both `expected_scenario_db` and `reference_answer` are derived from the same replay pass (one execution, two ground-truth signals — see plan §7 Q3). Subclasses must implement `_build_tool_map(state)` so `_gold_replay` can dispatch tau2 action records to the corresponding ported tool instances. ## Module Contents ### Classes | Name | Description | | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | [`Tau2BaseScenario`](#nemo_voice_agent-evaluation-scenarios-data-tau2_common-Tau2BaseScenario) | Base class for scenarios ported from tau2-bench voice-user-sim-v1.0. | ### Functions | Name | Description | | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | [`_load_tau2_voice_task_index`](#nemo_voice_agent-evaluation-scenarios-data-tau2_common-_load_tau2_voice_task_index) | Build `id → {"task", "persona_name"}` for one tau2 domain + split. | | [`_normalize_env_record`](#nemo_voice_agent-evaluation-scenarios-data-tau2_common-_normalize_env_record) | Translate upstream tau2 `env_type`-keyed records into our `side` shape. | ### API ```python class nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario() ``` **Bases:** [Scenario](/nemo/labs-voice-agent/nemo-voice-agent/nemo_voice_agent/evaluation/scenarios/classes#nemo_voice_agent-evaluation-scenarios-classes-Scenario) Base class for scenarios ported from tau2-bench voice-user-sim-v1.0. Subclasses must set: * `domain`: one of `"airline"`, `"retail"`, `"telecom"`. * `tau2_id`: the task id within that domain (key of `tasks_voice.json`). And must implement: * `_build_tool_map(state)`: return `{tool_name: tool_instance}` for the domain's full toolset, each tool instance bound to `state` as its `shared_state`. Used by `_gold_replay` to dispatch reference actions. Optional class attribute: * `has_user_state`: when True (telecom), `setup_shared_state` is called with `side="user"` during gold replay to seed `state["user_db"]`. Note: gold replay runs **in-process** so the same `state` dict holds both `db` (agent side) and `user_db` (user side) at the same time. In a **live** run each bot's shared\_state holds only its own DB at `state["db"]`; the agent-vs-user labeling lives at the bridge boundary. See `create_get_scenario_summary_action` for the live shape; gold replay deliberately diverges to keep the replay single-pass. Everything else (`tau2_task`, `persona_name`, `policy`, `db`, `expected_scenario_db`, `reference_answer`) is derived via cached properties from the upstream data files. **`_gold_replay`** Replay `evaluation_criteria.actions` against a fresh gold environment. Mirrors tau2-bench's `evaluator_env.py` flow: deepcopy the seeded DB, instantiate the full toolset bound to the gold state, dispatch each reference action by name, and capture the resulting `(final_db, final_user_db, recorded_actions)` tuple. `final_user_db` is `None` for single-side domains (airline / retail). **DB seeding bypasses `setup_shared_state` for the agent side.** `setup_shared_state` writes `state["db_path"]` (a small string) which the **bot server's** rtvi\_actions handler resolves into `state["db"]` by loading from disk — that resolution only happens in-process on the bot server, never in this code path. Gold replay runs entirely in-process so we load `self.db` directly here. `side` tagging: tool code itself records `{"type", "args", "result"}` with no side field — same `WriteScenarioTool` instance could be used in either bot. In the live run, the bridge stamps side based on which ws produced the record. Here in the gold replay, we stamp side based on the `requestor` field of the action entry — telecom-only since airline/retail omit the key (`.get()` returns `None` → mapped to `"agent"`, matching their single-side semantics). Failures during replay are logged but don't crash — a broken gold action just produces an empty/partial reference set, which the runner will then report as a mismatch. --- **`_index_entry`** --- **`_user_scenario`** Raw `user_scenario` from this task, or an empty stub if absent. Defensive: some tau2 tasks (or test fixtures) may omit fields. The `user_*` properties below dig with `.get(...) or ""` to handle that. --- **`agent_actions`** Stub Actions for the agent — instructions/guidelines live in policy.md. --- **`agent_persona`** Stub Persona for the agent — carries persona\_name label only. Not used to assemble the agent prompt (policy.md is the source of truth — see `get_agent_prompt` docstring). `name` carries the tau2 persona label for per-persona metric slicing later. --- **`agent_resources`** Stub Resources for the agent — tools are registered at the bot-server level. --- **`agent_task`** Stub Task for the agent — content lives in policy.md. --- **`db`** Initial DB state — loaded once per domain. Subclasses override for telecom (TOML). Returns the raw parsed dict. `setup_shared_state` deep-copies this on every scenario instantiation so per-scenario mutations don't leak. --- **`domain`** --- **`expected_scenario_db`** Post-replay agent-side DB — primary signal for runner DB-hash matching. --- **`expected_user_db`** Post-replay user-side DB (telecom only). `None` for single-side domains. --- **`has_user_state`** --- **`max_duration`** --- **`persona_name`** tau2 persona-name label (metric slicing). May be None for some tasks. --- **`policy`** The agent's system prompt — loaded once per domain from policy.md. Same content for every scenario in the domain. Subclasses that compose multiple policy files (telecom: main\_policy + tech\_support\_workflow + per-issue workflows) should override. --- **`reference_answer`** Wrapped action list — `{"actions": [...]}` shape matching eva\_airline. Wrapping makes eva and tau2 produce the same reference file shape so a single comparator path handles both: the runner's `check_if_task_success` Situation 2 (dict ref + list-of-dict pred → match the pred's last dict against the ref) consumes `{"actions": [...]}` references uniformly, regardless of domain. The bridge's prediction file `final_agent_response.json` is shaped `[{"actions": [...]}]` (a list-of-1 dict carrying the same actions key), so Situation 2 lines them up cleanly. The underlying flat list (the gold-replay's recorded actions, including the `side` field stamped by `_gold_replay`) is identity-shared with `_gold_replay[2]`. Same record schema as `WriteScenarioTool._record_action` emits during the live run, so the runner does apples-to-apples comparison without schema translation. Tasks with `evaluation_criteria.actions == []` (e.g. policy-refusal tasks) get `{"actions": []}` — the agent is expected to make no mutations. --- **`split`** --- **`tau2_id`** --- **`tau2_task`** The joined `tasks.json` entry for this scenario. --- **`user_actions`** Default user-side guidelines — voice readability rule applies to every tau2 domain. Every tau2 domain involves spoken alphanumeric IDs (confirmation numbers, user IDs, phone numbers, SIM PINs). Subclasses can extend by overriding and concatenating extra guidelines. --- **`user_persona`** Simulated-user persona derived from `user_scenario.instructions`. * `task_instructions` (behavioral guidance) → `personality`. * `name` is deliberately `None` — narrative identity (real reservation holder name, user\_id, or "you are a frequent flyer" framing) comes entirely from tau2's hand-authored `known_info`. Setting `name` to tau2's `persona_name` (e.g. `"lisa_brenner"`) would prepend an inconsistent "Your name is lisa\_brenner." line that contradicts the `known_info` content (e.g. `"Your user id is 'daiki_muller_1116'."`). `scenario.persona_name` is still available on the class for metric-slicing per plan §7 Q7; it just doesn't flow into the prompt. `known_info` and `unknown_info` are NOT placed in `background`. They live in `user_resources.info_sections` as `Things you know` / `Things you don't know` subsections. Reason: Persona is identity + style; these are facts. Separating them lets the prompt clearly signal which details the simulator should NOT invent (anything not in known\_info, and especially anything explicitly in unknown\_info). --- **`user_resources`** User-side resources — `known_info` + `unknown_info` as info\_sections. Renders into the user-sim prompt as:: ## Additional Information ### Things you know \ ### Things you don't know \ Telecom subclasses override to also register user-side tools. Why both subsections: the user simulator otherwise fabricates identifiers it doesn't have (e.g. tau2\_retail\_\_16 simulator invented `PEND456` / `WATCH001` instead of saying "I don't have my order IDs"). `unknown_info` is tau2's authored hint about what the user *explicitly does not know* — for task 16 it's "You do not remember your email address". Exposing it tells the simulator both what to share AND what to admit ignorance about. Combined with `GENERAL_PROMPT`'s anti-fabrication rule, this prevents the invent-plausible-IDs failure mode while preserving the agent's discovery path (the agent must still call `find_user_id_by_name_zip` → `get_user_details` → `get_order_details` to locate the actual orders). --- **`user_task`** Simulated-user task — `reason_for_call` is the user's goal. --- ```python nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario._build_tool_map( state: dict ) -> typing.Dict[str, typing.Any] ``` Return `{tool_name: tool_instance}` for this domain's toolset. Each tool instance must be bound to `state` as its `shared_state` so mutations from gold-replay land in the gold state (not a live bot's state). Subclasses implement this by instantiating their domain's full `Tool` set with `shared_state=state`. Used by `_gold_replay` to dispatch reference-action records to the matching tool implementation. ```python nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario.get_agent_prompt() -> str ``` Tau2 `policy.md` + a minimal voice-realization addendum. **Body (policy.md) is verbatim from tau2.** Sierra Research's published voice-leaderboard numbers assume the policy goes to the agent unchanged, so we don't splice or paraphrase. The abstract `agent_persona` / `agent_task` / `agent_actions` / `agent_resources` properties exist as Scenario-contract stubs only — they do NOT participate in prompt assembly (see `agent_persona` docstring). **However**, two voice-specific addenda are appended after policy.md: * `GENERAL_PROMPT`: "spoken aloud" guidance (concise, plain text, spell numbers as words). Without this the LLM produces written-text style replies that don't synthesize well. * `VOICE_ALPHANUMERIC_RULE`: how to spell confirmation numbers / IDs. Tau2's text-mode agent never needed this — the data round-trips through ASR/TTS in voice mode and benefits significantly. These additions don't conflict with policy.md; they're realization guidance, not policy content. They sit in a clearly-marked `## Additional Notes` section so a future reader can identify what's verbatim-tau2 vs added. Subclasses can further append (e.g. `self.policy + extra`) by overriding, but must call `super().get_agent_prompt()` if they want the addenda. ```python nemo_voice_agent.evaluation.scenarios.data.tau2_common.Tau2BaseScenario.setup_shared_state( state: dict, side: str ) -> None ``` Seed the agent side with a `db_path` pointing at the domain's `db.json`. **Why path-based, not inline:** the runner JSON-serializes `state` into `shared_state_init` and the bridge sends it through the WebSocket inside an `update_system_prompt` action. Tau2's airline `db.json` is \~7 MB; serialized + protobuf-wrapped it exceeds pipecat's default 1 MB WebSocket frame limit, which closes the connection with code `1009` before the action ever reaches the bot. Path-based seeding sends a short string instead; the bot server's `rtvi_actions.create_update_system_prompt_action` handler pops `db_path` and loads the file from disk on its side (relative to `EVAL_DATA_ROOT`). Gold replay bypasses this entirely — it runs in-process so it loads `self.db` directly without going through `db_path` resolution. Subclasses with dual-DB needs (telecom) should override to also handle `side == "user"` and populate `state["user_db"]`. ```python nemo_voice_agent.evaluation.scenarios.data.tau2_common._load_tau2_voice_task_index( domain: str, split: str = 'base' ) -> typing.Dict[str, typing.Dict[str, typing.Any]] ``` Build `id → {"task", "persona_name"}` for one tau2 domain + split. `domain` is the registry namespace string (`"tau2_airline"`, `"tau2_retail"`, `"tau2_telecom"`) — it also serves as the data subdirectory name under `nemo_voice_agent/evaluation/data/`. Filtering pipeline (intersection): 1. ids = `tasks_voice.json["configs"].keys()` (voice-eligible) 2. ids &= `split_tasks.json[split]` (split membership) 3. Each retained id is joined with its `tasks.json` entry. Banking has no `split_tasks.json` — for that domain, step 2 is skipped automatically when the file is absent. Other fields under `configs.<id>.configs.<preset>` (background noise, channel/source/speech effects, interruption flags) are deliberately discarded — see plan §1 non-goal. `persona_name` is read from `configs.<id>.configs.control.persona_name` and used as a metric-slicing label only (no voice binding). Cached via `functools.cache` so the join runs at most once per (domain, split) per process. The data dir (`nemo_voice_agent/evaluation/data/tau2_<domain>/`) must exist for the requested domain — until it's populated, this function raises `FileNotFoundError`, which is the desired behavior for not-yet-ported domains. ```python nemo_voice_agent.evaluation.scenarios.data.tau2_common._normalize_env_record( rec: typing.Dict[str, typing.Any] ) -> typing.Dict[str, typing.Any] ``` Translate upstream tau2 `env_type`-keyed records into our `side` shape. Upstream task JSON uses `env_type ∈ {"user", "assistant"}` on both `initial_state.initialization_actions[]` and `evaluation_criteria.env_assertions[]`. Our framework uses `side ∈ {"user", "agent"}` (matches the bridge's existing side-tagging on action records). This helper applies the paired rename — key `env_type → side`, value `"assistant" → "agent"` — at the scenario translation boundary so the runner, bridge, predicate registry, and init-function registry all see a uniform shape. Returns a new dict; doesn't mutate the input. Other fields pass through unchanged.