nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base

Tau2-telecom scenario base.

Tau2TelecomBaseScenario extends Tau2BaseScenario with three telecom-specific concerns the airline/retail bases don’t share:

Dual DB. A separate user-side TelecomUserDB (mock phone state + user surroundings) lives alongside the agent-side TelecomDB. has_user_state = True triggers user-side seeding in _gold_replay and tells the bridge to dual-pull.
Multi-file policy. The agent system prompt is two files concatenated: main_policy.md + tech_support_{manual,workflow}.md. We override policy to do the concat using a markdown horizontal-rule separator (no XML tags — matches airline/retail’s single-markdown convention; both upstream files are well-formed standalone markdown).
db_state_assertions + initialization_actions translation. Upstream uses env_type ∈ {"user", "assistant"} on both fields; the registry/framework uses side ∈ {"user", "agent"}. We translate at scenario load time so the runner / bridge / registries all see a uniform shape. The helper lives in tau2_common for reuse if another tau2 domain adds these fields.

Subclasses only need to set name and tau2_id; everything else derives from the upstream task JSON.

Module Contents

Classes

Name	Description
`Tau2TelecomBaseScenario`	Base class for scenarios ported from tau2-bench/telecom (voice-user-sim-v1.0).
`Tau2TelecomWorkflowBaseScenario`	Workflow policy variant of `Tau2TelecomBaseScenario`.

Data

TELECOM_AGENT_HOME_NETWORK_NOTE

TELECOM_AGENT_STAY_ON_TASK_GUIDELINE

TELECOM_AGENT_TOOL_AVAILABILITY_NOTE

TELECOM_PASSIVE_TOOL_USE_GUIDELINE

TELECOM_USER_INSTRUCTIONS

API

class nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.Tau2TelecomBaseScenario()

Bases: Tau2BaseScenario

Base class for scenarios ported from tau2-bench/telecom (voice-user-sim-v1.0).

Subclasses only declare name and tau2_id (the upstream task id string). For example::

@register_eval_scenario class Tau2TelecomMobileDataIssueAirplaneOnDataOff(Tau2TelecomBaseScenario): name = “tau2_telecom__mobile_data_issue__airplane_mode_on__data_mode_off” tau2_id = “[mobile_data_issue]airplane_mode_on|data_mode_off[PERSONA:None]”

agent_resources

Resources

Agent-side resources — full tau2_telecom tool surface + EndConversationTool.

OVERRIDES Tau2BaseScenario.agent_resources. Tool keys are snake_case cls.name strings matching upstream method names — the same identifiers the policy.md text references (e.g. get_customer_by_phone, send_payment_request). Single source of truth: the same key drives the bot’s tool_factory registry lookup, the LLM-visible function name, and gold-replay dispatch in _build_tool_map.

EndConversationTool is the voice-harness termination signal (registered under "default" in the global registry; the get_schema_tool_for_eval lookup falls back from "tau2_telecom" to "default" for shared harness tools). It uses its PascalCase registry key here intentionally — that’s how it’s registered.

db_state_assertions

Optional[List[Dict[str, Any]]]

Per-task db_state_assertions records, translated from upstream’s env_assertions.

Reads evaluation_criteria.env_assertions and runs each record through _normalize_env_record to rename env_type → side + "assistant" → "agent". Returns None (not []) when the task has no assertions so the runner’s if scenario.db_state_assertions: guard correctly skips verdict aggregation.

domain

str = 'tau2_telecom'

has_user_state

bool = True

initialization_actions

Optional[List[Dict[str, Any]]]

Per-task initialization_actions records, translated from upstream.

Source: initial_state.initialization_actions. Same translation as db_state_assertions — env_type → side, "assistant" → "agent"`. Returns `None when the task has none.

nl_assertions

Optional[List[str]]

Per-task nl_assertions for the LLM judge. None when absent.

policy

str

Concatenate main_policy.md + tech_support_{variant}.md.

Both files are well-formed standalone markdown (their own # / ## / ### header hierarchies). We join with a markdown horizontal-rule separator so the LLM has a clean section boundary without XML tags. The upstream telecom wraps each file in <main_policy>/<tech_support_policy> tags; we drop those because (a) airline / retail’s single-markdown-string convention doesn’t use XML, (b) markdown structure already provides hierarchy, (c) it saves ~30 tokens per request.

policy_variant

str = 'manual'

success_signals

tuple

Predicate-only gating; whole-DB hash and action match are informational.

Telecom has an open solution space — different valid action sequences land in different whole-DB states but satisfy the same outcome predicates. DB_STATE_ASSERTION (per-predicate, intent-focused) is the authoritative outcome signal. NL_ASSERTION is added when a task opts in. DB_STATE_MATCH and ACTION_MATCH are computed and saved for diagnostics but DO NOT gate the verdict (they appear in success_breakdown.excluded). JUDGE_PASSED (overall) is also non-gating — when the judge is enabled it contributes per-NL-assertion verdicts via NL_ASSERTION.

user_actions

Actions

User-side actions — telecom-specific procedural script + passive-tool-use constraint.

OVERRIDES Tau2BaseScenario.user_actions (which returns just the VOICE_ALPHANUMERIC_RULE guideline). Telecom is the first dual-side domain whose user-sim has LLM-callable tools, and the default tau2 task_instructions (preserved in user_persona.personality) was written for text-mode where the user-sim chains tool calls inside one turn. In voice mode the user-sim has to talk to the agent first — hence the explicit script + constraint here.

instructions carry the procedural sequence (describe → follow → report → close). The model uses these to anchor itself in the conversational rhythm.
guidelines carry behavioral constraints (passive tool use, voice-readable identifiers). Stay constraint-shaped so the model can check any single turn against them.

Inherits the parent’s guidelines (VOICE_ALPHANUMERIC_RULE) and appends the telecom-specific passive-tool-use rule.

user_db

Dict[str, Any]

Initial user-side DB state — loaded once per process from user_db.json.

The Pydantic round-trip in scripts/prepare_tau2_data/prepare_telecom.py materializes default fields (surroundings block, signal_strength per-network table, app_statuses defaults) that the raw TOML doesn’t carry, so downstream init / predicate code can read fields via dict access without defensive .get(default) calls.

setup_shared_state deepcopies this when seeding user-side state so per-scenario mutations don’t leak.

user_resources

Resources

User-side resources — phone-control LLM tools + known_info / unknown_info from the parent.

OVERRIDES the parent’s user_resources (which returns the same info-sections but with empty tools={}). Telecom is the first domain that populates user_resources.tools — the user-sim’s LLM needs phone-control tools (toggle_data, toggle_airplane_mode, etc.) to actually act on the device the agent is troubleshooting.

Tool kwargs are empty because tau2 tools take only shared_state which the bot server injects automatically.

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.Tau2TelecomBaseScenario._build_tool_map(
    state: dict
) -> typing.Dict[str, typing.Any]

Tool name → instance map for BOTH user-side and agent-side tools.

Consumed by Tau2BaseScenario._gold_replay to dispatch each reference action by name. Telecom is the first dual-side domain, so the map must cover both surfaces:

User-side tools operate on the user_db. In gold-replay the same state carries both state["db"] (agent’s TelecomDB) and state["user_db"] (user’s TelecomUserDB). We bridge by wrapping state in a user-view dict where state["db"] aliases state["user_db"] (gold replay) or falls back to state["db"] (live runtime, where the user bot’s state[“db”] IS the user_db).
Agent-side tools operate on the agent TelecomDB directly via state["db"] — no view wrapping needed.

The shared state["actions"] list collects both sides’ write records; the gold-replay caller (Tau2BaseScenario._gold_replay) side-tags them after replay so the runner’s comparator sees per-side action streams that align with the per-action side field in reference_answer.

Tool-name collisions between sides are impossible — upstream’s user-side and agent-side surfaces have disjoint method names (user: toggle_* / check_* / run_speed_test; agent: get_* / send_payment_request / refuel_data etc.).

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.Tau2TelecomBaseScenario.get_agent_prompt() -> str

Parent’s policy + voice notes + telecom-specific addenda.

Appends three telecom-only blocks to the parent’s policy + Additional Notes output:

TELECOM_AGENT_TOOL_AVAILABILITY_NOTE: explicit enumeration of which tools the agent CAN call vs which live on the user’s phone. Substitutes the actual snake_case tool names so the model can map policy.md references unambiguously.
TELECOM_AGENT_STAY_ON_TASK_GUIDELINE: prevents drift to unrelated discovered issues (e.g. overdue bills).
TELECOM_AGENT_HOME_NETWORK_NOTE: states the home country (US) and frontloads the “check roaming first when abroad” diagnostic rule — disambiguates user-reported locations (“France”, “overseas”) that policy.md mentions only generically as “outside the home network”.

See module-level docstrings on the three constants for the upstream-vs-voice-mode structural reasoning.

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.Tau2TelecomBaseScenario.setup_shared_state(
    state: dict,
    side: str
) -> None

Seed state["db_path"] per side (live-runtime path).

side="agent" → tau2_telecom/db.json (TelecomDB)
side="user" → tau2_telecom/user_db.json (TelecomUserDB)

Path-based seeding (not inline DB) — the bot-server’s rtvi_actions.create_update_system_prompt_action handler resolves the path against EVAL_DATA_ROOT and loads the file into shared_state["db"] on the bot side. Each bot ends up with its own DB at state["db"] (per-bot single-DB convention).

Gold replay runs in-process and bypasses this method’s user-side branch — the parent class’s _gold_replay inline-loads self.user_db directly into gold_state["user_db"].

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.Tau2TelecomBaseScenario.sync_state(
    agent_db: dict,
    user_db: dict
) -> typing.Dict[str, typing.Dict[str, typing.Any]]

Delegate to the pure sync_telecom_state function.

Called by the bridge after each write action on either bot. The actual reconciliation logic — propagating line.status, line.roaming_enabled, data-usage, and payment-request state between the two DBs — lives in sync_telecom_state so it can be unit-tested without bridge or scenario plumbing.

See Scenario.sync_state for the framework contract and sync_telecom_state’s docstring for the propagation paths.

class nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.Tau2TelecomWorkflowBaseScenario()

Bases: Tau2TelecomBaseScenario

Workflow policy variant of Tau2TelecomBaseScenario.

Mirrors upstream tau2’s --domain telecom-workflow registration: same task set, same reference actions, same predicates, same initialization actions, same agent / user tools, same data files. The ONLY difference is which tech_support policy file gets concatenated into the agent prompt — tech_support_workflow.md (procedural step-by-step) instead of tech_support_manual.md (long-form documentation).

domain stays "tau2_telecom" (inherited) so tool registry lookup, data-file paths (nemo_voice_agent/evaluation/data/tau2_telecom/), sync-applier dispatch, and predicate / init-function registries are all unchanged. Only scenario.name (the organizational-bucket key derived from name.split("__")[0]) and policy_variant (the policy file suffix) differ.

Subclasses set name and tau2_id exactly as for the manual variant — the scaffold generator emits both classes per upstream task:

@register_eval_scenario class Tau2TelecomMobileDataIssueDataModeOff(Tau2TelecomBaseScenario): name = “tau2_telecom__mobile_data_issue__data_mode_off” tau2_id = “[mobile_data_issue]data_mode_off[PERSONA:None]”

@register_eval_scenario class Tau2TelecomWorkflowMobileDataIssueDataModeOff(Tau2TelecomWorkflowBaseScenario): name = “tau2_telecom_workflow__mobile_data_issue__data_mode_off” tau2_id = “[mobile_data_issue]data_mode_off[PERSONA:None]”

Both variants resolve identical expected_scenario_db / reference_answer / db_state_assertions / initialization_actions from the upstream task — same scoring contract — so a side-by-side run yields a clean A/B comparison of which policy-prose variant produces better agent behavior.

policy_variant

str = 'workflow'

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.TELECOM_AGENT_HOME_NETWORK_NOTE = 'Home network context + location-probe rule. This telecom company operates in th...

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.TELECOM_AGENT_STAY_ON_TASK_GUIDELINE = "Stay on the user's stated problem. Address the PRIMARY issue the user reported ...

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.TELECOM_AGENT_TOOL_AVAILABILITY_NOTE = 'Tool availability for THIS voice scenario. The policy text above references man...

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.TELECOM_PASSIVE_TOOL_USE_GUIDELINE: str = 'Passive tool use only: never call a tool unless the agent has just instructed y...

nemo_voice_agent.evaluation.scenarios.data.tau2_telecom.base.TELECOM_USER_INSTRUCTIONS: List[str] = ["Greet the agent and explain your issue in natural language. Describe the sympt...