nemo_voice_agent.evaluation.scenarios.data.tau2_airline.base

Tau2-airline scenario base + seed scenarios.

Tau2AirlineBaseScenario glues together:

Tau2BaseScenario (cross-domain machinery: voice-task index, _gold_replay, Persona/Task stubs, structured user side, policy.md loader)
TAU2_AIRLINE_TOOL_NAME_TO_CLASS (the 14 ported tools)

Each scenario subclass only needs to set tau2_id (e.g. "0"). Everything else (tool map, db, policy, expected_scenario_db, reference_answer, user_persona / user_task / user_actions) derives from the upstream data files via cached properties.

There is no per-scenario clock. The whole domain is pinned to 2024-05-15 15:00:00 EST in two places that must agree: policy.md line 3 (what the agent is told) and _current_datetime() in nemo_voice_agent.evaluation.tools.tau2_airline_tools (what the tools return). No airline task sets initial_state, and unlike eva_airline — which reads a per-scenario _current_date out of its bound JSON — scenarios here expose no current_date attribute. Date-dependent policy checks (the 24h cancellation window, whether a segment has already flown) resolve against that fixed constant.

Hand-authored seed scenarios live at the bottom of this file. Auto-scaffolded scenarios land in group_Nx.py siblings via the scaffold generator at scripts/prepare_tau2_data/generate_airline_scaffolds.py.

Module Contents

Classes

Name	Description
`Tau2AirlineBaseScenario`	Base class for scenarios ported from tau2-bench/airline (voice-user-sim-v1.0).

Data

ADOPTED_NL_ASSERTIONS

GOLD_ACTION_DROPS

API

class nemo_voice_agent.evaluation.scenarios.data.tau2_airline.base.Tau2AirlineBaseScenario()

Bases: Tau2BaseScenario

Base class for scenarios ported from tau2-bench/airline (voice-user-sim-v1.0).

Subclasses must set tau2_id (the key from tasks.json). For example::

@register_eval_scenario class Tau2Airline0(Tau2AirlineBaseScenario): name = “tau2_airline__0” tau2_id = “0”

Everything else (tool registry, expected DB, reference action list, user persona/task) derives from the upstream files via Tau2BaseScenario machinery. The scenario clock is a module-level constant in the ported tools rather than a per-scenario attribute — see the module docstring.

agent_resources

Resources

Full tau2_airline tool surface (all 14 ported tools) + per-scenario info.

Tool keys are the Tool class names (matching the @register_schema_tool_for_eval registry keys); each maps to an empty kwargs dict because tau2 tools take only shared_state which the bot server injects automatically. The scenario DB flows in via setup_shared_state (seeded with a deep copy of db.json).

Note: this OVERRIDES Tau2BaseScenario.agent_resources (which returns an empty stub). The stub exists for Scenario-contract introspection; the live run needs the populated dict to actually register tools.

domain

str = 'tau2_airline'

has_user_state

bool = False

nl_assertions

Optional[List[str]]

Curated NL assertions for this task, or None when none are adopted.

None rather than [] because runner.py guards verdict aggregation with a truthiness check — an empty list would still be forwarded to the judge and produce an empty verdict array.

NOTE: this reads ADOPTED_NL_ASSERTIONS, NOT evaluation_criteria.nl_assertions. See that constant for why.

success_signals

tuple

DB-state hash + (when adopted) per-assertion NL judge verdicts.

Derived from self.nl_assertions rather than hard-coded per class, so the whitelist cannot drift from ADOPTED_NL_ASSERTIONS.

JUDGE_PASSED (the overall judge score) is deliberately NOT gating: DB_STATE_MATCH is deterministic, and where assertions are adopted NL_ASSERTION already carries per-claim verdicts, so the overall judge stays informational and the verdict does not depend on judge availability for the 26 tasks with no adopted assertions.

tau2_task

Dict[str, Any]

Upstream task entry, with GOLD_ACTION_DROPS applied.

OVERRIDES Tau2BaseScenario.tau2_task. Both expected_scenario_db and reference_answer derive from _gold_replay, which reads self.tau2_task, so patching here corrects the DB-hash target and the reference action list in one place.

The uncorrected path returns the shared index entry by identity (no copy) exactly as the base class does; only the corrected path pays for a deepcopy, and it must, because _index_entry is shared across every scenario instance built from the same task.

nemo_voice_agent.evaluation.scenarios.data.tau2_airline.base.Tau2AirlineBaseScenario._build_tool_map(
    state: dict
) -> typing.Dict[str, typing.Any]

Instantiate one of each ported tool, bound to the given state.

Used by Tau2BaseScenario._gold_replay to dispatch reference actions. For live runs the agent bot constructs its own tool instances via the usual register_schema_tools_to_llm path with shared_state=state.

nemo_voice_agent.evaluation.scenarios.data.tau2_airline.base.ADOPTED_NL_ASSERTIONS: Dict[str, Tuple[str, ...]] = {'0': ('Agent tells the user that reservation EHGLP3 cannot be cancelled.', 'Age...

nemo_voice_agent.evaluation.scenarios.data.tau2_airline.base.GOLD_ACTION_DROPS: Dict[str, FrozenSet[str]] = {'39': frozenset({'39_10'})}