tau2_airline

tau2_airline provides 50 airline customer-support scenarios ported from tau2-bench as full voice conversations. Tasks include cancellations, rebooking, upgrades, baggage changes, and compensation. The agent receives tau2’s policy.md unchanged, and scoring uses a path-independent hash of the end-state database.

At a Glance

The following table summarizes the domain’s registry, fixtures, tool surface, and state model.

Run It

Start the two bots and the bridge as described in the evaluation quickstart, then:

$
cd evaluation
$
python run_evaluation.py --domain tau2_airline

--domain filters on the tau2_airline__ name prefix. To run a subset, name scenarios explicitly:

$
cd evaluation
$
python run_evaluation.py --scenarios tau2_airline__11 tau2_airline__17

Two flags matter for a 50-scenario job. --duration is unset by default, so each scenario gets its own 900-second ceiling. Pass an integer to cap it globally. --min-agent-turns defaults to 3. Scenarios with fewer completed agent turns count as failures in the composite rate. The per-signal rates skip them instead of dropping them. Refer to the Evaluation Command-Line Interface (CLI) for all flags.

One Attribute Per Scenario

Every concrete class sets name and tau2_id and nothing else:

1
@register_eval_scenario
2
class Tau2Airline11(Tau2AirlineBaseScenario):
3
    name = "tau2_airline__11"
4
    tau2_id = "11"

Tau2BaseScenario derives the rest from the fixtures as cached properties:

There is no current_date or tool_map attribute. _build_tool_map(state) builds the tool map on demand by instantiating each TAU2_AIRLINE_TOOL_NAME_TO_CLASS entry with the given state dictionary.

The Agent Prompt

get_agent_prompt() returns policy.md verbatim, then appends one ## Additional Notes to Follow section containing four constants, in this order:

Sierra’s published voice numbers assume the policy reaches the agent unchanged, so nothing is spliced into the policy body. The agent_persona / agent_task / agent_actions stubs on the base class exist purely so that code iterating Scenario subclasses does not hit NotImplementedError — they do not participate in prompt assembly. agent_resources is the one agent-side property the base uses at runtime because the bot server reads it to register tools.

The Simulated User

The user side is assembled from the structured user_scenario block through the inherited get_user_prompt():

• task_instructions becomes the persona’s personality. reason_for_call becomes the task goal.
• known_info and unknown_info render as Things you know / Things you don't know info sections. Naming what the caller does not know is what stops the simulator from inventing plausible reservation IDs.
• user_persona.name is deliberately None. Identity comes from known_info (for example, a user ID like daiki_muller_1116). Injecting the tau2 persona_name would contradict it.
• The only user-side guideline is VOICE_ALPHANUMERIC_RULE. The user simulator gets no tools in this domain.

Tools

Fourteen tools are ported from tau2’s AirlineTools, registered under the tau2_airline namespace. The scenario also requests EndConversationTool, which resolves through the registry’s default namespace, for 15 registered tools in total. Only write tools call _record_action.

Class names are the registry keys. Action names match tau2’s method names and are what appear in recorded records. TAU2_AIRLINE_TOOL_NAME_TO_CLASS maps between the two. TransferToHumanAgentsTool mutates nothing but still records, and emits the <exit> marker after its result is delivered so the bridge can tear the session down cleanly.

Every tool subclasses _Tau2ReadTool or _Tau2WriteTool, both of which mix in _Tau2InvokeMixin. That mixin gives each tool a sync invoke(**kwargs) for gold replay and an async _execute(**kwargs) for live Pipecat calls, both routing into a single _do_work(p). New tools implement only _do_work, properties, required_properties, and DESCRIPTION. Refer to Authoring Tools.

Gold Replay, Expected DB, and reference_answer

Tau2BaseScenario._gold_replay deep-copies the seeded DB, instantiates the full tool map against it, and dispatches each entry of evaluation_criteria.actions through invoke(). One pass yields two ground truths: expected_scenario_db (the final DB) and reference_answer.

reference_answer is wrapped as {"actions": [...]} so it matches the eva shape and one comparator path serves both domains. One record, abridged to a single leg:

1
{
2
  "actions": [
3
    {
4
      "action_type": "update_reservation_flights",
5
      "name": "update_reservation_flights",
6
      "arguments": {
7
        "reservation_id": "GV1N64",
8
        "cabin": "basic_economy",
9
        "flights": [{"flight_number": "HAT003", "date": "2024-05-19"}],
10
        "payment_id": "gift_card_1642017"
11
      },
12
      "result": {"reservation_id": "GV1N64", "charged": -5244},
13
      "side": "agent"
14
    }
15
  ]
16
}

Because read tools record nothing, only the 27 tasks whose gold list contains at least one write produce a non-empty action list — 50 records across the domain. The other 23, including refusal tasks where the policy forbids the requested change, correctly produce {"actions": []}: the agent passes by making no mutation.

Scoring

success_signals = (DB_STATE_MATCH, CLEAN_EXIT). The bot hashes its own shared_state["db"] inside the get_scenario_summary handler and returns only the SHA-256 string. The runner hashes expected_scenario_db from its in-process replay and compares. Any tool sequence that lands on the correct end state passes.

ACTION_MATCH is still computed, because the scenario has a reference_answer — but it is not whitelisted, so it lands in success_breakdown.excluded as a diagnostic rather than gating the verdict. This domain sets no nl_assertions, no db_state_assertions, and no initialization_actions. Refer to the Scoring Model for how the composite is built and Reading Results for where each field is written.

Database Seeding and Key Casing

setup_shared_state writes state["db_path"] = "tau2_airline/db.json" for the agent side only. The bot resolves it against EVAL_DATA_ROOT in its apply_initialization handler. The path is sent instead of the DB itself because the airline database exceeds Pipecat’s WebSocket frame limit. On disk, it is sharded as db/flights.json, db/users.json, and db/reservations.json. load_db_artifact probes <name>.json and then the <name>/ directory. Both layouts produce an identical in-memory dictionary, so hashes are unchanged.

Automatic speech recognition (ASR) after letter-by-letter spelling returns inconsistent case, so the lookup helpers normalize:

Replay determinism also relies on three upstream behaviors reproduced verbatim. The created_at clock is frozen at 2024-05-15T15:00:00. New reservation IDs are allocated from HATHAT, HATHAU, and HATHAV in order, and certificate payment IDs come from a fixed triple.

Related

tau2_retail adds natural-language (NL) assertions on the same machinery. tau2_telecom is the dual-side variant with cross-side state sync. eva_airline is the other airline domain. To add scenarios, refer to Authoring Scenarios.