> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# eva_airline

`eva_airline` is a 50-scenario airline customer-service benchmark ported from
[ServiceNow/eva](https://github.com/ServiceNow/eva) (v0.1.3, MIT). In each scenario, a simulated passenger
calls the agent to change a flight, recover from a cancellation, request a refund, or challenge a fabricated
disruption claim.

Scoring is deterministic. Every scenario ships a gold post-run database from upstream, so the run is scored
by hashing the agent's final database rather than by a large language model (LLM) judge.

## Run the Domain

Use three terminals. `SERVER_CONFIG_PATH` resolves against the current working directory, so run
`cd evaluation` in each terminal first.

```bash
# Terminal 1 — simulated user bot
cd evaluation && WEBSOCKET_PORT=8766 SERVER_CONFIG_PATH=server_configs/user.yaml python bot_server.py

# Terminal 2 — agent under test
cd evaluation && WEBSOCKET_PORT=8765 SERVER_CONFIG_PATH=server_configs/agent.yaml python bot_server.py

# Terminal 3 — bridge + runner (all 50 scenarios)
cd evaluation && python run_evaluation.py --domain eva_airline
```

`--domain` filters registered scenarios by the `eva_airline__` name prefix. To run one scenario, pass its
registered name instead:

```bash
cd evaluation && python run_evaluation.py --scenarios eva_airline__voluntary_date_change
cd evaluation && python run_evaluation.py --list        # every registered scenario name
```

Refer to the [Evaluation Quickstart](/nemo/labs-voice-agent/evaluate-voice-agents/run-evaluations/quickstart) and the
[Evaluation Command-Line Interface (CLI) Reference](/nemo/labs-voice-agent/reference/evaluation/evaluation-cli)
for the full flag surface.

## Scenario Layout

Scenario classes live in the package `nemo_voice_agent/evaluation/scenarios/data/eva_airline/`. `base.py`
holds `EvaAirlineBaseScenario` plus five hand-authored seed scenarios. The `group_Nx.py` shards hold
scenarios auto-scaffolded from the dataset by `scripts/prepare_eva_data/generate_airline_scaffolds.py`.
The package `__init__.py` imports every shard so the `@register_eval_scenario` decorators run.

| Module | Scenarios | EVA IDs | Representative Content |
| --- | --- | --- | --- |
| `base.py` | 5 | 1.1.2, 2.1.1, 3.1.3, 5.1.1, 7.2.1 | Hand-authored seeds, one per major flow |
| `group_1x.py` | 8 | 1.1.3 – 1.3.2 | Voluntary date and segment changes on round trips |
| `group_2x.py` | 9 | 2.1.2 – 2.4.2 | IRROPS: airline cancellations and delays, rebooking plus vouchers |
| `group_3x.py` | 2 | 3.1.5, 3.3.4 | Missed-flight and itinerary-recovery cases |
| `group_4x.py` | 7 | 4.1.1 – 4.2.5 | Same-day changes and same-day fee waivers |
| `group_5x.py` | 7 | 5.1.2 – 5.2.6 | Cancellations, cash refunds, travel credits |
| `group_6x.py` | 4 | 6.1.1 – 6.3.4 | Rebooking with hotel vouchers, alternate airports, transfer requests |
| `group_7x.py` | 8 | 7.1.1 – 7.4.1 | Edge cases: missing auth details, fabricated disruptions, fee-waiver pressure |

Registered names are `eva_airline__voluntary_date_change`, `eva_airline__irrops_cancellation`,
`eva_airline__missed_flight_standby`, `eva_airline__cancellation_refund`,
`eva_airline__escalation_edge_case` for the seeds, and `eva_airline__<eva_id with dots as underscores>`
for the scaffolded ones (for example `eva_airline__3_1_5`).

## eva_id Drives Everything

A subclass declares `name`, `eva_id`, `description`, `user_persona`, `user_task`, and `user_actions` —
nothing else. `EvaAirlineBaseScenario` derives the rest lazily through `cached_property`.

| Member | Kind | Derived from |
| --- | --- | --- |
| `_scenario_db` | `cached_property` | `<data root>/eva_airline/<eva_id>.json`, read on first access |
| `current_date` | `cached_property` | the `_current_date` key of that JSON — single source of truth for "today" |
| `expected_scenario_db` | `cached_property` | `ground_truth.expected_scenario_db` in `eva_airline_dataset.jsonl` for the matching ID |
| `domain` | class attribute | fixed to `"eva_airline"` — the tool-registry namespace |
| `success_signals` | class attribute | `(SuccessSignal.DB_STATE_MATCH, SuccessSignal.CLEAN_EXIT)` |
| `max_duration` | class attribute | `900` seconds — voice round-trips run roughly ten times slower than text |
| `policy` | `cached_property` | the complete `instructions` field from the pinned upstream `airline_agent.yaml` |
| `agent_persona` / `agent_task` | properties | the upstream agent role and description, exposed for scenario-contract introspection |
| `agent_actions` / `agent_resources` | properties | an empty action stub and the fixed eva tool surface, respectively |

The dataset index is loaded one time per process by `_load_eva_airline_dataset_index()`, cached with
`functools.cache`. An `eva_id` with no dataset entry raises `KeyError` when `expected_scenario_db` is first
touched.

`setup_shared_state(state, side)` seeds the **agent** side only: it assigns the whole scenario database
inline to `state["db"]`. The eva fixtures are approximately 10–30 KB each, so they fit in the
`shared_state_init` argument of the `apply_initialization` real-time voice interface (RTVI) client message.
Unlike the tau2 domains,
`eva_airline` has no `db_path` indirection or user-side database.

## Agent Policy

The live agent prompt starts with the `role` and complete `instructions` from ServiceNow/eva 0.1.3's
`configs/agents/airline_agent.yaml`. A pinned copy lives at
`nemo_voice_agent/evaluation/data/eva_airline/airline_agent.yaml`. Only upstream trailing whitespace is
normalized. This keeps authentication, fees, rebooking, refunds, compensation, standby, elite-status, and
escalation rules in one auditable upstream-derived source instead of reconstructing a shorter policy in
`base.py`.

`get_agent_prompt()` preserves that policy content and then appends a clearly marked
`## Additional Notes to Follow` section containing only NeMo voice/runtime guidance:

- The scenario's current date.
- The general voice-agent prompt and spoken alphanumeric rule.
- The rule not to read internal journey IDs aloud.
- Clean conversation termination and execution-honesty guidance.

The YAML's tool declarations are retained for provenance, but the callable surface is still defined by
`agent_resources` and the NeMo eva tool implementations described below.

## Fixture Layout

Fixtures ship inside the installed library at `nemo_voice_agent/evaluation/data/eva_airline/`, resolved by
`get_eval_data_root()`. Set `EVAL_DATA_ROOT` to point at a different tree.

| Path | Contents |
| --- | --- |
| `eva_airline/<eva_id>.json` (50 files) | Per-scenario database: `_current_date`, `reservations`, `journeys`, `disruptions`, `travel_credits`, `meal_vouchers`, `refunds` |
| `eva_airline/eva_airline_dataset.jsonl` (50 lines) | Per-scenario metadata keyed by `id`: `user_goal`, `user_config`, `expected_flow`, `scenario_context`, `ground_truth` |
| `eva_airline/airline_agent.yaml` | Pinned upstream agent configuration; `role` and `instructions` form the policy portion of the live agent prompt |

Provenance and license notes are recorded in the data directory's `README.md`. Refer to
[Data provenance](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/fixture-data-provenance).

## Tool Surface

Every eva_airline scenario exposes the same fixed 15-tool eva surface plus the harness-generic
`EndConversationTool`. Implementations live in `nemo_voice_agent/evaluation/tools/eva_airline_tools.py`,
registered under the `eva_airline` domain namespace.

| Kind | Tools |
| --- | --- |
| Read (4) | `GetReservationTool`, `GetFlightStatusTool`, `GetDisruptionInfoTool`, `SearchRebookingOptionsTool` |
| Write (10) | `RebookFlightTool`, `CancelReservationTool`, `ProcessRefundTool`, `AssignSeatTool`, `AddBaggageAllowanceTool`, `AddMealRequestTool`, `AddToStandbyTool`, `IssueTravelCreditTool`, `IssueHotelVoucherTool`, `IssueMealVoucherTool` |
| System (1) | `TransferToAgentTool` |
| Harness (1) | `EndConversationTool` |

Write tools subclass `WriteAirlineTool`, which binds `ACTION_TYPES` to `AIRLINE_ACTION_TYPES`
(`rebook_flight`, `cancel_reservation`, `process_refund`, `issue_meal_voucher`, `issue_hotel_voucher`,
`issue_travel_credit`, `assign_seat`, `add_baggage_allowance`, `add_meal_request`, `add_to_standby`,
`transfer_to_agent`) and appends a record to `shared_state["actions"]` on success. Read tools record
nothing. To add or change a tool, refer to [Authoring Tools](/nemo/labs-voice-agent/evaluate-voice-agents/create-evaluations/authoring-tools).

### Ancillaries Carried Across a Rebook

`RebookFlightTool` copies `bags_checked` and `meal_request` from the replaced booking onto the new segments.
The tool deliberately does not carry `seat` because each aircraft has its own seat map. The gold replay
expects an explicit `AssignSeatTool` call. The rule is to carry what the dataset gives no
availability model for, and re-select what it does.

This is a **deliberate divergence from upstream**. ServiceNow's `eva` hard-codes both fields to
`0` / `None` on rebook, yet its own `expected_scenario_db` keeps the original checked-bag count for all
25 rebooking scenarios in the packaged dataset. With the upstream behavior, the gold state for 17 of the 50 scenarios
required an additional `AddBaggageAllowanceTool` call with the original count. Those scenarios do not request
that write. Both fields remain defaults rather than locks:
`AddBaggageAllowanceTool` and `AddMealRequestTool` still override afterwards. Carrying `meal_request` is
a no-op against the packaged fixtures, which leave it unset throughout.

## Scoring

The domain whitelists two of the six scoring signals:

| Signal | How It Is Produced |
| --- | --- |
| `db_state_match` | The bot hashes its own `shared_state["db"]` and returns the SHA-256 string in the `get_scenario_summary` response, alongside the recorded action list. The runner hashes `scenario.expected_scenario_db` from its in-process gold replay and compares strings. The database itself never crosses the WebSocket. |
| `clean_exit` | The agent called `EndConversationTool` and the conversation terminated normally. |

Hash matching is path-independent: any sequence of tool calls that lands on the gold end state passes.
Both sides import the same canonicalization module, `nemo_voice_agent/evaluation/db_hash.py`, so the two
hashes are comparable byte for byte.

No eva_airline scenario declares a `reference_answer`, so `is_action_match` does not participate — the
commented-out block in `VoluntaryDateChange` is kept only as a worked example of the action-list shape.
Scenarios that complete fewer than `--min-agent-turns` agent turns (default `3`) are counted as failures
in the composite rate and skipped in the per-signal rates. Details in [Scoring](/nemo/labs-voice-agent/evaluate-voice-agents/understand-scoring/scoring-model) and
[Metrics reference](/nemo/labs-voice-agent/reference/evaluation/metrics-dictionary).

## Voice-Readability Rule

Confirmation codes, flight numbers, and airport codes are the main failure surface in a spoken airline
call. `VOICE_ALPHANUMERIC_RULE` is a module-level constant in `nemo_voice_agent/utils/voice_prompts.py`.
`EvaAirlineBaseScenario` imports it into both the agent guidelines and each scenario's user guidelines.

The rule requires spelling each character one at a time. Speak letters as letters and digits as words.
Pronounce punctuation literally: `_` as "underscore", `-` as "dash", `@` as "at", `.` as "dot", `#` as
"hash", and `*` as "star".

The load-bearing clause: **speak ONLY the spelled-out form, never the canonical sequence alongside it in
the same utterance.**

- Correct: "E, P, X, Y, E, K"
- Wrong: "EPXYEK, spelled E, P, X, Y, E, K"

The `CODE (spelled out as ...)` notation that appears throughout scenario prose is instructional metadata
for the model, not a response template. One exception: proper names such as "Johnson" are real words, so
the model can say the name and then spell it.

The agent guidelines add one airline-specific companion rule — internal journey IDs such as
`FL_SK621_20260320` are never read aloud. Flights are referred to by flight number and date.

## Extending the Domain

Choose the extension path that matches whether you are adding scenario coverage, tools, or fixture data.

- Add a scenario: subclass `EvaAirlineBaseScenario`, set `name` / `eva_id` / `description` and the three
  user-side members, and decorate with `@register_eval_scenario`. Refer to
  [Authoring scenarios](/nemo/labs-voice-agent/evaluate-voice-agents/create-evaluations/authoring-scenarios).
- Regenerate the scaffolded shards from the dataset with
  `scripts/prepare_eva_data/generate_airline_scaffolds.py`. Generated prose is marked "Review prose before
  shipping" in the class docstring and is meant to be edited by hand.
- Build a new domain from a different upstream corpus:
  [Authoring Domains](/nemo/labs-voice-agent/evaluate-voice-agents/create-evaluations/adding-a-domain). For the tau2-based domains, refer to
  [tau2_airline](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/tau-2-airline), [tau2_retail](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/tau-2-retail), and [tau2_telecom](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/tau-2-telecom).