nemo_voice_agent.evaluation.scenarios.data.eva_airline.base

Module Contents

Classes

Name	Description
`CancellationRefund`	Voluntary cancellation with full refund to the original payment method.
`EvaAirlineBaseScenario`	-
`FabricatedDisruption`	Edge case: passenger claims a flight was canceled when it departed on time.
`IrropsCancellation`	IRROPS rebooking after airline-canceled flight (mechanical).
`MissedFlightStandby`	Passenger missed morning flight; wants free standby (driving to airport).
`VoluntaryDateChange`	Voluntary date change with cost cap and window-seat constraint.

Functions

Name	Description
`_load_eva_airline_agent_config`	Load the pinned eva `airline_agent.yaml` once per process.
`_load_eva_airline_dataset_index`	Index `eva_airline_dataset.jsonl` by scenario id, once per process.

API

class nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.CancellationRefund()

Bases: EvaAirlineBaseScenario

Voluntary cancellation with full refund to the original payment method.

description

eva_id

= '5.1.1'

name

= 'eva_airline__cancellation_refund'

user_actions

Actions

user_persona

Persona

user_task

Task

class nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.EvaAirlineBaseScenario()

Bases: Scenario

_scenario_db

dict

Load the bound eva scenario JSON. Bridge-side; cached after first read.

agent_actions

Actions

agent_persona

Persona

agent_resources

Resources

agent_task

Task

current_date

str

Scenario’s _current_date from the bound JSON. Single source of truth.

domain

str = 'eva_airline'

eva_id

str = ''

expected_scenario_db

dict

Eva-shipped expected post-run DB state for this scenario.

Sourced from eva_airline_dataset.jsonl’s ground_truth.expected_scenario_db for the matching eva_id. The runner SHA-256-hashes both this and the bridge-pulled final_scenario_db.json to score the scenario on end-state correctness (path-independent — any sequence of agent actions that lands here passes; see evaluation/db_hash.py).

Verified on 2026-05-11: a clean run of scenario 1.1.2 produces a DB whose canonical hash matches this expected state exactly. Hence we use eva’s expected_scenario_db as the ground truth for all airline scenarios rather than hand-authoring NeMo-specific expected states.

Raises KeyError if the eva_id isn’t in the dataset (e.g. a scenario we authored without a corresponding eva entry).

max_duration

= 900

policy

str

Upstream eva 0.1.3 airline instructions (content-verbatim).

success_signals

Base class for airline scenarios ported from eva.

Subclasses set only eva_id (e.g. "1.1.2") — everything else derives:

current_date — read lazily from the bound JSON’s _current_date.
DB seeding — setup_shared_state writes state["db_path"] for the action handler to resolve against EVAL_DATA_ROOT.

Subclasses also declare name, user_persona, user_task, user_actions, reference_answer (a list of expected actions, possibly empty for Q&A-only scenarios), and optionally override agent_actions / agent_resources if the scenario needs domain-specific tweaks.

The toolset is fixed: every airline scenario gets the full eva 15-tool surface plus EndConversationTool. The scenario action list and final DB state are pulled by the bridge at end-of-scenario via the get_scenario_summary RTVI action — no LLM-callable summary tool.

user_resources

Resources

nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.EvaAirlineBaseScenario.get_agent_prompt() -> str

Use eva’s airline policy as the prompt source of truth.

airline_agent.yaml supplies the role and complete policy. NeMo only appends voice-transport and evaluation-harness guidance; policy rules (fees, eligibility, compensation, escalation, and procedure) are never reconstructed locally.

nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.EvaAirlineBaseScenario.setup_shared_state(
    state: dict,
    side: str
) -> None

Seed the agent side with the scenario DB content (inline, not a path).

Symmetric with how the bridge pulls the final DB at end-of-scenario: full content travels both ways. See plan section 6.5 #8.

class nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.FabricatedDisruption()

Bases: EvaAirlineBaseScenario

Edge case: passenger claims a flight was canceled when it departed on time.

The agent should verify flight status before granting any IRROPS entitlements (fee waivers, free rebooking). Acceptable resolutions: paid rebook per voluntary policy, refund per policy, or graceful end-of-call. This scenario tests the agent’s discipline around verifying claims rather than just accommodating them.

description

eva_id

= '7.2.1'

name

= 'eva_airline__escalation_edge_case'

user_actions

Actions

user_persona

Persona

user_task

Task

class nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.IrropsCancellation()

Bases: EvaAirlineBaseScenario

IRROPS rebooking after airline-canceled flight (mechanical).

description

eva_id

= '2.1.1'

name

= 'eva_airline__irrops_cancellation'

user_actions

Actions

user_persona

Persona

user_task

Task

class nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.MissedFlightStandby()

Bases: EvaAirlineBaseScenario

Passenger missed morning flight; wants free standby (driving to airport).

description

eva_id

= '3.1.3'

name

= 'eva_airline__missed_flight_standby'

user_actions

Actions

user_persona

Persona

user_task

Task

class nemo_voice_agent.evaluation.scenarios.data.eva_airline.base.VoluntaryDateChange()

Bases: EvaAirlineBaseScenario

Voluntary date change with cost cap and window-seat constraint.

description

eva_id

= '1.1.2'

name

= 'eva_airline__voluntary_date_change'

user_actions

Actions

user_persona

Persona

user_task

Task

nemo_voice_agent.evaluation.scenarios.data.eva_airline.base._load_eva_airline_agent_config() -> dict

Load the pinned eva airline_agent.yaml once per process.

The checked-in file copies ServiceNow/eva tag 0.1.3 with trailing whitespace normalized. Keeping the upstream config intact makes the policy provenance auditable and avoids maintaining a second, hand-transcribed set of airline rules here.

nemo_voice_agent.evaluation.scenarios.data.eva_airline.base._load_eva_airline_dataset_index() -> dict

Index eva_airline_dataset.jsonl by scenario id, once per process.

The dataset.jsonl is the per-scenario metadata file shipped by eva alongside the scenario fixtures. Each line is a full dataset entry keyed by id (e.g. "1.1.2"). Callers pull whichever field they need: ground_truth.expected_scenario_db for DB-state hash matching, user_goal.decision_tree.must_have_criteria for the LLM judge, etc.

Cached via functools.cache — reads the file once across all scenario instances. The dataset is small (50 lines × ~15KB), and EVAL_DATA_ROOT doesn’t change within a process run.