Regenerating Benchmark Data

Benchmark fixtures and their scenario classes are committed artifacts, not runtime downloads. Two developer-only script directories produce them. Continuous integration (CI) and normal evaluation runs do not invoke these scripts. The directories have the following responsibilities:

For the resulting file locations and applicable licenses, refer to Fixture Data & Provenance.

Script Inventory

Use this inventory to identify the script that owns each generated artifact:

Paths in the Reads and Writes columns are relative to nemo_voice_agent/evaluation/. Only prepare_telecom.py needs an upstream checkout. Every other script reads data that is already committed to this repository.

Importing tau2 Telecom Data

prepare_telecom.py is the only import script. Airline and retail provide their databases as upstream .json files, which were copied directly when those domains were first ported. Telecom provides db.toml and user_db.toml, so a conversion step keeps the bot-side loader on a single json.load path.

$
uv run python scripts/prepare_tau2_data/prepare_telecom.py --source /path/to/tau2-bench

--source defaults to a developer-local absolute path, so pass it explicitly. Use --dest to redirect the output and compare a candidate import with the committed tree before overwriting it.

The script performs three transformations and copies the rest verbatim:

The run is idempotent. It prints a truncated SHA-256 hash for each output file and kept or dropped counts for filtered files. Compare these hashes with the previous run to identify upstream changes.

The repository does not currently track every file that the script can emit. audio_difficulty.json and workflows/ are traceability-only and are not consumed by the evaluation pipeline, so they are not committed. The agent policy is assembled from main_policy.md plus tech_support_manual.md or tech_support_workflow.md. For details, refer to tau2 Telecom.

Sharding an Oversized Database

The GitLab mirror enforces a 5 MB per-file cap, which the airline database exceeds. shard_db.py splits a top-level dictionary into one file per key:

$
uv run python scripts/prepare_tau2_data/shard_db.py \
>
    nemo_voice_agent/evaluation/data/tau2_airline/db.json

This writes db/flights.json, db/users.json, and db/reservations.json, and then deletes the source db.json. Pass --keep-source to retain it. load_db_artifact in nemo_voice_agent/evaluation/__init__.py probes <path>.json first and falls back to <path>/. Both layouts reassemble into an identical in-memory dictionary, and database hashes are unaffected. Shard the database again after any upstream pull that restores a single-file db.json.

Scaffolding tau2 Scenario Classes

The three tau2 scaffolders take no arguments. Each reads the imported data and divides the base-split IDs into groups of 10. It then overwrites every group_Nx.py in the target package and rewrites that package’s __init__.py with side-effect imports so @register_eval_scenario fires on import.

$
uv run python scripts/prepare_tau2_data/generate_airline_scaffolds.py   # 50 tasks  → 5 groups
$
uv run python scripts/prepare_tau2_data/generate_retail_scaffolds.py    # 114 tasks → 12 groups
$
uv run python scripts/prepare_tau2_data/generate_telecom_scaffolds.py   # 114 tasks → 12 groups

Generated classes contain only name and tau2_id. The domain base class derives everything else from the upstream task. Airline and retail use integer IDs and decade-based filenames (group_0x.py holds IDs 0–9). Telecom IDs are descriptive strings. Its filenames use a sequential chunk index, and the generator parses the class and scenario names from the ID:

tau2_id  [mms_issue]airplane_mode_on|data_mode_off[PERSONA:Hard]
name     tau2_telecom__mms_issue__airplane_mode_on__data_mode_off__hard
class    Tau2TelecomMmsIssueAirplaneModeOnDataModeOffHard

A PERSONA:None suffix is dropped from both the name and the class. The telecom generator emits two classes per task—one on Tau2TelecomBaseScenario (policy_variant="manual") and one on Tau2TelecomWorkflowBaseScenario (policy_variant="workflow")—so 114 tasks produce 228 classes across the tau2_telecom and tau2_telecom_workflow registrations. It also checks for class-name and scenario-name collisions across both variants before writing anything. It aborts rather than emitting a silently shadowed scenario.

Scaffolding eva_airline Scenarios

There is no eva import script. The eva fixtures were copied verbatim from upstream. The generator reads the packaged eva_airline_dataset.jsonl and streams to stdout instead of overwriting group files, because eva scenarios carry curated prose that the dataset alone cannot express.

$
# One major group, appended for hand review before committing.
$
uv run python scripts/prepare_eva_data/generate_airline_scaffolds.py --major 1 \
>
    >> nemo_voice_agent/evaluation/scenarios/data/eva_airline/group_1x.py
$
$
# Every major group in one pass, with splittable section markers on stdout.
$
uv run python scripts/prepare_eva_data/generate_airline_scaffolds.py > /tmp/all_scaffolds.py

--dataset points at an alternate JSONL. The per-group manifest goes to stderr so it never pollutes the generated Python. Five hand-authored seed scenarios (1.1.2, 2.1.1, 3.1.3, 5.1.1, 7.2.1) are skipped —they remain the reference for prose style. The generator applies the voice-readability rule from nemo_voice_agent/utils/voice_prompts.py, spelling confirmation numbers and airport codes out character-by-character, and folds each entry’s criteria and edge cases into the scenario guidelines.

Treat the output as a starting point: review the prose and prune bullets before committing.

Bumping the Upstream Pin

To update the tau2-bench source revision and regenerate affected artifacts, complete the following steps:

1. Check out sierra-research/tau2-bench at the new tag or commit.
2. Update PINNED_COMMIT in scripts/prepare_tau2_data/prepare_telecom.py. The check is intentionally nonblocking—a mismatch produces a warning and continues, so you can test an unpinned checkout first.
3. Re-run prepare_telecom.py, then re-shard any DB that grew past the file cap.
4. Re-run every tau2 scaffolder whose tasks.json or split_tasks.json changed.
5. Update the source and version fields in nemo_voice_agent/evaluation/data/README.md.
6. Format and lint the generated Python, then run the guard tests.

$
uv run ruff format nemo_voice_agent/evaluation/scenarios/data
$
uv run ruff check --fix nemo_voice_agent/evaluation/scenarios/data
$
uv run pytest tests/unit -m "not gpu"

tests/unit/test_tau2_telecom_scenarios.py asserts 114 manual plus 114 workflow registrations, tests/unit/test_tau2_retail_scenarios.py asserts 114 retail registrations, and the eva suites cover the airline tools and scenario metadata. A scaffolding regression shows up there first. Generated modules carry the SPDX header their templates emit, which keeps the copyright-check.yml gate green.

Related Pages

Use these pages for provenance requirements, authoring guidance, and validation details:

• Fixture Data & Provenance: On-disk layout, licenses, and EVAL_DATA_ROOT.
• Authoring Scenarios: How to write a scenario class manually.
• Adding a Domain: How to integrate a new benchmark end to end.
• Testing: Markers, suites, and CI behavior.