> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# Regenerating Benchmark Data

Benchmark fixtures and their scenario classes are **committed artifacts**, not runtime downloads. Two
developer-only script directories produce them. Continuous integration (CI) and normal evaluation runs do
not invoke these scripts. The directories have the following responsibilities:

| Directory | Purpose |
| --- | --- |
| `scripts/prepare_eva_data/` | Scaffold `eva_airline` scenario classes from the packaged eva dataset |
| `scripts/prepare_tau2_data/` | Import tau2-bench upstream data, shard oversized databases, and scaffold the three tau2 domains |

For the resulting file locations and applicable licenses, refer to
[Fixture Data & Provenance](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/fixture-data-provenance).

## Script Inventory

Use this inventory to identify the script that owns each generated artifact:

| Script | Reads | Writes |
| --- | --- | --- |
| `prepare_tau2_data/prepare_telecom.py` | A local `tau2-bench` checkout | `nemo_voice_agent/evaluation/data/tau2_telecom/` |
| `prepare_tau2_data/shard_db.py` | A single `db.json` | A sibling `db/` directory of per-table shards |
| `prepare_tau2_data/generate_airline_scaffolds.py` | `data/tau2_airline/` | `scenarios/data/tau2_airline/group_Nx.py` |
| `prepare_tau2_data/generate_retail_scaffolds.py` | `data/tau2_retail/` | `scenarios/data/tau2_retail/group_Nx.py` |
| `prepare_tau2_data/generate_telecom_scaffolds.py` | `data/tau2_telecom/` | `scenarios/data/tau2_telecom/group_Nx.py` |
| `prepare_eva_data/generate_airline_scaffolds.py` | `data/eva_airline/eva_airline_dataset.jsonl` | stdout |

Paths in the Reads and Writes columns are relative to `nemo_voice_agent/evaluation/`. Only
`prepare_telecom.py` needs an upstream checkout. Every other script reads data that is already committed to
this repository.

## Importing tau2 Telecom Data

`prepare_telecom.py` is the only import script. Airline and retail provide their databases as upstream
`.json` files, which were copied directly when those domains were first ported. Telecom provides `db.toml`
and `user_db.toml`, so a conversion step keeps the bot-side loader on a single `json.load` path.

```bash
uv run python scripts/prepare_tau2_data/prepare_telecom.py --source /path/to/tau2-bench
```

`--source` defaults to a developer-local absolute path, so pass it explicitly. Use `--dest` to redirect the
output and compare a candidate import with the committed tree before overwriting it.

The script performs three transformations and copies the rest verbatim:

| Step | Files | Behavior |
| --- | --- | --- |
| Convert | `db.toml`, `user_db.toml` | Parse with `tomllib`, validate with the `TelecomDB` and `TelecomUserDB` models in `nemo_voice_agent.evaluation.tools.tau2_telecom_params`, and serialize as indent-2 JSON. The Pydantic round trip materializes defaults that the raw TOML omits. Predicates and initialization functions do not need defensive lookups. |
| Filter | `tasks.json` | Keep only entries whose `id` appears in `split_tasks.json["base"]` (114 IDs). Raise an error if a base ID has no upstream task definition. |
| Filter | `tasks_voice.json` | Keep only the base IDs in its `configs` map. Raise an error if a base ID has no upstream voice configuration. |
| Copy | `split_tasks.json`, `audio_difficulty.json`, `main_policy.md`, `tech_support_manual.md`, `tech_support_workflow.md`, `workflows/` | Copy byte-for-byte. Warn about and skip missing optional inputs instead of failing. |

The run is idempotent. It prints a truncated SHA-256 hash for each output file and kept or dropped counts for
filtered files. Compare these hashes with the previous run to identify upstream changes.

The repository does not currently track every file that the script can emit. `audio_difficulty.json`
and `workflows/` are traceability-only and are not consumed by the evaluation pipeline, so they are not committed.
The agent policy is assembled from `main_policy.md` plus `tech_support_manual.md` or
`tech_support_workflow.md`. For details, refer to
[tau2 Telecom](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/tau-2-telecom).

## Sharding an Oversized Database

The GitLab mirror enforces a 5 MB per-file cap, which the airline database exceeds. `shard_db.py` splits a
top-level dictionary into one file per key:

```bash
uv run python scripts/prepare_tau2_data/shard_db.py \
    nemo_voice_agent/evaluation/data/tau2_airline/db.json
```

This writes `db/flights.json`, `db/users.json`, and `db/reservations.json`, and then **deletes the source
`db.json`**. Pass `--keep-source` to retain it. `load_db_artifact` in
`nemo_voice_agent/evaluation/__init__.py` probes `<path>.json` first and falls back to `<path>/`. Both layouts
reassemble into an identical in-memory dictionary, and database hashes are unaffected. Shard the database
again after any upstream pull that restores a single-file `db.json`.

## Scaffolding tau2 Scenario Classes

The three tau2 scaffolders take no arguments. Each reads the imported data and divides the base-split IDs
into groups of 10. It then **overwrites** every `group_Nx.py` in the target package and rewrites that package's
`__init__.py` with side-effect imports so `@register_eval_scenario` fires on import.

```bash
uv run python scripts/prepare_tau2_data/generate_airline_scaffolds.py   # 50 tasks  → 5 groups
uv run python scripts/prepare_tau2_data/generate_retail_scaffolds.py    # 114 tasks → 12 groups
uv run python scripts/prepare_tau2_data/generate_telecom_scaffolds.py   # 114 tasks → 12 groups
```

Generated classes contain only `name` and `tau2_id`. The domain base class derives everything else from the
upstream task. Airline and retail use integer IDs and decade-based filenames (`group_0x.py` holds IDs 0–9).
Telecom IDs are descriptive strings. Its filenames use a sequential chunk index, and the generator parses
the class and scenario names from the ID:

```text
tau2_id  [mms_issue]airplane_mode_on|data_mode_off[PERSONA:Hard]
name     tau2_telecom__mms_issue__airplane_mode_on__data_mode_off__hard
class    Tau2TelecomMmsIssueAirplaneModeOnDataModeOffHard
```

A `PERSONA:None` suffix is dropped from both the name and the class. The telecom generator emits **two**
classes per task—one on `Tau2TelecomBaseScenario` (`policy_variant="manual"`) and one on
`Tau2TelecomWorkflowBaseScenario` (`policy_variant="workflow"`)—so 114 tasks produce 228 classes across
the `tau2_telecom` and `tau2_telecom_workflow` registrations. It also checks for class-name and
scenario-name collisions across both variants before writing anything. It aborts rather than emitting a
silently shadowed scenario.

## Scaffolding eva_airline Scenarios

There is no eva import script. The eva fixtures were copied verbatim from upstream. The generator reads the
packaged `eva_airline_dataset.jsonl` and **streams to stdout** instead of overwriting group files, because
eva scenarios carry curated prose that the dataset alone cannot express.

```bash
# One major group, appended for hand review before committing.
uv run python scripts/prepare_eva_data/generate_airline_scaffolds.py --major 1 \
    >> nemo_voice_agent/evaluation/scenarios/data/eva_airline/group_1x.py

# Every major group in one pass, with splittable section markers on stdout.
uv run python scripts/prepare_eva_data/generate_airline_scaffolds.py > /tmp/all_scaffolds.py
```

`--dataset` points at an alternate JSONL. The per-group manifest goes to stderr so it never pollutes the
generated Python. Five hand-authored seed scenarios (`1.1.2`, `2.1.1`, `3.1.3`, `5.1.1`, `7.2.1`) are skipped
—they remain the reference for prose style. The generator applies the voice-readability rule from
`nemo_voice_agent/utils/voice_prompts.py`, spelling confirmation numbers and airport codes out
character-by-character, and folds each entry's criteria and edge cases into the scenario guidelines.

Treat the output as a starting point: review the prose and prune bullets before committing.

## Bumping the Upstream Pin

To update the tau2-bench source revision and regenerate affected artifacts, complete the following steps:

1. Check out `sierra-research/tau2-bench` at the new tag or commit.
2. Update `PINNED_COMMIT` in `scripts/prepare_tau2_data/prepare_telecom.py`. The check is intentionally
   nonblocking—a mismatch produces a warning and continues, so you can test an unpinned checkout first.
3. Re-run `prepare_telecom.py`, then re-shard any DB that grew past the file cap.
4. Re-run every tau2 scaffolder whose `tasks.json` or `split_tasks.json` changed.
5. Update the source and version fields in `nemo_voice_agent/evaluation/data/README.md`.
6. Format and lint the generated Python, then run the guard tests.

```bash
uv run ruff format nemo_voice_agent/evaluation/scenarios/data
uv run ruff check --fix nemo_voice_agent/evaluation/scenarios/data
uv run pytest tests/unit -m "not gpu"
```

`tests/unit/test_tau2_telecom_scenarios.py` asserts 114 manual plus 114 workflow registrations,
`tests/unit/test_tau2_retail_scenarios.py` asserts 114 retail registrations, and the eva suites cover the
airline tools and scenario metadata. A scaffolding regression shows up there first. Generated modules carry
the SPDX header their templates emit, which keeps the `copyright-check.yml` gate green.

## Related Pages

Use these pages for provenance requirements, authoring guidance, and validation details:

- [Fixture Data & Provenance](/nemo/labs-voice-agent/evaluate-voice-agents/domain-guides/fixture-data-provenance): On-disk layout, licenses, and `EVAL_DATA_ROOT`.
- [Authoring Scenarios](/nemo/labs-voice-agent/evaluate-voice-agents/create-evaluations/authoring-scenarios): How to write a scenario class manually.
- [Adding a Domain](/nemo/labs-voice-agent/evaluate-voice-agents/create-evaluations/adding-a-domain): How to integrate a new benchmark end to end.
- [Testing](/nemo/labs-voice-agent/resources/contribute/testing): Markers, suites, and CI behavior.