Fixture Data and Provenance

Every NeMo Labs Voice Agent evaluation domain uses fixture data copied from an upstream open-source benchmark. This page records each fixture’s source, license, disk location, and runtime resolution. Fixtures include scenario databases, task definitions, and agent policy prompts.

The machine-readable counterpart lives in nemo_voice_agent/evaluation/data/README.md (per-file source mapping) and THIRD_PARTY_NOTICES.md (reproduced license texts). Refer to Third-party notices.

Where Fixtures Live

Fixtures are packaged inside the library, not next to the eval scripts:

nemo_voice_agent/evaluation/data/
├── README.md            # per-file upstream source mapping
├── eva_airline/
├── tau2_airline/
├── tau2_retail/
└── tau2_telecom/

They ship in the wheel through [tool.setuptools.package-data] in pyproject.toml, which matches evaluation/data/**/*.json, **/*.jsonl, and **/*.md. A pip install of the package therefore carries the full eval surface. Nothing is downloaded at run time.

Resolution Order and EVAL_DATA_ROOT

get_eval_data_root() in nemo_voice_agent/evaluation/__init__.py returns the fixture root. It is a function, not a module constant, so an environment change after import still takes effect:

Set EVAL_DATA_ROOT when you want to point the runner and the bots at a scratch copy of the fixtures (for example, to test a re-import before committing it):

$
export EVAL_DATA_ROOT=/path/to/alternate/eval_data

Both the bridge and bot-server processes call get_eval_data_root() independently, so they can resolve to different absolute roots. Fixture paths stored in shared_state are therefore always relative to the root. The Environment Variables reference catalogs all variables.

How Fixtures Reach the Bots

Two database (DB) seeding styles are driven by Scenario.setup_shared_state(state, side):

Path seeding exists because Pipecat’s default WebSocket frame cap is 1 MB: inlining a multi-megabyte tau2 DB closes the connection with code 1009 before the payload arrives.

The bot-side resolution happens in the apply_initialization real-time voice interface (RTVI) client-message handler (create_apply_initialization_action in nemo_voice_agent/pipecat/processors/frameworks/rtvi_actions.py). It merges the shared_state_init JSON payload, then — only when db_path is present and db is not — loads the artifact and stores it under db. The step is idempotent: an inline db short-circuits the load and the redundant db_path key is dropped. A missing file raises FileNotFoundError with the currently resolved root printed in the message.

Sharded Databases and load_db_artifact

load_db_artifact resolves either of the supported database layouts.

load_db_artifact(path) (same module as get_eval_data_root()) accepts a path without the .json suffix and probes two on-disk layouts:

1. <path>.json — a single file, parsed with json.loads.
2. <path>/ — a directory. Every top-level *.json file becomes one key in the returned dict, using the file stem as the key. Sorted, non-recursive.

Neither form present raises FileNotFoundError. The two layouts are equivalent as in-memory dicts, so DB hashes and gold replays are identical in either layout. Refer to Scoring Signals for how the harness uses the hash.

tau2_airline is the only domain shipped in shards. Its upstream db.json is approximately 6.8 MB, which exceeds the 5 MB per-file cap on the GitLab mirror. It therefore ships as tau2_airline/db/flights.json, reservations.json, and users.json (approximately 4.6 MB combined). Re-shard after a fresh upstream pull:

$
python scripts/prepare_tau2_data/shard_db.py nemo_voice_agent/evaluation/data/tau2_airline/db.json

The source file is deleted after sharding unless you pass --keep-source.

Upstream Sources

The following table records the authoritative upstream project, pinned version, and license for each benchmark-derived fixture set.

voice-user-sim-v1.0 is an annotated tag, so its object SHA is not a commit SHA. Dereference it with git rev-parse voice-user-sim-v1.0^{commit} to get 17e07b1.

Adapted Python modules (tools, param models, scenario bases) each carry an inline # Adapted from <url> attribution at the top of the file. Data fixtures are verbatim copies except where noted below.

Per-Domain File Inventory

Each domain packages the files required to seed its scenarios and reproduce its scoring inputs.

eva_airline — 50 Scenarios

The EVA airline fixture directory contains the source dataset and its generated scenario index.

The harness reads this data one time per process and indexes it by scenario ID. Refer to eva_airline.

tau2_airline — 50 Tasks

The tau2 airline fixtures separate the policy, task split, and sharded database artifacts.

Refer to tau2_airline.

tau2_retail — 114 Tasks

Same file shape as airline, with db.json unsharded (~2.8 MB) and split_tasks.json carrying train (74), test (40), base (114). Of the 114 tasks, 73 are actions-only, 39 carry both actions and nl_assertions, 1 is nl-assertion-only, and 1 (task 57) is chitchat with neither. Refer to tau2_retail.

tau2_telecom — 114 Tasks

The only domain with an import script, because upstream ships its DBs as TOML.

Re-import from a local tau2-bench checkout:

$
python scripts/prepare_tau2_data/prepare_telecom.py --source /path/to/tau2-bench

The script verifies the checkout’s HEAD against its PINNED_COMMIT constant and warns on a mismatch. It fails if any base-split ID is missing upstream. It also round-trips the converted user DB through the ported Pydantic model so default fields omitted from the raw TOML are materialized on disk.

Both the tau2_telecom and tau2_telecom_workflow scenario registrations read this same directory — the workflow variant inherits domain = "tau2_telecom" and differs only in which tech_support_*.md file it renders. Refer to tau2_telecom.

Adding a New Source

Import scripts and scaffold generators live under scripts/. The workflow is documented in Importing benchmark data, and the scenario-class side in Authoring domains. When you add a source, also:

Complete the following provenance steps before committing generated fixtures.

1. Namespace its files under a new nemo_voice_agent/evaluation/data/<domain>/ subdirectory so fixtures from different upstream libraries cannot collide.
2. Append a section to nemo_voice_agent/evaluation/data/README.md recording the upstream URL, pinned version or commit, license, and per-file source mapping.
3. Append the license text to THIRD_PARTY_NOTICES.md if the license is not already reproduced there.
4. Add an inline # Adapted from <url> line to every ported Python module.