Adding a Domain

A domain in the NeMo Labs Voice Agent evaluation harness is a namespace that binds fixtures, a tool registry bucket, a base scenario class, and a scoring configuration. Refer to Authoring Scenarios for individual scenario classes and Authoring Tools for tool internals.

Pieces of a Domain

A complete domain can include the following fixture, runtime, and scoring components.

The harness uses two distinct domain strings:

• Scenario.domain — the registry namespace. Tools, predicates, init functions, and sync appliers are all keyed by it. The bridge ships it to both bots as the tool_domain argument of update_system_prompt.
• The --domain command-line interface (CLI) filter — a scenario-name prefix match on <domain>__. run_evaluation.py computes the available list from scenario names, not from Scenario.domain. Keep them identical to avoid confusion (tau2_telecom is the one deliberate exception: tau2_telecom_workflow__* scenarios carry domain = "tau2_telecom" so they share one tool bucket).

1. Add Fixtures

Fixtures live inside the installed package, under nemo_voice_agent/evaluation/data/<domain>/. get_eval_data_root() (in nemo_voice_agent/evaluation/__init__.py) resolves that directory, with the EVAL_DATA_ROOT environment variable as an override. Because the bridge and the bot servers can be different processes with different roots, every fixture reference stored in shared state is a relative path.

load_db_artifact() accepts either <name>.json or a <name>/ directory of per-table shards and returns the same in-memory dictionary, so DB hashes match across layouts. Shard when a single file exceeds a hosting size cap. tau2_airline therefore ships its DB as one file per table.

Record the upstream source, version, and license in nemo_voice_agent/evaluation/data/README.md under ## Sources & Licenses. Refer to Data Provenance.

2. Register Tools in the Domain Bucket

The registry maps domain → tool name → class. The same short class name can exist in two domains, but a duplicate inside one domain raises an error at import time.

1
from nemo_voice_agent.evaluation.tools import register_schema_tool_for_eval
2
from nemo_voice_agent.evaluation.tools._write_tool_base import WriteScenarioTool
3
4
MYDOMAIN_ACTION_TYPES = ["booking", "refund"]
5
6
7
@register_schema_tool_for_eval(domain="mydomain")
8
class BookThingTool(WriteScenarioTool):
9
    ACTION_TYPES = MYDOMAIN_ACTION_TYPES
10
11
    async def _execute(self, **kwargs):
12
        ...  # mutate self.state["db"]
13
        self._record_action({"action_type": "booking", "name": "book_thing", "arguments": kwargs})
14
        return {"status": "ok"}

Read-only tools subclass StandardSchemaTool directly and record nothing. _record_action appends to shared_state["actions"] (the bridge pulls that list at end of scenario) and pushes an action-applied real-time voice interface (RTVI) server message that drives cross-side sync. Set a class-level name attribute if you want snake_case large language model (LLM)-visible function names instead of the class name. Lookup falls back to the "default" bucket for shared harness tools such as EndConversationTool, so your scenarios receive them automatically. List them under their registered PascalCase key.

3. Write the Base Scenario

Define a base scenario that binds the registry namespace and derives shared behavior for the domain.

1
from functools import cached_property
2
from nemo_voice_agent.evaluation.scenarios.classes import Scenario, SuccessSignal
3
4
5
class MyDomainBaseScenario(Scenario):
6
    domain = "mydomain"
7
    success_signals = (SuccessSignal.DB_STATE_MATCH, SuccessSignal.CLEAN_EXIT)
8
9
    def setup_shared_state(self, state: dict, side: str) -> None:
10
        if side == "agent":
11
            state["db_path"] = f"{self.domain}/db.json"
12
13
    @cached_property
14
    def expected_scenario_db(self) -> dict:
15
        ...  # gold end state, hashed by the runner

success_signals is validated at class-definition time: any concrete scenario (one that declares name) must resolve a non-empty tuple of SuccessSignal members from itself or an ancestor. Declare it on the base class. When it depends on per-task opt-ins, use a cached_property. tau2_retail and tau2_telecom use this pattern to add NL_ASSERTION only when the task carries assertions. Refer to Scoring for what each signal means.

setup_shared_state(state, side) is called one time per side. Use one of these seeding styles:

Path seeding is mandatory above roughly 1 MB — Pipecat’s WebSocket frame cap closes the connection with code 1009 when a bigger payload is inlined.

4. Wire the Imports

Decorators run only if the modules are imported. Add your tool module to the import block at the bottom of nemo_voice_agent/evaluation/tools/__init__.py, and your scenario package to nemo_voice_agent/evaluation/scenarios/data/__init__.py. A domain package’s own __init__.py should side-import each group_Nx.py shard so every scaffolded scenario registers.

5. Understand the Runtime State Flow

The runtime initializes domain state and tools in the following order.

apply_initialization is the scenario-state initializer. Its handler merges shared_state_init into the bot’s shared_state (preserving runtime sentinels), resolves db_path into db if present, then dispatches initialization functions. The bridge calls it on both bots for every scenario, even when there are no init actions, because the merge and DB load must always run. Any failure aborts the scenario rather than running it half-seeded. update_system_prompt does no DB loading.

Each bot owns exactly one DB at shared_state["db"] and is side-agnostic. The bridge labels the pulls: the agent bot’s response becomes db_hash / db, the user bot’s becomes user_db_hash / user_db. db_hash is computed with nemo_voice_agent/evaluation/db_hash.py, which both the bot and the runner import, so canonicalization is identical on both ends. Refer to RTVI Messages for full payload shapes.

6. Optional: DB-State Assertion Predicates

Use these when the domain has an open solution space and several valid action sequences produce different whole-DB states that all satisfy the same outcome. Predicates are pure, side-agnostic, and dispatched runner-side through evaluate_db_state_assertion(domain, assertion, db, user_db).

1
from nemo_voice_agent.evaluation.db_state_predicates import register_db_state_predicate
2
3
4
@register_db_state_predicate(domain="mydomain")
5
def assert_line_active(db: dict, line_id: str) -> bool:
6
    return db["lines"][line_id]["status"] == "Active"

Each entry in Scenario.db_state_assertions has the shape side, func_name, arguments, assert_value, and optional message. The bridge sets include_db=True on the summary pull whenever db_state_assertions is truthy, so the actual dicts come back for the predicates to read.

7. Optional: Initialization Functions

These functions mutate the live DB before the conversation starts. The signature is (db: dict, **arguments) -> None, and dispatch occurs bot-side. The mutation must affect the same dictionary instance that the live tools use.

1
from nemo_voice_agent.evaluation.initialization_functions import register_initialization_function
2
3
4
@register_initialization_function(domain="mydomain")
5
def turn_roaming_off(db: dict, line_id: str) -> None:
6
    db["lines"][line_id]["roaming_enabled"] = False

Populate Scenario.initialization_actions with side, func_name, arguments records. The bridge filters by side before sending, and rejects any record whose side is neither "user" nor "agent".

8. Optional: Dual-Side Domains

A dual-side domain gives the user simulator its own tools and DB. tau2_telecom is the only current example. Refer to tau2-telecom. Complete these requirements:

1. Set has_user_state = True on the base scenario so gold replay seeds the user side and the bridge dual-pulls at end of scenario.
2. Seed both sides from setup_shared_state (side="agent" and side="user" branches).
3. Override sync_state(agent_db, user_db) to return per-side deltas, shaped agent and user keys mapping to delta dicts. Returning empty dicts is the base-class no-op that keeps single-side domains out of the pipeline entirely.
4. Provide _build_tool_map(state) returning name-to-instance pairs where each tool has a sync invoke(**kwargs) method. The bridge replays fired actions onto in-process shadow DBs through this map before calling sync_state. Tau2 tools get this from _Tau2InvokeMixin. EVA tools have only the async _execute and would need a sync wrapper first.
5. Register a sync applier if your deltas are not plain dotted paths:

1
from nemo_voice_agent.evaluation.sync_appliers import register_sync_applier
2
3
4
@register_sync_applier(domain="mydomain")
5
def apply_mydomain_sync_delta(db: dict, delta: dict) -> None:
6
    ...  # mutate db in place

The default applier handles dotted-path assignment such as surroundings.roaming_allowed. Anything richer — list-by-id lookups, post-apply re-derivation — needs your own applier.

Sync runs at two points: after apply_initialization (so the conversation starts from coherent cross-side state) and after every action-applied event from either bot. Both bots need llm.enable_tool_calling: true in their server config for a dual-side domain.

9. Verify

Run the focused registry, scoring, and documentation checks after wiring the new domain.

$
cd /path/to/Voice-Agent/evaluation
$
python run_evaluation.py --list-domains       # your domain, with its scenario count
$
python run_evaluation.py --scenarios mydomain__1   # smoke-run one scenario

Both bot servers must be running first, and SERVER_CONFIG_PATH resolves against the current working directory. Refer to the Evaluation Quickstart and the Evaluation CLI reference.