Authoring Tools

Evaluation tools are functions that the agent bot can call during a scenario. In dual-side domains, the user-simulator bot can also call tools. Tools live in nemo_voice_agent/evaluation/tools/, register in a per-domain registry, and are instantiated for each scenario. For interactive-server tools, refer to Custom Tools.

Base Classes

All evaluation tools derive from StandardSchemaTool (nemo_voice_agent/utils/tool_calling/base.py). Pick the base that matches the tool’s role.

A subclass implements three members. The properties property returns the JSON Schema properties dictionary. The required_properties property returns the names that the model must supply. async def _execute(**kwargs) receives the call arguments as keyword arguments and returns the result.

StandardSchemaTool.__call__ is the only Pipecat-facing entry point. It calls _execute, converts a raised exception into a structured {"error": ...} result, delivers the value through params.result_callback exactly once, and then runs the _after_result hook. Do not accept a FunctionCallParams argument in _execute and do not deliver the result yourself — doing both produced duplicate tool_call_id deliveries before delivery moved into __call__.

_normalize_empty_result wraps falsy results before delivery. Pipecat rewrites a falsy tool result to the literal string COMPLETED, which the model reads as success. An empty lookup instead reaches the large language model (LLM) as an explicit “No matching records found” envelope. Never apply that normalization inside _execute — the synchronous replay path needs the raw return shape.

The LLM-visible function name defaults to the Python class name. Set a class-level name attribute to override it, as the tau2 telecom tools do (name = "get_customer_by_phone"). The registry key, schema name, action record’s name, and replay dispatch key then share one snake_case identifier.

Shared State

Every tool in a scenario receives the same mutable shared_state dict, auto-injected into any constructor that declares shared_state: Optional[dict]. The convention is to store it as self.state.

Scenario fixture data arrives through the shared_state_init argument of the apply_initialization RTVI client message, not update_system_prompt. Dunder sentinels live on shared_state, not inside shared_state["db"], so they never reach the database (DB) hash. get_dict_hash hashes only the db dictionary and drops the top-level keys in HASH_EXCLUDED_KEYS (currently only session). Refer to RTVI Messages.

Termination Contracts

Two patterns coexist. New benchmarks should use the bridge-pull pattern.

Both patterns need EndConversationTool in the agent’s tool list. It emits <exit>, which stops the scenario early. Without it, the bridge waits out the scenario’s max_duration. CLEAN_EXIT is one of the six scoring signals. Refer to Scoring.

Terminal tools that record an action and end the call (TransferToHumanAgentsTool) emit the exit signal from _after_result, never from _execute. Pipecat must commit the tool-call record before the bridge ends the session. Otherwise, the captured llm_context.json loses the final tool call.

Add a Tool

Subclass the right base, register it with the domain, and side-import the module.

1
from typing import Any, Dict, List, Optional
2
3
from nemo_voice_agent.evaluation.tools import register_schema_tool_for_eval
4
from nemo_voice_agent.evaluation.tools._write_tool_base import WriteScenarioTool
5
6
7
COFFEE_ACTION_TYPES: List[str] = ["cancel_order"]
8
9
10
@register_schema_tool_for_eval(domain="coffee")
11
class CancelOrderTool(WriteScenarioTool):
12
    """Mutating tool — records an action the bridge pulls at scenario end."""
13
14
    ACTION_TYPES = COFFEE_ACTION_TYPES
15
16
    def __init__(self, *, shared_state: Optional[dict] = None, description: Optional[str] = None):
17
        super().__init__(description=description or "Cancel a pending coffee order.")
18
        self.state = shared_state if shared_state is not None else {}
19
20
    @property
21
    def properties(self) -> Dict[str, Any]:
22
        return {"order_id": {"type": "string", "description": "The order id, such as 'A123'."}}
23
24
    @property
25
    def required_properties(self) -> List[str]:
26
        return ["order_id"]
27
28
    async def _execute(self, order_id: str = "") -> Dict[str, Any]:
29
        # Normalize case: voice ASR of a spelled-out id is inconsistent.
30
        orders = (self.state.get("db") or {}).get("orders") or {}
31
        order = orders.get((order_id or "").upper())
32
        if order is None:
33
            return {"status": "error", "error_type": "not_found", "message": f"Order {order_id} not found"}
34
        order["status"] = "cancelled"
35
        self._record_action(
36
            {
37
                "action_type": "cancel_order",
38
                "name": "cancel_order",
39
                "arguments": {"order_id": order_id},
40
                "result": {"order_id": order_id},
41
            }
42
        )
43
        return {"status": "success", "order_id": order_id}

A read-only tool has the same shape with StandardSchemaTool as the base, no ACTION_TYPES, and no _record_action call. ACTION_TYPES is a ClassVar list that _record_action validates action_type against. A mismatch logs a warning rather than raising, so check the bot log when an action fails to score. The record’s name field is the upstream method name used for action-list comparison and is independent of the class name.

Add the side-import to nemo_voice_agent/evaluation/tools/__init__.py so the decorator runs at import time, then check the registration:

$
python -c "
>
from nemo_voice_agent.evaluation.tools import list_schema_tools_for_eval
>
print(sorted(list_schema_tools_for_eval('coffee')))
>
"

Constructor Auto-Injection

Scenarios reference tools by registry name in Resources.tools, as {name: constructor_kwargs}. The bridge serializes that mapping to JSON, and the bot server instantiates each entry through get_schema_tool_for_eval(name, domain=..., rtvi=..., shared_state=..., **tool_args).

That factory inspects the constructor signature and injects only parameters that the tool declares. SendRTVIMessageTool subclasses need rtvi, while WriteScenarioTool reads the processor from shared_state["__rtvi__"]. Every other parameter comes from the scenario’s per-tool kwargs, so one class can be reused with different fixtures (a different menu string per scenario, for example). Keep constructors keyword-only with Optional defaults so a tool stays constructible in a unit test with no arguments.

Per-Domain Registry

ALL_SCHEMA_TOOLS_FOR_EVAL is a dict of domain to a dict of name to class. The same short class name can exist in several domains. Within one domain, a duplicate name raises ValueError at decoration time.

1
@register_schema_tool_for_eval(domain="tau2_airline")   # keyword form
2
@register_schema_tool_for_eval("tau2_airline")          # positional shortcut
3
@register_schema_tool_for_eval                          # bare: registers into "default"

Use the bare form only for cross-domain harness tools such as EndConversationTool.

The domain key comes from Scenario.domain, which the bridge forwards as the tool_domain argument of update_system_prompt. Lookup tries that domain first, then falls back to "default" with a logged warning, and raises KeyError listing the available names if neither has the tool. The telecom user-side tools are exposed to the user simulator, which is why the eval user bot config sets llm.enable_tool_calling: true.

Synchronous invoke for Replay

The tau2 domains add a synchronous invoke(**kwargs) alongside the asynchronous _execute through _Tau2InvokeMixin. Both route through _do_work(p) after validating arguments with a Pydantic PARAMS_MODEL. A validation failure returns a structured error dictionary instead of a traceback. Airline and retail return error_type: "validation_error", while telecom returns error_type: "invalid_arguments". invoke is what the runner’s gold replay and the cross-side sync shadow-DB replay call in process. Any scenario that overrides Scenario.sync_state must also provide _build_tool_map(state) returning tools with a sync invoke. Single-side domains do not need it.

Next Steps

After implementing a tool, register it in a scenario and verify that live execution and any gold replay use the same state mutation.

• Author scenarios to expose the tool and select its success signals.
• Add a domain to register a new tool namespace and fixture set.