Evaluate Voice AgentsRun Evaluations

Evaluating an External Agent

View as Markdown

The evaluation harness evaluates an external agent over WebSockets without importing it. Any agent that implements the real-time voice interface (RTVI) wire protocol can run against the shipped domains. This page defines what your bot must implement, what the harness reads back, and how to verify the integration before a full benchmark run.

The reference implementation is evaluation/bot_server.py. Read it alongside this page — everything below is visible there in about 100 lines.

The Contract

An external agent must satisfy two hard requirements before the harness can initialize or score it.

1. A Pipecat WebSocket server transport. The bridge connects with the websockets client library and frames everything through ProtobufFrameSerializer, so use a Pipecat WS transport rather than reimplementing the framing. The shipped builder (build_ws_transport in nemo_voice_agent/pipecat/services/nemo/builders.py) constructs a SingleClientWebsocketServerTransport with audio_in_enabled, audio_out_enabled, no session timeout, and no WAV header. Bind it to the port in WEBSOCKET_PORT — the bridge defaults to ws://localhost:8765 for the agent and ws://localhost:8766 for the user sim, overridable with --agent-url / --user-url.

2. An RTVIProcessor with six client-message handlers registered. These are the entire control plane for a scenario.

Wire NameDirectionThe Harness Needs It for
update_system_promptbridge to botPer-scenario system prompt, tool surface, and tool_domain; clears prior shared_state.
apply_initializationbridge to botMerges shared_state_init, resolves db_path to db, applies init-function mutations.
apply_sync_deltabridge to botCross-side state sync (telecom only); harmless no-op elsewhere, but must be registered.
get_scenario_summarybot to bridgeEnd-of-scenario pull of actions plus db_hash, with opt-in include_db.
get_context_historybot to bridgeEnd-of-scenario large language model (LLM) context, saved as bot_logs_agent/llm_context.json and fed to the judge.
resetbridge to botClears conversation history and resets stateful services between scenarios.

All six factories live in nemo_voice_agent/pipecat/processors/frameworks/rtvi_actions.py. Each returns a (wire_name, handler) pair. Install them with one call to register_client_message_handlers. Per-handler argument and return shapes are documented in RTVI Control Plane. An unregistered type produces an error-response, which surfaces in bridge_log.txt as unknown message type.

Reuse the shipped factories when possible. They are pure functions over your pipeline objects and have no dependency on the NeMo services. For pipeline assembly around them, refer to Building Your Own Pipeline.

1from nemo_voice_agent.evaluation.tools import get_schema_tool_for_eval
2from nemo_voice_agent.pipecat.processors.frameworks.rtvi_actions import (
3    SharedStateRef,
4    TaskRef,
5    create_apply_initialization_action,
6    create_apply_sync_delta_action,
7    create_get_context_history_action,
8    create_get_scenario_summary_action,
9    create_reset_context_action,
10    create_update_system_prompt_action,
11    register_client_message_handlers,
12)
13from nemo_voice_agent.utils.tool_calling import register_schema_tools_to_llm
14
15task_ref = TaskRef()
16shared_state_ref = SharedStateRef()
17
18register_client_message_handlers(
19    rtvi,
20    [
21        create_reset_context_action(task_ref, user_agg, assistant_agg, original_messages, resettable),
22        create_update_system_prompt_action(
23            task_ref, user_agg, assistant_agg, original_messages, resettable,
24            system_role="system",
25            system_prompt_suffix="",
26            enable_tool_calling=True,
27            llm=llm,
28            context=context,
29            rtvi=rtvi,
30            tool_factory=get_schema_tool_for_eval,
31            register_schema_tools=register_schema_tools_to_llm,
32            shared_state_ref=shared_state_ref,
33        ),
34        create_get_context_history_action(task_ref, assistant_agg),
35        create_get_scenario_summary_action(task_ref, shared_state_ref),
36        create_apply_initialization_action(shared_state_ref),
37        create_apply_sync_delta_action(shared_state_ref),
38    ],
39)

If you supply a tool_factory instead of get_schema_tool_for_eval, keep the signature (name, domain=..., rtvi=..., shared_state=..., **tool_args). The bridge sends the scenario’s domain as tool_domain, and the factory must resolve names in that namespace.

Runtime Behaviors the Harness Assumes

Beyond the six handlers, the bridge relies on several behaviors that the reference server gets from run_bot_websocket_server in nemo_voice_agent/pipecat/bot_server.py and from the RTVIObserver in nemo_voice_agent/pipecat/processors/frameworks/rtvi.py.

BehaviorWhy It Matters
Answer client-ready with bot-ready (rtvi.set_bot_ready())The bridge waits up to 5 s for the handshake on every connection.
Survive disconnect and reconnect without losing scenario stateprepare_for_scenario opens a setup connection, sends the prompt and initialization, then closes it; the audio phase reconnects. Prompt, tools, and shared_state must persist.
Never end the pipeline on client disconnectThe WebSocket server lives inside the input transport — ending the pipeline kills the port and nothing can reconnect.
Accept the send-text kickoffAn RTVI send-text message with run_immediately starts the agent bot 1 s into the scenario. Pipecat’s RTVIProcessor handles this natively.
Emit bot-started-speaking, bot-tts-text, bot-stopped-speakingThe bridge builds conversation_log.txt, the segLST file, and per-turn latency from these events.
Emit metrics messages carrying token usagetoken_usage.agent.n_calls is the turn counter behind --min-agent-turns (default 3). A bot that never reports usage counts 0 turns and every scenario is scored a failure. Set enable_metrics=True and enable_usage_metrics=True on PipelineParams.
Push <exit> as an RTVI server message when the agent ends the callThis is the CLEAN_EXIT signal, which is in every domain’s whitelist. EndConversationTool in nemo_voice_agent/evaluation/tools/basic_tools.py does it; without it every scenario terminates on timeout and fails.
Push action-applied server messages from write toolsOnly needed for dual-side domains such as tau2_telecom, where the bridge uses them to drive cross-side sync.

Scoring reads two pieces of bot-owned state, both keyed off shared_state:

  • shared_state["actions"] — appended to by each write tool, returned by get_scenario_summary, and used for ACTION_MATCH and judge input.
  • shared_state["db"] — mutated in place by write tools and hashed with get_dict_hash for DB_STATE_MATCH, or returned inline when the bridge asks for include_db so DB_STATE_ASSERTION predicates can run.

If you reuse the tool classes under nemo_voice_agent/evaluation/tools/, they provide both values automatically. If you route tool calls through your own agent framework, mirror the same bookkeeping or those signals evaluate as not applicable. Refer to Scoring for the six signals and how they combine.

Running a Scenario Against Your Bot

SERVER_CONFIG_PATH is resolved against the current working directory, so run everything from evaluation/.

$cd evaluation
$
$# Terminal 1 — your agent bot on 8765
$WEBSOCKET_PORT=8765 python /path/to/your_bot_server.py
$
$# Terminal 2 — the stock simulated user on 8766
$WEBSOCKET_PORT=8766 SERVER_CONFIG_PATH=server_configs/user.yaml python bot_server.py
$
$# Terminal 3 — one fast scenario first
$python run_evaluation.py --scenarios restaurant__pizza_pepperoni

Point the bridge elsewhere if your bot is on another host or port:

$python run_evaluation.py --agent-url ws://10.0.0.7:9000 --scenarios restaurant__pizza_pepperoni

Then inspect eval_results/eval_<timestamp>/restaurant__pizza_pepperoni/:

ArtifactWhat a Healthy Run Looks Like
bridge_log.txtNo unknown message type lines. A [AGENT SERVER MESSAGE] line containing <exit>.
metrics.jsonis_successful is true, false, or "N/A" — never missing. token_usage.agent.n_calls is above --min-agent-turns.
conversation_log.txtAlternating user and agent turns. Empty means the speaking or text-to-speech (TTS) text events are not reaching the bridge.
bot_logs_agent/llm_context.jsonStarts with the scenario system prompt; tool calls appear as assistant tool_calls.
final_scenario_db_hash.txtContains a db_hash: line for database (DB)-scored domains.

Optional: trace_metrics.json Passthrough

Use trace_metrics.json to preserve architecture-specific diagnostics without changing the fixed metrics schema.

Architecture-specific diagnostics — internal handoff quality in a cascaded agent, router confidence, retry counts — do not belong in the evaluator’s fixed metric set, so the runner offers a passthrough instead. Write a JSON object to either path inside the scenario directory:

<output-dir>/eval_<timestamp>/<scenario-name>/trace_metrics.json
<output-dir>/eval_<timestamp>/<scenario-name>/bot_logs_agent/trace_metrics.json

The first path that exists is loaded verbatim into metrics.json["trace_metrics"]. The runner does not validate, interpret, or aggregate the contents, and the file is optional — when absent, the key is simply missing from metrics.json. The loader is _load_optional_trace_metrics in nemo_voice_agent/evaluation/runner.py.

Your bot must know where to write. The scenario directory uses the scenario name under the eval_<timestamp> session directory beneath --output-dir. Pass the directory to your bot out of band, or write from a post-run script that walks the session directory.

Non-Pipecat Agents

Agents built on other runtimes are outside the supported integration path. Such an agent must reimplement Pipecat’s WebSocket transport framing and the RTVI message layer before the bridge can communicate with it. For an unsupported custom integration, treat pipecat.transports.websocket.server and pipecat.serializers.protobuf as the wire specification, and RTVI Message Reference as the message-level one.

Use these pages to assemble an agent around the contract, run it, and interpret the resulting evidence.