About Voice Agent Evaluation

The NeMo Labs Voice Agent evaluation harness measures a voice agent in a live spoken conversation with a simulated user. A bridge routes audio, measures latency, seeds scenario state, captures each side’s actions, and sends the evidence to a scoring runner.

Nothing is scored from text alone. Each bot runs an automatic speech recognition (ASR), large language model (LLM), and text-to-speech (TTS) pipeline. Every turn goes through TTS on one side and ASR on the other. The score therefore captures ASR errors, barge-in, and turn-taking failures that a human caller would experience.

Workflow Overview

The evaluation workflow connects two independent bot servers through a bridge and then sends the captured conversation evidence to the runner for scoring.

Architecture

The following diagram shows the audio, state, and scoring flow between the four runtime components. The bots and bridge exchange control messages through the real-time voice interface (RTVI). For stateful domains, each bot can hold its own database (DB) in shared state.

┌──────────────────────┐      audio + RTVI     ┌──────────────────────┐     audio + RTVI    ┌──────────────────────┐
│   User bot server    │◄─────────────────────►│        Bridge        │◄───────────────────►│  Agent bot server    │
│  (simulated user)    │                       │                      │                     │  (agent under test)  │
│                      │                       │  audio routing       │                     │                      │
│  ASR → LLM → TTS     │                       │  latency metrics     │                     │  ASR → LLM → TTS     │
│  ws://localhost:8766 │                       │  transcript + WAV    │                     │  ws://localhost:8765 │
│                      │                       │  scenario seeding    │                     │                      │
│  shared_state["db"]  │                       │  exit detection      │                     │  shared_state["db"]  │
│  (user-side DB, only │                       │  cross-side sync     │                     │  (agent-side DB:     │
│   for dual-side      │                       │  end-of-run pull     │                     │   reservations,      │
│   domains)           │                       │                      │                     │   bills, lines, ...) │
└──────────────────────┘                       └──────────────────────┘                     └──────────────────────┘
                                                          │
                                                          ▼
                                                 runner → scoring → eval_results/

Both bot servers are the same script (evaluation/bot_server.py). The SERVER_CONFIG_PATH environment variable picks the role. Each runs its own Pipecat pipeline and holds a per-scenario shared_state dict.

Key Concepts

The bridge and scoring outputs are the two concepts to understand before you run or extend an evaluation.

What the Bridge Does

The bridge is responsible for the following runtime coordination and evidence-capture tasks.

• Audio routing. Audio is resampled at the source (matching browser-client behavior) rather than leaving small chunks for ASR to resample. Optional additive noise, configured per scenario, is applied on the user-to-agent path only — the agent hears a degraded channel, the simulated user does not.
• Latency measurement. Every measurement pairs the moment the user stopped speaking with the moment the agent started speaking. The bridge reports mean, P50, P95, min, and max per scenario and across the run.
• Transcript and audio capture. A timestamped conversation log, a segLST speaker-segment file, and a stereo WAV (left channel: user to agent, right channel: agent to user).
• Scenario setup. Per scenario the bridge sends update_system_prompt (prompt, tool registration, shared-state reset) followed by apply_initialization (merges the scenario’s shared_state_init payload, resolves db_path to a loaded DB, applies init-function mutations). Both bots always receive apply_initialization, because the DB-load step runs even when a scenario declares no init mutations.
• Termination detection. The agent ends a conversation by calling its end-conversation tool, which emits an <exit> tag. The bridge records stop reason [EXIT]. Hitting the time limit records [TIMEOUT].
• Cross-side state sync. For dual-side domains, each write tool emits an action-applied event. The bridge replays it onto shadow DBs and calls the scenario’s sync_state. It pushes the resulting delta to the other bot through apply_sync_delta. Single-side domains skip this step. Refer to tau2_telecom.
• End-of-scenario pull. The bridge pulls get_context_history and get_scenario_summary from each bot inside that bot’s own WebSocket scope. get_scenario_summary returns {actions, db_hash}. The inline DB comes back only when the bridge opts in with include_db (needed for DB-state assertions).

The bridge relies on both bots registering six RTVI actions — reset, update_system_prompt, get_context_history, get_scenario_summary, apply_initialization, and apply_sync_delta, all defined in nemo_voice_agent/pipecat/processors/frameworks/rtvi_actions.py. Any agent that implements them can be evaluated. Refer to External agents and the RTVI message reference.

What You Get Out

The runner scores each scenario with up to six independent signals. A scenario’s domain declares which signals gate the composite is_successful verdict. The rest are still computed and saved as informational.

CLEAN_EXIT is in every domain’s whitelist: an agent that does the right work but never stops talking is not a successful agent. Full semantics, the per-domain whitelist matrix, and the strict-conjunction rule are in Scoring model.

A run writes session artifacts to eval_results/eval_<TIMESTAMP>/. These include all_metrics.json, all_summary.txt, all_latencies.csv, evaluation_log.txt, and run_args.json. Each scenario subdirectory contains its transcript, stereo WAV, bridge log, pulled actions, DB hash, metrics, judge output, prompt and tool snapshots, and both bots’ full LLM context histories. Refer to Reading results and the metrics reference.

Run It

Use three terminals. SERVER_CONFIG_PATH resolves against the current working directory, so run cd evaluation first. Running from the repository root fails with FileNotFoundError.

$
# Terminal 1 — simulated user
$
cd evaluation
$
SERVER_CONFIG_PATH=server_configs/user.yaml WEBSOCKET_PORT=8766 CUDA_VISIBLE_DEVICES=0 python bot_server.py
$
$
# Terminal 2 — agent under test
$
cd evaluation
$
SERVER_CONFIG_PATH=server_configs/agent.yaml WEBSOCKET_PORT=8765 CUDA_VISIBLE_DEVICES=1 python bot_server.py
$
$
# Terminal 3 — bridge + runner
$
cd evaluation
$
python run_evaluation.py \
>
    --user-url ws://localhost:8766 \
>
    --agent-url ws://localhost:8765 \
>
    --domain restaurant

run_agent.sh and run_user.sh wrap terminals 1 and 2 with the environment already exported. Run them from evaluation/ for the same reason. Use --list-domains and --list to list registered domains and scenarios.

Two defaults to know before you compare runs:

• --min-agent-turns defaults to 3. Scenarios where the agent completed fewer turns are counted as failures in the composite success rate and skipped in the per-signal rates. This catches a hung LLM server. It also depresses the headline while shrinking each per-signal denominator. Check the warning line in all_summary.txt before reading the numbers. Pass 0 to disable.
• --duration defaults to unset, in which case each scenario’s own max_duration applies. Passing a value overrides every scenario.

The full flag list is in the eval CLI reference. Interrupted runs are picked up with --resume, described in Resuming a run.

Benchmark Domains

The harness includes large benchmark-derived domains and smaller in-repository verification domains.

Smaller in-repository domains (restaurant, customer_service, qa) provide basic verification sets and worked examples of the authoring pattern. Fixtures for all domains are packaged inside the library at nemo_voice_agent/evaluation/data/, resolved by get_eval_data_root(), which honors the EVAL_DATA_ROOT environment variable as an override. Upstream sources and licenses are recorded in Data provenance.

Next Steps

Continue with the first-run workflow, domain catalog, scoring details, or extension guides.

• Quickstart — first run, end to end.
• Benchmarks and domains — what each domain measures.
• Scoring model — the six signals and how the composite verdict is formed.
• Authoring scenarios, tools, and domains — extend the data layer.
• External agents — evaluate an agent that is not this pipeline.