Quickstart

Run your first NeMo Labs Voice Agent evaluation. The harness needs three processes: a simulated-user bot, the agent under test, and the bridge that connects them over audio and scores the result. Refer to the Evaluation Overview for how the components fit together.

Prerequisites

Before you start, complete these setup requirements.

1. Install the package and activate the virtual environment as described in Installation.

2. Start a vLLM server. Both eval configs (evaluation/server_configs/agent.yaml and user.yaml) set start_vllm_on_init: false and point llm.base_url at http://localhost:8000/v1, so nothing launches vLLM for you. Use the flags from the llm.vllm_server_params field of those configs — that field is the authoritative source if this code example becomes outdated:

   $
   vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
   >
       --trust-remote-code --enable-prefix-caching --max-num-seqs 1 \
   >
       --gpu-memory-utilization 0.85 --reasoning-parser deepseek_r1

   Both bots and the default large language model (LLM) judge share this endpoint. Refer to the vLLM Backend.

3. Only if you switch to the hosted configs (agent_nvidia.yaml, user_nvidia.yaml): copy evaluation/.env.example to evaluation/.env and fill in NVIDIA_API_KEY.

Quickstart Steps

Complete the following three-terminal workflow to start both bots, run a basic verification domain, and locate the saved evaluation evidence.

Use the Evaluation Directory

SERVER_CONFIG_PATH is resolved against the current working directory, not against the script’s directory. ConfigManager receives the raw string and calls os.path.exists on it. Every command below — including the helper shell scripts — must therefore be run from evaluation/:

FileNotFoundError: Server configuration file not found at server_configs/agent.yaml

That error means you ran the command from somewhere other than evaluation/. cd evaluation and retry, or export an absolute path.

Start the Simulated User Bot on Port 8766

In the first terminal, start the bot that plays the scenario’s simulated user.

$
cd evaluation
$
export SERVER_CONFIG_PATH=server_configs/user.yaml
$
export WEBSOCKET_PORT=8766
$
export FASTAPI_PORT=7861
$
export CUDA_VISIBLE_DEVICES=0
$
python bot_server.py

Start the Agent Under Test on Port 8765

In the second terminal, start the agent that the harness evaluates.

$
cd evaluation
$
export SERVER_CONFIG_PATH=server_configs/agent.yaml
$
export WEBSOCKET_PORT=8765
$
export FASTAPI_PORT=7860
$
export CUDA_VISIBLE_DEVICES=1
$
python bot_server.py

bot_server.py is the same script for both roles — SERVER_CONFIG_PATH is what selects the role. Automatic speech recognition (ASR) and text-to-speech (TTS) run locally on GPU in each process. Give the two bots separate devices when you can. Each process also starts a small FastAPI app on FASTAPI_PORT, so those two values must differ as well.

Wait until both processes log that they are serving before starting the bridge. Bot-side logs go to bot_user_server.log and bot_agent_server.log in evaluation/.

Helper Launch Scripts

run_user.sh and run_agent.sh wrap the two invocations above with the environment already exported. They resolve their own directory only to locate bot_server.py. The exported SERVER_CONFIG_PATH stays relative, so they must also be run from evaluation/:

$
cd evaluation
$
./run_user.sh    # user-sim role,  server_configs/user.yaml,  ws 8766, http 7861
$
./run_agent.sh   # agent role,     server_configs/agent.yaml, ws 8765, http 7860

Run the Bridge

After both bots report that they are serving, start the bridge with a small verification domain.

$
cd evaluation
$
python run_evaluation.py \
>
    --user-url ws://localhost:8766 \
>
    --agent-url ws://localhost:8765 \
>
    --domain restaurant

--user-url and --agent-url already default to those values, so the two flags are optional here. Swapping them silently inverts the roles — the user prompt lands on the agent bot — so keep 8766 on --user-url.

Discover what you can run before committing to a long job:

$
cd evaluation
$
python run_evaluation.py --list-domains   # domain names + scenario counts
$
python run_evaluation.py --list           # every scenario name, grouped by domain

restaurant, customer_service, and qa are small verification domains. fastbite and simple_qa_1 through simple_qa_3 carry no domain__ prefix, so --list files them under “Legacy scenarios” and you run them with --scenarios fastbite simple_qa_1 rather than --domain. The benchmark domains are much larger — eva_airline (50 scenarios), tau2_airline (50), tau2_retail (114), and tau2_telecom (114, plus a parallel tau2_telecom_workflow registration over the same 114 tasks). Refer to Benchmark Domains.

Flags Worth Knowing on Run One

Use these flags to select work, control output placement, and configure the first scoring run.

Refer to the Evaluation Command-Line Interface (CLI) Reference for all flags.

Verify the Saved Results

Each invocation creates a timestamped session directory under --output-dir, with one subdirectory per scenario. Database (DB) hashes appear for scenarios that use DB-based scoring. The following tree shows the saved artifacts:

evaluation/eval_results/eval_YYYYMMDD_HHMMSS/
├── evaluation_log.txt          # runner log
├── run_args.json               # the invocation(s) that produced this dir (judge key redacted)
├── all_metrics.json            # aggregated metrics across scenarios
├── all_latencies.csv           # every latency measurement
├── all_summary.txt             # human-readable per-scenario + overall summary
└── <scenario_name>/
    ├── conversation_log.txt        # transcript with latency annotations
    ├── conversation_log.seglst.json
    ├── conversation_log.wav        # stereo: L = user to agent, R = agent to user
    ├── bridge_log.txt
    ├── final_agent_response.json   # action records pulled from the bots
    ├── final_scenario_db_hash.txt  # post-run DB hash(es)
    ├── metrics.json                # all signals + the composite is_successful
    ├── judge_result.json           # present when the LLM judge ran
    ├── scenario_config/            # prompts, tools, reference_answer.json, metadata.json
    ├── bot_logs_agent/llm_context.json
    └── bot_logs_user/llm_context.json

Read all_summary.txt first, then open metrics.json and conversation_log.wav for any scenario that failed. eval_results/ is gitignored.

Next Steps

Use the scoring, results, resume, and external-agent guides to continue after the first successful run.