> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# Quickstart

Run your first NeMo Labs Voice Agent evaluation. The harness needs **three processes**: a simulated-user
bot, the agent under test, and the bridge that connects them over audio and scores the result. Refer to the
[Evaluation Overview](/nemo/labs-voice-agent/evaluate-voice-agents/overview) for how the components fit together.

## Prerequisites

Before you start, complete these setup requirements.

1. Install the package and activate the virtual environment as described in
   [Installation](/nemo/labs-voice-agent/get-started/installation).
2. Start a vLLM server. Both eval configs (`evaluation/server_configs/agent.yaml` and `user.yaml`) set
   `start_vllm_on_init: false` and point `llm.base_url` at `http://localhost:8000/v1`, so nothing launches
   vLLM for you. Use the flags from the `llm.vllm_server_params` field of those configs — that field is the
   authoritative source if this code example becomes outdated:

   ```bash
   vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
       --trust-remote-code --enable-prefix-caching --max-num-seqs 1 \
       --gpu-memory-utilization 0.85 --reasoning-parser deepseek_r1
   ```

   Both bots and the default large language model (LLM) judge share this endpoint. Refer to the
   [vLLM Backend](/nemo/labs-voice-agent/build-voice-agents/model-serving/serving-with-v-llm).
3. Only if you switch to the hosted configs (`agent_nvidia.yaml`, `user_nvidia.yaml`): copy
   `evaluation/.env.example` to `evaluation/.env` and fill in `NVIDIA_API_KEY`.

## Quickstart Steps

Complete the following three-terminal workflow to start both bots, run a basic verification domain, and locate the saved
evaluation evidence.

### Use the Evaluation Directory

`SERVER_CONFIG_PATH` is resolved **against the current working directory**, not against the script's
directory. `ConfigManager` receives the raw string and calls `os.path.exists` on it. Every command below —
including the helper shell scripts — must therefore be run from `evaluation/`:

```
FileNotFoundError: Server configuration file not found at server_configs/agent.yaml
```

That error means you ran the command from somewhere other than `evaluation/`. `cd evaluation` and retry, or
export an absolute path.

### Start the Simulated User Bot on Port 8766

In the first terminal, start the bot that plays the scenario's simulated user.

```bash
cd evaluation
export SERVER_CONFIG_PATH=server_configs/user.yaml
export WEBSOCKET_PORT=8766
export FASTAPI_PORT=7861
export CUDA_VISIBLE_DEVICES=0
python bot_server.py
```

### Start the Agent Under Test on Port 8765

In the second terminal, start the agent that the harness evaluates.

```bash
cd evaluation
export SERVER_CONFIG_PATH=server_configs/agent.yaml
export WEBSOCKET_PORT=8765
export FASTAPI_PORT=7860
export CUDA_VISIBLE_DEVICES=1
python bot_server.py
```

`bot_server.py` is the same script for both roles — `SERVER_CONFIG_PATH` is what selects the role. Automatic
speech recognition (ASR) and text-to-speech (TTS) run locally on GPU in each process. Give the two bots
separate devices when you can. Each process also
starts a small FastAPI app on `FASTAPI_PORT`, so those two values must differ as well.

| Variable | Default | Purpose |
| --- | --- | --- |
| `SERVER_CONFIG_PATH` | `server_configs/agent.yaml` | Selects the role; resolved against the CWD |
| `WEBSOCKET_PORT` | `8765` | Port the bridge connects to |
| `FASTAPI_PORT` | `7860` | HTTP side-car port |
| `SERVER_HOST` | `0.0.0.0` | Bind address |
| `SERVER_PUBLIC_HOST` | `127.0.0.1` | Host advertised in the connect URL |
| `WEBSOCKET_SCHEME` | `ws` | `ws` or `wss` |

Wait until both processes log that they are serving before starting the bridge. Bot-side logs go to
`bot_user_server.log` and `bot_agent_server.log` in `evaluation/`.

### Helper Launch Scripts

`run_user.sh` and `run_agent.sh` wrap the two invocations above with the environment already exported. They
resolve their own directory only to locate `bot_server.py`. The exported `SERVER_CONFIG_PATH` stays relative,
so **they must also be run from `evaluation/`**:

```bash
cd evaluation
./run_user.sh    # user-sim role,  server_configs/user.yaml,  ws 8766, http 7861
./run_agent.sh   # agent role,     server_configs/agent.yaml, ws 8765, http 7860
```

### Run the Bridge

After both bots report that they are serving, start the bridge with a small verification domain.

```bash
cd evaluation
python run_evaluation.py \
    --user-url ws://localhost:8766 \
    --agent-url ws://localhost:8765 \
    --domain restaurant
```

`--user-url` and `--agent-url` already default to those values, so the two flags are optional here. Swapping
them silently inverts the roles — the user prompt lands on the agent bot — so keep 8766 on `--user-url`.

Discover what you can run before committing to a long job:

```bash
cd evaluation
python run_evaluation.py --list-domains   # domain names + scenario counts
python run_evaluation.py --list           # every scenario name, grouped by domain
```

`restaurant`, `customer_service`, and `qa` are small verification domains. `fastbite` and `simple_qa_1`
through `simple_qa_3` carry no `domain__` prefix, so `--list` files them under "Legacy scenarios" and you run
them with `--scenarios fastbite simple_qa_1` rather than `--domain`. The benchmark domains are much larger —
`eva_airline` (50 scenarios), `tau2_airline` (50), `tau2_retail` (114), and `tau2_telecom` (114, plus a
parallel `tau2_telecom_workflow` registration over the same 114 tasks). Refer to
[Benchmark Domains](/nemo/labs-voice-agent/evaluate-voice-agents/understand-scoring/benchmarks-domains).

### Flags Worth Knowing on Run One

Use these flags to select work, control output placement, and configure the first scoring run.

| Flag | Default | Notes |
| --- | --- | --- |
| `--domain` | none | Runs every scenario whose name starts with `<domain>__` |
| `--scenarios` | none | Explicit scenario names; overrides `--domain` |
| `--output-dir` | `./eval_results` | Relative to the CWD, so it lands under `evaluation/` |
| `--duration` | unset | Per-scenario cap in seconds; when unset, each scenario's own `max_duration` applies |
| `--min-agent-turns` | `3` | Scenarios with fewer completed agent turns are counted as **failures** in the composite rate and skipped in the per-signal rates. Pass `0` to disable |
| `--judge-url` | `http://localhost:8000/v1/chat/completions` | The LLM judge is on by default and reuses your vLLM server |
| `--judge-api-key-name` | `JUDGE_API_KEY` | Environment variable read when `--judge-api-key` is not passed |

Refer to the [Evaluation Command-Line Interface (CLI) Reference](/nemo/labs-voice-agent/reference/evaluation/evaluation-cli) for all flags.

### Verify the Saved Results

Each invocation creates a timestamped session directory under `--output-dir`, with one subdirectory per
scenario. Database (DB) hashes appear for scenarios that use DB-based scoring. The following tree shows the
saved artifacts:

```
evaluation/eval_results/eval_YYYYMMDD_HHMMSS/
├── evaluation_log.txt          # runner log
├── run_args.json               # the invocation(s) that produced this dir (judge key redacted)
├── all_metrics.json            # aggregated metrics across scenarios
├── all_latencies.csv           # every latency measurement
├── all_summary.txt             # human-readable per-scenario + overall summary
└── <scenario_name>/
    ├── conversation_log.txt        # transcript with latency annotations
    ├── conversation_log.seglst.json
    ├── conversation_log.wav        # stereo: L = user to agent, R = agent to user
    ├── bridge_log.txt
    ├── final_agent_response.json   # action records pulled from the bots
    ├── final_scenario_db_hash.txt  # post-run DB hash(es)
    ├── metrics.json                # all signals + the composite is_successful
    ├── judge_result.json           # present when the LLM judge ran
    ├── scenario_config/            # prompts, tools, reference_answer.json, metadata.json
    ├── bot_logs_agent/llm_context.json
    └── bot_logs_user/llm_context.json
```

Read `all_summary.txt` first, then open `metrics.json` and `conversation_log.wav` for any scenario that
failed. `eval_results/` is gitignored.

## Next Steps

Use the scoring, results, resume, and external-agent guides to continue after the first successful run.

| Guide | Use It to |
| --- | --- |
| [Scoring Model](/nemo/labs-voice-agent/evaluate-voice-agents/understand-scoring/scoring-model) | Understand the six signals and how the composite verdict is computed. |
| [Reading Results](/nemo/labs-voice-agent/evaluate-voice-agents/run-evaluations/reading-results) | Review the generated artifacts field by field. |
| [Resume and Manage Long Runs](/nemo/labs-voice-agent/evaluate-voice-agents/run-evaluations/resuming-long-runs) | Pick up a killed run with `--resume <timestamp>`. |
| [Evaluating an External Agent](/nemo/labs-voice-agent/evaluate-voice-agents/run-evaluations/evaluating-an-external-agent) | Point the bridge at a non-Pipecat agent. |