> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# Extension Overview

NeMo Labs Voice Agent is a Pipecat pipeline assembled from a small set of builder functions. You can change
the models that builders instantiate, the processors in the chain, or the assembly code. Choose the lowest
tier that supports your requirement.

## Workflow Overview

Choose one of three extension tiers, starting with the least invasive option that supports your change.

| Tier | What You Change | Python Needed | Use When |
|---|---|---|---|
| 1. YAML swap | A file under `server_configs/` | None | A different model or endpoint that the existing builders already support. |
| 2. Custom frame processor | One `FrameProcessor` subclass, inserted into `pipeline_list` | ~20 lines | You need a transform between two existing stages. |
| 3. New pipeline | Your own `run_bot_websocket()` | ~150 lines | Different services or a different pipeline shape entirely. |

## Key Concepts

Both shipped entrypoints use the same pipeline shape, builders, and extension boundaries.

### Two Entrypoints, One Pipeline

Everything on this page applies to both shipped entrypoints. They differ only in their config directory and
the number of real-time voice inference (RTVI) control-plane handlers that they register.

| Entrypoint | Script | Config Directory | RTVI Handlers |
|---|---|---|---|
| Demo server (browser client) | `examples/generic_voice_agent/server/server.py` | `examples/generic_voice_agent/server/server_configs/` | `reset` only |
| Eval bot (agent or user-sim role) | `evaluation/bot_server.py` | `evaluation/server_configs/` | all six |

Both define `run_bot_websocket()`, both import the same `build_*` functions from `nemo_voice_agent/pipecat/services/nemo/builders.py`, and both assemble the same shape:

```text
ws.input -> VAD -> RTVI -> STT -> [Diar] -> [TurnTaking] -> [UserAudioBuffer]
  -> UserAggregator -> LLM -> [LLMTextProcessor] -> TTS -> ws.output -> AssistantAggregator
```

Bracketed stages are dropped when their builder returns `None`. These stages include `Diar` when
`diar.enabled` is false and `TurnTaking` when turn taking is disabled. They also include `LLMTextProcessor`
when `tts.use_text_aggregator` is false and `UserAudioBuffer` for non-omni models
(`llm.is_omni_model`). Refer to
[How It Works](/nemo/labs-voice-agent/about/architecture) for the runtime view.

## Three Tiers

The following tiers describe what each extension approach changes and where to implement it.

### Tier 1 — Swap Models via YAML

`ConfigManager` (`nemo_voice_agent/utils/config_manager.py`) reads the file named by the `SERVER_CONFIG_PATH`
environment variable. It gives each section to the matching builder: `stt:` to `build_stt`, `diar:` to
`build_diar`, `llm:` to `build_llm`, and `tts:` to `build_tts`. It also maps `vad:` to `build_vad_analyzer`,
`turn_taking:` to `build_turn_taking`, and `transport:` to `build_ws_transport`.

The two config directories use different structures:

- **Demo configs** (`default.yaml`, `default_nvidia.yaml`) delegate per-model parameters to a sub-YAML through
  `model_config:`. The field points into `llm_configs/`, `tts_configs/`, or `stt_configs/`. The sub-YAML
  **overrides** the top-level file, so its value wins when both files set the same key. Editing that key in
  `default.yaml` alone has no effect.
- **Eval configs** (`agent.yaml`, `agent_nvidia.yaml`, `agent_nvidia_omni.yaml`, `user.yaml`,
  `user_nvidia.yaml`) are flat and self-contained. They have no `model_config:` key or sub-config directories
  under `evaluation/server_configs/`. Sub-config paths resolve against the entrypoint's directory, so a
  relative path cannot point an eval config at a demo sub-YAML. Edit the eval file directly.

Reasoning mode adds another condition. The automatic swap to a sibling `*_think.yaml` fires only when
`server.use_model_registry: true`, the model appears in `model_registry.yaml`, **and** that entry sets
`reasoning_supported: true`. An explicit `llm.model_config:` short-circuits the registry lookup, so the
shipped `default.yaml` never swaps. Point `model_config:` at the `_think.yaml` yourself. The eval configs set
`use_model_registry: false` and pass the toggle directly to vLLM by interpolating `${llm.enable_reasoning}`
into `vllm_generation_params`. For details, refer to
[Reasoning Mode](/nemo/labs-voice-agent/about/core-concepts/language-models/reasoning-mode) and
[Model Registry](/nemo/labs-voice-agent/build-voice-agents/configure/model-registry).

What Tier 1 covers without any Python:

| Change | Key |
|---|---|
| LLM backend (`auto`, `hf`, `vllm`, `nvidia`) | `llm.type` |
| Swap the served model | `llm.model`, `llm.base_url`, `llm.vllm_server_params` |
| Swap TTS voice or engine | `tts.model`, `tts.main_model_id`, `tts.sub_model_id` |
| Swap the streaming ASR model | `stt.model`, `stt.att_context_size` |
| Toggle diarization | `diar.enabled` |
| Backchannel handling | `turn_taking.backchannel_phrases_path` |
| VAD sensitivity and endpointing | `vad.confidence`, `vad.stop_secs` |
| Tool calling | `llm.enable_tool_calling` |

Run a demo server against your edited config:

```bash
cd examples/generic_voice_agent/server
SERVER_CONFIG_PATH=server_configs/default_nvidia.yaml python server.py
```

Run an eval bot against yours. `SERVER_CONFIG_PATH` resolves against the current working directory, so the `cd` is required:

```bash
cd evaluation
WEBSOCKET_PORT=8765 SERVER_CONFIG_PATH=server_configs/agent.yaml python bot_server.py
```

Full key reference: [Server Config](/nemo/labs-voice-agent/build-voice-agents/configure/server-config) and [Config Schema](/nemo/labs-voice-agent/reference/runtime/server-config-schema).

### Tier 2 — Insert a Frame Processor

When no builder exposes the required transformation between two stages, write a `FrameProcessor` subclass.
Then insert it into `pipeline_list` before the `Pipeline(...)` call. Both entrypoints build that list inline,
so the change is localized.

| Insertion Point | Typical Use |
|---|---|
| After `ws_transport.input()` | Raw-audio processing: noise suppression, a custom VAD. |
| Between STT and the user aggregator | Transcript transforms: ASR error correction, redaction, language detection. |
| Between the user aggregator and LLM | Context shaping: memory injection, intent classification. |
| Between LLM and TTS | Output transforms: Markdown stripping, profanity filtering. |
| Between TTS and `ws_transport.output()` | Output-audio processing: gain, resampling, earcons. |

Three rules cover most processors: call `super().process_frame(frame, direction)` first, filter with
`isinstance` so unrelated frame types pass through untouched, and always finish by pushing the frame onward.
For worked examples, refer to [Custom Frame Processors](/nemo/labs-voice-agent/build-voice-agents/extend/pipelines/custom-frame-processors).

### Tier 3 — Replace the Assembly

At this tier you write your own `run_bot_websocket()` and choose every service yourself. Two things stay fixed.

**A Pipecat WebSocket transport.** `build_ws_transport` returns a `SingleClientWebsocketServerTransport` with a
`ProtobufFrameSerializer`. That pairing is the wire contract that browser clients and the evaluation bridge
both use. For protocol details, refer to [Client Protocol](/nemo/labs-voice-agent/build-voice-agents/extend/protocols/client-protocol).

**An `RTVIProcessor` with handlers registered.** Handler factories live in
`nemo_voice_agent/pipecat/processors/frameworks/rtvi_actions.py`. Each returns a
`(message_type, handler)` pair that you pass to `register_client_message_handlers`. A demo server needs only
`reset`. An eval bot needs all six because the bridge drives scenario lifecycle over them.

| Message Type | Factory | Purpose |
|---|---|---|
| `reset` | `create_reset_context_action` | Clear conversation history and reset stateful services between scenarios. |
| `update_system_prompt` | `create_update_system_prompt_action` | Set the system prompt and register tools from the `tool_domain` registry. |
| `apply_initialization` | `create_apply_initialization_action` | Merge `shared_state_init`, resolve `db_path` to a loaded `db`, apply init-function mutations. |
| `apply_sync_delta` | `create_apply_sync_delta_action` | Apply a cross-side state delta (dual-side domains only; harmless elsewhere). |
| `get_scenario_summary` | `create_get_scenario_summary_action` | Return `actions` plus `db_hash`, and the inline `db` when `include_db` is requested. |
| `get_context_history` | `create_get_context_history_action` | Return the LLM conversation history for the run artifacts. |

An unregistered type produces an `error-response` rather than a hang, so a missing handler shows up
immediately in the bridge log. For skeleton code, refer to
[Building Your Own Pipeline](/nemo/labs-voice-agent/build-voice-agents/extend/pipelines/building-your-own-pipeline). For message payloads, refer to
[RTVI Control Plane](/nemo/labs-voice-agent/build-voice-agents/extend/protocols/rtvi-control-plane) and [RTVI Messages](/nemo/labs-voice-agent/reference/runtime/rtvi-message-reference).

## Verify Your Customization

Whichever tier you used, exercise it against one fast scenario before a full run. Three terminals, all started from `evaluation/`:

```bash
cd evaluation
WEBSOCKET_PORT=8766 SERVER_CONFIG_PATH=server_configs/user.yaml  python bot_server.py
WEBSOCKET_PORT=8765 SERVER_CONFIG_PATH=server_configs/agent.yaml python bot_server.py
python run_evaluation.py --scenarios restaurant__pizza_pepperoni
```

Then read the per-scenario output directory:

| Artifact | Check |
|---|---|
| `metrics.json` | Present and populated. Missing means the bot never returned a usable `get_scenario_summary` payload. |
| `bridge_log.txt` | Grep for `unknown message type` — that is a handler you forgot to register. |
| `bot_logs_agent/llm_context.json` | The scenario system prompt should be the first message, and tool calls should appear as `assistant.tool_calls`. |
| `final_scenario_db_hash.txt` | Should carry a hash line; absence means `get_scenario_summary` returned the wrong shape. |

## Related Pages

Use these pages for implementation details and evaluation integration:

- [The Builder API](/nemo/labs-voice-agent/build-voice-agents/extend/pipelines/the-builder-api) — every `build_*` function and the config keys it reads.
- [Writing Your Own Tools](/nemo/labs-voice-agent/build-voice-agents/tools/writing-your-own-tools) — extending the agent's capabilities rather than its pipeline.
- [Evaluating External Agents](/nemo/labs-voice-agent/evaluate-voice-agents/run-evaluations/evaluating-an-external-agent) — pointing the harness at a bot you built yourself.