> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/labs-voice-agent/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/labs-voice-agent/_mcp/server.

# Text to Speech

NeMo Labs Voice Agent synthesizes the bot's speech with a local text-to-speech (TTS) model that runs in the server
process. The service classes live in `nemo_voice_agent/pipecat/services/nemo/tts.py`. The factory
`get_tts_service_from_config` in that file dispatches on `tts.model` and is what
`build_tts` (in `nemo_voice_agent/pipecat/services/nemo/builders.py`) calls.

## Available Models

Three local models ship with configurations under
`examples/generic_voice_agent/server/server_configs/tts_configs/`. The `tts.model` value is the
dispatch key — it must be exactly one of the three strings below, or the factory raises `ValueError`.

| `tts.model` | Sub-Configuration | Weights | Output Sample Rate | Voices |
| --- | --- | --- | --- | --- |
| `kokoro` (default) | `kokoro_82M.yaml` | `hexgrad/Kokoro-82M` | 24,000 Hz | `sub_model_id`, such as `af_heart`, `af_bella`, `am_fenrir`, or `am_michael` |
| `fastpitch-hifigan` | `nemo_fastpitch-hifigan.yaml` | `nvidia/tts_en_fastpitch` + `nvidia/tts_hifigan` | 22,050 Hz | Single voice |
| `magpie` | `magpie_tts_multilingual_357m.yaml` | `nvidia/magpie_tts_multilingual_357m` | 22,050 Hz | `speaker`: `Sofia`, `Aria`, `John`, `Jason`, or `Leo` |

A fourth option, `tts.type: nvidia`, routes to a hosted NVIDIA Riva/NIM endpoint instead of a local
model. For endpoint configuration, refer to
[NVIDIA NIM endpoints](/nemo/labs-voice-agent/build-voice-agents/model-serving/nvidia-nim-riva).

For local models, `main_model_id` is the primary checkpoint and `sub_model_id` is the secondary one:
the Kokoro voice, the HiFi-GAN vocoder for FastPitch, and `null` for Magpie. Both accept a
Hugging Face or NGC identifier or a local `.nemo` path (FastPitch, HiFi-GAN, and Magpie call
`restore_from` when the string ends in `.nemo`).

## Select a Model

`tts.model` and `tts.model_config` live in the top-level configuration
(`examples/generic_voice_agent/server/server_configs/default.yaml`). `ConfigManager` merges the sub-YAML
named by `model_config` after the top-level block, so any key in the sub-YAML takes precedence. For details,
refer to [Server configuration](/nemo/labs-voice-agent/build-voice-agents/configure/server-config).

```yaml
# server_configs/default.yaml
tts:
  type: nemo
  model: "magpie"
  model_config: "./server_configs/tts_configs/magpie_tts_multilingual_357m.yaml"
  device: "cuda"
```

Then restart the server:

```bash
cd examples/generic_voice_agent/server
python server.py
```

`tts.model_config` short-circuits the registry: `_configure_tts` in
`nemo_voice_agent/utils/config_manager.py` only consults
`examples/generic_voice_agent/server/model_registry.yaml` when `model_config` is unset *and*
`server.use_model_registry` is true. The registry's `tts_models` section lists only
`fastpitch-hifigan` and `hexgrad/Kokoro-82M`. Setting `model_config` explicitly, as the shipped
default does, is the reliable path. For registry behavior, refer to
[Model registry](/nemo/labs-voice-agent/build-voice-agents/configure/model-registry).

## Shared TTS Keys

The following keys apply to every local model:

| Key | Where It Is Read | Default | Effect |
| --- | --- | --- | --- |
| `type` | `get_tts_service_from_config` | `nemo` | One of `nemo`, `nvidia`, or `nemotron`. `nvidia` builds the hosted service. The other two use the local dispatch on `model`. |
| `device` | factory | `cuda` | Torch device for the model. |
| `think_tokens` | `BaseNemoTTSService` | `["<think>", "</think>"]` in all three sub-configs | Must be a list of exactly **two** strings (asserted at construction) or `null`. Text between them is never spoken. |
| `ignore_strings` | `BaseNemoTTSService` and the aggregator | `["*", "<unk>"]` in `default.yaml` | Substrings stripped from the text before synthesis. The aggregator strips them too, falling back to `*` alone when the key is unset. |
| `extra_separator` | `build_text_aggregator` | `[',', '\n', '.', '?', '!', ';']` in all three sub-configs | Punctuation that closes a chunk, so speech starts earlier. Setting it to `null` leaves the aggregator with no punctuation marks at all, so chunking then depends on `use_legacy_eos_detection`. |
| `use_text_aggregator` | `build_text_aggregator` | `true` | Set to `false` to drop the `LLMTextProcessor` stage entirely and let Pipecat's plain sentence splitting run. |
| `min_sentence_length` | `build_text_aggregator` | `5` | Chunks shorter than this are held back and merged with the next text. |
| `use_legacy_eos_detection` | `build_text_aggregator` | `false` | Fall back to Pipecat's `match_endofsentence` when this repository's punctuation search finds no chunk end. |

The last three have no entry in the shipped YAML files — add them under `tts:` if you need them.

### Chunking and Latency

Text aggregation is **not** part of the TTS service. Since Pipecat 1.0, it belongs to an
`LLMTextProcessor` that `build_llm_text_processor` inserts between the LLM and the TTS service, built
from `SimpleSegmentedTextAggregator` (`nemo_voice_agent/pipecat/utils/text/simple_text_aggregator.py`).
That aggregator is why `extra_separator` includes `,`. It emits a chunk at the last valid comma, so
audio starts before the sentence is finished. Its period and comma heuristics avoid splitting on decimals
(`3.14`), bullet numbering (`1.`), abbreviations (`e.g.`, `Dr.`), and times (`p.m.`).

Pipecat silently ignores the aggregator when you pass it to the TTS service because it drops unknown
constructor keyword arguments. Segmentation then degrades without an error. In a custom pipeline,
keep the processor upstream of TTS. For the construction pattern, refer to
[Builders](/nemo/labs-voice-agent/build-voice-agents/extend/pipelines/the-builder-api).

If TTS playback stutters on the client, raise `transport.audio_out_10ms_chunks` in
`default.yaml`. The shipped value is `8`, and Pipecat's WebSocket default is `4`.

### Reasoning Models

`think_tokens` keeps a reasoning model's reasoning text out of the audio. `_handle_think_tokens`
tracks the open/close markers across streamed chunks and returns only the text after the closing
token, so the user hears the answer and not the deliberation. Set it to `null` to think out loud.
When the LLM runs on vLLM with a reasoning parser, the parser strips the reasoning upstream. For details,
refer to [Reasoning mode](/nemo/labs-voice-agent/about/core-concepts/language-models/reasoning-mode).

## Kokoro-Specific Keys

The following settings control Kokoro voice selection, speed, and text normalization behavior.

| Key | Default | Effect |
| --- | --- | --- |
| `speed` | `1.25` in `kokoro_82M.yaml` (`1.0` in code) | Speaking rate multiplier. Must be greater than zero. |
| `sub_model_id` | `af_heart` | Voice id passed to the Kokoro pipeline. |

`KokoroTTSService` preloads both English pipelines (`a` = American, `b` = British) at startup so voice
switches at conversation time do not pay a download cost.

Kokoro is also the canonical component-owned tool provider. `setup_tool_calling` registers six direct
functions that the LLM can call mid-conversation:

- `tool_tts_speak_faster` and `tool_tts_speak_slower`, each applying a 15% relative change.
- `tool_tts_set_speed` and `tool_tts_reset_speed`.
- `tool_tts_set_voice` and `tool_tts_reset_voice`. The setter accepts `American English` or
  `British English` plus gender and maps to `af_heart`, `am_michael`, `bf_emma`, or `bm_george`.

The functions are only registered when
`llm.enable_tool_calling` is true. `server.py` passes the TTS service in the `tool_mixins` list. For registration
details, refer to [Tool calling](/nemo/labs-voice-agent/build-voice-agents/tools/tool-calling). FastPitch-HiFiGAN and
Magpie register no tools.

An RTVI context reset also resets the service: Kokoro's `reset()` restores the original speed, voice,
accent, and pipeline, so a new session never inherits the previous caller's "speak faster".

## Magpie-Specific Keys

The following settings control Magpie language and text normalization behavior.

| Key | Default | Effect |
| --- | --- | --- |
| `language` | `en` | Language code passed to `do_tts`. |
| `speaker` | `Sofia` | One of `Sofia`, `Aria`, `John`, `Jason`, or `Leo`. An unknown name raises `ValueError` at construction. |
| `apply_TN` | `false` | Run the model's text normalization before synthesis. |

Magpie warms up at load time by synthesizing a fixed sentence, so the first real turn is not slowed by
lazy CUDA initialization.

## Licensing Note for Kokoro

Kokoro's upstream grapheme-to-phoneme stack falls back to espeak-ng through `phonemizer`, both GPL-3.0,
which this repository excludes. `_espeak_gpl_shim.py` installs no-op substitutes so `kokoro` and `misaki` import
cleanly, and `_g2p_fallback.py` supplies an Apache-2.0 replacement built on `g2p_en` that maps ARPAbet
to misaki's phoneme inventory. Without that fallback, out-of-vocabulary words would be silently
dropped from the audio. Both modules are in `nemo_voice_agent/pipecat/services/nemo/`.

## Record Synthesized Audio

Audio logging captures synthesized output alongside the other session audio when recording is enabled.

`build_tts` accepts the audio logger built by `build_audio_logger`, so bot audio is captured when
recording is enabled. For recording configuration, refer to
[Audio logging](/nemo/labs-voice-agent/build-voice-agents/configure/audio-capture-logging).

## Related Topics

Use these pages to understand the upstream language-model stream and configure speech output behavior.

- [LLM Backends](/nemo/labs-voice-agent/about/core-concepts/language-models/llm-backends) — select and configure the upstream language model.
- [Reasoning Mode](/nemo/labs-voice-agent/about/core-concepts/language-models/reasoning-mode) — prevent reasoning text from reaching speech output.
- [Server Configuration](/nemo/labs-voice-agent/build-voice-agents/configure/server-config) — understand configuration precedence.
- [Audio Logging](/nemo/labs-voice-agent/build-voice-agents/configure/audio-capture-logging) — capture synthesized output.