Text to Speech | NeMo Labs Voice Agent

NeMo Labs Voice Agent synthesizes the bot’s speech with a local text-to-speech (TTS) model that runs in the server process. The service classes live in nemo_voice_agent/pipecat/services/nemo/tts.py. The factory get_tts_service_from_config in that file dispatches on tts.model and is what build_tts (in nemo_voice_agent/pipecat/services/nemo/builders.py) calls.

Available Models

Three local models ship with configurations under examples/generic_voice_agent/server/server_configs/tts_configs/. The tts.model value is the dispatch key — it must be exactly one of the three strings below, or the factory raises ValueError.

`tts.model`	Sub-Configuration	Weights	Output Sample Rate	Voices
`kokoro` (default)	`kokoro_82M.yaml`	`hexgrad/Kokoro-82M`	24,000 Hz	`sub_model_id`, such as `af_heart`, `af_bella`, `am_fenrir`, or `am_michael`
`fastpitch-hifigan`	`nemo_fastpitch-hifigan.yaml`	`nvidia/tts_en_fastpitch` + `nvidia/tts_hifigan`	22,050 Hz	Single voice
`magpie`	`magpie_tts_multilingual_357m.yaml`	`nvidia/magpie_tts_multilingual_357m`	22,050 Hz	`speaker`: `Sofia`, `Aria`, `John`, `Jason`, or `Leo`

A fourth option, tts.type: nvidia, routes to a hosted NVIDIA Riva/NIM endpoint instead of a local model. For endpoint configuration, refer to NVIDIA NIM endpoints.

For local models, main_model_id is the primary checkpoint and sub_model_id is the secondary one: the Kokoro voice, the HiFi-GAN vocoder for FastPitch, and null for Magpie. Both accept a Hugging Face or NGC identifier or a local .nemo path (FastPitch, HiFi-GAN, and Magpie call restore_from when the string ends in .nemo).

Select a Model

tts.model and tts.model_config live in the top-level configuration (examples/generic_voice_agent/server/server_configs/default.yaml). ConfigManager merges the sub-YAML named by model_config after the top-level block, so any key in the sub-YAML takes precedence. For details, refer to Server configuration.

1 # server_configs/default.yaml
2 tts:
3   type: nemo
4   model: "magpie"
5   model_config: "./server_configs/tts_configs/magpie_tts_multilingual_357m.yaml"
6   device: "cuda"

Then restart the server:

$ cd examples/generic_voice_agent/server
$ python server.py

tts.model_config short-circuits the registry: _configure_tts in nemo_voice_agent/utils/config_manager.py only consults examples/generic_voice_agent/server/model_registry.yaml when model_config is unset and server.use_model_registry is true. The registry’s tts_models section lists only fastpitch-hifigan and hexgrad/Kokoro-82M. Setting model_config explicitly, as the shipped default does, is the reliable path. For registry behavior, refer to Model registry.

Shared TTS Keys

The following keys apply to every local model:

Key	Where It Is Read	Default	Effect
`type`	`get_tts_service_from_config`	`nemo`	One of `nemo`, `nvidia`, or `nemotron`. `nvidia` builds the hosted service. The other two use the local dispatch on `model`.
`device`	factory	`cuda`	Torch device for the model.
`think_tokens`	`BaseNemoTTSService`	`["<think>", "</think>"]` in all three sub-configs	Must be a list of exactly two strings (asserted at construction) or `null`. Text between them is never spoken.
`ignore_strings`	`BaseNemoTTSService` and the aggregator	`["*", "<unk>"]` in `default.yaml`	Substrings stripped from the text before synthesis. The aggregator strips them too, falling back to `*` alone when the key is unset.
`extra_separator`	`build_text_aggregator`	`[',', '\n', '.', '?', '!', ';']` in all three sub-configs	Punctuation that closes a chunk, so speech starts earlier. Setting it to `null` leaves the aggregator with no punctuation marks at all, so chunking then depends on `use_legacy_eos_detection`.
`use_text_aggregator`	`build_text_aggregator`	`true`	Set to `false` to drop the `LLMTextProcessor` stage entirely and let Pipecat’s plain sentence splitting run.
`min_sentence_length`	`build_text_aggregator`	`5`	Chunks shorter than this are held back and merged with the next text.
`use_legacy_eos_detection`	`build_text_aggregator`	`false`	Fall back to Pipecat’s `match_endofsentence` when this repository’s punctuation search finds no chunk end.

The last three have no entry in the shipped YAML files — add them under tts: if you need them.

Chunking and Latency

Text aggregation is not part of the TTS service. Since Pipecat 1.0, it belongs to an LLMTextProcessor that build_llm_text_processor inserts between the LLM and the TTS service, built from SimpleSegmentedTextAggregator (nemo_voice_agent/pipecat/utils/text/simple_text_aggregator.py). That aggregator is why extra_separator includes ,. It emits a chunk at the last valid comma, so audio starts before the sentence is finished. Its period and comma heuristics avoid splitting on decimals (3.14), bullet numbering (1.), abbreviations (e.g., Dr.), and times (p.m.).

Pipecat silently ignores the aggregator when you pass it to the TTS service because it drops unknown constructor keyword arguments. Segmentation then degrades without an error. In a custom pipeline, keep the processor upstream of TTS. For the construction pattern, refer to Builders.

If TTS playback stutters on the client, raise transport.audio_out_10ms_chunks in default.yaml. The shipped value is 8, and Pipecat’s WebSocket default is 4.

Reasoning Models

think_tokens keeps a reasoning model’s reasoning text out of the audio. _handle_think_tokens tracks the open/close markers across streamed chunks and returns only the text after the closing token, so the user hears the answer and not the deliberation. Set it to null to think out loud. When the LLM runs on vLLM with a reasoning parser, the parser strips the reasoning upstream. For details, refer to Reasoning mode.

Kokoro-Specific Keys

The following settings control Kokoro voice selection, speed, and text normalization behavior.

Key	Default	Effect
`speed`	`1.25` in `kokoro_82M.yaml` (`1.0` in code)	Speaking rate multiplier. Must be greater than zero.
`sub_model_id`	`af_heart`	Voice id passed to the Kokoro pipeline.

KokoroTTSService preloads both English pipelines (a = American, b = British) at startup so voice switches at conversation time do not pay a download cost.

Kokoro is also the canonical component-owned tool provider. setup_tool_calling registers six direct functions that the LLM can call mid-conversation:

tool_tts_speak_faster and tool_tts_speak_slower, each applying a 15% relative change.
tool_tts_set_speed and tool_tts_reset_speed.
tool_tts_set_voice and tool_tts_reset_voice. The setter accepts American English or British English plus gender and maps to af_heart, am_michael, bf_emma, or bm_george.

The functions are only registered when llm.enable_tool_calling is true. server.py passes the TTS service in the tool_mixins list. For registration details, refer to Tool calling. FastPitch-HiFiGAN and Magpie register no tools.

An RTVI context reset also resets the service: Kokoro’s reset() restores the original speed, voice, accent, and pipeline, so a new session never inherits the previous caller’s “speak faster”.

Magpie-Specific Keys

The following settings control Magpie language and text normalization behavior.

Key	Default	Effect
`language`	`en`	Language code passed to `do_tts`.
`speaker`	`Sofia`	One of `Sofia`, `Aria`, `John`, `Jason`, or `Leo`. An unknown name raises `ValueError` at construction.
`apply_TN`	`false`	Run the model’s text normalization before synthesis.

Magpie warms up at load time by synthesizing a fixed sentence, so the first real turn is not slowed by lazy CUDA initialization.

Licensing Note for Kokoro

Kokoro’s upstream grapheme-to-phoneme stack falls back to espeak-ng through phonemizer, both GPL-3.0, which this repository excludes. _espeak_gpl_shim.py installs no-op substitutes so kokoro and misaki import cleanly, and _g2p_fallback.py supplies an Apache-2.0 replacement built on g2p_en that maps ARPAbet to misaki’s phoneme inventory. Without that fallback, out-of-vocabulary words would be silently dropped from the audio. Both modules are in nemo_voice_agent/pipecat/services/nemo/.

Record Synthesized Audio

Audio logging captures synthesized output alongside the other session audio when recording is enabled.

build_tts accepts the audio logger built by build_audio_logger, so bot audio is captured when recording is enabled. For recording configuration, refer to Audio logging.

Use these pages to understand the upstream language-model stream and configure speech output behavior.

LLM Backends — select and configure the upstream language model.
Reasoning Mode — prevent reasoning text from reaching speech output.
Server Configuration — understand configuration precedence.
Audio Logging — capture synthesized output.