Server Config

The NeMo Labs Voice Agent server is driven entirely by one YAML file. This page walks through the top-level blocks of examples/generic_voice_agent/server/server_configs/default.yaml and names the builder that consumes each one.

How the File Is Loaded

ConfigManager (nemo_voice_agent/utils/config_manager.py) loads the top-level YAML, then for the stt, llm, and tts blocks merges in a model-specific sub-config. examples/generic_voice_agent/server/server.py passes server_base_path=os.path.dirname(__file__) and reads the config path from the SERVER_CONFIG_PATH environment variable, defaulting to <server dir>/server_configs/default.yaml.

cd examples/generic_voice_agent/server
# Uses default server_configs/default.yaml if SERVER_CONFIG_PATH not specified: 
python server.py
# Or use a specific server cofnig pointed by SERVER_CONFIG_PATH:                                                   
# SERVER_CONFIG_PATH=./server_configs/default_nvidia.yaml python server.py

Three rules govern the merge:

• The sub-config wins. For every key present in the sub-YAML, ConfigManager assigns it over the top-level value and logs an ... is overridden from ... to ... line. Editing a key in default.yaml that the sub-YAML also sets has no effect.
• Only the basename of model_config matters. The directory is fixed per component: server_configs/stt_configs/, server_configs/llm_configs/, server_configs/tts_configs/.
• Interpolation in the top-level file is resolved eagerly at load (OmegaConf.to_container(..., resolve=True)), while sub-config values are copied over unresolved and resolved lazily against the merged config. That is how nemotron_3.5_lightning.yaml can write temperature: ${llm.temperature} — it resolves to the temperature: 1.0 the same sub-config contributed, since default.yaml defines no llm.temperature at all.

If a component omits model_config and server.use_model_registry is true, the model name is looked up in model_registry.yaml instead. Refer to Model Registry.

server

The server block controls model-registry resolution and the connection lifecycle.

transport

Consumed by build_ws_transport and build_audio_logger. The transport is SingleClientWebsocketServerTransport from Pipecat: while a client is connected, a second connection is closed with code 1013 and the incumbent is kept.

vad

Consumed by build_vad_analyzer, which always constructs a SileroVADAnalyzer — VAD cannot be turned off on this pipeline, and type: silero is descriptive only. build_vad_processor wraps the analyzer in the processor placed right after transport.input().

All four are read as direct attributes and passed to Pipecat’s VADParams, so all four must be present.

stop_secs is only part of the end-of-turn latency. With turn_taking.type: speech_timeout, the Pipecat stop strategy starts its own timers when the VAD frame arrives, so the user turn ends at least vad.stop_secs + turn_taking.user_speech_timeout after the last speech and can take longer while it waits for a transcript. With turn_taking.type: nemo, only stop_secs applies. Change the two keys together, and refer to Turn Taking for the exact expression.

stt

Consumed by build_stt using get_stt_service_from_config. type accepts nemo, nemo_speechlm, or nvidia. Any other value raises. For details, refer to ASR.

For type: nemo_speechlm, the block instead points at a vLLM endpoint with base_url, generation_kwargs, system_prompt, and user_prompt, as server_configs/default_salm.yaml shows. Add max_tokens_per_sec there to cap the decode budget by audio duration, which contains the repetition hallucinations that a flat generation_kwargs.max_tokens cannot. It is unset by default. Refer to ASR.

For type: nvidia, the relevant keys are language, model, and function_id — the model name and function id address one specific NVCF deployment and must be changed together. NVIDIA_API_KEY from the environment takes precedence over any api_key in YAML.

diar

Consumed by build_diar, which returns None when enabled is false, so the diarization stage drops out of the pipeline entirely. Refer to Diarization.

Two keys in the shipped file are inert: diar.device is not read — build_diar runs the diarizer on stt.device — and diar.type has no consumer.

turn_taking

Consumed by build_turn_taking, which builds a NeMoTurnTakingService for type: nemo and returns None for type: speech_timeout. The type also decides who emits user-turn frames. With nemo, the service emits them and build_context_and_aggregators selects ExternalUserTurnStrategies for the aggregator. With speech_timeout, the aggregator drives turns from VAD itself. Refer to Turn Taking.

This block used a boolean enabled key before. A config that still sets it gets a startup warning naming the mapping and falls back to the nemo default: use type: nemo in place of enabled: true, and type: speech_timeout in place of enabled: false.

llm

Consumed by build_llm using get_llm_service_from_config, and by build_context_and_aggregators for the system message. Refer to Large Language Model Backends.

Backend-specific keys arrive from the sub-config. nemotron_3.5_lightning.yaml adds system_role, system_prompt_suffix, enable_tool_calling, inject_dummy_user_message, sampling knobs (temperature, top_k, top_p, min_p, max_new_tokens), the vLLM endpoint (api_key, base_url), start_vllm_on_init, vllm_server_params, and the vllm_generation_params block sent to the OpenAI API. Because start_vllm_on_init is false, python server.py does not work by itself. Start vLLM first with the flags from vllm_server_params:

vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --trust-remote-code --tensor-parallel-size 1 --enable-prefix-caching \
  --max-num-seqs 1 --gpu-memory-utilization 0.8 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

Setting llm.is_omni_model inserts a UserAudioBuffer before the user aggregator, which reads the pre_cache_duration_secs, use_stt_transcript, keep_only_last_audio_turn, text_prompt_for_audio, and text_prompt_for_transcript keys. Refer to Multimodal.

tts

Consumed by build_tts (the synthesis service) and build_llm_text_processor (the upstream text segmenter). type accepts nemo, nvidia, or nemotron. For type: nemo, the supported model values are kokoro, fastpitch-hifigan, and magpie. Refer to TTS.

The NIM Variant

server_configs/default_nvidia.yaml runs the same pipeline against hosted NVIDIA endpoints. The stt.type, llm.type, and tts.type keys are all nvidia, and NVIDIA_API_KEY must be set. diar.enabled is false and turn_taking.type is speech_timeout, so the pipeline has neither a diarization nor a turn-taking stage and VAD alone drives turns. Its vad.stop_secs: 0.2 and user_speech_timeout: 0.6 end a user turn 0.8 to 1.0 seconds after the last speech, the spread coming from the strategy’s transcript timer. Refer to Turn Taking. The llm block carries base_url, system_role, system_prompt_suffix, and a nvidia_generation_params block instead of the vLLM ones. Because none of its models appear in model_registry.yaml and none set model_config, ConfigManager logs a not-in-registry warning per component and every key is taken from this one file. Refer to NVIDIA NIM.

Next Steps

Continue with the guide for the component or behavior you need to refine:

• Model Registry — how a model name resolves to a sub-config.
• Prompts — system prompt files and suffixes.
• Config Schema — full key reference.
• Builders — the functions that turn each block into a pipeline stage.