NVIDIA NIM and Riva

NeMo Labs Voice Agent can run automatic speech recognition (ASR), the large language model (LLM), and text-to-speech (TTS) against hosted NVIDIA endpoints instead of local GPU models. This configuration-only change sets stt.type, llm.type, and tts.type to nvidia. The builders in nemo_voice_agent/pipecat/services/nemo/ construct Pipecat’s NVIDIA services from the same YAML blocks.

Prerequisites

Before you connect the agent to hosted NVIDIA endpoints, complete the following preparation:

1. Install NeMo Labs Voice Agent by following the Installation guide.
2. Obtain the NVIDIA API key required by the endpoint you plan to use.
3. Choose the demo-server or evaluation configuration for your run.

Run It

server_configs/default_nvidia.yaml is the ready-made top-level config. All three component types are nvidia, and both diar.enabled and turn_taking.enabled are false. There is no diarization NIM, so voice activity detection (VAD) alone drives turn boundaries.

$
export NVIDIA_API_KEY="nvapi-..."
$
cd examples/generic_voice_agent/server
$
SERVER_CONFIG_PATH=./server_configs/default_nvidia.yaml python server.py

SERVER_CONFIG_PATH is resolved against the current working directory, so cd first. The browser client is unchanged. Refer to Quickstart. Unlike the default vLLM path, this config has no local model server to start in a second terminal.

The two-bot eval harness has matching configs (evaluation/server_configs/agent_nvidia.yaml, user_nvidia.yaml, agent_nvidia_omni.yaml). Refer to Evaluation Quickstart.

Credentials

Hosted NVIDIA services read credentials from environment variables rather than the server YAML.

Each component also accepts an api_key key in its YAML block, but the environment variable wins: the builders read os.getenv("NVIDIA_API_KEY", config.get("api_key", "None")). server.py calls load_dotenv(override=True), so a .env file found from the server directory upward is applied and overrides variables already exported in the shell.

STT Keys

Read by get_stt_service_from_config in nemo_voice_agent/pipecat/services/nemo/stt.py, which builds Pipecat’s NvidiaSTTService.

Transient stream drops need no configuration: Pipecat’s NvidiaSTTService reconnects on gRPC errors itself, and defers the reconnect until the user stops speaking.

LLM Keys

Read by get_llm_service_from_config in nemo_voice_agent/pipecat/services/nemo/llm.py, which builds Pipecat’s NvidiaLLMService (an OpenAI-compatible client).

system_prompt, system_role, and system_prompt_suffix behave exactly as on the local backends — refer to Prompts.

TTS Keys

Read by get_tts_service_from_config in nemo_voice_agent/pipecat/services/nemo/tts.py, which builds ResilientNvidiaTTSService from nemo_voice_agent/pipecat/services/nvidia/tts.py — a thin subclass of Pipecat’s NvidiaTTSService.

Why the subclass exists: upstream treats every synthesis exception as terminal, so the NVCF cold-start failure DEADLINE_EXCEEDED: failed to establish link to worker silently drops a whole bot turn. The subclass replays the buffered text and retries — but only when the attempt produced no audio, since re-running mid-utterance would splice a duplicate prefix into the speech. The output sample rate on this path is fixed at 22050 Hz by the builder. tts.sample_rate is not consulted.

Tool Calling

Tool calling works on this backend. default_nvidia.yaml already sets llm.enable_tool_calling: true, which is the only thing server.py checks before registering tools — there is no backend gate.

One difference from the local default: component-owned tools come from services that mix in ToolCallingMixin, and the NVIDIA TTS service does not. The Kokoro-only tools (“speak faster”, “switch accent”) are therefore absent, register_direct_tools_to_llm logs is not a ToolCallingMixin, skipping, and only the direct function tool_get_city_weather is registered. Refer to Tool Calling and Custom Tools.

Reasoning

There is no _think.yaml swap on this path — the config interpolates the switch straight into the request body, so flipping one boolean is enough:

1
llm:
2
  enable_reasoning: false
3
  nvidia_generation_params:
4
    extra:
5
      extra_body:
6
        chat_template_kwargs:
7
          enable_thinking: ${llm.enable_reasoning}
8
        thinking_token_budget: 3000

Reasoning text does not reach TTS. Pipecat’s NvidiaLLMService pulls reasoning_content from the streaming delta and emits it as thought frames instead of spoken text. For models that emit leading <think> spans inline, the service strips those spans. Refer to Reasoning Mode.

Self-Hosted NIM and Riva

The same three blocks target a NIM you host yourself: set llm.base_url to your endpoint’s /v1 URL, and both stt.server and tts.server to your Riva host and gRPC port. Consider two limits before you try a plaintext local deployment:

• The builders do not forward Pipecat’s use_ssl flag, which defaults to True. A Riva server without TLS cannot be reached by YAML alone. It needs a builder change. Refer to Builders.
• stt.language and tts.language are read from YAML and passed to the constructor, but Pipecat 1.6 takes the language from its settings object instead. The value is therefore discarded, and both services stay on en-US.

Gotchas

Use these symptoms to distinguish credential, endpoint, and tool-calling configuration problems.

Related

Continue with the concept or configuration reference that matches the backend you are using:

• LLM Backends — the auto, hf, vllm, nvidia selection.
• ASR and TTS — the local counterparts of these blocks.
• Server Configuration — every top-level block.
• Troubleshooting.