About

NeMo Voice Agent

View as Markdown

NeMo Voice Agent is an open source Python framework for building and evaluating real-time voice agents. It combines streaming automatic speech recognition (ASR), turn taking, large language model (LLM) inference, and text-to-speech (TTS) services in a Pipecat pipeline. The repository also provides a browser client and a spoken-conversation evaluation harness.

Skip Ahead

Use the following paths to find the information that matches your goal.

I Want toStart Here
Understand the systemArchitecture
Run the default voice agentChoose your path
Configure or extend an agentBuild Voice Agents
Measure an agentEvaluate Voice Agents
Resolve a problemTroubleshooting

Use Cases

NeMo Voice Agent supports the following voice-agent development and evaluation use cases.

  • Run a local voice pipeline with configurable speech, language, and text-to-speech models.
  • Connect supported stages to hosted NVIDIA endpoints.
  • Add application tools or custom frame processors.
  • Evaluate an in-repository or external voice agent through live audio conversations.

Core Concepts

Start with these explanations to understand how the framework processes speech and generates responses.

  • Speech pipeline covers recognition, diarization, turn taking, and synthesis.
  • Language models covers backends, reasoning, and multimodal execution.
  • Architecture shows how the pipeline stages and control messages fit together.

Core Components

The following components work together to provide the runtime and evaluation experiences.

ComponentRole
Browser clientCaptures microphone audio, opens the WebSocket session, and plays assistant audio.
Voice-agent serverLoads configuration, builds the Pipecat pipeline, and serves one client connection.
Model servicesProvide ASR, LLM, diarization, turn-taking, and TTS behavior.
Evaluation harnessConnects a simulated user and agent, records artifacts, and scores scenario outcomes.

For dated project updates, refer to Release Notes.