Client Protocol

Connecting to NeMo Labs Voice Agent requires a two-step handshake. A client first calls POST /connect on the FastAPI port to discover the media socket. It then opens that WebSocket and exchanges Protobuf-encoded frames. The frames carry raw PCM audio in both directions and JSON control messages in the real-time voice inference (RTVI) envelope.

The two servers are started together by run_bot_with_fastapi in nemo_voice_agent/pipecat/bot_server.py.

Prerequisites

Before you connect a custom client, complete the following preparation:

1. Start the voice-agent server by following the Quickstart.
2. Choose whether the client uses the /connect discovery endpoint or connects directly to the WebSocket port.
3. Use a client that can serialize and deserialize Pipecat Protobuf frames.

Step 1: POST /connect

The endpoint takes no meaningful request body and returns one key:

$
curl -s -X POST http://127.0.0.1:7860/connect
$
# {"ws_url":"ws://127.0.0.1:8765"}

The URL is assembled by build_websocket_url in nemo_voice_agent/utils/websocket_url.py from three server-side values — it never echoes anything the client sent:

The app is created with permissive CORS (allow_origins=["*"]), so a browser on another origin can call it directly. The /ws route on the FastAPI app is an unimplemented stub — ignore it. Full variable list is in Environment variables.

Step 2: The WebSocket

The transport is Pipecat’s SingleClientWebsocketServerTransport, built by build_ws_transport in nemo_voice_agent/pipecat/services/nemo/builders.py with a ProtobufFrameSerializer. Every WebSocket message — in both directions — is a binary Protobuf Frame with a oneof:

Audio is 16-bit signed little-endian PCM, mono. Inbound audio is expected at transport.audio_in_sample_rate (default 16000). Outbound audio follows the TTS service’s rate unless you set transport.audio_out_sample_rate. The JavaScript transport plays back at 24 kHz by default.

RTVI Messages

Control traffic rides inside MessageFrame.data as a JSON object with label, type, id, and data keys. Two exchanges matter to every client:

1. client-ready — the client must send it after connecting. The server replies with bot-ready, and, because the example server passes talk_first=True, queues an LLMRunFrame so the bot speaks first. A client that never sends client-ready remains silent.
2. client-message — a request/response call whose data is {"t": "<type>", "d": {...}}. The server answers with server-response (payload under d) or error-response on an unknown type or a raised exception. Dispatch is installed by register_client_message_handlers in nemo_voice_agent/pipecat/processors/frameworks/rtvi_actions.py.

The example server registers exactly one custom type, reset, which clears the conversation context back to the original system prompt. The evaluation bots register five more. Refer to RTVI Control Plane and RTVI Message Reference.

Server-to-client event messages (user-transcription, bot-transcription, bot-llm-text, bot-tts-text, bot-started-speaking, bot-stopped-speaking, metrics, server-message, …) are emitted by the RTVIObserver attached to the pipeline worker.

Browser Client

examples/generic_voice_agent/client/ is a vanilla-TypeScript Vite app built on @pipecat-ai/client-js and @pipecat-ai/websocket-transport. The whole connection is three calls in src/app.ts:

1
const transport = new WebSocketTransport();
2
const client = new PipecatClient({ transport, enableMic: true, enableCam: false, callbacks });
3
await client.initDevices();
4
await client.connect({ endpoint: "http://<host>:7860/connect" });

connect() issues the POST, reads ws_url out of the JSON response, and gives it to the transport. You do not connect to the socket directly. The reset message goes through client.sendClientRequest("reset", {}), which resolves with the handler’s return value.

Run it with npm install && npm run dev. The dev server listens on port 5173 on all interfaces and proxies /connect to http://0.0.0.0:7860. The demo page has a Server dropdown. Leave it on “WebSocket Server”, which points at port 7860. Startup details are in the Quickstart.

Non-Browser Client

Any WebSocket client works. nemo_voice_agent/evaluation/bridge.py is a complete Python implementation that drives both bots in the eval harness. Use it as the implementation reference. A minimal client must:

1. POST /connect and read ws_url (or skip discovery and dial ws://host:8765 directly).
2. Send a client-ready RTVI message and wait for bot-ready.
3. Stream 16 kHz mono PCM as serialized OutputAudioRawFrames, paced in real time.
4. Deserialize incoming frames: audio arrives as InputAudioRawFrame, RTVI JSON as InputTransportMessageFrame with the parsed dict on .message.

1
import asyncio
2
import json
3
4
import requests
5
import websockets
6
from pipecat.frames.frames import OutputAudioRawFrame
7
from pipecat.serializers.protobuf import MessageFrame, ProtobufFrameSerializer
8
9
10
async def main() -> None:
11
    serializer = ProtobufFrameSerializer()
12
    ws_url = requests.post("http://127.0.0.1:7860/connect", timeout=10).json()["ws_url"]
13
14
    async with websockets.connect(ws_url, ping_timeout=None) as ws:
15
        ready = {
16
            "label": "rtvi-ai",
17
            "type": "client-ready",
18
            "id": "client-ready-1",
19
            "data": {"version": "1.1.0", "about": {"library": "my-client", "library_version": "0.1.0"}},
20
        }
21
        await ws.send(await serializer.serialize(MessageFrame(data=json.dumps(ready))))
22
23
        # Then, per 20 ms tick: send microphone PCM.
24
        chunk = b"\x00" * 640  # 320 samples of int16 at 16 kHz
25
        await ws.send(await serializer.serialize(
26
            OutputAudioRawFrame(audio=chunk, sample_rate=16000, num_channels=1)
27
        ))
28
29
        async for message in ws:
30
            frame = await serializer.deserialize(message)
31
            print(type(frame).__name__)
32
33
34
asyncio.run(main())

Send OutputAudioRawFrame even though the server sees it as input — that is the direction the serializer’s tables expect.

Connection Rules

Clients must follow these lifecycle rules to connect without replacing or corrupting the active session:

• One client at a time. While a client is connected, a second connection is closed immediately with WebSocket code 1013 and reason Server already has a connected client. The incumbent keeps talking.
• The pipeline outlives the connection. On disconnect the server deliberately does not end the pipeline task. The WebSocket listener lives inside the input transport, so ending the task would leave /connect advertising a dead port. The next client enters the same running pipeline.
• Context survives reconnects. Because the pipeline persists, so does the LLM conversation history. Send the reset message to clear it explicitly.

Next

Continue with the protocol guide or reference that matches your integration work:

• RTVI control plane — add your own client message types.
• RTVI message reference — every message the server understands.
• How it works — where the transport sits in the pipeline.
• Troubleshooting — connection failures and what causes them.