RTVI Message Reference

Use this reference to look up the custom real-time voice interface (RTVI) client messages that a NeMo Labs Voice Agent bot handles. The handlers live in nemo_voice_agent/pipecat/processors/frameworks/rtvi_actions.py. For handler behavior and extension instructions, refer to RTVI Control Plane. For the audio interface on the same socket, refer to Client Protocol.

Envelope

Control messages travel on the same WebSocket as audio. Pipecat wraps each message in a MessageFrame, and ProtobufFrameSerializer encodes the frame. The frame body is a JSON string in Pipecat’s RTVI envelope:

1
{
2
  "label": "rtvi-ai",
3
  "type": "client-message",
4
  "id": "get_scenario_summary_1754500000.0",
5
  "data": { "t": "get_scenario_summary", "d": { "include_db": false } }
6
}

t is the message type (the wire name in the table below) and d is the argument object. id is caller-chosen and echoed back — match on it, because responses to concurrent requests arrive interleaved.

The d field of a server-response contains the handler return value:

1
{
2
  "label": "rtvi-ai",
3
  "type": "server-response",
4
  "id": "get_scenario_summary_1754500000.0",
5
  "data": { "t": "get_scenario_summary", "d": { "actions": [], "db_hash": null } }
6
}

An unknown t or a handler exception produces an error-response. This response lets callers fail fast instead of blocking until a read timeout:

1
{
2
  "label": "rtvi-ai",
3
  "type": "error-response",
4
  "id": "get_scenario_summary_1754500000.0",
5
  "data": { "error": "KeyError: 'db'" }
6
}

For browser clients built on @pipecat-ai/client-js, sendClientRequest(type, args) builds the envelope and resolves with the d payload. The evaluation bridge (nemo_voice_agent/evaluation/bridge.py) constructs the envelope directly, as shown in the examples on this page.

Message Types

The following table summarizes the custom request types registered by the example and evaluation servers.

examples/generic_voice_agent/server/server.py registers reset only. evaluation/bot_server.py registers all six. There is no schema or defaults layer on this path: every handler reads arguments with arguments.get(name, default), so unknown keys are ignored and missing keys fall back silently.

reset

Send "d": {}. Response d is a bare boolean — true on success, false if the aggregator reset raised.

update_system_prompt

Use the following arguments to replace the evaluation bot’s system prompt and register scenario tools.

Response d is a bare boolean. The tools value is a JSON string, not a nested object.

1
{"t": "update_system_prompt",
2
 "d": {"prompt": "You are a support agent.",
3
       "tools": "{\"GetUserDetailsTool\": {}}",
4
       "add_suffix": false,
5
       "tool_domain": "tau2_airline"}}

Scenario fixture data does not travel here — shared_state_init is an argument of apply_initialization.

get_context_history

Send "d": {}. Response d is {"context": "..."} where the value is the assistant aggregator’s message list rendered with Python str(), after sanitize_context_for_transport has replaced inline audio, image, and file blobs with placeholder tags. On an internal error the handler returns context as an empty list.

get_scenario_summary

Use the following argument to control whether the response includes the bot’s inline database.

1
{"actions": [{"name": "book_reservation", "action_type": "...", "arguments": {}}],
2
 "db_hash": "3f2a…",
3
 "db": null}

db_hash is the SHA-256 of the canonicalized DB from nemo_voice_agent.evaluation.db_hash.get_dict_hash, or null when the bot holds no DB. The db key is present only when include_db was true. Keep it off for large databases, which overflow Pipecat’s 1 MB WebSocket frame cap and close the connection with code 1009. Each bot returns only its own DB. The caller applies agent-versus-user labels based on which socket the response arrived on.

apply_initialization

Use the following arguments to seed shared state, load fixture data, and apply initialization actions.

1
{"t": "apply_initialization",
2
 "d": {"domain": "tau2_telecom",
3
       "shared_state_init": "{\"db_path\": \"tau2_telecom/db.json\"}",
4
       "actions": [{"func_name": "set_data_usage", "arguments": {"usage_gb": 15.1}}]}}

Response d is {"success": bool, "errors": [string]}. success is false, with an explanatory entry in errors, under any of these conditions:

• shared_state_init is not valid JSON or does not decode to an object.
• actions is not a list.
• Actions were supplied, but no db could be resolved.

A db_path is resolved against the eval data root (nemo_voice_agent/evaluation/data/, overridable with EVAL_DATA_ROOT) and is skipped when db is already present.

apply_sync_delta

Use the following arguments to apply a cross-side state update to the bot’s database.

1
{"t": "apply_sync_delta",
2
 "d": {"domain": "tau2_telecom",
3
       "delta": {"surroundings.line_active": true,
4
                 "bills[B1002].status": "Paid"}}}

Response d is {"success": bool, "errors": [string]}. It is false when delta is not a dict or the bot has no db in shared_state. Failures are informational for the caller — a malformed delta must not stall the conversation.

Server-to-Client Messages

The bot also pushes unsolicited messages that the evaluation bridge monitors. These are Pipecat server-message frames with a custom data payload, plus the standard RTVI lifecycle and transcript events.

The bridge also uses two Pipecat built-ins on the outbound path. The client-ready handshake awaits bot-ready. The send-text message injects a text turn into the bot’s large language model (LLM) with the run_immediately and audio_response options. Both are Pipecat protocol messages, not handlers defined in this repository.

The evaluation harness captures wire traffic in bridge_log.txt inside each scenario result directory. For the result workflow, refer to Evaluation.