nemo_voice_agent.pipecat.processors.frameworks

Bases: _RTVIObserver

An observer that processes RTVI frames and pushes them to the transport.

nemo_voice_agent.pipecat.processors.frameworks.rtvi.RTVIObserver.on_push_frame(
    data: pipecat.observers.base_observer.FramePushed
)

nemo_voice_agent.pipecat.processors.frameworks.rtvi.RTVIObserver.reset()

Mutable handle to a PipelineWorker and its running flag.

Construct early, hand to RTVI action factories, then populate once the task exists. running is flipped by the bot runner during shutdown so handlers can avoid queueing frames onto a dead task.

Build the context.get_context_history action.

Returns the assistant aggregator’s full message list, stringified to match the shape evaluation clients expect.

Race-safety against in-flight function calls. When the bridge fetches the context immediately after receiving the <exit> server message (the SendExitMessageTool / EndConversationTool flow), the agent’s final tool_call may still be in-flight in the aggregator’s pipeline — the exit message arrives via the bot_server_message channel while the tool_call frame is still being committed via the FunctionCallInProgressFrame → FunctionCallResultFrame cycle. If we read the context before those frames commit, the captured message list is missing the final assistant turn. The judge then sees a stale context that doesn’t contain the EndConversationTool tool_call and (incorrectly) deducts points for “didn’t call EndConversationTool”.

Fix: poll the aggregator until its in-progress map drains, with a hard deadline so a stuck tool can never deadlock scenario cleanup. The common case (no pending calls) returns immediately — zero added latency.

Build the context.reset action.

original_messages is captured by reference so the action always resets to whatever update_system_prompt last wrote.

Build the context.update_system_prompt action.

Tool registration is optional. When enable_tool_calling is True and a tools JSON string is supplied by the caller, tool_factory is invoked per tool to produce schema tools, then register_schema_tools swaps them onto llm / context. This keeps the factory decoupled from evaluation-specific tool registries.

Scenario-start lifecycle gate. This handler is the bot-side signal that a new scenario is starting. In addition to swapping the prompt + tool surface, it RESETS shared_state_ref.state so any prior scenario’s data (db, actions log, etc.) doesn’t bleed into the new one. The reset is done via dict.clear() rather than reassignment so the dict identity is preserved — any tool that already holds a reference to shared_state continues to see the same object after the clear, and subsequent mutations by apply_initialization propagate correctly.

Scenario fixture data (db, db_path, actions list, custom keys from Scenario.setup_shared_state) is NOT loaded here — that moves to create_apply_initialization_action which the bridge calls immediately after this one. This handler only stashes bot-side runtime sentinels (__rtvi__, __tool_domain__) into the freshly-cleared dict so write tools have them available at call time.

Install one on_client_message dispatcher for handlers.

This is the pipecat 1.x replacement for calling rtvi.register_action(...) once per action. Handler return values are sent back as the d payload of a server-response keyed to the request’s message id; an unknown type or a raised exception produces an error-response instead, so callers fail loudly rather than blocking until their read timeout expires.

Return a deep copy of an LLM-context structure with raw media blobs dropped.

Omni models keep audio (and, in future, image/file) payloads inline in their message list. Serializing those verbatim produces a multi-MB payload that overflows pipecat’s WebSocket frame cap, so get_context_history silently fails and the saved agent context ends up empty.

Only the raw encoded data is dropped (replaced with an <audio> / <image> / <file> / <binary> tag) — small metadata is kept so the snapshot stays useful for debugging:

• dropped: bytes / bytearray values; data: base64 URI strings (e.g. image_url.url); file_data / b64_json leaves; and the data leaf when nested inside a media container (input_audio.data, assistant audio.data).
• kept: text / refusal strings, format, filename, file_id, detail, audio transcript, id, and plain url / file-path strings (only base64 data: URIs are stripped, not real URLs/paths).

_media_key carries the nearest enclosing media-container key name down the recursion so a stripped leaf gets the right tag. Pure and non-mutating — the caller’s live context is never altered; a fresh structure is returned.