Integrate into Frameworks

Handle Non-Serializable Data

View as Markdown

Use this guide when a framework exposes SDK clients, streams, callbacks, file handles, or custom classes at the same boundary where you need NeMo Relay instrumentation.

What You Build

You will keep non-serializable framework objects in framework-owned storage and pass only stable JSON projections through NeMo Relay middleware and event payloads.

Before You Start

You need:

  • A stable request ID, tool call ID, or framework request object that can key external storage.
  • A JSON-compatible projection of the data that guardrails, intercepts, and subscribers need.
  • A cleanup point for framework-owned object maps.

The Constraint

NeMo Relay middleware surfaces operate on JSON-compatible data. Frameworks do not always expose tool or model requests in that form.

These strategies keep provider and framework data JSON-compatible before it reaches NeMo Relay.

  • Convert provider payloads into a stable request shape before NeMo Relay sees them.
  • Preserve opaque framework objects outside the middleware path and pass only the serializable projection into the runtime.
  • Store external object references in framework-owned maps keyed by request IDs, not inside NeMo Relay event payloads.
  • Use typed wrappers for your application boundary, then serialize at the last responsible moment.

Concrete Projection Pattern

Keep framework objects in your own map, but send only the JSON projection through NeMo Relay.

1from typing import TypedDict
2
3import nemo_relay
4
5class SearchArgs(TypedDict):
6    client_id: str
7    query: str
8
9class SearchResult(TypedDict):
10    hits: int
11
12framework_clients: dict[str, object] = {}
13framework_clients["client-1"] = object()
14
15async def invoke(args: SearchArgs) -> SearchResult:
16    client = framework_clients[args["client_id"]]
17    _ = client  # framework-owned object stays outside NeMo Relay payloads
18    return {"hits": 2}
19
20result = await nemo_relay.tools.execute(
21    "search",
22    SearchArgs(client_id="client-1", query="weather"),
23    invoke,
24)

Codec Pattern For Provider Payloads

Use an LLM codec when the framework payload is structurally different from the annotated request model you want intercepts to reason about.

  • Decode the provider payload into a normalized annotated request.
  • Let request intercepts edit the normalized shape.
  • Encode the annotated request back to the provider payload before the real call.
  • Keep sockets, streams, callbacks, and SDK objects outside the codec result.

What Not to Do

Avoid these patterns because they make runtime payloads difficult to serialize or observe.

  • Do not place client instances, callbacks, streams, sockets, or open file handles inside JSON event payloads.
  • Do not rely on Python or JavaScript object identity surviving the middleware boundary.
  • Do not leak framework-internal classes into a plugin config document.

Practical Workarounds

Use these workarounds when framework data cannot be passed directly through NeMo Relay.

  • Replace large objects with IDs and look them up later.
  • Emit summarized metadata instead of full request bodies.
  • Use request codecs to normalize provider payloads.
  • Use manual lifecycle APIs when the framework does not expose a clean execution wrapper.

Common Failure Cases

These failure cases are common signs that non-serializable data crossed the runtime boundary.

  • A request intercept tries to return a framework SDK object instead of JSON.
  • A plugin config stores a callable, client instance, or file handle.
  • A subscriber assumes Python or JavaScript object identity survives event serialization.
  • A worker thread receives a scope UUID but not the corresponding framework-owned object lookup table.

Validation Checklist

Use this checklist to confirm the implementation preserves the expected runtime contract.

  • Middleware receives only JSON-compatible values.
  • The framework callback can still resolve the original SDK client or stream by ID.
  • Subscribers and exporters receive enough metadata to debug the call.
  • Cleanup removes object-map entries after the request finishes.
  • Redaction happens before payloads reach production subscribers.

Next Steps

Use these links to continue from this workflow into the next related task.