Tool Execution Intercept Outcomes

A managed tool callback and every execution-intercept continuation return one canonical application-visible result:

{
  "result": {}
}

result is required and contains the application-owned tool payload. annotation is optional opaque adjacent metadata. It can be any JSON value. An absent or JSON null annotation means that the result has no annotation; an empty object ({}) is a present annotation. Relay transports the annotation without interpreting its schema.

A tool execution intercept wraps or short-circuits that callback and returns a canonical outcome:

{
  "result": {},
  "annotation": {"source": "cache"},
  "pending_marks": []
}

result and annotation form the application-visible result passed to the remaining middleware. pending_marks defaults to an empty list and remains Relay-owned lifecycle data. Each pending mark contains name, optional category and category_profile, and optional data, data_schema, metadata, and typed log severity. Relay assigns event UUIDs, parent UUIDs, and timestamps.

Continuation Semantics

The default tool callback and an intercept’s next(args) continuation both return ToolExecutionResult. The continuation exposes the downstream result and annotation, but not downstream pending marks. An intercept can preserve, replace, or remove the annotation in its returned outcome. Relay retains pending marks separately, combines them in effective middleware order, and materializes them only after the final outcome succeeds.

Use the existing global, scope-local, or plugin-context tool execution registration APIs to produce pending marks, and return the canonical outcome from every registered callback. Use a mark event sanitizer when the emitted pending mark’s data, category_profile, or metadata must be sanitized. Public APIs, native API 1 plugins, and grpc-v1 workers require canonical tool callback and intercept returns. Relay 0.8 does not provide a legacy raw-result adapter.

Event and Exporter Projection

For a successful tool end event, Relay records result as the event output and places a non-null annotation at category_profile.tool_result_annotation. Tool sanitize-response guardrails receive and sanitize only result. They cannot read or rewrite the annotation. The scope-end event sanitizer runs after the tool response sanitizer and can replace or remove category_profile.tool_result_annotation before subscribers or exporters receive the event.

ATOF preserves the annotation in the tool end event’s category profile. ATIF places it in the observation result’s extra.tool_result_annotation field. The full and openinference OpenTelemetry projections serialize it as one opaque JSON-valued nemo_relay.tool.result.annotation attribute rather than flattening application-defined keys. The standards-only gen_ai projection does not emit Relay-private attributes.

Relay returns the unsanitized ToolExecutionResult to the application. Event sanitizers affect observability only.

Normalized Tool Results

Framework integrations can use one JSON format for tool results. This wrapper holds the display content, a failure flag, and the encoded result. Store it in ToolExecutionResult.result. The following example shows its fields:

{
  "schema": "nemo.relay.tool_result@1",
  "content": "The tool's answer",
  "is_error": false,
  "result": {"framework_owned": "payload"}
}

All four fields are required:

The wrapper can also include a cache object. Set cache.replayable to true only when the codec can restore the result in a later call. The codec can keep source call IDs in this object. It must bind the restored result to the new call. A result that needs objects from the original call must set cache.replayable to false.

This flag does not enable caching. The tool must also meet the cache policy. The built-in cache does not store or replay wrappers unless cache.replayable is true.

Relay reads the content and failure flag only when the wrapper has the expected schema and field types. Other JSON values pass through unchanged. The optional ToolExecutionResult.annotation field stays separate. Relay copies that field without reading its contents.

Tool end events include the full wrapper. ATIF uses content for the observation when it can represent the text or image parts. It keeps the full wrapper in extra.tool_result. For failed results, ATIF also adds extra.is_error: true. Successful results omit this flag.

When is_error is true, Relay sets otel.status_code to ERROR for the tool span. This applies to both managed and manual tool end helpers. The tool still returns its result. Relay does not raise an exception or emit an exception event for this handled failure.

Tool response sanitizers receive the full wrapper. Event sanitizers can still change or remove fields before export.

Execution intercepts can pass the wrapper through or edit its content and failure flag. For LangChain and LangGraph, the codec applies these edits to the matching ToolMessage. It keeps the message’s other fields and the graph state. If an intercept returns a cached result without calling the tool, it must return the full wrapper for that framework’s codec.

The LangChain and LangGraph tool codec can save and restore JSON values, standard framework messages, commands, and graph destinations across calls. On replay, it binds matching tool messages to the current tool-call ID. It clears their old graph message IDs so the graph can assign new ones. Other messages keep their IDs.

Custom message subclasses and other Python objects, such as artifacts and state objects, stay in the original call’s memory. The codec marks these results as not replayable. The built-in cache runs the tool again instead of reusing them.

Managed Lifecycle

On successful managed execution, Relay emits the tool end event before any pending marks. The end timestamp is no earlier than one microsecond after the tool start timestamp. Relay emits pending marks in their resolved middleware order, assigns each mark the managed tool UUID as its parent, and gives each mark a timestamp after the tool end event.

If execution or an intercept fails, Relay emits the error end event without a tool result annotation and discards accumulated pending marks. Pending marks are never included in the application-visible tool result or passed to tool sanitize-response guardrails. They pass through the mark sanitizer registry when materialized.

Tool Execution Context

Beginning with Relay 0.9, every tool execution intercept receives the tool call’s context instead of separate tool-name and argument parameters. The context carries three fields:

tool_call_id is the same identifier that Relay records on tool lifecycle events. An intercept that completes execution itself, without invoking the remaining chain, uses it to associate its result with the originating tool call.

The existing registration names now use the context callback. Scope-local and plugin-context registrations follow the same naming.

The callback takes two parameters because the tool name and arguments move onto the context:

from collections.abc import Awaitable, Callable
import nemo_relay
from nemo_relay import Json, ToolExecutionContext, ToolExecutionInterceptOutcome, ToolExecutionResult
async def measure(
    context: ToolExecutionContext,
    next_call: Callable[[Json], Awaitable[ToolExecutionResult[Json]]],
) -> ToolExecutionInterceptOutcome:
    downstream = await next_call(context.args)
    record(context.tool_call_id, context.tool_name, downstream.result)
    return ToolExecutionInterceptOutcome(result=downstream.result)
nemo_relay.intercepts.register_tool_execution("measure", 10, measure)

registerToolExecutionIntercept('measure', 10, async (context, next) => {
  const downstream = await next(context.args);
  record(context.toolCallId, context.toolName, downstream.result);
  return { result: downstream.result };
});

This is a breaking callback change: migrate Rust, Python, and Go callbacks from (tool_name, arguments, next) to (context, next), and migrate Node.js and C callbacks from their argument-only shape to the context shape. In Node.js the context object uses toolName, args, and toolCallId, where toolCallId is string | null. The public C callback receives a single context_json string carrying tool_name, args, and tool_call_id; callbacks must ignore unknown fields. The grpc-v1 ToolInvocation message and its optional tool_call_id field are unchanged.

Binding Contract

Managed callbacks, continuations, and execute helpers use the same logical result in their native type or object shape:

Execution intercepts add lifecycle-owned pending marks to that shape:

Canonical JSON uses annotation, pending_marks, and category_profile across bindings.

Relay 0.8 establishes this canonical result as the native API 1 and grpc-v1 baseline. Rebuild every dynamic plugin and declare a compat.relay range that excludes Relay versions before 0.8. The recommended range is >=0.8.0,<1.0; open-ended ranges such as >=0.8.0 are valid. The native ABI for that result-shape change is v4. Workers retain the grpc-v1 identifier and nemo.relay.worker.v1 package, but Relay 0.8 changes the ToolNext response type and tool-execution outcome field type. Regenerate worker protobuf bindings as part of the required rebuild. Future incompatible contract changes must increment the corresponding native API or worker protocol version.

Relay 0.9 replaces the tool execution callback shape without changing the grpc-v1 package or result messages. Worker plugins that register this surface must declare compat.relay = ">=0.9,<1.0" or another range that excludes Relay 0.8. Native plugins use ABI v5 for the context-carrying callback slot; Relay rejects the registration when the plugin admits an incompatible Relay version.

Migration

This release deliberately replaces raw tool execution results. Make these changes together:

1. Wrap every managed tool callback return in ToolExecutionResult.
2. Treat every execution intercept’s next(args) return as ToolExecutionResult.
3. Return the downstream result and annotation from forwarding intercepts, or deliberately replace or remove the annotation.
4. Read .result when an application or framework needs the original business payload.
5. Wrap the result passed to a manual tool end helper in ToolExecutionResult.

Return-Shape Errors

Python distinguishes ToolExecutionResult from ToolExecutionInterceptOutcome. A callback that returns a raw dictionary is rejected; return ToolExecutionResult(payload) instead. A forwarding intercept must unwrap the downstream result:

downstream = await next(args)
return ToolExecutionInterceptOutcome(
    result=downstream.result,
    annotation=downstream.annotation,
)

Do not pass downstream itself as the outcome result or return it directly. The resulting RuntimeError names the required wrapper and this unwrapping form.

Node.js represents both values as compatible object shapes. Therefore, return await next(args) is a valid forwarding intercept in Node.js. Raw tool callback returns and raw intercept outcomes are still rejected with guidance to return { result: payload } or { result: downstream.result, annotation: downstream.annotation }.

Rebuild native plugins and workers against the same NeMo Relay release that hosts them. For release-specific upgrade actions, refer to the Migration Guides.

LLM request intercepts use a different outcome because they must preserve an annotated request and codec authority. That contract is described in LLM Request Intercept Outcomes. Worker authors can trace both outcome envelopes through the grpc-v1 protocol reference.