Add Middleware

Use this guide when instrumentation is working and you want NeMo Relay to enforce policy, transform requests, wrap execution, or sanitize observability payloads around tool and LLM calls.

What You Build

You will add middleware to an instrumented application and verify that it runs in the expected part of the pipeline:

• Request intercepts transform the real request before execution.
• Sanitize guardrails transform only the payload recorded on events.
• Conditional-execution guardrails can block execution.
• Execution intercepts wrap the callback and can add timing, retries, routing, or fallback behavior.

Before You Start

Complete Instrument a Tool Call or Instrument an LLM Call. Middleware only runs when the call goes through a NeMo Relay managed lifecycle API.

Choose the Middleware Type

Use this table to match the behavior you need with the correct middleware family.

Use the narrowest middleware type that matches the behavior. For example, do not use a request intercept when you only need to hide a secret from exported events.

Control a Global Registration

Use a conditional middleware guardrail when an operational policy must temporarily exclude existing global middleware or a subscriber. Discover the target from the active runtime, and match its registration kind and effective name. Do not persist or construct the effective name.

The gate affects future middleware and event-publication snapshots. It does not change an in-flight chain, a queued event, the target’s ownership, or a same-named scope-local registration. Every matching gate must allow the target. If a gate callback fails, Relay fails open and keeps the target eligible.

Choose one of the following lifecycle patterns:

• Register a global gate when application code owns the timer, health check, or operational control. Remove the gate in a guaranteed cleanup path.
• Register an activation-owned gate when a plugin component owns the control for its complete active lifetime. Relay removes the gate during rollback or teardown.

Gate callbacks must read already-available state. They must not make reentrant Relay calls that produce middleware, flush subscribers, wait for exporters, or clear plugins. Refer to Conditional Middleware Guardrails for the complete semantic model and Code Examples for binding-specific registration and cleanup.

Add a Tool Policy

This example adds three behaviors around a search tool:

• Redact api_key from emitted request events.
• Reject empty queries before execution.
• Measure execution duration.

1
import time
2
3
import nemo_relay
4
5
def redact_api_key(tool_name, args):
6
    safe_args = dict(args)
7
    if "api_key" in safe_args:
8
        safe_args["api_key"] = "<redacted>"
9
    return safe_args
10
11
def require_query(tool_name, args):
12
    if not args.get("query"):
13
        return "search.query is required"
14
    return None
15
16
async def measure_tool(tool_name, args, next_call):
17
    started = time.perf_counter()
18
    try:
19
        execution_result = await next_call(args)
20
        return nemo_relay.ToolExecutionInterceptOutcome(
21
            execution_result.result,
22
            annotation=execution_result.annotation,
23
        )
24
    finally:
25
        elapsed_ms = round((time.perf_counter() - started) * 1000, 2)
26
        print(f"{tool_name} completed in {elapsed_ms} ms")
27
28
nemo_relay.guardrails.register_tool_sanitize_request("search.redact_api_key", 10, redact_api_key)
29
nemo_relay.guardrails.register_tool_conditional_execution("search.require_query", 20, require_query)
30
nemo_relay.intercepts.register_tool_execution("search.measure", 30, measure_tool)

Tool execution intercepts return an outcome even when they only forward the canonical result from next. A forwarding intercept must preserve both result and annotation; pending marks remain lifecycle-owned. Refer to Tool Execution Intercept Outcomes for annotation, pending-mark, lifecycle, and binding-specific return semantics.

Scope Middleware to One Request

Use scope-local middleware when a policy applies only to one request, tenant, experiment, or agent run.

1. Create or receive the active scope handle.
2. Register middleware with the scope-local helper for that handle.
3. Execute tools or LLM calls inside that scope.
4. Let the scope end remove the scope-local registrations automatically.

Use global middleware for process-wide behavior, such as organization-wide redaction. Use scope-local middleware for request-specific policy, such as tenant routing or an A/B test.

Middleware Registration Families

NeMo Relay exposes the same core middleware families for tools and LLMs:

Sanitize guardrails affect only the payload recorded on emitted events. Tool sanitize-response guardrails receive only ToolExecutionResult.result; a scope-end event sanitizer can separately sanitize the projected tool result annotation. Request intercepts affect the real request that reaches the tool or provider. Execution intercepts wrap the callback itself and are only available when the invocation uses managed execution.

Node.js Callback Failures

Node.js guardrail and request-intercept callbacks can return direct values or Promises. A thrown error or rejected Promise from a conditional-execution guardrail or request intercept rejects the managed call before protected execution or later middleware runs. A thrown error or rejected Promise from a sanitize guardrail or event sanitizer fails closed, withholding the governed emitted payload and recording the error for getLastCallbackError().

Scope-local variants are available through nemo_relay.scope_local.register_*, Node.js scopeRegister* helpers, and Rust scope_register_* functions.

Validate the Middleware

Run one allowed request and one rejected request:

• The allowed request should return the same business result as before.
• The rejected request should fail before the tool callback executes.
• Subscriber output should show redacted api_key values.
• The timing intercept should print once for each executed tool call.

Debug Middleware Order

Middleware runs by ascending priority inside each middleware family. Families and lifecycle emission run in this order for managed tool calls:

1. Conditional-execution guardrails.
2. Request intercepts.
3. Sanitize-request guardrails and start-event emission.
4. Execution intercepts and the real callback or replacement.
5. Sanitize-response guardrails and end-event emission.

If a later middleware does not run, check whether an earlier conditional-execution guardrail rejected the call or a request intercept raised an error.

Common Issues

Check these symptoms first when the workflow does not behave as expected.

• Sanitized data reaches the real tool: Use a sanitize guardrail only for event payloads. Use a request intercept when the real request should change.
• Middleware affects unrelated requests: Register it scope-locally instead of globally.
• Duplicate names replace behavior: Middleware names are registry keys. Use stable, unique names for each behavior.
• Execution intercept never prints: Confirm that the application uses the managed execute helper and that no guardrail rejected the request.

Next Steps

Use these links to continue from this workflow into the next related task.

• Use Middleware to review execution order.
• Use Code Examples for direct registration and partial-execution examples.
• Use Handle Non-Serializable Data if middleware needs to work with framework objects.