> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Add Middleware

Use this guide when instrumentation is working and you want NeMo Relay to enforce policy, transform requests, wrap execution, or sanitize observability payloads around tool and LLM calls.

## What You Build

You will add middleware to an instrumented application and verify that it runs in the expected part of the pipeline:

* Request intercepts transform the real request before execution.
* Sanitize guardrails transform only the payload recorded on events.
* Conditional-execution guardrails can block execution.
* Execution intercepts wrap the callback and can add timing, retries, routing, or fallback behavior.

## Before You Start

Complete [Instrument a Tool Call](/instrument-applications/instrument-tool-call) or [Instrument an LLM Call](/instrument-applications/instrument-llm-call). Middleware only runs when the call goes through a NeMo Relay managed lifecycle API.

## Choose the Middleware Type

Use this table to match the behavior you need with the correct middleware family.

| Need                                       | Middleware Type                                 | Changes Real Execution |
| ------------------------------------------ | ----------------------------------------------- | ---------------------- |
| Redact event payloads                      | Sanitize-request or sanitize-response guardrail | No                     |
| Normalize tool arguments or model requests | Request intercept                               | Yes                    |
| Block unsafe or invalid work               | Conditional-execution guardrail                 | Yes, by rejecting      |
| Add timing, retries, routing, or fallback  | Execution intercept                             | Yes                    |
| Wrap streaming model output                | LLM stream execution intercept                  | Yes                    |

Use the narrowest middleware type that matches the behavior. For example, do not use a request intercept when you only need to hide a secret from exported events.

## Control a Global Registration

Use a conditional middleware guardrail when an operational policy must temporarily
exclude existing global middleware or a subscriber. Discover the target from the active
runtime, and match its registration kind and effective name. Do not persist or construct
the effective name.

The gate affects future middleware and event-publication snapshots. It does not change
an in-flight chain, a queued event, the target's ownership, or a same-named scope-local
registration. Every matching gate must allow the target. If a gate callback fails, Relay
fails open and keeps the target eligible.

Choose one of the following lifecycle patterns:

* Register a global gate when application code owns the timer, health check, or
  operational control. Remove the gate in a guaranteed cleanup path.
* Register an activation-owned gate when a plugin component owns the control for its
  complete active lifetime. Relay removes the gate during rollback or teardown.

Gate callbacks must read already-available state. They must not make reentrant Relay
calls that produce middleware, flush subscribers, wait for exporters, or clear plugins.
Refer to
[Conditional Middleware Guardrails](/about-nemo-relay/concepts/conditional-middleware-guardrails)
for the complete semantic model and [Code Examples](/instrument-applications/code-examples)
for binding-specific registration and cleanup.

## Add a Tool Policy

This example adds three behaviors around a `search` tool:

* Redact `api_key` from emitted request events.
* Reject empty queries before execution.
* Measure execution duration.

#### Python

```python
import time
from collections.abc import Awaitable, Callable

import nemo_relay

def redact_api_key(_tool_name: str, args: nemo_relay.Json) -> nemo_relay.Json:
    safe_args = dict(args)
    if "api_key" in safe_args:
        safe_args["api_key"] = "<redacted>"
    return safe_args

def require_query(_tool_name: str, args: nemo_relay.Json) -> str | None:
    if not args.get("query"):
        return "search.query is required"
    return None

async def measure_tool(
    context: nemo_relay.ToolExecutionContext,
    next_call: Callable[[nemo_relay.Json], Awaitable[nemo_relay.ToolExecutionResult[nemo_relay.Json]]],
) -> nemo_relay.ToolExecutionInterceptOutcome:
    started = time.perf_counter()
    try:
        execution_result = await next_call(context.args)
        return nemo_relay.ToolExecutionInterceptOutcome(
            execution_result.result,
            annotation=execution_result.annotation,
        )
    finally:
        elapsed_ms = round((time.perf_counter() - started) * 1000, 2)
        print(f"{context.tool_name} completed in {elapsed_ms} ms")

nemo_relay.guardrails.register_tool_sanitize_request("search.redact_api_key", 10, redact_api_key)
nemo_relay.guardrails.register_tool_conditional_execution("search.require_query", 20, require_query)
nemo_relay.intercepts.register_tool_execution("search.measure", 30, measure_tool)
```

#### Node.js

```js
const {
  registerToolConditionalExecutionGuardrail,
  registerToolExecutionIntercept,
  registerToolSanitizeRequestGuardrail,
} = require("nemo-relay-node");

registerToolSanitizeRequestGuardrail("search.redact_api_key", 10, (_toolName, args) => {
  if (!args.api_key) {
    return args;
  }
  return { ...args, api_key: "<redacted>" };
});

registerToolConditionalExecutionGuardrail("search.require_query", 20, (_toolName, args) => (
  args.query ? null : "search.query is required"
));

registerToolExecutionIntercept("search.measure", 30, async (context, next) => {
  const started = performance.now();
  try {
    const executionResult = await next(context.args);
    return {
      result: executionResult.result,
      annotation: executionResult.annotation,
    };
  } finally {
    console.log(`search completed in ${Math.round(performance.now() - started)} ms`);
  }
});
```

#### Rust

```rust
use nemo_relay::api::registry::{
    register_tool_conditional_execution_guardrail,
    register_tool_execution_intercept,
    register_tool_sanitize_request_guardrail,
};
use nemo_relay::api::tool::ToolExecutionInterceptOutcome;
use serde_json::json;
use std::sync::Arc;
use std::time::Instant;

register_tool_sanitize_request_guardrail(
    "search.redact_api_key",
    10,
    Arc::new(|_tool_name, mut args| {
        Box::pin(async move {
            if let Some(object) = args.as_object_mut() {
                if object.contains_key("api_key") {
                    object.insert("api_key".into(), json!("<redacted>"));
                }
            }
            Ok(args)
        })
    }),
)?;

register_tool_conditional_execution_guardrail(
    "search.require_query",
    20,
    Arc::new(|_tool_name, args| {
        Box::pin(async move {
            Ok(match args.get("query").and_then(|value| value.as_str()) {
                Some(query) if !query.is_empty() => None,
                _ => Some("search.query is required".into()),
            })
        })
    }),
)?;

register_tool_execution_intercept(
    "search.measure",
    30,
    Arc::new(|context, next| {
        let name = context.tool_name().to_owned();
        Box::pin(async move {
            let started = Instant::now();
            let result = next(context.into_args()).await;
            println!("{name} completed in {:?}", started.elapsed());
            Ok(ToolExecutionInterceptOutcome::from(result?))
        })
    }),
)?;
```

Tool execution intercepts return an outcome even when they only forward the
canonical result from `next`. A forwarding intercept must preserve both
`result` and `annotation`; pending marks remain lifecycle-owned. Refer to
[Tool Execution Intercept Outcomes](/reference/tool-execution-intercept-outcomes)
for annotation, pending-mark, lifecycle, and binding-specific return semantics.

## Scope Middleware to One Request

Use scope-local middleware when a policy applies only to one request, tenant, experiment, or agent run.

1. Create or receive the active scope handle.
2. Register middleware with the scope-local helper for that handle.
3. Execute tools or LLM calls inside that scope.
4. Let the scope end remove the scope-local registrations automatically.

Use global middleware for process-wide behavior, such as organization-wide redaction. Use scope-local middleware for request-specific policy, such as tenant routing or an A/B test.

## Middleware Registration Families

NeMo Relay exposes the same core middleware families for tools and LLMs:

| Family                     | Tool Registration                     | LLM Registration                     | Changes Real Execution |
| -------------------------- | ------------------------------------- | ------------------------------------ | ---------------------- |
| Sanitize request           | `register_tool_sanitize_request`      | `register_llm_sanitize_request`      | No                     |
| Sanitize response          | `register_tool_sanitize_response`     | `register_llm_sanitize_response`     | No                     |
| Conditional execution      | `register_tool_conditional_execution` | `register_llm_conditional_execution` | Yes, by rejecting      |
| Request intercept          | `register_tool_request`               | `register_llm_request`               | Yes                    |
| Execution intercept        | `register_tool_execution`             | `register_llm_execution`             | Yes                    |
| Stream execution intercept | Not applicable                        | `register_llm_stream_execution`      | Yes                    |

Sanitize guardrails affect only the payload recorded on emitted events. Tool
sanitize-response guardrails receive only `ToolExecutionResult.result`; a
scope-end event sanitizer can separately sanitize the projected tool result
annotation. Request intercepts affect the real request that reaches the tool or
provider. Execution intercepts wrap the callback itself and are only available
when the invocation uses managed execution.

A tool execution intercept receives a `ToolExecutionContext` carrying the tool
name, the arguments, and the managed `tool_call_id`. Use it when an intercept
short-circuits execution and must still correlate its result with the
originating tool call. See
[Tool Execution Intercept Outcomes](/reference/tool-execution-intercept-outcomes)
for the per-binding contract.

### Node.js Callback Failures

Node.js guardrail and request-intercept callbacks can return direct values or
Promises. A thrown error or rejected Promise from a conditional-execution
guardrail or request intercept rejects the managed call before protected
execution or later middleware runs. A thrown error or rejected Promise from a
sanitize guardrail or event sanitizer fails closed, withholding the governed
emitted payload and recording the error for `getLastCallbackError()`.

Scope-local variants are available through `nemo_relay.scope_local.register_*`, Node.js `scopeRegister*` helpers, and Rust `scope_register_*` functions.

## Validate the Middleware

Run one allowed request and one rejected request:

* The allowed request should return the same business result as before.
* The rejected request should fail before the tool callback executes.
* Subscriber output should show redacted `api_key` values.
* The timing intercept should print once for each executed tool call.

## Debug Middleware Order

Middleware runs by ascending priority inside each middleware family. Families
and lifecycle emission run in this order for managed tool calls:

1. Conditional-execution guardrails.
2. Request intercepts.
3. Sanitize-request guardrails and start-event emission.
4. Execution intercepts and the real callback or replacement.
5. Sanitize-response guardrails and end-event emission.

If a later middleware does not run, check whether an earlier conditional-execution guardrail rejected the call or a request intercept raised an error.

## Common Issues

Check these symptoms first when the workflow does not behave as expected.

* **Sanitized data reaches the real tool**: Use a sanitize guardrail only for event payloads. Use a request intercept when the real request should change.
* **Middleware affects unrelated requests**: Register it scope-locally instead of globally.
* **Duplicate names replace behavior**: Middleware names are registry keys. Use stable, unique names for each behavior.
* **Execution intercept never prints**: Confirm that the application uses the managed execute helper and that no guardrail rejected the request.

## Next Steps

Use these links to continue from this workflow into the next related task.

* Use [Middleware](/about-nemo-relay/concepts/middleware) to review execution order.
* Use [Code Examples](/instrument-applications/code-examples) for direct registration and partial-execution examples.
* Use [Handle Non-Serializable Data](/integrate-into-frameworks/non-serializable-data) if middleware needs to work with framework objects.