> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Middleware

This page explains the runtime behavior that runs around managed tool and LLM
calls and sanitizes emitted mark and scope events.

## What Middleware Is

Middleware controls or transforms tool and LLM execution and sanitizes emitted
events. NeMo Relay applies each surface at a specific lifecycle point.

Middleware is organized by lifecycle meaning rather than as one undifferentiated
hook system.

## Registration Levels

Middleware and subscribers can be registered at different levels depending on their
lifetime and visibility.

### Global Registrations

Global registrations stay active for the whole process until they are removed.
Use them for defaults that should apply broadly.

### Scope-Local Registrations

Scope-local registrations are owned by one active scope and disappear
automatically when that scope closes.

Use them when behavior should stay local to one request, workflow, or nested
unit of work.

### Plugin-Installed Registrations

Plugins can install middleware during initialization. This is the reusable,
configuration-driven path for shipping middleware bundles without hand-registering
everything in application code.

## Middleware Families

NeMo Relay has two major middleware families:

* **Intercepts** change the real execution path
* **Guardrails** block work or rewrite emitted observability payloads

## Intercepts

Intercepts are middleware that change the real request or execution path.

### Request Intercepts

Request intercepts rewrite the real request before execution continues.

Use them when the next stage of execution should receive changed input, such as:

* Header injection
* Request normalization
* Argument enrichment
* Provider-specific request rewriting

### Execution Intercepts

Execution intercepts wrap or replace the real callback.

Use them when behavior belongs around the invocation boundary itself, such as:

* Retries
* Timing
* Routing
* Wrapper logic
* Framework integration

### Stream Execution Intercepts

LLM streaming has a stream execution path for wrappers that need to run around
chunk delivery and finalization rather than only around a single response
object.

## Guardrails

Guardrails are middleware that block execution or sanitize observability payloads.

### Conditional Execution

Conditional-execution guardrails run before the real callback. They decide
whether execution may proceed.

Use them when the runtime should block work based on policy, budget, or context.

### Sanitize Request

Sanitize-request guardrails rewrite the payload recorded on emitted start events.

Use them when the event stream should hide or reduce sensitive request data.

### Sanitize Response

Sanitize-response guardrails rewrite the payload recorded on emitted end events.

Use them when the event stream should hide or reduce sensitive response data.

### Sanitize Mark and Scope Events

Event sanitizers cover observability fields that are outside the specialized
tool and LLM payload APIs. Separate registries apply to marks, scope starts,
and scope ends. They can rewrite `data`, `category_profile`, and `metadata`
while receiving the complete event as immutable context.

Scope event sanitizers run for every category. On tool and LLM scope events,
they run after the specialized request or response sanitizer. Mark sanitizers
cover explicit marks and marks materialized by middleware, plugins, and
streaming lifecycle helpers.

Register event sanitizers globally, on an owning scope, or through a plugin
context. For the callback contract and binding APIs, refer to
[Event Sanitizers](/reference/event-sanitizers).

Sanitize guardrails are observability-oriented. They do not rewrite the real
arguments passed to the callback or the real value returned to the caller.

## Managed Execution Order

For managed execution, NeMo Relay applies middleware and emits lifecycle events
in this order:

```mermaid
sequenceDiagram
    autonumber
    actor Caller as Application / Framework
    participant Runtime as NeMo Relay Runtime
    participant Cond as Conditional Guardrails
    participant Req as Request Intercepts
    participant Exec as Execution Intercepts
    participant Callback as Real Callback
    participant San as Tool / LLM Sanitizers
    participant EventSan as Mark / Scope Event Sanitizers
    participant Dispatch as Async Subscriber Dispatcher
    participant Consumers as Subscribers / Exporters

    Caller->>Runtime: managed tool or LLM call
    Runtime->>Cond: decide whether work may proceed

    alt blocked
        Runtime->>EventSan: sanitize guardrail scopes and rejection mark
        Cond-->>Caller: reject execution
    else allowed
        Runtime->>Req: rewrite the real request
        Runtime->>San: sanitize emitted start payload
        Runtime->>EventSan: sanitize start event fields
        Runtime->>Dispatch: enqueue start event before execution
        Dispatch-->>Consumers: deliver start event later
        Runtime->>Exec: wrap execution
        Exec->>Callback: invoke callback
        Callback-->>Exec: return real result
        Exec-->>Runtime: continue
        Runtime->>San: sanitize emitted end payload
        Runtime->>EventSan: sanitize end event fields
        Runtime->>Dispatch: enqueue end event
        Dispatch-->>Consumers: deliver end event later
        Runtime-->>Caller: return real result
    end
```

1. Conditional-execution guardrails
2. Request intercepts
3. Tool or LLM sanitize-request guardrails
4. Scope-start event sanitizers and start-event emission
5. Execution intercepts
6. The real callback, unless an execution intercept replaces it
7. Tool or LLM sanitize-response guardrails
8. Scope-end event sanitizers and end-event emission

For streaming LLM flows, the same pre-execution order applies: the runtime
applies `sanitize-request` guardrails and emits the LLM start event before the
stream execution intercept chain runs. Stream execution intercepts are the
execution family for streaming provider callbacks. The runtime then collects
chunks and finalizes the stream before `sanitize-response` guardrails rewrite
the emitted end-event payload and scope-end event sanitizers run at items 7 and
8\.

This ordering is what makes the semantic split between intercepts and
guardrails important:

* If you need to change the real execution path, use an intercept
* If you need to change only the emitted payload, use a sanitize guardrail

## Detailed Execution Flow

The simplified sequence above is the right mental model for most readers. The
diagram below expands the same flow to show where guardrail rejections, event
subscribers, execution-intercept chaining, and streaming collection/finalization
fit into the runtime path.

```mermaid
flowchart TB
    Request([Request])

    subgraph Execution
        direction TB
        ConditionalExecutionGuardrails{{Conditional-Execution Guardrail}}
        RequestIntercepts[/Request Intercepts/]
        RaiseException[Raise Exception]
        subgraph Invocation
            direction TB
            HasExecutionIntercept{{Has Valid Execution Intercept}}
            ExecutionIntercepts[/Execution Intercepts/]
            DefaultCallable[Default Callable]
            InterceptResult[Execution Result]
        end

        subgraph Streaming
            direction TB
            Finalizer[Finalizer]
            Collector[Collector]
        end

        subgraph Observability
            direction TB
            SanitizeRequestGuardrails[/Sanitize Request Guardrail/]
            SanitizeResponseGuardrails[/Sanitize Response Guardrail/]
            MarkSanitizers[/Mark Event Sanitizers/]
            ScopeStartSanitizers[/Scope-Start Event Sanitizers/]
            ScopeEndSanitizers[/Scope-End Event Sanitizers/]
            StartEvent[Emit Start Event]
            EndEvent[Emit End Event]
            Dispatcher[["Async Subscriber Dispatcher"]]
            EventConsumers[["Subscribers / Exporters"]]
        end
    end

    Response([Response])

    Request --> ConditionalExecutionGuardrails
    RequestIntercepts -->|Transformed Request| SanitizeRequestGuardrails
    ConditionalExecutionGuardrails -->|"(rejection mark)"| MarkSanitizers
    MarkSanitizers -->|Sanitized Mark Fields| Dispatcher
    ConditionalExecutionGuardrails -->|"(rejected)"| RaiseException
    ConditionalExecutionGuardrails -->|"(passed)"| RequestIntercepts
    SanitizeRequestGuardrails -->|Sanitized Start Payload| ScopeStartSanitizers
    ScopeStartSanitizers -->|Sanitized Event Fields| StartEvent
    StartEvent --> Dispatcher
    Dispatcher --> EventConsumers
    StartEvent -->|Before Execution Intercepts| HasExecutionIntercept
    RequestIntercepts -.->|Real Request| HasExecutionIntercept

    HasExecutionIntercept -->|No| DefaultCallable
    HasExecutionIntercept -->|Yes| ExecutionIntercepts
    ExecutionIntercepts -.->|calls next| HasExecutionIntercept
    ExecutionIntercepts -->|returns or replaces| InterceptResult
    DefaultCallable -->|returns| InterceptResult

    InterceptResult -->|Response| SanitizeResponseGuardrails
    InterceptResult -->|Response| Response

    InterceptResult -.->|stream chunks| Collector
    Collector -..->|stream chunks| Response
    InterceptResult -.->|"(stream ends)"| Finalizer
    Finalizer -.->|Aggregated Response| SanitizeResponseGuardrails
    Finalizer o--o|shared state| Collector

    SanitizeResponseGuardrails -->|Sanitized End Payload| ScopeEndSanitizers
    ScopeEndSanitizers -->|Sanitized Event Fields| EndEvent
    EndEvent --> Dispatcher

    class Execution,Invocation,Streaming,Observability,Request,Response grey-lightest;
    class Dispatcher,EventConsumers,StartEvent,EndEvent teal-lightest;
    class RequestIntercepts,HasExecutionIntercept,ExecutionIntercepts yellow-lightest;
    class ConditionalExecutionGuardrails,SanitizeRequestGuardrails,SanitizeResponseGuardrails,MarkSanitizers,ScopeStartSanitizers,ScopeEndSanitizers green-lightest;
    class RaiseException red-lightest;
    class DefaultCallable,InterceptResult,Collector,Finalizer magenta-lightest;
```

## Choosing the Right Surface

Use these comparisons to pick the middleware surface that matches the behavior you need.

* Use a **conditional-execution guardrail** when the work should be allowed or
  rejected.
* Use a **request intercept** when the real request must change before the call.
* Use an **execution intercept** when behavior belongs around the invocation
  boundary.
* Use a **sanitize guardrail** when only subscribers and exporters should see
  rewritten data.
* Use a **mark or scope event sanitizer** when the sensitive fields are in
  `data`, `category_profile`, or `metadata` rather than the managed tool or LLM
  request/response payload.
* Use a **stream execution intercept** when you need streaming-specific
  behavior applied across the lifecycle of a long-lived or chunked response,
  such as per-chunk transformation, incremental authorization, logging or
  metrics per event, backpressure handling, or cancellation and cleanup,
  rather than an execution intercept that only surrounds a single call
  boundary.

## Practical Guidance

Use these practices when applying the concept in application or integration code.

* Keep process-wide defaults global.
* Keep request-local policy scope-local.
* Use plugins when the middleware bundle should be reusable and
  configuration-driven.
* Treat execution intercepts as the preferred wrapper point for framework
  integrations.