> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Migration Guides

> Upgrade NeMo Relay integrations and migrate async middleware, plugins, workers, LLM sanitizers, and PII policy.

Use this page to plan an upgrade from NeMo Relay 0.6 to 0.7. It will group the
actions from the release notes by the surface you operate. If you skip one or
more releases, review the migration guides and release notes for every
intervening release in sequence.

## Upgrade to NeMo Relay 0.7

NeMo Relay 0.7 makes the Rust middleware callback contract asynchronous and
adds awaitable middleware support across the in-process bindings, native
plugins, and worker plugins. It also changes the LLM
observability sanitizer contract. Complete the following migrations before you
run existing middleware or a sanitizer with a 0.7 host.

A 0.6 native plugin can still load through the legacy v2 table fallback, but it
is not compatible with the changed middleware and LLM sanitizer callback
contracts or other changed ABI and schema behavior. Rebuild plugins and workers
for 0.7 before using those surfaces. NeMo Relay does not adapt synchronous Rust
middleware callbacks or one-argument LLM sanitizer callbacks.

Rust code that constructs `AdaptiveConfig` with an exhaustive struct literal
must add `response_cache: None`. Prefer `..AdaptiveConfig::default()` when the
literal should remain compatible with new optional fields.

### Update Exhaustive FlowError Matches

Rust callers that exhaustively match `FlowError` must add the new
`FlowError::CallbackException` variant or use a wildcard arm. The variant
retains the originating Python or JavaScript exception type for observability
while preserving the existing `internal_error` classification.

### Reinstall Coding-Agent Enforcement Hooks

Generated enforcement hooks now fail closed when Relay cannot start,
authenticate, evaluate, or deliver a response. Generated lifecycle and
after-the-fact hooks continue to fail open.

After upgrading, reinstall every persistent coding-agent integration so its
generated hooks use the 0.7 policy:

```bash
nemo-relay install claude-code --force
nemo-relay install codex --force
nemo-relay install hermes --force
```

Custom enforcement hooks must pass `--fail-closed` or set
`NEMO_RELAY_FAIL_CLOSED=1`. Do not apply fail-closed behavior to lifecycle or
after-the-fact hooks that cannot prevent the action they observe.

### Migrate Middleware Callbacks

The following callback families are now asynchronous: conditional execution
guardrails, request intercepts, execution intercepts, tool and LLM sanitizers,
and event sanitizers. Relay awaits each registered callback sequentially in
priority order. A callback that rejects or returns an error preserves the
existing error behavior for its middleware family.

| Surface        | 0.6 Callback           | 0.7 Callback                                                 |
| -------------- | ---------------------- | ------------------------------------------------------------ |
| Rust           | `Fn(...) -> Result<T>` | `Fn(...) -> Pin<Box<dyn Future<Output = Result<T>> + Send>>` |
| Python         | Direct return value    | Direct return value or awaitable                             |
| Node.js        | Direct return value    | Direct return value or `Promise`                             |
| Go / raw C FFI | Synchronous callback   | Synchronous callback                                         |

Conditional-execution guardrails and request or execution intercepts retain a
fail-closed exception contract in 0.7. Relay stops at the failing middleware
stage and surfaces the error to the managed caller. Conditional guardrail and
request-intercept failures prevent the real callback from running. This
behavior is the same for callbacks that return directly and callbacks that
complete asynchronously. If an execution intercept fails after invoking `next`,
Relay cannot undo completed downstream work. Relay rejects `next` calls that
remain unfinished and calls that begin after the interceptor settles.

The experimental Go and raw C FFI callbacks remain synchronous. Relay waits
for each callback on a native thread, so blocking I/O and other long-running
callback work occupy that thread. No completion-based registration API is
provided for these bindings.

Python's standalone middleware helpers preserve their direct synchronous
return when called without a running `asyncio` loop. In that mode, registered
callbacks must also return direct values; an awaitable callback raises a clear
runtime error. Call the helper from async Python and await its result when any
entry may return an awaitable.

For Rust, wrap the existing result in a ready async future, or use an async
block when the callback needs to await work:

```rust
use std::sync::Arc;

use nemo_relay::api::registry::register_tool_conditional_execution_guardrail;

register_tool_conditional_execution_guardrail(
    "policy",
    10,
    Arc::new(|_name, _args| {
        Box::pin(async move {
            // Await policy I/O here when needed.
            Ok(None) // Return Some(reason) to block execution.
        })
    }),
)?;
```

Rust middleware inputs that previously borrowed call data are now owned so
they can live for the duration of the returned future. Update explicitly typed
closures and named callback functions as follows:

* Event sanitizers receive `Arc<Event>` instead of `&Event`.
* Tool sanitizers, conditional guardrails, and request intercepts receive an
  owned `String` tool name; tool conditional guardrails also receive an owned
  `Json` payload instead of `&Json`.
* LLM conditional guardrails receive an owned `LlmRequest` instead of
  `&LlmRequest`, and LLM request intercepts receive an owned `String` call
  name.

Python and Node.js registration names are unchanged. Mark a Python callback
`async def`, or return a Promise from Node.js, only when it needs asynchronous
work; existing direct-value callbacks remain supported.

Do not introduce recursive middleware calls while converting callbacks to
async. Event sanitizers, conditional-execution guardrails, request intercepts,
execution intercepts, and subscribers must not call NeMo Relay APIs that run
middleware, flush subscribers, wait on exporters, or clear plugins. Within
these callbacks, scope operations remain supported at every nesting level,
including replacement of the active stack with an arbitrary scope stack. Event
emission is the only supported operation that can enqueue additional callback
work. An execution intercept may also invoke its supplied `next` continuation.

When queued Python middleware has no live captured event loop—including when
it was registered outside a loop or its registration loop has closed—the
fallback uses `asyncio.run` with a fresh loop. Bare coroutine results are
supported in that fallback; `Task` and `Future` instances bound to a different
loop are not.

Scope, mark, and manual tool/LLM lifecycle APIs remain synchronous. This
includes `push_scope`, `pop_scope`, mark APIs, `tool_call`, `tool_call_end`,
`llm_call`, and `llm_call_end`. These APIs snapshot the event and visible
sanitizer/subscriber chain, then enqueue sanitization and publication on a
serial dispatcher. Event subscribers and exporters therefore receive sanitized
events later, in emission order. Do not add `await` to these lifecycle calls.
Only enqueue-time validation and runtime-state errors are returned directly.
Sanitizer errors discovered during queued publication are logged and fail
closed; codec errors retain their documented fallback behavior.

Python subscriber flushing is context-sensitive. Continue to call
`nemo_relay.subscribers.flush()` from synchronous code. From a running
`asyncio` event loop, replace that call with
`await nemo_relay.subscribers.flush_async()` so queued async middleware can
continue running; the synchronous flush raises in that context.

### Update LLM Sanitizer Callbacks

The registration names remain unchanged for global, plugin-context, and
scope-local sanitizers. Update the callback itself as follows:

| Surface  | 0.6 Contract                 | 0.7 Contract                                                    |
| -------- | ---------------------------- | --------------------------------------------------------------- |
| Request  | `(LlmRequest) -> LlmRequest` | `(LlmRequest, LlmSanitizeRequestContext) -> Option<LlmRequest>` |
| Response | `(Json) -> Json`             | `(Json, LlmSanitizeResponseContext) -> Option<Json>`            |

The payload is always the first argument. The directional context is always the
second argument. Update request sanitizers with the following pattern:

#### Rust

```rust
use std::sync::Arc;

use nemo_relay::api::registry::register_llm_sanitize_request_guardrail;

register_llm_sanitize_request_guardrail(
    "redact-request",
    10,
    Arc::new(|request, context| {
        Box::pin(async move {
            let _active_codec = context.resolve_codec();
            // Apply policy, using _active_codec when normalized access is required.
            Ok(Some(request))
        })
    }),
)?;
```

#### Python

```python
import nemo_relay
from nemo_relay import LLMRequest
from nemo_relay import guardrails

def redact_request(
    request: LLMRequest,
    context: nemo_relay.LlmSanitizeRequestContext,
) -> LLMRequest | None:
    _active_codec = context.resolve_codec()
    # Apply policy, using _active_codec when normalized access is required.
    return request

guardrails.register_llm_sanitize_request(
    "redact-request",
    10,
    redact_request,
)
```

#### Node.js

```js
const relay = require("nemo-relay-node");

relay.registerLlmSanitizeRequestGuardrail(
  "redact-request",
  10,
  (request, context) => {
    const _activeCodec = context.resolveCodec();
    // Apply policy, using _activeCodec when normalized access is required.
    return request;
  },
);
```

Apply the same change to response sanitizers with
`LlmSanitizeResponseContext`. Return the response payload to keep it in the
observability event.

### Handle Payload Omission

In 0.7, `None` or `null` has a specific fail-closed meaning for an LLM
sanitizer: Relay omits the observability payload and its annotation. Omission
short-circuits the remaining LLM sanitizer chain but does not change the
request or response returned to the client.

Check every callback for an implicit empty return. In particular:

* A Python function that reaches the end without `return` omits the payload.
* A JavaScript callback that returns `null` or `undefined` omits the payload.
* A Rust callback must return `Some(payload)` to retain the payload.
* A sanitizer error or panic fails closed: Relay omits the payload and
  annotation, logs or records the callback error, and continues publication.

Use omission only when recording the payload would be unsafe.

### Resolve the Active Codec

The request and response contexts expose `codec`, which has one of the
following identities:

| Identity      | Meaning                                        | `resolve_codec()`         |
| ------------- | ---------------------------------------------- | ------------------------- |
| `none`        | No codec is active for this payload direction. | Returns no codec.         |
| `builtin(id)` | A built-in codec is active.                    | Returns the active codec. |
| `runtime(id)` | A named runtime-registered codec is active.    | Returns the active codec. |
| `opaque`      | A codec without a public ID is active.         | Returns the active codec. |

Treat identity as descriptive and codec resolution as authoritative. Do not
infer a provider from the request shape. A request codec supports
`decode(request)` and `encode(annotated, original)`. A response codec supports
`decode_response(response)` in in-process Rust and Python,
`decode(response).await` in Rust workers, `await decode(response)` in Python
workers, and `decodeResponse(response)` in Node.js; response encoding is not
available.

Worker codec proxies are scoped to the sanitizer callback and must not be
stored for later work. Worker codec operations are asynchronous because they
call the host over `grpc-v1`. In-process resolvers return the active codec
facade directly.

For complete in-process examples, refer to
[Codec-Aware LLM Sanitizers](/about-nemo-relay/concepts/middleware#codec-aware-llm-sanitizers).

### Migrate Worker Sanitizers

All Rust worker middleware registrations now require callbacks that return
futures. This includes conditional guardrails, request and execution intercepts,
mark and scope event sanitizers, tool request/response sanitizers, and LLM
request/response sanitizers. Python worker middleware can return either an
immediate value or an awaitable. Python LLM sanitizers must still accept both
the payload and directional context.

Update Rust worker callbacks to return `Box::pin(async move { ... })` and
resolve to `Result` from the future.

#### Rust Worker

```rust
ctx.register_llm_sanitize_request_guardrail(
    "redact-request",
    10,
    |request, context| {
        Box::pin(async move {
            if let Some(codec) = context.resolve_codec() {
                let annotated = codec.decode(&request).await?;
                let request = codec.encode(&annotated, &request).await?;
                return Ok(Some(request));
            }
            Ok(Some(request))
        })
    },
);
```

#### Python Worker

```python
from nemo_relay_plugin import LlmSanitizeRequestContext

async def redact_request(request, context: LlmSanitizeRequestContext):
    codec = context.resolve_codec()
    if codec is None:
        return request
    annotated = await codec.decode(request)
    return await codec.encode(annotated, request)

ctx.register_llm_sanitize_request_guardrail(
    "redact-request",
    redact_request,
    priority=10,
)
```

Regenerate worker bindings from the 0.7
`nemo.relay.worker.v1` definition and deploy the 0.7 host and worker SDK
together. The protocol identifier remains `grpc-v1`, but the wire contract now
includes:

* `LlmCodecIdentity`
* `LlmSanitizeRequestContext`
* `LlmSanitizeResponseContext`
* Host RPCs for request decode, request encode, and response decode

The previous codec fields on `LlmInvocation` are reserved. Read codec identity
and capability data from the directional sanitizer context instead.

The codec capability ID in the protocol is SDK-internal. Do not expose or
persist it in plugin code. The host rejects forged, expired, unauthorized, and
wrong-direction capability IDs.

### Rebuild Native Plugins and Raw FFI Consumers

NeMo Relay 0.7 uses native ABI v3. Recompile native plugins against the 0.7
`nemo-relay-plugin` crate and rebuild raw FFI consumers against the generated
0.7 header.

The v3 table preserves the v2 prefix, and Relay retries a legacy v2 table when
loading a plugin that rejects v3. That fallback supports loading, not
compatibility with changed middleware, LLM sanitizer, ABI, or schema contracts.
Rebuild native plugins that use the raw plugin ABI callbacks: native ABI v3
adds completion-based async middleware registration, async execution
continuations, and explicit cancellation/late-settlement behavior. This is
separate from the synchronous `nemo-relay-ffi` middleware registration API.

The plugin manifest value remains `compat.native_api = "1"`. This manifest
contract version is separate from the host ABI version; do not change it to
`"2"`.

Request and response callbacks now receive distinct context structures. Each
structure contains structured codec identity and a borrowed directional codec
handle. The request handle supports decode and encode host operations. The
response handle supports decode. A successful null sanitizer output omits the
observability payload and annotation.

Do not retain the codec handle, SDK facade, input pointers, or borrowed codec ID
after the callback returns. Release host-owned output strings with the standard
host string release operation.

For the complete ABI contract, refer to
[Native ABI v3](/build-plugins/dynamic-plugins/native-dynamic/about#native-abi-v3).

### Update PII Redaction Configuration

The PII `codec` field is now an optional compatibility fallback. Relay uses it
only for a legacy or manual managed LLM call with no active codec. Any active
codec takes precedence, including runtime and opaque codecs.

For a mixed-provider gateway, remove the fixed `codec` field:

#### plugins.toml

```toml
[[components]]
kind = "pii_redaction"
enabled = true

[components.config]

[[components.config.profiles]]
mode = "builtin"
priority = 80

[components.config.profiles.builtin]
action = "redact"
detector = "email"
target_paths = ["/messages/0/content", "/message"]
```

Provider-agnostic all-leaf and `trajectory_context` policies do not require
`codec`. A normalized `target_paths` policy fails closed when Relay has neither
a recognized active codec nor a compatible fallback: Relay omits the LLM
payload and annotation instead of recording unsanitized data. Normalized
`target_paths` response policies for runtime and opaque codecs also omit the
payload because response codec capabilities are decode-only.

For the complete policy behavior, refer to
[PII Redaction Configuration](/configure-plugins/pii-redaction/configuration).

### Verify the Upgrade

Before deployment:

1. Search for every LLM sanitize-request and sanitize-response registration.
2. Add the required directional context argument and optional payload result.
3. Check that no callback implicitly returns `None`, `null`, or `undefined`.
4. Convert every Rust worker sanitizer callback to an asynchronous callback.
5. Rebuild native plugins and raw FFI consumers against 0.7.
6. Regenerate and redeploy workers with the matching 0.7 protocol and SDK.
7. Remove fixed PII `codec` values from mixed-provider configurations.
8. Test buffered and streaming calls for every provider and custom codec that
   your deployment uses.

### Observability Configuration Version 3

Observability configuration version 3 replaces the two version-2 OTLP sections
with one typed, multi-endpoint `opentelemetry` section. OpenTelemetry and
OpenInference support is always built in and no longer uses the `otel` or
`openinference` Cargo features.

Version 3 does not normalize version-2 OTLP fields. Migrate the complete
configuration before changing `components.config.version` to `3`.

The following example shows the version-2 shape:

```toml
[components.config]
version = 2

[components.config.opentelemetry]
enabled = true
endpoint = "http://localhost:4318/v1/traces"
transport = "http_binary"
service_name = "relay-service"

[components.config.opentelemetry.headers]
x-tenant-id = "demo"

[components.config.openinference]
enabled = true
endpoint = "http://localhost:6006/v1/traces"
transport = "http_binary"
service_name = "relay-service"
```

Replace it with typed endpoints:

```toml
[components.config]
version = 3

[components.config.opentelemetry]
enabled = true

[[components.config.opentelemetry.endpoints]]
type = "full"
endpoint = "http://localhost:4318/v1/traces"
transport = "http_binary"
service_name = "relay-service"

[components.config.opentelemetry.endpoints.headers]
x-tenant-id = "demo"

[components.config.opentelemetry.endpoints.header_env]
authorization = "OTEL_AUTHORIZATION"

[[components.config.opentelemetry.endpoints]]
type = "openinference"
endpoint = "http://localhost:6006/v1/traces"
transport = "http_binary"
service_name = "relay-service"
```

Set `OTEL_AUTHORIZATION` to the complete authorization header value before
activation. Process-global `OTEL_EXPORTER_OTLP_HEADERS` and
`OTEL_EXPORTER_OTLP_TRACES_HEADERS` are no longer accepted because their values
cannot be isolated between multiple exporters. Move non-sensitive version-2 headers to the
endpoint's `headers` map. Move `resource_attributes` and all service,
transport, instrumentation-scope, and timeout settings into the corresponding
endpoint.

Use the following endpoint type for each projection:

| Version-2 Source                   | Version-3 Endpoint Type |
| ---------------------------------- | ----------------------- |
| `opentelemetry`                    | `full`                  |
| `openinference`                    | `openinference`         |
| New standardized GenAI-only export | `gen_ai`                |

Every endpoint requires a nonblank `endpoint` and explicit `type`. The defaults
also change:

| Field                   | Version 2                                    | Version 3         |
| ----------------------- | -------------------------------------------- | ----------------- |
| `service_name`          | `nemo-relay`                                 | `unknown_service` |
| `instrumentation_scope` | Exporter-specific or omitted                 | `opentelemetry`   |
| `endpoint`              | Exporter or environment default when omitted | Required          |

Remove `mark_projection`, `mark_exclude_names`, `attribute_mappings`,
`semantic_selector`, and `capture_content`. Projection behavior is fixed. The
`full` and `openinference` types retain their previous default mark behavior.
The `gen_ai` type omits marks and emits minimal internal spans for Relay scope
types without GenAI semantics, preserving the original span parentage. It does
not emit `nemo_relay.*` attributes. It targets the
[OpenTelemetry GenAI semantic-conventions v1.42-era snapshot](https://github.com/open-telemetry/semantic-conventions-genai/tree/43633a68ef8f8ed87a1d5eb205990311ca708bf1).

NeMo Relay also upgrades to OpenTelemetry Rust `0.32`.

Top-level component `config` lists now concatenate across layers, with
higher-precedence entries first. The observability destination lists
`atof.sinks`, `opentelemetry.endpoints`, and `atif.storage` follow the same
rule. Arbitrary lists nested inside structured values continue to use
replacement semantics. Edit the layer that declares an inherited entry when
you need to change or remove it.

#### Binding API Changes

The standalone OpenInference config and subscriber APIs are removed. Construct
one independently managed OpenTelemetry subscriber from one typed endpoint.

| Surface | Version-2 API                                                                                                                | Version-3 API                                                                                                             |
| ------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Rust    | `OpenInferenceConfig`, `OpenInferenceSubscriber`, or untyped `OpenTelemetryConfig` constructors                              | `OpenTelemetryConfig::new(OpenTelemetryType::OpenInference, endpoint)` and `OpenTelemetrySubscriber`                      |
| Python  | `OpenInferenceConfig`, `OpenInferenceSubscriber`, or `OpenTelemetryConfig()`                                                 | `OpenTelemetryConfig("openinference", endpoint)` and `OpenTelemetrySubscriber`                                            |
| Node.js | `OpenInferenceSubscriber` or an OpenTelemetry config without a type                                                          | `new OpenTelemetrySubscriber({ type: "openinference", endpoint })`                                                        |
| Go      | `NewOpenInferenceConfig`, `NewOpenInferenceSubscriber`, or `NewOpenTelemetryConfig()`                                        | `NewOpenTelemetryConfig(OpenTelemetryTypeOpenInference, endpoint)` and `NewOpenTelemetrySubscriber`                       |
| C FFI   | `nemo_relay_openinference_subscriber_*`, the untyped `nemo_relay_otel_subscriber_create`, and attribute-mapping constructors | `nemo_relay_otel_subscriber_create(otel_type, ..., endpoint, ...)` and `nemo_relay_otel_subscriber_*` lifecycle functions |

For plugin-owned multi-endpoint configuration, use
`OpenTelemetryEndpointConfig` and `OpenTelemetrySectionConfig` in Python,
`openTelemetryEndpoint` and `openTelemetryConfig` in Node.js, or
`ObservabilityOpenTelemetryEndpointConfig` and
`ObservabilityOpenTelemetryConfig` in Go.

Direct subscribers still own one tracer provider each. Register the subscriber
before instrumented work, then deregister, flush, and shut it down during
graceful teardown.

## Related Release Information

For release highlights, compatibility updates, and current known issues, refer
to the [Release Notes](/about-nemo-relay/release-notes). Use [GitHub Releases](https://github.com/NVIDIA/NeMo-Relay/releases) for the complete
release history and notes for a specific tag.