Migration Guides

Use this page to plan an upgrade from NeMo Relay 0.6 to 0.7. It will group the actions from the release notes by the surface you operate. If you skip one or more releases, review the migration guides and release notes for every intervening release in sequence.

Upgrade to NeMo Relay 0.7

NeMo Relay 0.7 makes the Rust middleware callback contract asynchronous and adds awaitable middleware support across the in-process bindings, native plugins, and worker plugins. It also changes the LLM observability sanitizer contract. Complete the following migrations before you run existing middleware or a sanitizer with a 0.7 host.

Rust code that constructs AdaptiveConfig with an exhaustive struct literal must add response_cache: None. Prefer ..AdaptiveConfig::default() when the literal should remain compatible with new optional fields.

Update Exhaustive FlowError Matches

Rust callers that exhaustively match FlowError must add the new FlowError::CallbackException variant or use a wildcard arm. The variant retains the originating Python or JavaScript exception type for observability while preserving the existing internal_error classification.

Reinstall Coding-Agent Enforcement Hooks

Generated enforcement hooks now fail closed when Relay cannot start, authenticate, evaluate, or deliver a response. Generated lifecycle and after-the-fact hooks continue to fail open.

After upgrading, reinstall every persistent coding-agent integration so its generated hooks use the 0.7 policy:

nemo-relay install claude-code --force
nemo-relay install codex --force
nemo-relay install hermes --force

Custom enforcement hooks must pass --fail-closed or set NEMO_RELAY_FAIL_CLOSED=1. Do not apply fail-closed behavior to lifecycle or after-the-fact hooks that cannot prevent the action they observe.

Migrate Middleware Callbacks

The following callback families are now asynchronous: conditional execution guardrails, request intercepts, execution intercepts, tool and LLM sanitizers, and event sanitizers. Relay awaits each registered callback sequentially in priority order. A callback that rejects or returns an error preserves the existing error behavior for its middleware family.

Conditional-execution guardrails and request or execution intercepts retain a fail-closed exception contract in 0.7. Relay stops at the failing middleware stage and surfaces the error to the managed caller. Conditional guardrail and request-intercept failures prevent the real callback from running. This behavior is the same for callbacks that return directly and callbacks that complete asynchronously. If an execution intercept fails after invoking next, Relay cannot undo completed downstream work. Relay rejects next calls that remain unfinished and calls that begin after the interceptor settles.

The experimental Go and raw C FFI callbacks remain synchronous. Relay waits for each callback on a native thread, so blocking I/O and other long-running callback work occupy that thread. No completion-based registration API is provided for these bindings.

Python’s standalone middleware helpers preserve their direct synchronous return when called without a running asyncio loop. In that mode, registered callbacks must also return direct values; an awaitable callback raises a clear runtime error. Call the helper from async Python and await its result when any entry may return an awaitable.

For Rust, wrap the existing result in a ready async future, or use an async block when the callback needs to await work:

use std::sync::Arc;
use nemo_relay::api::registry::register_tool_conditional_execution_guardrail;
register_tool_conditional_execution_guardrail(
    "policy",
    10,
    Arc::new(|_name, _args| {
        Box::pin(async move {
            // Await policy I/O here when needed.
            Ok(None) // Return Some(reason) to block execution.
        })
    }),
)?;

Rust middleware inputs that previously borrowed call data are now owned so they can live for the duration of the returned future. Update explicitly typed closures and named callback functions as follows:

• Event sanitizers receive Arc<Event> instead of &Event.
• Tool sanitizers, conditional guardrails, and request intercepts receive an owned String tool name; tool conditional guardrails also receive an owned Json payload instead of &Json.
• LLM conditional guardrails receive an owned LlmRequest instead of &LlmRequest, and LLM request intercepts receive an owned String call name.

Python and Node.js registration names are unchanged. Mark a Python callback async def, or return a Promise from Node.js, only when it needs asynchronous work; existing direct-value callbacks remain supported.

When queued Python middleware has no live captured event loop—including when it was registered outside a loop or its registration loop has closed—the fallback uses asyncio.run with a fresh loop. Bare coroutine results are supported in that fallback; Task and Future instances bound to a different loop are not.

Scope, mark, and manual tool/LLM lifecycle APIs remain synchronous. This includes push_scope, pop_scope, mark APIs, tool_call, tool_call_end, llm_call, and llm_call_end. These APIs snapshot the event and visible sanitizer/subscriber chain, then enqueue sanitization and publication on a serial dispatcher. Event subscribers and exporters therefore receive sanitized events later, in emission order. Do not add await to these lifecycle calls. Only enqueue-time validation and runtime-state errors are returned directly. Sanitizer errors discovered during queued publication are logged and fail closed; codec errors retain their documented fallback behavior.

Python subscriber flushing is context-sensitive. Continue to call nemo_relay.subscribers.flush() from synchronous code. From a running asyncio event loop, replace that call with await nemo_relay.subscribers.flush_async() so queued async middleware can continue running; the synchronous flush raises in that context.

Update LLM Sanitizer Callbacks

The registration names remain unchanged for global, plugin-context, and scope-local sanitizers. Update the callback itself as follows:

The payload is always the first argument. The directional context is always the second argument. Update request sanitizers with the following pattern:

use std::sync::Arc;
use nemo_relay::api::registry::register_llm_sanitize_request_guardrail;
register_llm_sanitize_request_guardrail(
    "redact-request",
    10,
    Arc::new(|request, context| {
        Box::pin(async move {
            let _active_codec = context.resolve_codec();
            // Apply policy, using _active_codec when normalized access is required.
            Ok(Some(request))
        })
    }),
)?;

Apply the same change to response sanitizers with LlmSanitizeResponseContext. Return the response payload to keep it in the observability event.

Handle Payload Omission

In 0.7, None or null has a specific fail-closed meaning for an LLM sanitizer: Relay omits the observability payload and its annotation. Omission short-circuits the remaining LLM sanitizer chain but does not change the request or response returned to the client.

Check every callback for an implicit empty return. In particular:

• A Python function that reaches the end without return omits the payload.
• A JavaScript callback that returns null or undefined omits the payload.
• A Rust callback must return Some(payload) to retain the payload.
• A sanitizer error or panic fails closed: Relay omits the payload and annotation, logs or records the callback error, and continues publication.

Use omission only when recording the payload would be unsafe.

Resolve the Active Codec

The request and response contexts expose codec, which has one of the following identities:

Treat identity as descriptive and codec resolution as authoritative. Do not infer a provider from the request shape. A request codec supports decode(request) and encode(annotated, original). A response codec supports decode_response(response) in in-process Rust and Python, decode(response).await in Rust workers, await decode(response) in Python workers, and decodeResponse(response) in Node.js; response encoding is not available.

Worker codec proxies are scoped to the sanitizer callback and must not be stored for later work. Worker codec operations are asynchronous because they call the host over grpc-v1. In-process resolvers return the active codec facade directly.

For complete in-process examples, refer to Codec-Aware LLM Sanitizers.

Migrate Worker Sanitizers

All Rust worker middleware registrations now require callbacks that return futures. This includes conditional guardrails, request and execution intercepts, mark and scope event sanitizers, tool request/response sanitizers, and LLM request/response sanitizers. Python worker middleware can return either an immediate value or an awaitable. Python LLM sanitizers must still accept both the payload and directional context.

Update Rust worker callbacks to return Box::pin(async move { ... }) and resolve to Result from the future.

ctx.register_llm_sanitize_request_guardrail(
    "redact-request",
    10,
    |request, context| {
        Box::pin(async move {
            if let Some(codec) = context.resolve_codec() {
                let annotated = codec.decode(&request).await?;
                let request = codec.encode(&annotated, &request).await?;
                return Ok(Some(request));
            }
            Ok(Some(request))
        })
    },
);

Regenerate worker bindings from the 0.7 nemo.relay.worker.v1 definition and deploy the 0.7 host and worker SDK together. The protocol identifier remains grpc-v1, but the wire contract now includes:

• LlmCodecIdentity
• LlmSanitizeRequestContext
• LlmSanitizeResponseContext
• Host RPCs for request decode, request encode, and response decode

The previous codec fields on LlmInvocation are reserved. Read codec identity and capability data from the directional sanitizer context instead.

The codec capability ID in the protocol is SDK-internal. Do not expose or persist it in plugin code. The host rejects forged, expired, unauthorized, and wrong-direction capability IDs.

Rebuild Native Plugins and Raw FFI Consumers

NeMo Relay 0.7 uses native ABI v3. Recompile native plugins against the 0.7 nemo-relay-plugin crate and rebuild raw FFI consumers against the generated 0.7 header.

The v3 table preserves the v2 prefix, and Relay retries a legacy v2 table when loading a plugin that rejects v3. That fallback supports loading, not compatibility with changed middleware, LLM sanitizer, ABI, or schema contracts. Rebuild native plugins that use the raw plugin ABI callbacks: native ABI v3 adds completion-based async middleware registration, async execution continuations, and explicit cancellation/late-settlement behavior. This is separate from the synchronous nemo-relay-ffi middleware registration API.

The plugin manifest value remains compat.native_api = "1". This manifest contract version is separate from the host ABI version; do not change it to "2".

Request and response callbacks now receive distinct context structures. Each structure contains structured codec identity and a borrowed directional codec handle. The request handle supports decode and encode host operations. The response handle supports decode. A successful null sanitizer output omits the observability payload and annotation.

Do not retain the codec handle, SDK facade, input pointers, or borrowed codec ID after the callback returns. Release host-owned output strings with the standard host string release operation.

For the complete ABI contract, refer to Native ABI v3.

Update PII Redaction Configuration

The PII codec field is now an optional compatibility fallback. Relay uses it only for a legacy or manual managed LLM call with no active codec. Any active codec takes precedence, including runtime and opaque codecs.

For a mixed-provider gateway, remove the fixed codec field:

[[components]]
kind = "pii_redaction"
enabled = true
[components.config]
[[components.config.profiles]]
mode = "builtin"
priority = 80
[components.config.profiles.builtin]
action = "redact"
detector = "email"
target_paths = ["/messages/0/content", "/message"]

Provider-agnostic all-leaf and trajectory_context policies do not require codec. A normalized target_paths policy fails closed when Relay has neither a recognized active codec nor a compatible fallback: Relay omits the LLM payload and annotation instead of recording unsanitized data. Normalized target_paths response policies for runtime and opaque codecs also omit the payload because response codec capabilities are decode-only.

For the complete policy behavior, refer to PII Redaction Configuration.

Verify the Upgrade

Before deployment:

1. Search for every LLM sanitize-request and sanitize-response registration.
2. Add the required directional context argument and optional payload result.
3. Check that no callback implicitly returns None, null, or undefined.
4. Convert every Rust worker sanitizer callback to an asynchronous callback.
5. Rebuild native plugins and raw FFI consumers against 0.7.
6. Regenerate and redeploy workers with the matching 0.7 protocol and SDK.
7. Remove fixed PII codec values from mixed-provider configurations.
8. Test buffered and streaming calls for every provider and custom codec that your deployment uses.

Observability Configuration Version 3

Observability configuration version 3 replaces the two version-2 OTLP sections with one typed, multi-endpoint opentelemetry section. OpenTelemetry and OpenInference support is always built in and no longer uses the otel or openinference Cargo features.

The following example shows the version-2 shape:

[components.config]
version = 2
[components.config.opentelemetry]
enabled = true
endpoint = "http://localhost:4318/v1/traces"
transport = "http_binary"
service_name = "relay-service"
[components.config.opentelemetry.headers]
x-tenant-id = "demo"
[components.config.openinference]
enabled = true
endpoint = "http://localhost:6006/v1/traces"
transport = "http_binary"
service_name = "relay-service"

Replace it with typed endpoints:

[components.config]
version = 3
[components.config.opentelemetry]
enabled = true
[[components.config.opentelemetry.endpoints]]
type = "full"
endpoint = "http://localhost:4318/v1/traces"
transport = "http_binary"
service_name = "relay-service"
[components.config.opentelemetry.endpoints.headers]
x-tenant-id = "demo"
[components.config.opentelemetry.endpoints.header_env]
authorization = "OTEL_AUTHORIZATION"
[[components.config.opentelemetry.endpoints]]
type = "openinference"
endpoint = "http://localhost:6006/v1/traces"
transport = "http_binary"
service_name = "relay-service"

Set OTEL_AUTHORIZATION to the complete authorization header value before activation. Process-global OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS are no longer accepted because their values cannot be isolated between multiple exporters. Move non-sensitive version-2 headers to the endpoint’s headers map. Move resource_attributes and all service, transport, instrumentation-scope, and timeout settings into the corresponding endpoint.

Use the following endpoint type for each projection:

Every endpoint requires a nonblank endpoint and explicit type. The defaults also change:

Remove mark_projection, mark_exclude_names, attribute_mappings, semantic_selector, and capture_content. Projection behavior is fixed. The full and openinference types retain their previous default mark behavior. The gen_ai type omits marks and emits minimal internal spans for Relay scope types without GenAI semantics, preserving the original span parentage. It does not emit nemo_relay.* attributes. It targets the OpenTelemetry GenAI semantic-conventions v1.42-era snapshot.

NeMo Relay also upgrades to OpenTelemetry Rust 0.32.

Top-level component config lists now concatenate across layers, with higher-precedence entries first. The observability destination lists atof.sinks, opentelemetry.endpoints, and atif.storage follow the same rule. Arbitrary lists nested inside structured values continue to use replacement semantics. Edit the layer that declares an inherited entry when you need to change or remove it.

Binding API Changes

The standalone OpenInference config and subscriber APIs are removed. Construct one independently managed OpenTelemetry subscriber from one typed endpoint.

For plugin-owned multi-endpoint configuration, use OpenTelemetryEndpointConfig and OpenTelemetrySectionConfig in Python, openTelemetryEndpoint and openTelemetryConfig in Node.js, or ObservabilityOpenTelemetryEndpointConfig and ObservabilityOpenTelemetryConfig in Go.

Direct subscribers still own one tracer provider each. Register the subscriber before instrumented work, then deregister, flush, and shut it down during graceful teardown.

Related Release Information

For release highlights, compatibility updates, and current known issues, refer to the Release Notes. Use GitHub Releases for the complete release history and notes for a specific tag.