Migration Guides

Use this page to upgrade from NeMo Relay 0.9.0 to the current 0.10 release. If you already use 0.10.0, apply the patch-upgrade steps below.

If you upgrade directly from 0.9.0, apply both the base-release and patch-upgrade steps. The base-release steps also cover configuration and telemetry changes from the 0.9 patch releases.

Upgrade to NeMo Relay 0.10

Update from NeMo Relay 0.10.0 to 0.10.1

Update the Relay packages you use to 0.10.1. Review the following steps for hook delivery and authentication, daemon workers, request intercepts, dynamic plugins, Python typed values, provider endpoints, and HTTP errors.

Restore Credentials for Fail-Closed Daemon Hooks

Make sure each user’s hooks can resolve NEMO_RELAY_CLIENT_TOKEN or the per-user token file before starting a session. If the token file is missing, run the following command as that user:

nemo-relay daemon token ensure

Do not run this command as root. Creating the token file does not register a route for an MCP server that already started without a credential. Start a new harness session and confirm that its MCP server registered the token with the daemon before retrying hooks. Refer to Diagnose Common Failures for credential and route diagnostics.

Check custom hook wrappers and scripts that handle exit codes: delivery failures under a fail-closed policy now return 2 instead of 1. Fail-open delivery failures still return 0. Guardrail rejections still return 2 under either policy. Public HTTP requests without a bound credential retain their separate pass-through rules. Refer to Requests Without a Bound Credential.

Refresh Personal Hooks and Review Timeouts

Gateway hook delivery now waits up to 30 seconds instead of two seconds. Daemon and daemon-worker delivery use the same limit. Set LLM judge guardrail evaluation timeouts to leave room for hook delivery within that limit. Timed-out requests are not retried.

Refresh personal Relay-managed hooks before starting new sessions:

nemo-relay integrations refresh

To refresh one agent, use nemo-relay install <agent> --force. For administrator-managed daemon bundles, follow Upgrade and Roll Back instead. Keep the existing bundle files; do not regenerate them with personal install or refresh commands. Refer to Hook Forwarding for failure policies.

Review Daemon-Worker LLM Middleware

Daemon workers now support all six large language model (LLM) middleware types for OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, including streaming calls. Review plugins that register request intercepts, execution intercepts, or conditional execution guardrails. These now use the managed LLM pipeline and can change or block requests and responses.

Check clients that depend on provider success headers or exact stream formatting. When request intercepts, execution intercepts, or conditional execution guardrails are configured, successful calls return HTTP 200 without the provider’s original success headers or HTTP trailers. Streaming responses still arrive in parts, but their formatting can change. Provider HTTP errors keep their status, supported headers, and body. LLM sanitizers alone forward the provider response unchanged, including its success status and headers. Other provider APIs and WebSocket connections bypass LLM middleware.

The response-header timeout includes middleware execution. For calls without streaming, it also includes the wait for the full response. It does not limit stream duration after response headers arrive.

For responses forwarded unchanged, the default stream observation copy limit increases from 4 MiB to 16 MiB. If needed, set NEMO_RELAY_DAEMON_OBSERVATION_CAPTURE_BYTES before starting the worker to choose another limit. This setting does not cut off the client’s stream. Refer to LLM Middleware.

Review Request-Intercept Edits

If your intercept edits several messages or array entries, review the rules for keeping provider fields. Relay keeps those fields on unchanged entries and on tool calls or results with a clear call-ID match. Entries without a match use the values you return. New entries do not inherit fields from old entries. Do not rely on a generic id field to preserve data in arbitrary JSON arrays. Refer to Editing Arrays and Mappings for matching rules and the limited fallback for a single edit.

For LangChain managed model calls, request intercepts can edit function-tool descriptions and input schemas or provider-specific tool definitions. Keep tool count, order, and function-tool names unchanged. Review intercepts that modify those fields, and verify that the selected model adapter accepts the edited definitions. Refer to LangChain.

Refresh Pi Extensions for Authenticated Hooks

After upgrading the gateway, refresh a Relay-managed Pi extension with nemo-relay install pi --force. Replace manually installed copies with the updated extension source. The extension now forwards hooks through nemo-relay hook-forward pi; direct requests without valid credentials receive HTTP 401. Requests for sessions owned by another client also receive 401.

For standalone Pi, make the updated nemo-relay executable available on PATH or set NEMO_RELAY_PI_COMMAND to its path. Use a loopback HTTP gateway URL and run Pi as the same local user as the gateway so the forwarder can load bootstrap credentials. The Relay launcher sets the executable path automatically. Refer to Pi.

Review Pi Named Provider Endpoints

Pi named upstreams must use public or loopback addresses. Private, link-local, shared, and other special-use addresses are refused, including addresses a hostname resolves to when the gateway connects. Remote endpoints require HTTPS; HTTP is allowed only for loopback hosts.

If a Pi model uses a private provider endpoint, configure it as the gateway’s upstream with --openai-base-url or --anthropic-base-url, or in the daemon’s system upstream configuration. Do not pass it as a named endpoint. Refer to Model Redirection.

Review the ChatGPT Codex Endpoint

The default ChatGPT Codex endpoint is unchanged. To use another endpoint, set chatgpt_codex_base_url under [upstream]. Relay forwards the caller’s ChatGPT credential to that endpoint, including for voice call creation, so use only an endpoint trusted with those credentials. Remote hosts require HTTPS; HTTP is allowed for loopback hosts.

Personal gateways also accept NEMO_RELAY_CHATGPT_CODEX_BASE_URL and --chatgpt-codex-base-url, with CLI flags taking precedence. Managed workers read the endpoint from system TOML configuration and ignore that environment variable. Refer to Codex Responses Compatibility.

Review HTTP Error Handling

Remote NeMo Guardrails failures now use FlowError::Upstream with the received HTTP status instead of FlowError::Internal. Update Rust matches that handle these errors. Gateway and forwarded lifecycle-hook errors preserve received statuses from 400 through 599, including failures to read an error body. Existing response envelopes and hook failure policies remain unchanged.

Plugin error strings beginning exactly with HTTP <status>: <message> now carry HTTP meaning. The status must be three ASCII digits from 400 through 599, followed by a colon and a space. Review existing callback error strings that match this form. To adopt it, use http_error_message in the Rust native SDK, Rust worker SDK, or Python worker SDK. Relay adds the host-verified reporting plugin identity to recognized messages. Returning a continuation error unchanged preserves its prefix; adding text before HTTP prevents recognition.

Recognized plugin errors use FlowError::Upstream with empty headers and non-retryable Other classification. Callback signatures, the native ABI, and worker protobuf fields remain unchanged. Errors after streaming headers have been sent remain stream errors and cannot change the HTTP response status. Refer to Build Native Plugins, Worker Middleware and Continuations, and NeMo Guardrails.

Review Dynamic Plugin Trust and Native Dependencies

Failed integrity or signature verification now always prevents plugin activation. With startup = "optional", the host skips the failed plugin and continues. With startup = "required", the failure stops host startup. Review plugin validation and lifecycle reports and correct artifact digests, signatures, or trusted keys before restarting. Refer to Discoverable Plugins.

Application embeddings load native libraries from private temporary copies of the verified bytes. Native plugins used in embeddings must not rely on adjacent files relative to the loaded library’s location. Review those dependencies before upgrading. The CLI retains its activation snapshot, including adjacent native dependencies. Native library digest checks reject symbolic links and special files; use regular files for native library artifacts. Refer to Package Discoverable Plugins.

Select Explicit Codecs for Stored Python Typed Values

BestEffortAnyCodec no longer imports a Python type named by a dataclass or Pydantic tag. It reconstructs only types registered by to_json() in the same process. Unregistered tags remain JSON dictionaries. For stored or cross-process payloads, use an explicit DataclassCodec or PydanticCodec to select the type.

LangChain and LangGraph tool arguments now use JsonPassthrough, even when a tag names a registered type. Review tool handlers that expect Relay to reconstruct objects from tagged arguments. Tool result codecs remain unchanged. Refer to Using Codecs.

Update Packages and Plugin SDKs

When upgrading from 0.9, update the Relay CLI, application bindings, and plugin SDK dependencies to the current 0.10 release. Rebuild Rust applications and native or worker plugins against the updated dependencies. If you use the OpenClaw plugin, update its package too. It requires the matching Node.js binding version.

The optional Python LangChain NVIDIA integration now requires langchain-nvidia-ai-endpoints>=1.4.2,<2.0.0. Update older version requirements and regenerate the environment’s lockfile before installing the integration. For package names and installation commands, refer to Installation.

Move Primary Daemon Logging Settings

Move the daemon service’s logging settings to [daemon.logging]. Keep [logging] for other Relay processes, such as agent launches. CLI options and NEMO_RELAY_LOG* environment variables still override TOML settings for both. The following example enables JSON Lines logging to standard error:

[daemon.logging]
level = "info"
stderr_enabled = true
stderr_format = "jsonl"

For the order in which Relay applies logging settings, refer to Operational Logging.

Require Observability Configuration Version 4

Change version = 3 to version = 4 inside the observability component’s configuration, including programmatic configuration objects. Keep the existing typed trace endpoints. Adding log or metric sections is optional. The outer plugin configuration remains version = 1. The following example uses observability version 4 with a trace endpoint:

version = 1
[[components]]
kind = "observability"
enabled = true
[components.config]
version = 4
[[components.config.opentelemetry.endpoints]]
type = "gen_ai"
endpoint = "http://localhost:4318/v1/traces"

For the complete schema, refer to Observability Configuration.

Review Automatic OTLP Export and Collector URLs

OpenTelemetry Protocol (OTLP) endpoint environment variables now enable automatic export when their values are not empty or whitespace. The generic OTEL_EXPORTER_OTLP_ENDPOINT enables traces, logs, and metrics. Each endpoint variable for a specific signal enables only that signal. Automatic traces use the gen_ai projection, which maps Relay events to GenAI attributes. Relay can also export to its configured destinations, so existing settings can produce extra exports after the upgrade.

Review the launch environment and choose the exports you need. Remove unintended endpoint variables. To disable automatic and configured OTLP export, set opentelemetry.enabled = false in the observability component. To disable every Relay OTLP exporter, use OTEL_SDK_DISABLED=true. Resource and batch settings alone do not enable export.

If you configure Relay endpoints, move the process’s global OTLP header variables to each endpoint’s headers, header_env, or header_file settings. Relay skips a configured endpoint if global OTLP headers are set when it starts. The underlying exporter cannot keep those credentials separate. Automatic exporters can still use the global header variables.

Use the collector’s final URL. All OTLP/HTTP exporters now reject redirects, including log and metric exporters without headers. Remote automatic trace endpoints require HTTPS. Automatic log and metric endpoints with headers also require HTTPS. Plain HTTP remains available for localhost and loopback IPs. For supported variables and the order in which Relay applies them, refer to Automatic OpenTelemetry Configuration.

Update LLM Execution Intercepts

Relay 0.10.0 changes every large language model (LLM) execution-intercept callback, including streaming callbacks. Update their arguments as follows:

The new context identifies the selected request codec. A codec converts data between a provider’s format and Relay’s common format. When a request codec is available, the context provides decode and encode operations. Non-streaming execution also identifies the response codec and provides decoding. Streaming execution has no response codec because chunks are not complete provider responses.

Codec access lasts only for the current call. Do not store the context or selected codec for later calls. Non-streaming access ends when the callback finishes. Streaming request access ends when the returned stream closes.

Existing execution-intercept code and compiled plugins need updates. Make the following changes:

• Update callback argument order in the bindings you use. Recompile affected applications that use compiled bindings.
• Rebuild every native plugin against the 0.10 SDK. Native plugins continue to declare compat.native_api = "1", but must set compat.relay to ">=0.10.0,<1.0" or another range that excludes Relay 0.9. Relay 0.10 uses version 7 of the internal application binary interface (ABI). This interface defines how compiled plugins call the host. Relay rejects compiled layouts from versions 2 through 6.
• Regenerate and rebuild gRPC workers that register an LLM execution intercept. The protocol remains grpc-v1, but those workers must also exclude Relay 0.9 in compat.relay.

Raw native asynchronous intercepts also need the registration APIs for LLM execution. Replace non-streaming LLM execution uses of PluginContext::register_async_middleware_raw with register_async_llm_execution_intercept_raw, backed by the ABI v7 plugin_context_register_async_llm_execution_intercept host function. Use register_async_stream_middleware_raw for streaming execution. Its callback now receives codec context. The generic asynchronous middleware registration rejects both LLM execution kinds.

Raw asynchronous streaming code must keep a reference to the output stream while it uses the request codec. Use the stream’s decode and encode operations. Release the reference after its last use. The typed SDK manages these references for its codec wrappers.

LLM request intercepts and conditional middleware do not change. For the full context and lifetime contract, refer to Build Plugins, Native ABI Reference, and gRPC-v1 Worker Protocol.

Update Custom Worker Protocol Bindings

The worker protocol remains grpc-v1. Regenerate custom bindings. Rust code that lists every field in a protobuf message literal must add the following fields. Use generated defaults if you do not need the new values. The following table lists each field and its value when unused:

Custom implementations of the RelayHostRuntime service must also implement the new Log remote procedure call (RPC). The PluginWorker service does not gain a method. Workers that need logging through the host or earlier scope timestamps must declare a compat.relay range beginning at 0.10.0. Older hosts can lack Log or ignore the optional timestamps. Ordinary SDK scope calls still use the host’s current time. Refer to gRPC-v1 Worker Protocol.

Review Python Exception Handling

In Python, managed LLM and tool calls and intercepts now raise the original callback exception. The exception keeps its type, attributes, and traceback, which shows where the error occurred. If your error handlers catch only RuntimeError for callback failures, update them to catch the exceptions that your callbacks raise. Runtime failures and guardrail rejections still raise RuntimeError. This change does not apply to worker plugins.

Callbacks and stream iteration preserve asyncio.CancelledError. If the iterator raises it during aclose() cleanup, Relay reports RuntimeError. Handle cancellation separately from ordinary exceptions when needed. A streaming callback that returns an object that is not an async iterator raises TypeError.

Review Linux CLI Requirements

The musl CLI binaries now require musl 1.2 or newer on the host to load native plugins. Install the musl runtime, which includes its loader and C library (libc). Earlier musl binaries included the C library and could run without this runtime. If you used those binaries on a glibc-based system, switch to the GNU binary or install the required musl runtime.

The Unix shell installer now selects the GNU binary when it detects glibc. Otherwise, it selects the musl binary. Refer to Installation for target names and requirements.

Update Rust and Go Struct Literals

The following public Rust types gained fields after 0.9.0. If your code lists every field in a struct literal, add these fields. Use builders or defaults where the type provides them. The following table lists the added fields:

The Rust enum variant FlowError::CallbackException also gains a source field. When creating this error without an underlying exception, add source: None. When matching this error in Rust, include source or use .. to ignore fields that you don’t need.

In Go observability or propagation struct literals, use field names if your code currently lists values without names. Saved configuration and propagation payloads remain valid when they omit the new optional fields. The propagation wire version remains 1.

Set promote_resource_metadata_prefixes to an empty list when you do not want log or metric endpoints to copy root-scope metadata into resource attributes. An omitted field defaults to an empty list in TOML and JSON. Log and metric endpoints derived from a trace endpoint inherit its setting. Choose metadata values from a small, controlled set to limit the number of resources. For setting priorities and resource limits, refer to OpenTelemetry.

Preserve Trace Context Across Boundaries

Send the complete captured propagation context when continuing work in another scope stack or process. This context links work across boundaries. Its optional W3C fields now preserve an imported upstream trace, sampling flags, and vendor state. Sampling flags show whether the upstream trace was sampled. A valid imported W3C parent overrides trace IDs that Relay derives, including a supplied explicit root. Relay ignores malformed W3C values and continues with valid Relay propagation data.

Capture without a root removes the trace derived from a local Relay sender. It keeps a previously imported W3C parent. If you always need a new OpenTelemetry trace, remove traceparent and tracestate from that context. PropagationContext.to_traceparent() now forwards a valid imported parent unchanged. Capture and import context before leaving a trace if later calls must continue it. A scope handle outside the current stack does not keep its earlier W3C context. Refer to Scope Concepts.

Managed LLM execution replaces Relay’s traceparent and tracestate headers on provider requests. Import the upstream propagation context to preserve that parent instead of setting those headers manually. Capture the managed LLM context inside its provider callback if you need that span as a later parent. After the call returns, capture uses the enclosing active scope or imported context. The completed LLM span is no longer current.

Review Token Usage and Cost Reports

The optional Usage.uncached_input_tokens count is present only when the provider reports how many input tokens were not cached. Do not treat an absent value as zero. Anthropic compaction usage that combines multiple models no longer receives a model-based cost estimate. Review cost reports that assumed every response could be priced from its response model. Refer to Provider Response Codecs.

Update GenAI Attributes and Metric Dimensions

The GenAI projection replaces the old attribute gen_ai.usage.cache_creation.input_tokens with gen_ai.usage.cache_write.input_tokens. Update collector queries, dashboards, and alerts that use the old attribute name.

The projection also no longer adds nemo_relay.session.instance_id to metric attributes. Update dashboards that group or filter on this session attribute. Matching measurements now combine into the same time series across sessions.

Review GenAI Tool Content and Completion Marks

GenAI tool spans now preserve every JSON argument and successful result except null, after sanitizers process the content. They use a consistent JSON text format, including for arrays, strings, numbers, and booleans. Update consumers that assumed all values were objects. Review collector access and sanitizers for the newly kept content. enable_full_payloads = false does not disable tool-content export. For the projection contract, refer to GenAI Projection.

Managed streaming telemetry adds gen_ai.response.time_to_first_chunk and the gen_ai.client.operation.time_to_first_chunk histogram in seconds. These measure the first provider protocol chunk, not the first text token. Keep those measurements separate in latency dashboards.

Coding-agent end hooks without a matching start now produce agent_end_without_start, turn_end_without_start, subagent_end_without_start, or tool_end_without_start marks. Consumers must allow start_observed = false, absent start times and durations, and an absent parent when no existing parent can be verified. ATIF and the GenAI projection omit these partial observations. Use ATOF, the full OpenTelemetry projection, or OpenInference when you need them.

Relay reserves tool_end_without_start. Any application or plugin that emits that mark must provide a string in data.tool_name that is not empty or whitespace. Otherwise, emission fails. Its arguments and result now pass through tool sanitizers before event sanitizers. Rename any unrelated custom mark that used that name. Refer to Completion Hooks Without a Start.

Refresh Relay-Managed Coding-Agent Integrations

After upgrading the CLI, run the following command to refresh personal Relay-managed coding-agent integrations before starting new sessions:

nemo-relay integrations refresh

On Windows, this command updates installed Claude Code hooks to use paths that work in Bash. It also updates Codex hooks to use PowerShell commands that preserve input, output, and exit codes. Existing installed hooks keep their old commands until you refresh them.

To refresh only Codex, use nemo-relay install codex --force. Refreshing Codex switches persistent routing to the built-in OpenAI provider. This preserves session identity and routes image generation through Relay. It replaces URLs that contain credentials with a plain gateway URL. It stores managed authentication in $CODEX_HOME/.env. Quit and restart Codex after refreshing to load the header. The gateway still accepts older capability URLs for older clients.

For daemon bundles managed by an administrator, follow the daemon upgrade workflow. Close sessions and let workers finish pending requests. Then stop the service and replace the binary at the same path. Keep the saved identity state and existing bundle files.

Update Codex Trajectory Consumers

Codex now exports one Agent Trajectory Interchange Format (ATIF) file for each codex-task instead of one file for the whole session. A task groups parent turns and delegated agent work. Relay writes the file when the task closes. Separate tasks in the same Codex session produce separate files and traces.

If you use the default filename template, update file lookups from nemo-relay-atif-<session-id>.json to nemo-relay-atif-<task-scope-id>.json. Use extra.nemo_relay.session_id in each trajectory to group tasks from the same Codex session. Update tools that expect a cumulative file after each parent Stop hook. A task can stay open while delegated agents finish and the parent prepares its final response. Refer to Codex.

Review Managed Provider Credentials

Managed Codex routes now forward ChatGPT OAuth to the native ChatGPT Codex backend when no administrator OpenAI credential is configured. If the daemon has openai_auth_header or OPENAI_API_KEY, Relay replaces the incoming OAuth credential and uses the administrator’s configured OpenAI-compatible upstream. ChatGPT voice calls to POST /backend-api/codex/realtime/calls are an exception. They keep the caller’s ChatGPT credential and use the native ChatGPT backend even when a replacement credential is configured. Review this exception if your deployment relies on administrator credentials to select the upstream. Review those settings to choose the intended provider and credential source. Refer to Codex Responses Compatibility.

Managed Claude model routing requires an Anthropic API key or an administrator’s Anthropic-compatible gateway credential. Claude.ai OAuth is not supported with ANTHROPIC_BASE_URL. If your deployment uses only OAuth, keep native routing instead of setting the managed base URL. Refer to Daemon Configuration.

Choose Managed Daemon Startup Policy

Model Context Protocol (MCP) initialization can now finish after the client authenticates and registers. A worker starts in the background. By default, authenticated requests use passthrough during startup and recovery. Passthrough forwards requests without worker plugins. New requests switch to the worker after runtime, plugin, control, and HTTP readiness checks pass.

If every request must use worker guardrails, plugins, or observability, add --require-worker to the daemon service command. Configure clients to handle HTTP 503 while a worker is unavailable. This flag conflicts with --pass-through, which disables workers. Neither /healthz nor an MCP handshake proves that worker policies are active. Verify worker status and an observable plugin result.

Worker startup has no deadline or retry limit. Update monitoring that assumed a fixed startup timeout. Check worker_startup_pending, worker_added, and worker_activation_failed for progress, elapsed time, and retry delays. Retry delays increase up to 60 seconds. Active requests and streams keep their original destination during route changes. Refer to Daemon Operations.

On Windows, managed workers request breakaway when the host Job Object permits it. Otherwise they start inside the host job and emit worker_lifetime_limited. This respects the host’s job policy. Closing that host can stop the worker and require route reactivation after the broker’s recovery grace period. For process ownership and log destinations, refer to Register the MCP Lifecycle Client.

Revalidate Shared Python Worker Environments

Older managed Python environments without .nemo-relay-environment.key now require the original installer’s bootstrap key to verify their signed installation records. An environment that previously validated without that key can fail validation or startup after upgrading. If the original key is missing or unreadable, recreate the environment from the trusted plugin source. Newly provisioned environments keep their verification key with the environment.

For system-installed workers, make sure each daemon user can read the configuration, manifest, lifecycle state, and environment. Users must also be able to access their directories. Only administrators should have write access. Lifecycle state records the plugin’s installation and validation status. Startup and inspection validate shared state without rewriting it. Run nemo-relay plugins validate <plugin-id> to check the installation. This command saves its validation result. Refer to Discoverable Plugins.

Review Anthropic Compaction Cache Support

Compaction summarizes older conversation content to reduce its size. Relay 0.9.1 bypassed the response cache for all Anthropic compaction traffic. Relay 0.10.0 can cache validated paused-threshold and on-demand compaction responses and their supported continuations with anthropic-version: 2023-06-01 and the matching compaction beta headers. Default non-paused threshold compaction and unsupported or malformed traffic still run live.

No new cache setting is required. The existing rules for requests that can produce different results still apply. Requests without explicit temperature = 0 need cache_nondeterministic = true to be eligible. Review expected cache hits and anthropic_compaction bypass marks. Compare them with the supported cases in LLM Eligibility and Limits.

Related Release Information

For release highlights, compatibility updates, and current known issues, refer to the Release Notes. Use GitHub Releases for the complete release history and notes for a specific tag.