Skip to navigation
About NVIDIA NeMo Relay

Release Notes for NVIDIA NeMo Relay

View as Markdown

This page contains the release notes for NVIDIA NeMo Relay.

For the full release history, including individual pull requests, refer to GitHub Releases.

Release 0.11.0

NeMo Relay 0.11.0 includes the changes below. They cover updates since 0.10.0 and fixes from the 0.10 patch releases. For the full release history, refer to GitHub Releases.

Compatibility and Migration

Review the Upgrade Steps

Upgrades from 0.10.0 can change hook failure behavior, provider access, worker configuration, and stream error handling. Review the Migration Guides before updating a deployment.

The main compatibility changes affect these areas:

  • Node.js applications and package workflows require Node.js 24.13.1 or later.
  • LangChain and LangGraph tool results use the nemo.relay.tool_result@1 JSON wrapper. It holds display content, failure status, and result values with their types. Custom Python objects stay with the call that returned them. Review how custom middleware and subscribers read tool calls and results. Refer to Normalized Tool Results.
  • Worker messages now have size limits for sending and receiving data. The default is 4 MiB per message. You can set limits by callback name. Upgrade the host and worker SDKs together before using these settings. Refer to Configure Execution Message Size.
  • Automatic OpenTelemetry Protocol (OTLP) logs and metrics now require HTTPS for remote destinations, just like traces. This rule applies even without credentials. Make configured Transport Layer Security (TLS) files available when Relay checks plugins. A fatal TLS error stops an HTTP trace exporter. Fix the TLS problem, then recreate the exporter to send traces again.
  • If you create FileLogSinkConfig directly in Rust, add the failure_policy field or use ..Default::default() to fill in missing fields.
  • Adaptive hints now use the settings in inject_header and inject_body_path. Check these settings because they can now change requests sent to the provider. Refer to Adaptive Hints.
  • Python object decoding now requires an explicit list of allowed classes to restore tagged dataclass or Pydantic values.
  • Node.js streams now report provider, collector, and finalizer failures. Callers must handle errors and await cleanup when they stop reading early.
  • Workers now load layered user and system configuration. Policy checks should run before response caches. The default cache priority is now 150.
  • Local hooks use verified TLS connections. Fail-closed delivery failures return exit code 2. Refresh personal hooks and restart affected processes.
  • Native plugins can use application binary interface (ABI) v8 scope snapshots. Existing v7 callback layouts remain compatible. Worker snapshots require the new grpc-v1 extension.
  • Rust worker SDK error matches must handle WorkerSdkError::Downstream. Python worker error checks must allow WorkerSdkError subclasses. Preserve the original error so diagnostics can identify where the failure began.
  • Embedded diagnostics settings are fixed by the first initialization. Restart the process to change them. Diagnostic read access can expose private data and needs separate authorization.

Integrations and Coding Agents

These changes improve framework support and coding-agent traces:

  • The Python Google Agent Development Kit (ADK) integration records agent runs, large language model (LLM) calls, and tool calls. Relay guardrails and intercepts apply to supported model and tool calls. Setup guides include ADK CLI and workflow examples. For the full support list, refer to Supported Models and Providers.
  • Deep Agents supports position="outer" to record tool results after your middleware changes them. The default remains "inner". Refer to Middleware Ordering.
  • Codex and Claude Code hook sessions can continue the launcher’s trace through TRACEPARENT and TRACESTATE. Refer to Continue the Launcher Trace.
  • LangChain tool schema edits now reach the model call.
  • Framework tool results retain content and handled error status. Manual LLM completion preserves an explicit error status. Cache replay keeps framework tool result objects and content intact.
  • Tool classification now recognizes load_skill as a skill tool.
  • CLI turn spans now inherit the name of their owning Agent scope. Paired tool hooks use that name unless they supply an explicit name. OpenTelemetry agent names and span names can change as a result. Refer to Turn Identity.

Diagnostics and Observability

These changes help you inspect runtime behavior and manage telemetry:

  • Runtime diagnostics stores counts and recent events in memory. Set storage limits at startup. Readers can inspect gateways, daemons, and embedded applications. Native and worker plugins can record their own internal work. Memory views show diagnostic storage use and optional resource samples. Collection is disabled by default. Refer to Diagnostics.
  • Doctor checks OTLP file sinks, disabled components and destinations, references to credentials, and personally identifiable information (PII) settings. A passing HTTP check only means the collector answered. It does not prove that the collector accepts your credentials or trace data. Doctor does not test whether redaction removed sensitive data. Refer to Check Configuration and Exporters.
  • OTLP endpoints support custom certificate authorities and mutual TLS. Refer to TLS and Mutual TLS.
  • Python applications can configure local OTLP trace files with OpenTelemetryFileSinkConfig. Refer to Trace File Sinks.
  • Agent Trajectory Interchange Format (ATIF) exports can omit raw events and full provider requests to reduce file size. Both remain included by default. Remote uploads use compact JSON, with fewer spaces and line breaks. Refer to Reduce Output Size.
  • ATIF output uses valid content parts. Text-only step messages use strings. Delivery-failure warnings identify the affected trajectory.
  • Diagnostic HTTP errors now use consistent status codes and reason fields. Worker inspection reports whether a worker is missing or its diagnostics endpoint is unavailable. Refer to Diagnostics API.
  • Gemini cost estimates account for thinking and supported tool-use tokens. Pricing now counts each group of input tokens once, including cached tokens. It uses all input tokens to choose a pricing tier. If token counts conflict, Relay does not estimate a cost. Refer to Model Pricing.

Providers and Request Handling

These changes improve provider access and request handling:

  • Daemon workers support the full middleware pipeline for LLM calls. This covers OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Refer to LLM Middleware.
  • Provider API key helpers can supply a fresh credential for each request attempt. Refer to Generate API Keys Per Request.
  • OpenAI- and Anthropic-compatible model upstreams can use remote HTTP when their hosts are explicitly trusted. These requests send no credential headers. Prefer HTTPS. Refer to Credential-Free HTTP Model Upstreams.
  • Provider routing checks access before using Relay-held credentials.
  • Request codecs preserve provider fields across multiple edits. Gemini request edits preserve text placement. Relay also merges streamed function-call arguments correctly.
  • OpenAI Responses streams retain usage and report missing terminal events.

Configuration and Deployment

These changes improve setup, logging, and deployment guidance:

  • Operational file log sinks now support failure_policy = "optional". This lets Relay start even if it cannot open that log file. Refer to File Sink Startup Failure Policy.
  • Logging overrides preserve configured file sinks.
  • The configuration editor exposes more gateway, provider, logging, and plugin settings. Daemon guidance now covers Kubernetes deployment planning. Refer to Interactive Setup and Kubernetes Deployment Guidance.
  • New quick starts cover application bindings and supported frameworks and coding agents. Plugin setup guidance includes Switchyard. Refer to Quick Starts and Switchyard.

Runtime and Plugin Reliability

These changes improve scope inspection, streaming, and resource cleanup:

  • Scope snapshots expose the visible scope IDs and types across bindings and plugin SDKs. Refer to Inspect the Visible Stack.
  • Plugin integrity and signature failures always prevent activation.
  • Streaming preserves Unicode characters split across transport chunks. Python streams retain pending exceptions through native wakeups. Node.js streams retain errors when closing overlaps a read. Node.js also catches errors thrown by codec callbacks. Worker streams keep middleware errors. Errors before the first chunk keep their HTTP status. Failed streams no longer send a marker that reports successful completion.
  • Daemon, worker, and hook paths use less memory or avoid repeated work. Worker cancellation and session-lock cleanup release resources when calls end.

Known Issues in 0.11.0

The following known issues and limitations apply to NeMo Relay 0.11.0:

  • Diagnostics stores recent evidence in memory and can replace older events when storage fills. It does not keep history after process exit. Diagnostic detail can contain private data that has not been sanitized. Daemon worker caches can return stale or partial pages. Check coverage, freshness, and continuation fields before drawing conclusions. Refer to Diagnostics and CLI and Daemon Diagnostics.
  • Google ADK live model calls through Runner.run_live or RunConfig.support_cfc have no Relay model events or middleware. Relay’s response cache skips ADK model calls. While personally identifiable information (PII) redaction is active, Relay omits model response payloads from exported model end events. Tools run by the provider cannot be blocked or changed by Relay. Use wrap_node to record full scopes for workflow steps, except steps run with parallel_worker. Those steps have completion events rather than full step scopes. Refer to Google ADK Limitations.
  • ChatGPT voice calls and sidebands use separate upstream settings. Call creation uses the ChatGPT backend and keeps the caller’s ChatGPT credential. Sideband attachment uses openai_base_url and never receives Relay’s provider credentials. A custom OpenAI upstream might not support a call created on ChatGPT. Relay also removes recognized ChatGPT tokens and chatgpt-account-id from unmanaged voice sideband requests to that custom upstream. Use a sideband destination that can access the call without relying on the caller’s ChatGPT credential. Codex Desktop connects its sideband directly to OpenAI by default. Refer to Codex Responses Compatibility.
  • OTLP collectors can return a successful response while rejecting individual spans, log records, or metric data points. Relay does not report these partial successes in runtime diagnostics, and flush or shutdown can still succeed. Monitor the collector’s rejection metrics and logs. Refer to OpenTelemetry.
  • Go and the raw C foreign function interface (FFI) remain experimental. Build them from source. Generated API pages focus on Rust, Python, and Node.js.
  • Local coding-agent observability depends on host hooks and provider traffic reaching the local gateway. Relay cannot fully capture remote or cloud execution that bypasses the local host.
  • Persistent Codex and Claude Code integrations use each user’s configuration and a shared loopback gateway. Use nemo-relay run when a launch must retain project-specific configuration.
  • On Windows, a restrictive host Job Object can limit gateway reuse or prevent persistent bootstrap. A Job Object controls a group of processes. If it prevents a managed worker from leaving the group, the worker starts inside the group and emits worker_lifetime_limited. Closing the host can stop that worker. MCP clients can then need to reactivate the route after the broker’s recovery grace period. Refer to Register the MCP Lifecycle Client. During startup, Codex can request /models before required MCP servers start. Relay retries the request.
  • Codex 0.143 does not expose SessionEnd, and Codex multi-agent v2 encrypts delegated task payloads. Relay cannot recover the task text from encrypted payloads. Parent-child links depend on the agent identities and lifecycle events that Codex provides. Refer to Multi-Agent Delegation Traces.
  • Pi support is a proof of concept tested with 0.84.x. By default, its personal extension lets work continue if the gateway is unavailable. Later extensions can change arguments after Relay checks them. Unsupported provider routes produce no LLM spans. This includes Google’s native Gemini and Vertex APIs. To route Gemini model traffic through Relay, use a provider or proxy that accepts an OpenAI- or Anthropic-compatible API. Nested Pi processes appear as separate sessions. SIGINT in headless mode can stop Pi before queued events are sent. For details, refer to Pi Limitations.
  • Authenticated managed daemon routes use passthrough while workers start or recover unless the daemon uses --require-worker. Passthrough forwards requests without worker plugins. Failed starts retry automatically, but /healthz and MCP readiness alone do not prove policy enforcement or exporter activity. Verify worker status and an observable plugin result, or require a worker and handle HTTP 503 during unavailability. Refer to Daemon Operations.
  • Harness model requests do not read the per-user token file. Codex and Claude Code send the route credential header only from their own provider configuration, which they read at startup. Without that header, model requests pass through without Relay telemetry. The managed v1 Pi extension reads only NEMO_RELAY_CLIENT_TOKEN and fails to start without it, so nemo-relay doctor --managed-bundle rejects a file-only credential for Pi bundles. On Windows, nemo-relay daemon token ensure does not check the access list of an existing token directory. Refer to Credential Resolution.
  • Remote daemons and client workers must be able to reach each other. Proxies must support WebSockets. Loopback control connections do not send periodic pings. An idle hung process can go undetected until an operation times out.
  • OpenClaw provides telemetry through public hooks. Its security and optimization coverage is partial because Relay does not manage its execution.
  • The built-in nemo_guardrails plugin remains deprecated but is still available in 0.11.0. No replacement is included. The remote backend depends on its configured service’s availability, latency, and policy behavior. The local backend requires Python 3.11 or later and nemoguardrails==0.22.0. Refer to NeMo Guardrails.
  • The personally identifiable information (PII) redaction plugin supports its local backend based on fixed rules. Local-model backend configuration is reserved for future work.
  • Pricing and optimization estimates depend on model names, token data, pricing sources, and how current those sources are. Missing or conflicting data produces partial or absent cost fields rather than zero values.
  • ATOF stream sinks and remote ATIF storage require reachable, correctly configured destinations. A failed stream sink does not stop file output or other active sinks.
  • ATIF omits point-in-time marks. Use ATOF for the complete mark stream. The gen_ai OpenTelemetry projection also omits marks. The full and openinference projections retain their fixed native mark handling.
  • Native dynamic plugins run in the Relay process and are not sandboxed. A grpc-v1 worker runs in a separate process, but that process is not a security sandbox.
  • Do not change Python LLMRequest objects in place. Request middleware that changes content must return a new request object.
  • Native subscriber callbacks arrive asynchronously. Flush subscribers before depending on their changes, captured events, files, or exporter output.
  • OpenTelemetry endpoints use batch queues with a size limit. If a burst fills a queue, the endpoint drops completed spans instead of slowing the sender. This can leave an incomplete trace, including a missing root span. The SDK warns on the first drop and reports the exact dropped-span count during graceful shutdown. Configure max_queue_size, max_export_batch_size, and scheduled_delay_millis independently on each endpoint, or use the standard OTEL_BSP_* environment variables as defaults for the process. A larger queue does not guarantee lossless telemetry.
  • Operational logging covers Relay-managed hooks, gateway requests, the daemon, and workers. It does not yet cover every CLI command.
  • A parent handle outside the current scope stack does not keep its earlier W3C context. If it belongs to a different trace, managed-call headers can differ from the parent context kept by an exporter. Capture and import the parent’s propagation context before leaving that trace. Reusing only the handle does not preserve this context.
  • Codex’s built-in openai provider attempts a WebSocket connection before falling back to HTTP streaming. Relay does not support that transport, so a new session can report connection errors and take several seconds longer to start. Refer to Codex.
  • Available resource measurements depend on the platform. Initial rates can be unavailable, and copies of one polling sample under multiple Agent scopes must not be added together. Refer to Resource Metrics.
  • Default non-paused threshold compaction and unsupported or malformed Anthropic compaction traffic bypass the response cache. Selected compaction support does not let Relay replay every provider response. Refer to LLM Eligibility and Limits.

Previous Releases

For previous release notes, release artifacts, and the complete pull request history, refer to GitHub Releases.

For more information, refer to the following pages: