Skip to navigation
About NVIDIA NeMo Relay

Release Notes for NVIDIA NeMo Relay

View as Markdown

This page contains the release notes for NVIDIA NeMo Relay.

For the full release history, including individual pull requests, refer to GitHub Releases.

Release 0.11.0

NeMo Relay 0.11.0 is in development. The following sections summarize changes since 0.10.0, including fixes from the 0.10 patch releases. For the full history of a published version, refer to GitHub Releases.

Highlights

This release expands plugin control, deployment options, and setup guidance:

  • Daemon workers support the full middleware pipeline for large language model (LLM) calls. This covers OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Refer to LLM Middleware.
  • Scope snapshots expose the visible scope IDs and types across bindings and plugin SDKs. Refer to Inspect the Visible Stack.
  • OpenTelemetry Protocol (OTLP) endpoints support custom certificate authorities and mutual Transport Layer Security (TLS). Refer to TLS and Mutual TLS.
  • Provider API key helpers can supply a fresh credential for each request attempt. Refer to Generate API Keys Per Request.
  • New quick starts cover application bindings and supported frameworks and coding agents. Plugin setup guidance includes Switchyard. Refer to Quick Starts and Switchyard.
  • The configuration editor exposes more gateway, provider, logging, and plugin settings. Daemon guidance now covers Kubernetes deployment planning. Refer to Interactive Setup and Kubernetes Deployment Guidance.

Compatibility and Migration

Review the Upgrade Steps

Upgrades from 0.10.0 can change hook failure behavior, provider access, worker configuration, and stream error handling. Review the Migration Guides before updating a deployment.

The main compatibility changes affect these areas:

  • Python object decoding now requires an explicit list of allowed classes to restore tagged dataclass or Pydantic values.
  • Node.js streams now report provider, collector, and finalizer failures. Callers must handle errors and await cleanup when they stop reading early.
  • Workers now load layered user and system configuration. Policy checks should run before response caches. The default cache priority is now 150.
  • Local hooks use verified TLS connections. Fail-closed delivery failures return exit code 2. Refresh personal hooks and restart affected processes.
  • Native plugins can use application binary interface (ABI) v8 scope snapshots. Existing v7 callback layouts remain compatible. Worker snapshots require the new grpc-v1 extension.

Fixes and Other Changes

The main fixes improve request handling, telemetry, and process cleanup:

  • Provider routing checks access before using Relay-held credentials. Plugin integrity and signature failures always prevent activation.
  • Request codecs preserve provider fields across multiple edits. LangChain tool schema edits now reach the model call.
  • Framework tool results retain content and handled error status. Manual LLM completion preserves an explicit error status.
  • OpenAI Responses streams retain usage and report missing terminal events. Gemini cost estimates account for thinking and supported tool-use tokens.
  • Agent Trajectory Interchange Format (ATIF) output uses valid content parts. Text-only step messages use strings. Delivery-failure warnings identify the affected trajectory.
  • Logging overrides preserve configured file sinks. Go bindings retain native handles for the duration of foreign function interface (FFI) calls.
  • Daemon, worker, and hook paths use less memory or avoid repeated work. Worker cancellation and session-lock cleanup release resources when calls end.

For upgrade actions and links to each affected workflow, refer to the Migration Guides.

Known Issues in 0.11.0

The following known issues and limitations apply to NeMo Relay 0.11.0:

  • ChatGPT voice sidebands are separate connections used to control a voice call. Routing them through Relay can fail when Relay has a replacement OpenAI credential. Call creation at POST /backend-api/codex/realtime/calls keeps the caller’s ChatGPT credential and uses the ChatGPT backend. Attachment at /live/CALL_ID or /v1/live/CALL_ID instead uses the replacement credential and configured API upstream, which cannot find that ChatGPT call. Connect the sideband directly to OpenAI for this setup. Codex Desktop’s default direct sideband avoids this issue. Refer to Codex.
  • OTLP collectors can return a successful response while rejecting individual spans, log records, or metric data points. Relay does not report these partial successes in runtime diagnostics, and flush or shutdown can still succeed. Monitor the collector’s rejection metrics and logs. Refer to OpenTelemetry.
  • Go and the raw C foreign function interface (FFI) remain experimental. Build them from source. Generated API pages focus on Rust, Python, and Node.js.
  • Local coding-agent observability depends on host hooks and provider traffic reaching the local gateway. Relay cannot fully capture remote or cloud execution that bypasses the local host.
  • Persistent Codex and Claude Code integrations use each user’s configuration and a shared loopback gateway. Use nemo-relay run when a launch must retain project-specific configuration.
  • On Windows, a restrictive host Job Object can limit gateway reuse or prevent persistent bootstrap. A Job Object controls a group of processes. If it prevents a managed worker from leaving the group, the worker starts inside the group and emits worker_lifetime_limited. Closing the host can stop that worker. MCP clients can then need to reactivate the route after the broker’s recovery grace period. Refer to Register the MCP Lifecycle Client. During startup, Codex can request /models before required MCP servers start. Relay retries the request.
  • Codex 0.143 does not expose SessionEnd, and Codex multi-agent v2 encrypts delegated task payloads. Relay cannot recover the task text from encrypted payloads. Parent-child links depend on the agent identities and lifecycle events that Codex provides. Refer to Multi-Agent Delegation Traces.
  • Pi support is a proof of concept tested with 0.84.x. By default, its personal extension lets work continue if the gateway is unavailable. Later extensions can change arguments after Relay checks them. Unsupported provider routes produce no LLM spans. This includes Google’s native Gemini and Vertex APIs. To route Gemini model traffic through Relay, use a provider or proxy that accepts an OpenAI- or Anthropic-compatible API. Nested Pi processes appear as separate sessions. SIGINT in headless mode can stop Pi before queued events are sent. For details, refer to Pi Limitations.
  • Authenticated managed daemon routes use passthrough while workers start or recover unless the daemon uses --require-worker. Passthrough forwards requests without worker plugins. Failed starts retry automatically, but /healthz and MCP readiness alone do not prove policy enforcement or exporter activity. Verify worker status and an observable plugin result, or require a worker and handle HTTP 503 during unavailability. Refer to Daemon Operations.
  • Harness model requests do not read the per-user token file. Codex and Claude Code send the route credential header only from their own provider configuration, which they read at startup. Without that header, model requests pass through without Relay telemetry. The managed v1 Pi extension reads only NEMO_RELAY_CLIENT_TOKEN and fails to start without it, so nemo-relay doctor --managed-bundle rejects a file-only credential for Pi bundles. On Windows, nemo-relay daemon token ensure does not check the access list of an existing token directory. Refer to Credential Resolution.
  • Remote daemons and client workers must be able to reach each other. Proxies must support WebSockets. Loopback control connections do not send periodic pings. An idle hung process can go undetected until an operation times out.
  • The Node.js binding and package workflows require Node.js 24 or later.
  • OpenClaw provides telemetry through public hooks. Its security and optimization coverage is partial because Relay does not manage its execution.
  • The built-in nemo_guardrails plugin remains deprecated but is still available in 0.11.0. No replacement is included. The remote backend depends on its configured service’s availability, latency, and policy behavior. The local backend requires Python 3.11 or later and nemoguardrails==0.22.0. Refer to NeMo Guardrails.
  • The personally identifiable information (PII) redaction plugin supports its local backend based on fixed rules. Local-model backend configuration is reserved for future work.
  • Pricing and optimization estimates depend on model names, token data, pricing sources, and how current those sources are. Missing or conflicting data produces partial or absent cost fields rather than zero values.
  • ATOF stream sinks and remote ATIF storage require reachable, correctly configured destinations. A failed stream sink does not stop file output or other active sinks.
  • ATIF omits point-in-time marks. Use ATOF for the complete mark stream. The gen_ai OpenTelemetry projection also omits marks. The full and openinference projections retain their fixed native mark handling.
  • Native dynamic plugins run in the Relay process and are not sandboxed. A grpc-v1 worker runs in a separate process, but that process is not a security sandbox.
  • Do not change Python LLMRequest objects in place. Request middleware that changes content must return a new request object.
  • Native subscriber callbacks arrive asynchronously. Flush subscribers before depending on their changes, captured events, files, or exporter output.
  • OpenTelemetry endpoints use batch queues with a size limit. If a burst fills a queue, the endpoint drops completed spans instead of slowing the sender. This can leave an incomplete trace, including a missing root span. The SDK warns on the first drop and reports the exact dropped-span count during graceful shutdown. Configure max_queue_size, max_export_batch_size, and scheduled_delay_millis independently on each endpoint, or use the standard OTEL_BSP_* environment variables as defaults for the process. A larger queue does not guarantee lossless telemetry.
  • Operational logging covers Relay-managed hooks, gateway requests, the daemon, and workers. It does not yet cover every CLI command.
  • A parent handle outside the current scope stack does not keep its earlier W3C context. If it belongs to a different trace, managed-call headers can differ from the parent context kept by an exporter. Capture and import the parent’s propagation context before leaving that trace. Reusing only the handle does not preserve this context.
  • Codex’s built-in openai provider attempts a WebSocket connection before falling back to HTTP streaming. Relay does not support that transport, so a new session can report connection errors and take several seconds longer to start. Refer to Codex.
  • Available resource measurements depend on the platform. Initial rates can be unavailable, and copies of one polling sample under multiple Agent scopes must not be added together. Refer to Resource Metrics.
  • Default non-paused threshold compaction and unsupported or malformed Anthropic compaction traffic bypass the response cache. Selected compaction support does not let Relay replay every provider response. Refer to LLM Eligibility and Limits.

Previous Releases

For previous release notes, release artifacts, and the complete pull request history, refer to GitHub Releases.

For more information, refer to the following pages: