About NVIDIA NeMo Relay

Release Notes for NVIDIA NeMo Relay

View as Markdown

This page contains the release notes for NVIDIA NeMo Relay.

Release 0.7

NVIDIA NeMo Relay 0.7 adds asynchronous middleware, codec-aware observability sanitization, typed OpenTelemetry exporters, cross-boundary context propagation, and opt-in exact-match LLM response caching. It also expands CLI configuration, packaging, logging, and diagnostics workflows.

Highlights

The following highlights summarize the most significant user-facing changes in 0.7:

  • Asynchronous middleware and ordered event delivery: Rust middleware callbacks can await work directly, Python callbacks can return awaitables, and Node.js callbacks can return Promises. Scope and mark APIs remain synchronous while Relay publishes sanitized events through an ordered background dispatcher.
  • Codec-aware privacy controls: LLM sanitizers receive the active per-call codec across in-process, native-plugin, and worker boundaries. The PII redaction plugin can protect mixed OpenAI Chat, OpenAI Responses, and Anthropic Messages traffic without a fixed provider codec.
  • Unified OpenTelemetry export: One typed, multi-endpoint configuration supports full, gen_ai, and openinference projections. The gen_ai projection implements a pinned OpenTelemetry GenAI semantic-conventions v1.42-era snapshot.
  • Portable Relay context: Transport-neutral propagation can seed isolated scope stacks across process and runtime boundaries. Python applications can also fork the active scope stack for concurrent asyncio tasks.
  • Opt-in LLM response caching: The Adaptive plugin can serve eligible exact-match calls from in-memory or Redis stores. The cache remains disabled unless response_cache is configured.
  • CLI and operational improvements: The CLI adds structured configuration editing, PyPI distribution, size-based log rotation, and offline doctor probes. Python, Node.js, and Go bindings initialize default operational logging from the same environment controls as the CLI.

Support Matrix and Compatibility Updates

LLM sanitizer callbacks require migration in 0.7. Callbacks now receive the payload first and a directional context second, and they return an optional payload. There is no one-argument compatibility adapter. Rust worker mark, scope, tool, and LLM sanitizer callbacks are also asynchronous in 0.7. Native plugins and workers must be rebuilt with the matching 0.7 SDK and protocol definitions.

The Support Matrix is the canonical reference for supported platforms and architectures, worker runtimes, coding agents, and integrations. It also records current limitations, including platform-specific worker requirements.

Before upgrading from 0.6, follow the 0.7 Migration Guide. For the new callback contract and codec operations, refer to Codec-Aware LLM Sanitizers.

Rust callers that exhaustively match FlowError must handle the new CallbackException variant. It retains Python and JavaScript exception types for OpenTelemetry projection while preserving the existing internal_error classification.

Observability configuration version 3 is a breaking change. It removes the standalone openinference section and standalone OpenInference subscriber APIs, and requires a type and endpoint for every OpenTelemetry exporter. Legacy mark and attribute-alias controls remain available on typed full and openinference endpoints. Refer to Migration Guides for before-and-after configuration and binding API changes.

Each opentelemetry.endpoints entry can resolve sensitive header values from environment variables through header_env. Plugin activation rejects missing variables, blank values, and duplicate header names, then snapshots each valid value for the endpoint. Process-global OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS now fail construction or activation because their values cannot be isolated between endpoints.

The experimental Relay-owned Switchyard plugin is deprecated in 0.7. It is excluded from default CLI builds and is scheduled for removal in 0.8, when a Switchyard-owned native plugin is expected to replace it.

Fixed Known Issues in 0.7

The following known issues are fixed in 0.7:

  • Sanitizer callback errors, panics, rejected awaitables or Promises, and invalid return values now fail closed. Relay withholds the governed observability payload instead of emitting the original unsanitized value.
  • Relay removes known LLM credential headers before emitting request events, and the PII redaction plugin rejects malformed RFC 6901 target_paths before activation.
  • Generated coding-agent enforcement hooks now fail closed for transport, startup, authentication, evaluation, malformed-response, and policy-delivery failures. Lifecycle and after-the-fact hooks continue to fail open.
  • OpenTelemetry exporters now report finite batch-queue span loss through operational logs and plugin runtime diagnostics, including an exact dropped span count during graceful shutdown.
  • Typed OpenTelemetry configuration now rejects colliding destinations after canonical endpoint resolution, validates attribute mappings before activation, and completes bare OTLP HTTP endpoints with /v1/traces.
  • OpenTelemetry and OpenInference prefer response-side model attribution, preserve binding callback exception types, and map structured FlowError values to useful error.type attributes. OpenInference emits system content as ordered input messages.
  • ATIF no longer duplicates user steps for same-turn continuations. Model fallback matches the span exporters, and metadata-routing and remote-delivery failures remain visible and recoverable through flush and teardown.
  • Explicit configuration editor targets are preserved, file layers follow one documented priority model, disabled plugin entries do not contribute configuration, and programmatic component enablement takes precedence over discovered files.
  • Library initialization reports inherited plugins.toml files without exposing their values. nemo-relay doctor reports invalid explicit and logging configuration, supports offline exporter validation, and reports effective upstream authentication presence without exposing credentials.
  • Transparent gateways preserve non-success streaming provider responses and have bounded, interruptible shutdown behavior.
  • Node.js preserves asynchronous propagation context, includes queued callback delivery in flush completion, and isolates subscriber exceptions instead of aborting the process.
  • Python plugin teardown is safe from a running asyncio loop, cancelled scopes are reported as errors, and completion callbacks no longer target a closed event loop during cancellation.
  • Prebuilt CLI binaries carry the intended release version, so an upgraded CLI does not silently reuse a gateway started by a differently identified build.
  • object_store is upgraded to 0.14.1, which removes the temporary RUSTSEC-2026-0194 and RUSTSEC-2026-0195 advisory exceptions.
  • Embedded Python hosts can use plugin.load_dynamic_plugin_activation_specs(path) to convert the standard [[plugins.dynamic]] records in one explicitly selected plugins.toml into the activation specs accepted by initialize_with_dynamic_plugins(). This scoped 0.7 compatibility helper removes host-side TOML and manifest parsing; a future unified file-backed initializer is expected to replace it.
  • Programmatically declared plugin components now apply their enabled value over discovered file configuration. When code re-enables a component that a discovered file disabled, initialization reports a warning that names the component and source file.
  • Library plugin initialization now emits a warning for each discovered plugins.toml file inherited by the caller configuration. The warning is available in operational logs, initialization results, and the active plugin report without exposing configuration values or credentials.
  • LLM payload redaction now follows the codec active for each call instead of a codec captured from plugin configuration. Codec-dependent policies omit the observability payload and annotation when Relay cannot safely normalize the payload. This change resolves the mixed-provider PII sanitization issue #526.

Known Issues in 0.7

  • Go and the raw C FFI remain experimental and source-first. Generated API pages focus on Rust, Python, and Node.js.
  • Local coding-agent observability depends on host hooks and provider traffic reaching the local gateway. Relay cannot fully capture remote or cloud execution that bypasses the local host.
  • Persistent Codex, Claude Code, and Hermes integrations use user-scoped configuration and a shared loopback gateway. Use nemo-relay run when a launch must retain project-specific configuration.
  • On Windows, a restrictive host Job Object can limit gateway reuse or prevent persistent bootstrap. Codex can also make a cold-start /models request before required MCP servers start; Relay retries the request.
  • Codex 0.143 does not expose SessionEnd, and Codex multi-agent v2 encrypts delegated-task payloads that Relay cannot decrypt or reliably link.
  • The Node.js binding and package workflows require Node.js 24 or later.
  • The deprecated Switchyard plugin is experimental, excluded from default CLI builds, and requires a separately managed compatible service and a named authenticated ATOF HTTP stream sink. It is scheduled for removal in 0.8.
  • OpenClaw has public hook-backed telemetry. Its security and optimization coverage is partial because it does not own a managed execution path.
  • The NeMo Guardrails remote backend inherits its configured service’s availability, latency, and policy behavior. The local backend requires Python 3.11 or later and nemoguardrails==0.22.0.
  • The PII redaction plugin currently supports its deterministic local backend; local-model backend configuration is reserved for future work.
  • Pricing and optimization estimates depend on model names, token data, pricing sources, and freshness evidence. Missing or inconsistent evidence produces partial or absent cost fields rather than zero values.
  • ATOF stream sinks and remote ATIF storage require reachable, correctly configured destinations. A failed stream sink does not stop file output or other active sinks.
  • ATIF omits point-in-time marks. Use ATOF for the canonical mark stream; the gen_ai OpenTelemetry projection also omits marks. The full and openinference projections retain their fixed native mark handling.
  • Native dynamic plugins run in the Relay process and are not sandboxed. A grpc-v1 worker runs in a separate process, but that process is not a security sandbox.
  • Treat Python LLMRequest objects as immutable. Request middleware that changes content must return a new request object.
  • Native subscriber callbacks arrive asynchronously. Flush subscribers before depending on their side effects, captured events, files, or exporter output.
  • OpenTelemetry endpoints use finite batch queues. A burst that fills an endpoint queue drops completed spans without applying backpressure and can leave an incomplete trace, including a missing root span. The SDK warns on the first drop and reports the exact dropped-span count during graceful shutdown. Increase OTEL_BSP_MAX_QUEUE_SIZE before plugin activation to reduce the risk for a known burst size; endpoint-specific batch sizing is not available, and a larger finite queue does not guarantee lossless telemetry.
  • Operational logging configuration and sink lifecycle are available, but broad operational log coverage across commands is not yet available.

Previous Releases

For previous release notes, release artifacts, and the complete PR-by-PR history, refer to GitHub Releases.