About NVIDIA NeMo Relay

Release Notes for NVIDIA NeMo Relay

View as Markdown

This page contains the release notes for NVIDIA NeMo Relay.

Release 0.8

NeMo Relay 0.8 changes the typed native Rust SDK’s guardrail, sanitizer, and intercept callbacks to return futures. The SDK runs these futures on one plugin-owned Tokio executor and provides async, cloneable unary and streaming continuations. This is a source-breaking change for typed native Rust plugins; rebuild them with nemo-relay-plugin 0.8.0 and declare compat.relay = ">=0.8.0,<1.0".

Native ABI v4 adds completion-scoped codec operations and pull-based LLM stream continuations. Relay also negotiates the frozen ABI v3 and v2 table layouts for Relay 0.8-built plugins that target them, and native subscribers remain synchronous. Follow the native Rust migration guide for callback and executor changes.

Managed and manual tool execution now uses the canonical ToolExecutionResult shape across Rust, Python, Node.js, Go, the public C surface, native plugins, and grpc-v1 workers. The required result remains application-owned, while an optional opaque annotation travels beside it and is projected consistently into sanitized observability.

Highlights

The following updates expand native Rust plugin capabilities:

  • Typed native Rust middleware can await Tokio timers, I/O, codecs, unary continuations, and downstream LLM streams on an SDK-owned executor.
  • Native ABI v4 provides completion-scoped codec operations and pull-based LLM stream continuations.
  • Tool callbacks and execution-intercept continuations share one canonical result contract with an optional opaque annotation.
  • ATOF, ATIF, and the full and openinference OpenTelemetry projections preserve sanitized tool result annotations without flattening their application-defined schemas.

Support Matrix and Compatibility Updates

The Support Matrix is the canonical reference for supported platforms and architectures, worker runtimes, coding agents, and integrations. It also records current limitations, including platform-specific worker requirements.

Migration guidance for upgrading from 0.7 to 0.8 is available in the Migration Guides.

Breaking Changes

  • Typed native Rust middleware callbacks now return futures and receive owned arguments. Native subscribers and raw synchronous ABI entry points are unchanged.
  • Managed tool callbacks, execution-intercept continuations, execute-helper returns, and manual tool end APIs now use ToolExecutionResult instead of a raw JSON result. Applications that need the original business payload must read its result field. Refer to the tool result migration guide.
  • Relay 0.8 resets native API 1 and the grpc-v1 tool-result contract to the canonical ToolExecutionResult shape. Rebuild every native and worker plugin and declare a compat.relay range that excludes Relay versions before 0.8. The native ABI v4 table remains unchanged. The worker package and RPC method names remain nemo.relay.worker.v1, but the ToolNext response and tool-execution outcome field now use structured protobuf messages. Regenerate custom worker bindings before rebuilding.
  • Repository-local .nemo-relay/config.toml, plugins.toml, and .dynamic-plugins.json files are no longer discovered or activated. Runtime resolution is explicit-or-user followed by higher-precedence system policy.
  • Project setup scopes and the --project and config --reset --scope interfaces have been removed. Setup and reset now target XDG user configuration; --user, --global, --config, and --plugin-config-path remain supported.
  • nemo-relay doctor reports ignored ancestor project configuration as warning-only migration diagnostics. Relay does not automatically move, rewrite, or delete those legacy files.
  • Removed Hermes-specific support from the NeMo Relay CLI, including the hermes shortcut, run --agent hermes, and Hermes-specific install, uninstall, doctor, configuration, MCP selection, and /hooks/hermes paths. NeMo Relay is built into Hermes Agent, and Hermes Agent understands NeMo Relay plugin configurations. No separate observability plugin or Relay CLI setup is required.

Refer to Migration Guides for destination paths and explicit-file alternatives.

Fixed Known Issues in 0.8

  • Restored a trailing / on an OTLP/HTTP trace endpoint as an explicit root-path destination. A bare HTTP authority still defaults to /v1/traces; when version 4 implicitly derives log or metric destinations from either form, it uses /v1/logs or /v1/metrics respectively. Explicit trace, log, and metric endpoint paths remain unchanged.

Known Issues in 0.8

  • Go and the raw C FFI remain experimental and source-first. Generated API pages focus on Rust, Python, and Node.js.
  • Local coding-agent observability depends on host hooks and provider traffic reaching the local gateway. Relay cannot fully capture remote or cloud execution that bypasses the local host.
  • Persistent Codex and Claude Code integrations use user-scoped configuration and a shared loopback gateway. Use nemo-relay run when a launch must retain project-specific configuration.
  • On Windows, a restrictive host Job Object can limit gateway reuse or prevent persistent bootstrap. Codex can also make a cold-start /models request before required MCP servers start; Relay retries the request.
  • Codex 0.143 does not expose SessionEnd, and Codex multi-agent v2 encrypts delegated-task payloads that Relay cannot decrypt or reliably link.
  • The Node.js binding and package workflows require Node.js 24 or later.
  • The deprecated Switchyard plugin is experimental, excluded from default CLI builds, and requires a separately managed compatible service and a named authenticated ATOF HTTP stream sink. It is scheduled for removal in 0.8.
  • OpenClaw has public hook-backed telemetry. Its security and optimization coverage is partial because it does not own a managed execution path.
  • The built-in nemo_guardrails plugin is deprecated and scheduled for removal in NeMo Relay 0.9. It remains available in 0.8: the remote backend inherits its configured service’s availability, latency, and policy behavior, and the local backend requires Python 3.11 or later and nemoguardrails==0.22.0. A replacement is not included in 0.8 and will target 0.9 or later. Removal will include the built-in component kind, the public nemo_relay::plugins::nemo_guardrails Rust module, its CLI editor entry, and the guardrails-remote Cargo feature.
  • The PII redaction plugin currently supports its deterministic local backend; local-model backend configuration is reserved for future work.
  • Pricing and optimization estimates depend on model names, token data, pricing sources, and freshness evidence. Missing or inconsistent evidence produces partial or absent cost fields rather than zero values.
  • ATOF stream sinks and remote ATIF storage require reachable, correctly configured destinations. A failed stream sink does not stop file output or other active sinks.
  • ATIF omits point-in-time marks. Use ATOF for the canonical mark stream; the gen_ai OpenTelemetry projection also omits marks. The full and openinference projections retain their fixed native mark handling.
  • Native dynamic plugins run in the Relay process and are not sandboxed. A grpc-v1 worker runs in a separate process, but that process is not a security sandbox.
  • Treat Python LLMRequest objects as immutable. Request middleware that changes content must return a new request object.
  • Native subscriber callbacks arrive asynchronously. Flush subscribers before depending on their side effects, captured events, files, or exporter output.
  • OpenTelemetry endpoints use finite batch queues. A burst that fills an endpoint queue drops completed spans without applying backpressure and can leave an incomplete trace, including a missing root span. The SDK warns on the first drop and reports the exact dropped-span count during graceful shutdown. Configure max_queue_size, max_export_batch_size, and scheduled_delay_millis independently on each endpoint, or use the standard OTEL_BSP_* environment variables as process-wide fallbacks. A larger finite queue does not guarantee lossless telemetry.
  • Operational logging configuration and sink lifecycle are available, but broad operational log coverage across commands is not yet available.

Previous Releases

For previous release notes, release artifacts, and the complete PR-by-PR history, refer to GitHub Releases.