About NVIDIA NeMo RelayRelease Notes

Known Issues

View as Markdown

Use this page to check whether a current limitation affects your deployment and to plan an upgrade from an earlier release. It also keeps the complete list of fixes recorded in prior release-note pages.

NVIDIA NeMo Relay 0.6

The following limitations and migration steps apply to NVIDIA NeMo Relay 0.6.

Current Limitations and Support Notes

  • Go and the raw C FFI remain experimental and source-first. Generated API pages focus on Rust, Python, and Node.js, and the new C and Go dynamic-plugin activation entry points are not ready for production use.
  • The NeMo Relay CLI remains experimental. Local coding-agent observability depends on host hooks and provider traffic reaching the local gateway. Relay cannot fully capture remote or cloud execution that bypasses the machine.
  • Persistent Codex, Claude Code, and Hermes integrations share a loopback gateway and user-scoped Relay configuration. Use nemo-relay run when one launch must keep its project configuration isolated.
  • On Windows, a restrictive host Job Object can prevent a coding-agent plugin’s Relay gateway from breaking away. When the host permits nested assignment, the gateway remains scoped to the host job, so its 300-second idle reuse window can end early. If the host rejects nested assignment, persistent bootstrap stops and reports the conflict.
  • Codex can request /models during provider discovery before it launches the required plugin MCP servers. Codex retries this cold-start request if it fails. The captured turn and /responses traffic still wait for verified Relay gateway readiness.
  • The Codex 0.143 plugin hook schema does not expose SessionEnd. Relay updates the cumulative ATIF snapshot after each delivered Stop, so the final Stop captures the full session.
  • Codex multi-agent v2 encrypts inter-agent message payloads. Relay can observe parent and child activity but cannot decrypt delegated task text or reliably link their traces. Managed sessions request the readable legacy path, but higher-precedence configuration or the selected model can still choose v2. Check the captured payload before relying on readable delegation text.
  • Node.js binding and package workflows require Node.js 24 or newer.
  • The Switchyard integration is experimental, is not part of default CLI builds, and needs a separately managed compatible service. ATOF-backed profiles also need one named, authenticated HTTP stream sink. A local file sink does not populate the Switchyard accumulator.
  • OpenClaw uses public hook-backed telemetry, so its security and optimization coverage is partial. Security stops at pre-tool conditional guardrails, and optimization stops at adaptive telemetry unless the integration owns a managed execution path.
  • The NeMo Guardrails remote backend inherits the availability, latency, and policy behavior of its configured service. The local backend requires Python 3.11 or newer and nemoguardrails==0.22.0.
  • The PII redaction plugin currently supports its deterministic local backend. Local-model backend configuration is reserved for future work.
  • Pricing and optimization estimates are only as complete as the model name, token data, pricing source, and freshness evidence that Relay receives. When evidence is missing or inconsistent, cost fields remain partial or absent rather than defaulting to zero.
  • ATOF stream sinks and remote ATIF storage need reachable, correctly configured destinations. If one ATOF stream sink fails, file output and the other active sinks continue.
  • ATIF models steps and leaves out point-in-time marks. Use ATOF when you need the canonical mark stream. Tool-style mark projection in OpenTelemetry and OpenInference is an opt-in visualization aid, not a replacement event format.
  • Native dynamic plugins run in the Relay process and are not sandboxed. A grpc-v1 worker runs in a separate process, but that process is not a security sandbox.
  • Treat Python LLMRequest objects as immutable. Request middleware that changes content should return a new request object.
  • Native subscriber callbacks arrive asynchronously. Flush subscribers before you depend on callback side effects, captured events, files, or exporter output. Deregistration stops future emissions, but callbacks that were already queued can still run.
  • Version 0.6 adds operational logging configuration and sink lifecycle, but doesn’t yet provide broad operational log coverage across commands.

Compatibility and Migration Notes

Update Managed LLM Stream Consumers

Managed LLM streams now have one explicit close contract across bindings. When you stop consuming a stream before natural exhaustion, close it so Relay can stop the producer, finalize the partial response, and emit the interrupted end event. Natural exhaustion is unchanged.

Rust LlmJsonStream is no longer the former boxed-stream type alias. Replace direct construction and alias-dependent type usage with LlmJsonStream::new or LlmJsonStream::from_closeable; call stream.close().await on early exit. Go LlmStream.Close now returns an error, so update call sites to handle its result. Python keeps await stream.aclose(), Node.js provides await stream.close(), and C callers can call nemo_relay_stream_close before nemo_relay_stream_free.

The Rust Library Reference documents the current Rust API. Go and raw C FFI remain experimental and source-first. PR #465 introduced the unified close behavior.

Reinstall Coding-Agent Integrations

The installed integration format changed. Existing Codex, Claude Code, and Hermes integrations do not automatically pick up the MCP-owned gateway, agent-owned hooks, generation fencing, or user-scoped configuration.

Run nemo-relay install <agent> --force for each installed integration, or run nemo-relay install all --force. Otherwise, integration state generated by 0.5 can fail the 0.6 readiness and doctor checks.

Follow the installation and doctor workflow in Plugin Installation. PR #395 introduced the new lifecycle.

Migrate ATOF Configuration and Manual Exporters

ATOF configuration that uses the 0.5 schema no longer validates. Set components.config.version = 2, then replace the legacy output_directory, filename, mode, and endpoints fields with a tagged atof.sinks list. Relay rejects the old fields instead of translating them.

If you construct an ATOF exporter directly, pass one typed file or stream sink. A stream-backed exporter no longer exposes a file path.

The ATOF Export guide shows the version 2 configuration and current binding examples. PR #416 introduced the multi-sink API and configuration.

Update OTLP Attribute Queries

OpenTelemetry and OpenInference no longer emit raw *_json event-payload attributes. Move trace queries, dashboards, and processors to the typed top-level paths. If a backend still needs an older key, configure an attribute_mappings alias. Nested values remain JSON strings.

Use the OpenTelemetry Export and OpenInference Export guides to find the current attribute paths. PR #414 changed the attribute schema.

Move Mark Consumers to ATOF or Trace Projection

ATIF no longer turns Relay marks into synthetic system steps or populates AtifStepExtra.event_payload for them. Use ATOF when you need the canonical mark stream. If a trace viewer needs marks as visible tree nodes, set mark_projection = "tool" for OpenTelemetry or OpenInference and review the default llm.chunk exclusion.

The ATIF Export and ATOF Export guides explain the format boundary and canonical mark stream. Use the OpenTelemetry or OpenInference guide to configure trace projection. PR #382 changed the ATIF behavior and added trace projection.

Recheck ATIF Model Attribution

ATIF now uses the effective model from the normalized provider response for step.model_name. That value can differ from the requested or profile model for routed and cross-protocol calls.

Audit any consumer that groups, compares, or prices trajectories by step.model_name. When the response does not identify a model, Relay still uses the previous fallback.

The ATIF Export guide explains the trajectory semantics. PR #399 changed the model attribution.

Treat LLM Event Histories as Bounded Observations

Do not treat every LLM start event as a complete replay log. After the first LLM start records an agent’s complete history, later event inputs and annotations can omit earlier turns. A new agent and the first start after a compaction mark keep the full sanitized history; later starts keep the system instructions, latest user turn, and messages that follow it.

This projection changes observability events only. Relay sends the original request to the provider.

The Events guide explains the freshness and scope-ownership rules. PR #398 introduced the bounded-history behavior.

Handle Sanitizer Failures Deliberately

A runtime error from a sanitizer now drops the affected event. Subscribers and exporters can therefore have a gap instead of receiving unsanitized data.

If your policy requires fail-open behavior, catch the error inside the sanitizer and return the fields that are safe to emit.

The Event Sanitizers guide explains registration, lifetime, and field replacement. PR #371 made sanitizer failures fail-closed.

Update Rust Editor Metadata Consumers

If your Rust code consumes editor metadata, handle EditorFieldKind::List and EditorFieldKind::TaggedUnion in exhaustive matches. Direct EditorFieldSpec literals must also initialize the required list_item and tagged_union fields.

Schema consumers should now expect typed collections such as learners, config_ids, and target_paths to report EditorFieldKind::List instead of EditorFieldKind::Json. Serialized plugin configuration remains compatible.

The Rust Library Reference lists the current metadata types, and Plugin Configuration Files explains the interactive editor workflow. PR #410 changed the Rust metadata API.

Update Rust Runtime Matches and Struct Construction

Add FlowError::Upstream to downstream exhaustive matches. Replace direct mark and ATOF struct literals with EmitMarkEventParams::builder() and the current typed sink constructors when possible. If you keep direct literals, initialize the new mark data_schema, category, and category_profile fields and the ATOF stream header_env field. The version 2 ATOF migration above supersedes legacy AtofEndpointSectionConfig construction.

The Rust Library Reference lists the current public types, and Adding Scopes and Marks shows builder-based mark emission. PR #369 introduced the Rust source-compatibility change.

Update Public Skill Paths

Update any automation or agent instructions that open 0.5 skill directories by name. Use nemo-relay-install and nemo-relay-get-started for setup; nemo-relay-instrument-context-isolation and nemo-relay-instrument-typed-wrappers for application instrumentation; nemo-relay-plugin-build for plugin creation; and the consolidated nemo-relay-plugin-observability and nemo-relay-plugin-adaptive-tuning skills for exporter and adaptive workflows.

The NeMo Relay User Skills catalog lists the current paths and task routing. PR #407 reorganized the catalog.

Fixed in NVIDIA NeMo Relay 0.6

  • Installed coding-agent sessions now acquire the shared gateway through MCP before hooks or routed provider traffic. This prevents cold-start loss and coordinates concurrent startup and recovery.
  • Transparent Codex routing now recognizes at-... access tokens and sends them to the ChatGPT Codex backend without rewriting unrelated bearer tokens or provider API keys.
  • Relay now asks Codex to prefer the readable legacy multi-agent path during managed runs, then restores the user’s previous setting during uninstall.
  • ATIF now reports the normalized provider response model when a routed or translated call runs on a different model from the request.
  • The NeMo Flow migration skill now skips credential-bearing dotenv files and symbolic links, asks for exact project-root confirmation, and refuses write mode for filesystem roots and home directories.

Fixed in Prior Releases

NVIDIA NeMo Relay 0.5

  • Gateway shutdown flushes queued events before it exits.
  • Dynamic worker invocations support cancellation, and the CLI manages Python dynamic-plugin environments.
  • ATOF file export creates a missing output directory before opening JSONL output.
  • CLI status output reports daemon binding, exporter status, and remote ATIF storage destinations more reliably.
  • Codex host-plugin discovery uses the supported text output from the Codex plugin command.

NVIDIA NeMo Relay 0.4

  • ATIF shutdown no longer deadlocks queued subscribers.
  • Sanitized LLM requests are resolved from annotations for observability output.
  • Structured ATIF tool results and Hermes tool-result observations are preserved more reliably.
  • Hermes routed-provider spans, wrapped ATIF fidelity, subagent lineage, and error-path export consistency are covered and corrected.
  • OpenClaw observability output is more consistent for nested subagents, model timing diagnostics, hook-backed provenance, placeholder replay, and hook-only fallback exports.
  • LangChain serialization handles wrapped integration payloads more reliably.
  • Plugin context-manager teardown avoids the previous deadlock path.
  • Node.js withScope callbacks receive a real ScopeHandle.
  • Deep Agents model responses are annotated for downstream observability.

NVIDIA NeMo Relay 0.3

  • Managed LLM start events are emitted before execution intercepts.
  • Coding-agent trace scopes are aligned with NeMo Relay agent scope semantics.
  • ATIF tool observations are correlated with their matching tool calls.
  • OpenClaw tool call replay visibility is preserved.
  • LangChain serialization handles wrapped integration payloads more reliably.

Earlier Releases

  • Enabled TLS support for OTLP HTTP export.
  • Preserved Go scope stacks across OS threads.

For complete PR-by-PR history and release artifacts, refer to GitHub Releases.