> This page is for NeMo Relay, version v0.8.4. > For other versions, use one of these documentation indexes: > - Latest (v0.10.0) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt > - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt > - v0.10.0: https://docs.nvidia.com/nemo/relay/v0.10.0/llms.txt > - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt > - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt > - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt > - v0.6.0: https://docs.nvidia.com/nemo/relay/v0.6.0/llms.txt > - v0.5.0: https://docs.nvidia.com/nemo/relay/v0.5.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Subscribers > Understand NeMo Relay subscribers, delivery behavior, scope visibility, and exporter roles. This page explains how subscribers consume lifecycle events without changing runtime execution. ## What Subscribers Are Subscribers are consumers of the NeMo Relay event stream. They receive emitted lifecycle events and use them for observation, forwarding, export, or analysis. In the native Rust, Python, Node.js, and FFI bindings, event-producing calls enqueue subscriber delivery on a process-wide background dispatcher and return without waiting for subscriber callbacks or exporter work. ## How Subscribers Relate to Events Events describe what happened. Subscribers are the components that watch those events. In this documentation, an event is **emitted** when the runtime submits it for subscriber dispatch. An event is **delivered** to a subscriber when its callback runs. An event is **exported** when an exporter completes the downstream work defined by its API. On native targets, these are separate milestones. That separation matters: * The runtime can emit one ATOF event stream. * Native event calls stay non-blocking for subscriber work. * Many subscribers can consume the same stream. * Observability behavior stays separate from call execution. ## Registration Levels Middleware and subscribers can be registered at different levels depending on their lifetime and visibility. ### Global Subscribers Global subscribers remain active process-wide until they are removed. ### Scope-Local Subscribers Scope-local subscribers are owned by one active scope and disappear when that scope closes. Deregistering a subscriber affects future emissions. Events that were already emitted carry a subscriber snapshot, so queued callbacks from that snapshot can still run after deregistration. ### Plugin-Installed Subscribers Plugins can install subscribers as reusable, configuration-driven runtime components. ## What Subscribers Consume Subscribers consume the ATOF event stream. They do not define the event model. They react to it. This lets plain subscribers, exporters, and tracing adapters share one runtime source of truth. ### ATOF Events and ATIF Trajectories ATOF and ATIF describe different stages of the observability path: | Format | What It Represents | What Subscribers or Exporters Do With It | | ------ | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | ATOF | The canonical raw scope, tool, LLM, middleware, and mark event stream | Subscribers receive these events directly. The ATOF JSONL exporter writes one raw event per line. | | ATIF | A trajectory projection assembled from related ATOF events | The ATIF exporter maps model and tool activity into trajectory steps for analysis, replay, or evaluation. | Marks are point-in-time ATOF events, not ATIF trajectory steps, so ATIF `steps` are not a lossless event log. ATIF is designed primarily for evaluation and visualization. Its steps do not represent the complete parent-child graph needed for perfect session replay. This limitation comes from the ATIF data model, not from converting ATOF events. Plugin-managed ATIF files preserve nested scope lineage and the raw events associated with the trajectory under `extra.observed_events`, including observed marks, for analysis and debugging. Those fields do not change the ATIF step model or make `steps` a complete replay log. Refer to [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) for the complete projection and file contract. ## Common Subscriber Roles Subscribers are commonly used for in-process observation, counters, debugging, and exporter handoff. ### In-Process Observation Some subscribers stay inside the process and power custom logging, analytics, or debugging logic. #### Host Integration Event JSON For host integrations that need a serialized event payload, use the event object's JSON serialization helpers instead of reconstructing payloads from native attributes. Python subscribers can call `event.to_dict()` or `event.to_json()` from the callback, while continuing to use the normal subscriber registration API. This pattern is useful when an agent runtime, framework adapter, or plugin host already has its own lifecycle hooks but wants NeMo Relay to be the shared ATOF event representation. The host integration maps those hooks into NeMo Relay scopes, LLM calls, tool calls, or marks. NeMo Relay emits the ATOF event stream, and each subscriber chooses whether to consume the native event object, serialized event JSON, or an exporter-specific translation. ```mermaid flowchart Host[Host Integration] subgraph NeMoFlow[NeMo Relay] direction TB Binding[Binding API] Core[Rust Core Runtime] Events[Canonical ATOF Event Stream] Dispatcher[Async Subscriber Dispatcher] Observer[In-Process Subscriber] Json[Canonical Event JSON] Exporters[Exporter Subscribers] Backends[JSONL / ATIF / OTLP] Binding -->|emits scopes, tools, LLMs, marks| Core Core --> Events Events --> Dispatcher Dispatcher --> Observer Observer -->|to_dict / to_json / JSON| Json Dispatcher --> Exporters Exporters --> Backends end Host -->|maps lifecycle hooks| Binding Json -. host consumes canonical telemetry .-> Host ``` Subscribers do not define the event schema. They receive the runtime event and can serialize it through a binding helper when they need JSON. Exporter subscribers, such as the ATOF JSONL exporter, consume the same event stream and serialize it for their target backend. Native subscribers are invoked by one process-wide worker thread in FIFO event order and subscriber snapshot order. > **Warning** > > Subscriber callbacks are not re-entrant. Do not invoke a NeMo Relay API that > runs middleware, flushes subscriber delivery, waits on an exporter, or clears > plugins from a subscriber callback. Scope APIs remain supported: callbacks may > create, push, or pop scopes at any nesting level and may replace the active > scope stack with an arbitrary stack. Emitting a new event is the only > supported operation that can enqueue additional callback work; Relay queues > that event behind the active publication instead of recursively delivering it. ## Waiting for Delivery An event-producing call returning is not a delivery barrier. These guarantees apply when you call a barrier outside a native subscriber callback. Use the following milestones to choose the barrier that matches the output you need to observe: | Milestone | What It Establishes | What It Does Not Establish | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Event-producing call returns | The runtime call completed. On native targets, the runtime normally queues subscriber work for background dispatch. | Subscriber callbacks ran, exporter work completed, or output is visible or durable. | | Subscriber flush returns | All native subscriber callbacks queued before the flush call have completed. | Exporter-owned workers drained or a remote backend accepted the data. | | Exporter barrier returns | The runtime drained the shared subscriber queue, and the exporter completed its documented `force_flush()`, `export()`, or equivalent operation. For a worker-backed exporter, that operation can stop after its configured timeout. | Exactly-once delivery, machine-crash durability, confirmed completion by a worker that timed out, or success beyond the exporter's documented error and timeout behavior. | | Plugin clear or exporter shutdown returns | NeMo Relay completed the component's documented graceful teardown steps, or reported an error. | Retention if the process terminates before teardown completes or completion by a worker that timed out during teardown. | | Process terminates before a barrier | You can observe only callback side effects and exporter work that completed before termination. | Retention of queued subscriber callbacks or pending exporter work. | Use the subscriber flush API when application shutdown, tests, or examples must observe side effects from callbacks that were already queued before the barrier: * Rust: `nemo_relay::api::subscriber::flush_subscribers()?` * Python: `nemo_relay.subscribers.flush()` from synchronous code, or `await nemo_relay.subscribers.flush_async()` from an `asyncio` task * Node.js: `await flushSubscribers()`, then await an event-loop tick for JavaScript callback side effects * FFI: `nemo_relay_flush_subscribers()` Events emitted concurrently after the flush barrier are not covered by that call. Exporters with their own workers or batch processors require the exporter-specific flush, export, or shutdown operation described in [Observability](/configure-plugins/observability/about). > **Warning** > > Do not invoke a subscriber flush, an exporter barrier, or plugin clear from a > subscriber, event-sanitizer, conditional-guardrail, request-intercept, or > execution-intercept callback. These operations are not supported there and do > not establish a valid delivery barrier; depending on the binding or exporter, > they can also create a wait cycle. Run them after the callback returns. > > Queued tool and LLM observability sanitizers are a narrow exception: Python and > Node.js permit a subscriber flush call there, but it returns without waiting > for the sanitizer's own publication. > > If the process terminates before subscriber delivery and exporter teardown > complete, queued telemetry can be lost. Returning from the event-producing API > does not make that telemetry crash-safe. ### Forwarding and Export Some subscribers translate the event stream into external formats or transport it to another system. ### Analytics and Diagnostics Some subscribers derive measurements, trajectories, or diagnostics from the event stream without affecting execution behavior. ## Built-In Subscriber Examples These examples show how built-in subscriber patterns relate to custom subscribers and exporters. ### Custom Subscribers A plain custom subscriber is the right choice when you want in-process handling of the canonical event stream. ### Agent Trajectory Interchange Format (ATIF) Exporter The [Agent Trajectory Interchange Format (ATIF) exporter](/configure-plugins/observability/atif) collects lifecycle events and emits trajectory artifacts for offline analysis, replay, or debugging. ### Agent Trajectory Observability Format (ATOF) JSONL Exporter The [Agent Trajectory Observability Format (ATOF) JSONL exporter](/configure-plugins/observability/atof) writes the canonical event stream to a native filesystem path as one raw ATOF event per line. ### OpenTelemetry Subscribers The trace subscriber maps runtime events into OTLP traces. Each trace subscriber uses one `full`, `gen_ai`, or `openinference` projection. Select `openinference` for model-centric OpenInference semantics; there is no separate OpenInference subscriber. Independent log and metric subscribers consume the same sanitized event stream. The log subscriber exports non-metric marks with typed severity and scope correlation. Valid marks that use the Relay-owned `nemo.relay.metric_measurements` schema route exclusively to the metric subscriber and are not projected as traces or logs. Invalid reserved metric marks reach no OTLP signal and produce an operational diagnostic. Scope events maintain log lineage but do not become logs or metrics. Operational stderr and file records are not input to either telemetry subscriber. Each direct trace, log, and metric subscriber also exposes a bounded runtime diagnostics snapshot. Rust and Python use `runtime_diagnostics()`, Node.js uses `runtimeDiagnostics()`. The C FFI exposes the corresponding `*_runtime_diagnostics_json` APIs, which return a caller-owned JSON array released with `nemo_relay_string_free`. These snapshots report stable codes, occurrence counts, and the most recent message for delivery and event-processing failures. See [OpenTelemetry](/configure-plugins/observability/opentelemetry#direct-subscribers) for the complete API and lifecycle contract. Detailed setup, configuration, and API behavior for these subscribers belongs in [Observability](/configure-plugins/observability/about). For configuration-driven setup, use the built-in [`observability` plugin](/configure-plugins/observability/configuration) to install ATOF, ATIF, and typed OpenTelemetry exporters from one plugin component. ## Practical Guidance Use these practices when applying the concept in application or integration code. * Use a plain subscriber when you want in-process custom behavior. * Flush subscribers before inspecting printed output, captured lists, or other custom callback side effects. * Use the exporter's documented barrier before inspecting exporter output. * Clear plugin-managed exporters during graceful shutdown. * Use `event.to_dict()` or `event.to_json()` when a host runtime or exporter needs the ATOF event as JSON in process. * Use a scope-local subscriber when the observation should disappear with the owning scope. * Use a plugin-installed subscriber when the behavior should be reusable and config-driven. * Use an exporter-oriented subscriber when the event stream should leave the process.