> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Observability

> Configure NeMo Relay observability exporters for events, traces, and trajectories.

Use the Observability plugin when you need to inspect NeMo Relay lifecycle events
in process or export agent activity to tracing, trajectory, or analysis
systems from one plugin configuration document.

## Shortest Path

Start with one plugin-managed exporter before layering in additional formats or
manual subscribers. A common first path is:

1. Instrument one scope, tool call, or LLM call in your application.
2. Enable the Observability plugin with only one exporter, such as ATOF JSONL.
3. Run one request through the instrumented path.
4. Confirm the emitted output before adding more exporters or sanitization rules.

Observability in NeMo Relay starts with events. Scopes, marks, managed tool
calls, managed LLM calls, middleware, and manual lifecycle APIs emit the
canonical Agent Trajectory Observability Format (ATOF) event stream.
Subscribers consume that stream in process, and exporter-oriented subscribers
write raw ATOF JSONL or translate events into Agent Trajectory Interchange
Format (ATIF), OpenTelemetry, or OpenInference.

That makes observability downstream from the runtime contract:

* Scopes decide ownership and parentage.
* Middleware decides whether execution continues or what payloads are sanitized.
* Codecs provide normalized request and response data when available.
* Events record the canonical runtime stream.
* Subscribers and exporters consume or project that stream.

Exporters should not redefine execution semantics. They should preserve the
meaning of the underlying events while translating them into the shape expected
by the downstream system.

Configuration and activation failures are setup failures for the observability
component. Delivery failures after activation are downstream exporter failures:
application work continues, and explicit exporter barriers such as flush or
shutdown can report stored delivery problems. Refer to
[Observability Configuration](/configure-plugins/observability/configuration) for the
failure table.

The first-party plugin component has kind `observability`. It can install:

* Agent Trajectory Observability Format (ATOF) JSONL export for raw lifecycle events.
* Agent Trajectory Interchange Format (ATIF) trajectory export for each top-level
  Agent scope or supported coding-agent turn scope.
* OpenTelemetry OTLP trace export.
* OpenInference-oriented OTLP trace export.

## Plugin-Managed Versus Manual Export

Use the Observability plugin for process-level exporter setup that should be
activated from config, `plugins.toml`, or a shared plugin document.

Use manual subscriber or exporter APIs when a test, script, or application
needs direct control over registration names, collection windows, explicit
flush timing, or per-run exporter objects. The plugin owns subscriber names and
teardown for the sections it enables.

Both paths consume the same runtime event contract. Choose plugin-managed export
for reusable process configuration, and choose manual APIs only when the caller
needs direct control over registration or lifetime.

## Use Observability When

Start here when you need to:

* Verify that instrumentation is attached to the right scope.
* Inspect tool and LLM inputs and outputs after sanitization.
* Correlate concurrent agent runs by root scope.
* Export traces to OTLP-compatible infrastructure.
* Produce trajectory data for analysis, replay, or evaluation workflows.

If you have not instrumented scopes, tools, or LLM calls yet, start with
[Instrument Applications](/instrument-applications/about).

## Exporter Selection

Choose the exporter based on the downstream system:

| Need                                                 | Use                                                                                   |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Raw canonical event stream                           | [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) |
| Offline analysis, replay, or evaluation trajectories | [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif)   |
| Generic OTLP traces                                  | [OpenTelemetry](/configure-plugins/observability/opentelemetry)                       |
| OpenInference-oriented agent and LLM spans           | [OpenInference](/configure-plugins/observability/openinference)                       |

Start with in-process event inspection before exporting externally. Add sanitize
guardrails before exporters receive sensitive payloads.

## What Success Looks Like

Your first exporter path works correctly when:

* The instrumented request still completes successfully.
* The chosen exporter receives scope data for that request, plus tool or LLM
  lifecycle data when the request exercises those paths.
* The exported output shows the same NeMo Relay scope hierarchy you expect from that request.

If that basic check fails, use the
[Trace Incident Runbook](/resources/troubleshooting/trace-incident-runbook)
before adding more exporters or extra config layers.

## Correlating Trajectories and Traces

When ATIF and trace exporters observe the same NeMo Relay events, they share
NeMo Relay UUIDs for cross-format joins. Plugin-managed ATIF uses the top-level
trajectory root scope UUID as the trajectory `session_id`. ATIF step lineage stores the
event UUID as `step.extra.ancestry.function_id` and the parent UUID as
`step.extra.ancestry.parent_id`.

OpenTelemetry and OpenInference spans carry the same values as
`nemo_relay.uuid` and `nemo_relay.parent_uuid` span attributes. Mark events use
`nemo_relay.mark.uuid` and `nemo_relay.mark.parent_uuid`. Native backend
`trace_id` and `span_id` values are still generated by the tracing backend and
are not written into ATIF.

## Pages

* [Observability Configuration](/configure-plugins/observability/configuration) documents the whole plugin
  component shape, activation, validation, and teardown.
* [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) covers raw JSONL event stream export.
* [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) covers
  trajectory export for top-level Agent scopes and supported coding-agent turn
  scopes.
* [OpenTelemetry](/configure-plugins/observability/opentelemetry) covers generic OTLP trace export.
* [OpenInference](/configure-plugins/observability/openinference) covers OpenInference-oriented OTLP trace
  export.