> This page is for NeMo Relay, version v0.10.0.
> For other versions, use one of these documentation indexes:
> - Latest (v0.10.0) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt
> - v0.10.0: https://docs.nvidia.com/nemo/relay/v0.10.0/llms.txt
> - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt
> - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt
> - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt
> - v0.6.0: https://docs.nvidia.com/nemo/relay/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/nemo/relay/v0.5.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Observability

> Configure NeMo Relay observability exporters for events, traces, logs, metrics, and trajectories.

Use the Observability plugin to inspect NeMo Relay lifecycle events in your
application. You can also export agent activity to tracing, trajectory, or
analysis systems. One plugin configuration controls these outputs.

## Shortest Path

Start with one plugin-managed exporter. Add other formats or manual subscribers
after you confirm that it works. Follow these steps to set up your first exporter:

1. Instrument one scope, tool call, or LLM call in your application.
2. Enable the Observability plugin with only one exporter, such as ATOF JSONL.
3. Run one request through the instrumented path.
4. Close the activation, or call the manual exporter's documented barrier.
5. Confirm the output before adding more exporters or sanitization rules.

Observability in NeMo Relay starts with events. Scopes, marks, managed tool
calls, managed LLM calls, middleware, and manual lifecycle APIs emit the
canonical Agent Trajectory Observability Format (ATOF) event stream.
Subscribers read that stream within the application. Exporters write raw ATOF
JSONL or convert events to Agent Trajectory Interchange Format (ATIF) or a typed
OpenTelemetry projection.

The runtime and its components determine what exporters receive:

* Scopes decide ownership and parentage.
* Middleware decides whether execution continues or what payloads are sanitized.
* Codecs convert request and response data to a common format when available.
* Events record the canonical runtime stream.
* Subscribers and exporters read that stream and convert it to other formats.

Exporters preserve the meaning of runtime events. They convert those events to
the format expected by the receiving system.

Configuration and activation errors mean that observability setup failed.
If an exporter fails to deliver data after activation, application work
continues. Exporter operations such as flush or shutdown can report those
delivery errors. Refer to
[Observability Configuration](/configure-plugins/observability/configuration) for the
failure table.

The Observability plugin uses the component kind `observability` and supports
these outputs:

* Agent Trajectory Observability Format (ATOF) JSONL export for raw lifecycle events.
* Agent Trajectory Interchange Format (ATIF) trajectory export for each top-level
  Agent scope or supported coding-agent turn scope.
* Typed OpenTelemetry OTLP trace export using the `full`, `gen_ai`, or
  `openinference` projection.
* OpenTelemetry OTLP log export for sanitized non-metric marks.
* OpenTelemetry OTLP metric export for marks that use the Relay-owned metric
  schema.

## Plugin-Managed Versus Manual Export

Use the Observability plugin to set up exporters for a process through
configuration, `plugins.toml`, or a shared plugin document.

Use manual subscriber or exporter APIs when a test, script, or application
needs direct control over registration names, collection windows, explicit
flush timing, or per-run exporter objects. The plugin owns subscriber names and
teardown for the sections it enables.

Both paths use the same runtime events. Choose plugin-managed export for
reusable process configuration. Choose manual APIs when you need direct control
over registration or lifetime.

## Delivery and Shutdown

All built-in exporters use the same asynchronous native subscriber dispatcher.
An API that emits an event can return before the exporter handles it.
The subscriber flush API waits for callbacks already in its queue.
Some exporters then send that work to a separate stream worker, trajectory
builder, batch processor, or remote system. Those exporters have their own
operations to wait for delivery.

Use the barrier that matches how you manage the exporter lifecycle:

* For a custom subscriber, call the subscriber flush API before inspecting
  callback side effects.
* For a manual exporter, call its documented `force_flush()`, `export()`, or
  `shutdown()` operation.
* For plugin-managed export, retain the `PluginHostActivation` returned by
  initialization and call its `close()` operation during graceful shutdown.

If the process terminates before those operations complete, the output can omit
queued telemetry even when the instrumented application work completed
successfully. Refer to [Subscribers](/about-nemo-relay/concepts/subscribers#waiting-for-delivery)
for the complete delivery-barrier contract.

## Use Observability When

Use observability for these tasks:

* Verify that instrumentation is attached to the right scope.
* Inspect tool and LLM inputs and outputs after sanitization.
* Correlate concurrent agent runs by root scope.
* Export traces, logs, and metrics to OTLP-compatible infrastructure.
* Produce trajectory data for analysis, replay, or evaluation workflows.

If you have not instrumented scopes, tools, or LLM calls yet, start with
[Instrument Applications](/instrument-applications/about).

## Exporter Selection

Choose the exporter based on the downstream system:

| Need                                                                     | Use                                                                                   |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| Raw canonical event stream                                               | [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) |
| Offline analysis, replay, or evaluation trajectories                     | [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif)   |
| Complete NeMo Relay or standardized GenAI OTLP traces, logs, and metrics | [OpenTelemetry](/configure-plugins/observability/opentelemetry)                       |
| OpenInference-oriented agent and LLM spans                               | [OpenInference Projection](/configure-plugins/observability/openinference)            |

Start with in-process event inspection before exporting externally. Add sanitize
guardrails before exporters receive sensitive payloads.

## What Success Looks Like

Use these checks to confirm that your first exporter works:

* The instrumented request still completes successfully.
* After the exporter or plugin barrier returns, the output contains scope data
  for that request. It also contains tool or LLM lifecycle data if the request
  uses those operations.
* The exported output shows the same NeMo Relay scope hierarchy you expect from that request.

An application can finish its work before exporters deliver the telemetry.
Check the exported output to confirm delivery.

If that basic check fails, use the
[Trace Incident Runbook](/resources/troubleshooting/trace-incident-runbook)
before adding more exporters or extra config layers.

## Correlating Trajectories and Traces

ATIF and trace exporters share NeMo Relay UUIDs when they observe the same
events. Use those UUIDs to match records across formats. Plugin-managed ATIF
uses the top-level trajectory scope UUID as `trajectory_id`. If a propagation
root exists, it uses that root's UUID as the run's `session_id`. Each ATIF step
stores the event UUID in `step.extra.ancestry.function_id` and the parent UUID
in `step.extra.ancestry.parent_id`.

The `full` and `openinference` OpenTelemetry projections carry the same values
as `nemo_relay.uuid` and `nemo_relay.parent_uuid` span attributes. Mark events
use `nemo_relay.mark.uuid` and `nemo_relay.mark.parent_uuid`. The `gen_ai`
projection intentionally omits `nemo_relay.*` attributes. Native backend
`trace_id` and `span_id` values are not written into ATIF.

For coding-agent sessions, `full` and `openinference` OTLP trace roots also
include `session.id`, optional `user.id`, and `nemo_relay.session.instance_id`.
The instance value is the root scope UUID. Trace roots created by the same
runtime session share this value.
The `gen_ai` projection uses the standard `gen_ai.conversation.id` when the
event supplies a conversation, session, or thread identifier. Coding-agent LLM
events use the provider's conversation identifier if one is supplied.
Otherwise, they use the harness session identifier. Typed endpoints create
the same native trace and span IDs from the same Relay lifecycle UUIDs.
Those IDs also let you match records across exporters. ATIF filenames and
storage keys use the trajectory scope UUID. Coding-session values are also
available under `extra.nemo_relay`.

OTLP metrics do not automatically add `nemo_relay.session.instance_id` to
measurement attributes. Measurements with the same resource, instrumentation
scope, and metric attributes can aggregate across sessions.

## Pages

Use these pages for exporter setup and reference details:

* [Observability Configuration](/configure-plugins/observability/configuration) documents the whole plugin
  component shape, activation, validation, and teardown.
* [Agent Trajectory Observability Format (ATOF)](/configure-plugins/observability/atof) covers raw JSONL event stream export.
* [Agent Trajectory Interchange Format (ATIF)](/configure-plugins/observability/atif) covers
  trajectory export for top-level Agent scopes and supported coding-agent turn
  scopes.
* [OpenTelemetry](/configure-plugins/observability/opentelemetry) covers typed
  OTLP trace, log, and metric export.
* [OpenInference Projection](/configure-plugins/observability/openinference)
  covers OpenInference-oriented OTLP trace export.

## Completion Hooks Without a Start

A worker can receive an end hook without its start hook. This can happen when
traffic switches from passthrough to a worker while an operation is running.
If the worker has a matching start, it ends the existing lifecycle. Otherwise,
it emits a mark that records completion at one point in time.

Relay uses four general mark names for endings without a matching start:

* `agent_end_without_start`
* `turn_end_without_start`
* `subagent_end_without_start`
* `tool_end_without_start`

Claude Code also uses `claude_agent_stop_unclassified` when an unmatched
`SubagentStop` has the session's agent type. Claude's
[SubagentStop input documentation](https://code.claude.com/docs/en/hooks#subagentstop-input)
explains that internal agents can emit these stops. The type match does not
prove the stop came from an internal agent; see
[Unmatched Subagent Stops](/nemo-relay-cli/claude-code#unmatched-subagent-stops).

Each mark includes the available completion data and identifiers. It sets
`start_observed=false` and leaves out unknown start times and run durations.
Relay attaches the mark to an existing parent scope if it can verify that parent.
Otherwise, the mark has no parent. An unmatched ending does not create a lasting
session ownership record.

Tool completion marks run the tool request and response sanitizers before the
normal event sanitizers. They do not create a tool-start event.

Relay reserves `tool_end_without_start` for tool completion marks. These marks
must include a nonblank string in `data.tool_name`. Relay rejects malformed marks
before sending them to subscribers. Tool sanitizers also apply when the mark
has no `start_observed` metadata.

Each worker keeps a completion cache to handle duplicate or delayed hooks.
The cache follows these rules:

* Store up to 4,096 completed operations for five minutes.
* Skip duplicate endings and late starts only when a stable source identifier
  matches a completed operation.
* Keep separate records for different lifecycle identifiers, agent kinds,
  sessions, and owners.
* Keep events without stable identifiers, even if their contents are similar.

Ending a turn does not release the enclosing agent's ownership.

ATOF, full OpenTelemetry, and OpenInference handle these marks as they handle
other marks. ATIF and the GenAI projection leave them out. Those formats do not
export these partial observations as execution spans or trajectory steps.