Observability

Use the Observability plugin to inspect NeMo Relay lifecycle events in your application. You can also export agent activity to tracing, trajectory, or analysis systems. One plugin configuration controls these outputs.

Shortest Path

Start with one plugin-managed exporter. Add other formats or manual subscribers after you confirm that it works. Follow these steps to set up your first exporter:

Instrument one scope, tool call, or LLM call in your application.
Enable the Observability plugin with only one exporter, such as ATOF JSONL.
Run one request through the instrumented path.
Close the activation, or call the manual exporter’s documented barrier.
Confirm the output before adding more exporters or sanitization rules.

Observability in NeMo Relay starts with events. Scopes, marks, managed tool calls, managed LLM calls, middleware, and manual lifecycle APIs emit the canonical Agent Trajectory Observability Format (ATOF) event stream. Subscribers read that stream within the application. Exporters write raw ATOF JSONL or convert events to Agent Trajectory Interchange Format (ATIF) or a typed OpenTelemetry projection.

The runtime and its components determine what exporters receive:

Scopes decide ownership and parentage.
Middleware decides whether execution continues or what payloads are sanitized.
Codecs convert request and response data to a common format when available.
Events record the canonical runtime stream.
Subscribers and exporters read that stream and convert it to other formats.

Exporters preserve the meaning of runtime events. They convert those events to the format expected by the receiving system.

Configuration and activation errors mean that observability setup failed. If an exporter fails to deliver data after activation, application work continues. Exporter operations such as flush or shutdown can report those delivery errors. Refer to Observability Configuration for the failure table.

The Observability plugin uses the component kind observability and supports these outputs:

Agent Trajectory Observability Format (ATOF) JSONL export for raw lifecycle events.
Agent Trajectory Interchange Format (ATIF) trajectory export for each top-level Agent scope or supported coding-agent turn scope.
Typed OpenTelemetry OTLP trace export using the full, gen_ai, or openinference projection.
OpenTelemetry OTLP log export for sanitized non-metric marks.
OpenTelemetry OTLP metric export for marks that use the Relay-owned metric schema.

Plugin-Managed Versus Manual Export

Use the Observability plugin to set up exporters for a process through configuration, plugins.toml, or a shared plugin document.

Use manual subscriber or exporter APIs when a test, script, or application needs direct control over registration names, collection windows, explicit flush timing, or per-run exporter objects. The plugin owns subscriber names and teardown for the sections it enables.

Both paths use the same runtime events. Choose plugin-managed export for reusable process configuration. Choose manual APIs when you need direct control over registration or lifetime.

Delivery and Shutdown

All built-in exporters use the same asynchronous native subscriber dispatcher. An API that emits an event can return before the exporter handles it. The subscriber flush API waits for callbacks already in its queue. Some exporters then send that work to a separate stream worker, trajectory builder, batch processor, or remote system. Those exporters have their own operations to wait for delivery.

Use the barrier that matches how you manage the exporter lifecycle:

For a custom subscriber, call the subscriber flush API before inspecting callback side effects.
For a manual exporter, call its documented force_flush(), export(), or shutdown() operation.
For plugin-managed export, retain the PluginHostActivation returned by initialization and call its close() operation during graceful shutdown.

If the process terminates before those operations complete, the output can omit queued telemetry even when the instrumented application work completed successfully. Refer to Subscribers for the complete delivery-barrier contract.

Use Observability When

Use observability for these tasks:

Verify that instrumentation is attached to the right scope.
Inspect tool and LLM inputs and outputs after sanitization.
Correlate concurrent agent runs by root scope.
Export traces, logs, and metrics to OTLP-compatible infrastructure.
Produce trajectory data for analysis, replay, or evaluation workflows.

If you have not instrumented scopes, tools, or LLM calls yet, start with Instrument Applications.

Exporter Selection

Choose the exporter based on the downstream system:

Need	Use
Raw canonical event stream	Agent Trajectory Observability Format (ATOF)
Offline analysis, replay, or evaluation trajectories	Agent Trajectory Interchange Format (ATIF)
Complete NeMo Relay or standardized GenAI OTLP traces, logs, and metrics	OpenTelemetry
OpenInference-oriented agent and LLM spans	OpenInference Projection

Start with in-process event inspection before exporting externally. Add sanitize guardrails before exporters receive sensitive payloads.

What Success Looks Like

Use these checks to confirm that your first exporter works:

The instrumented request still completes successfully.
After the exporter or plugin barrier returns, the output contains scope data for that request. It also contains tool or LLM lifecycle data if the request uses those operations.
The exported output shows the same NeMo Relay scope hierarchy you expect from that request.

An application can finish its work before exporters deliver the telemetry. Check the exported output to confirm delivery.

If that basic check fails, use the Trace Incident Runbook before adding more exporters or extra config layers.

Correlating Trajectories and Traces

ATIF and trace exporters share NeMo Relay UUIDs when they observe the same events. Use those UUIDs to match records across formats. Plugin-managed ATIF uses the top-level trajectory scope UUID as trajectory_id. If a propagation root exists, it uses that root’s UUID as the run’s session_id. Each ATIF step stores the event UUID in step.extra.ancestry.function_id and the parent UUID in step.extra.ancestry.parent_id.

The full and openinference OpenTelemetry projections carry the same values as nemo_relay.uuid and nemo_relay.parent_uuid span attributes. Mark events use nemo_relay.mark.uuid and nemo_relay.mark.parent_uuid. The gen_ai projection intentionally omits nemo_relay.* attributes. Native backend trace_id and span_id values are not written into ATIF.

For coding-agent sessions, full and openinference OTLP trace roots also include session.id, optional user.id, and nemo_relay.session.instance_id. The instance value is the root scope UUID. Trace roots created by the same runtime session share this value. The gen_ai projection uses the standard gen_ai.conversation.id when the event supplies a conversation, session, or thread identifier. Coding-agent LLM events use the provider’s conversation identifier if one is supplied. Otherwise, they use the harness session identifier. Typed endpoints create the same native trace and span IDs from the same Relay lifecycle UUIDs. Those IDs also let you match records across exporters. ATIF filenames and storage keys use the trajectory scope UUID. Coding-session values are also available under extra.nemo_relay.

OTLP metrics do not automatically add nemo_relay.session.instance_id to measurement attributes. Measurements with the same resource, instrumentation scope, and metric attributes can aggregate across sessions.

Completion Hooks Without a Start

A worker can receive an end hook without its start hook. This can happen when traffic switches from passthrough to a worker while an operation is running. If the worker has a matching start, it ends the existing lifecycle. Otherwise, it emits a mark that records completion at one point in time.

Relay uses four general mark names for endings without a matching start:

agent_end_without_start
turn_end_without_start
subagent_end_without_start
tool_end_without_start

Claude Code also uses claude_agent_stop_unclassified when an unmatched SubagentStop has the session’s agent type. Claude’s SubagentStop input documentation explains that internal agents can emit these stops. The type match does not prove the stop came from an internal agent; see Unmatched Subagent Stops.

Each mark includes the available completion data and identifiers. It sets start_observed=false and leaves out unknown start times and run durations. Relay attaches the mark to an existing parent scope if it can verify that parent. Otherwise, the mark has no parent. An unmatched ending does not create a lasting session ownership record.

Tool completion marks run the tool request and response sanitizers before the normal event sanitizers. They do not create a tool-start event.

Relay reserves tool_end_without_start for tool completion marks. These marks must include a nonblank string in data.tool_name. Relay rejects malformed marks before sending them to subscribers. Tool sanitizers also apply when the mark has no start_observed metadata.

Each worker keeps a completion cache to handle duplicate or delayed hooks. The cache follows these rules:

Store up to 4,096 completed operations for five minutes.
Skip duplicate endings and late starts only when a stable source identifier matches a completed operation.
Keep separate records for different lifecycle identifiers, agent kinds, sessions, and owners.
Keep events without stable identifiers, even if their contents are similar.

Ending a turn does not release the enclosing agent’s ownership.

ATOF, full OpenTelemetry, and OpenInference handle these marks as they handle other marks. ATIF and the GenAI projection leave them out. Those formats do not export these partial observations as execution spans or trajectory steps.