Observability Configuration

Use this page when an application should install standard observability exporters from one plugin configuration document instead of manually registering each subscriber.

The plugin kind is observability. The core runtime registers it, so applications do not need to register a plugin implementation before validation or initialization.

For plugin file discovery, precedence, merge behavior, editor controls, and gateway conflict rules, refer to Plugin Configuration Files.

Observability plugin configuration uses the generic NeMo Relay plugin document shape, so field names are snake_case in every binding. This differs from Node.js runtime classes such as OpenTelemetrySubscriber, which use Node-native camelCase option names outside the plugin system.

What It Installs

Every exporter section is optional and defaults to disabled. A section is active only when it includes enabled = true.

The following table lists the runtime behavior that each section installs:

Section	Runtime behavior
`atof`	Registers a global Agent Trajectory Observability Format (ATOF) JSONL exporter for raw lifecycle events.
`atif`	Registers one Agent Trajectory Interchange Format (ATIF) dispatcher that writes one trajectory file for each top-level Agent scope or supported coding-agent turn scope.
`opentelemetry`	Registers a global OpenTelemetry OTLP subscriber.
`openinference`	Registers a global OpenInference OTLP subscriber.

subscriber_name is not part of this config. The runtime infers component-local subscriber names and registers them under the observability plugin namespace:

Agent Trajectory Observability Format (ATOF): __nemo_relay_plugin__observability__atof
Agent Trajectory Interchange Format (ATIF) dispatcher: __nemo_relay_plugin__observability__atif
Per-trajectory-root ATIF scope subscriber: __nemo_relay_plugin__observability__atif-{root_scope_uuid}
OpenTelemetry: __nemo_relay_plugin__observability__opentelemetry
OpenInference: __nemo_relay_plugin__observability__openinference

`plugins.toml` Example

Add the following observability component configuration to plugins.toml:

1 version = 1
2 
3 [[components]]
4 kind = "observability"
5 enabled = true
6 
7 [components.config]
8 version = 1
9 
10 [components.config.atof]
11 enabled = true
12 output_directory = "logs"
13 filename = "events.jsonl"
14 mode = "overwrite"
15 
16 [[components.config.atof.endpoints]]
17 url = "http://localhost:8080/events"
18 transport = "http_post"
19 timeout_millis = 3000
20 
21 [components.config.atof.endpoints.headers]
22 authorization = "Bearer <token>"
23 
24 [components.config.atif]
25 enabled = true
26 output_directory = "logs"
27 filename_template = "trajectory-{session_id}.json"
28 
29 [components.config.opentelemetry]
30 enabled = true
31 transport = "http_binary"
32 endpoint = "http://localhost:4318/v1/traces"
33 service_name = "nemo-relay"
34 service_namespace = "agent"
35 service_version = "0.5.0"
36 instrumentation_scope = "nemo-relay-observability"
37 timeout_millis = 3000
38 
39 [components.config.opentelemetry.headers]
40 authorization = "Bearer <token>"
41 
42 [components.config.opentelemetry.resource_attributes]
43 "deployment.environment" = "dev"
44 "service.instance.id" = "local"
45 
46 [components.config.openinference]
47 enabled = true
48 transport = "http_binary"
49 endpoint = "http://localhost:6006/v1/traces"
50 service_name = "nemo-relay"
51 service_namespace = "agent"
52 service_version = "0.5.0"
53 instrumentation_scope = "nemo-relay-openinference"
54 timeout_millis = 3000
55 
56 [components.config.openinference.headers]
57 authorization = "Bearer <token>"
58 
59 [components.config.openinference.resource_attributes]
60 "deployment.environment" = "dev"
61 "service.instance.id" = "local"
62 
63 [components.config.policy]
64 unknown_component = "warn"
65 unknown_field = "warn"
66 unsupported_value = "error"

Include only the sections you want to configure. In layered plugins.toml files, omission inherits lower-precedence values; write enabled = false to disable an inherited section.

Failure Behavior

NeMo Relay treats exporter configuration and exporter delivery failures differently. Configuration and activation failures are fail-closed for the observability setup: validation or initialization returns an error before the new plugin configuration becomes active. Runtime delivery failures are fail-open for application work: the tool, LLM, or agent run continues while the affected exporter records, logs, or reports the delivery problem.

The following table describes how each failure affects application work:

Failure	Behavior
Invalid `plugins.toml`, duplicate component kinds, malformed component shapes, unsupported values, unavailable exporter features, ATOF file-open failures, invalid ATOF endpoint config, unavailable ATOF streaming support, or ATOF endpoint worker startup failures	Validation or initialization fails. If a previous plugin configuration was active, NeMo Relay attempts to restore it after a failed replacement.
ATOF event serialization or file write/flush failure after activation	Application work continues. The exporter stores the failure, stops accepting later events for that file, and returns the stored error from `force_flush()` or `shutdown()`.
ATOF streaming endpoint connection or send failure after activation	File output and other already-started endpoints continue. Endpoint failures are logged with the endpoint index; endpoint flush and close timeouts are logged instead of blocking shutdown indefinitely.
ATIF local file write, HTTP storage, or S3-compatible storage failure	Application work continues. The ATIF exporter records the failed sink as unhealthy and skips it for later trajectories. Other configured sinks continue to receive writes.
ATIF dispatcher serialization or subscriber-management failure	The ATIF dispatcher records a fatal exporter error and stops observing later ATIF events. Other observability sections continue to run.
OpenTelemetry or OpenInference construction failure	Plugin initialization fails before the subscriber is registered.
OpenTelemetry or OpenInference export failure after registration	Application work continues. The OTLP exporter reports failures through its runtime logging and flush or shutdown path.

Missing or delayed telemetry is represented as absence of exporter output, not as synthetic success or failure events. NeMo Relay does not backfill events for subscribers that register late. If the plugin is cleared while an Agent scope or supported coding-agent turn scope is still open, the ATIF dispatcher writes the partial trajectory it has already observed.

Use nemo-relay doctor to validate local ATOF and ATIF output directories, OTLP HTTP endpoints, and ATOF streaming endpoints. Validate ATIF remote storage destinations, including HTTP storage and S3-compatible storage, with exporter logs and backend-side checks because nemo-relay doctor does not probe atif.storage. For local artifact paths, verify that the running process can create the output directory and that teardown calls plugin.clear() or clear_plugin_configuration() before the process exits.

Per-Language Plugin Configuration

The following examples configure and activate the Observability component through each supported language binding:

validate() checks only the supplied in-memory object. initialize() also layers discovered plugins.toml configuration. For effective file-backed validation, refer to Plugin Configuration Files and run the gateway with the same configuration path that production uses.

Python

Node.js

Rust

1 import asyncio
2 
3 from nemo_relay import plugin, scope, ScopeType
4 from nemo_relay.observability import (
5     AtifConfig,
6     AtofConfig,
7     ComponentSpec,
8     ObservabilityConfig,
9     OtlpConfig,
10 )
11 
12 config = plugin.PluginConfig(
13     components=[
14         ComponentSpec(
15             ObservabilityConfig(
16                 atof=AtofConfig(
17                     enabled=True,
18                     output_directory="logs",
19                     filename="events.jsonl",
20                     mode="overwrite",
21                 ),
22                 atif=AtifConfig(
23                     enabled=True,
24                     output_directory="logs",
25                     filename_template="trajectory-{session_id}.json",
26                 ),
27                 opentelemetry=OtlpConfig(
28                     enabled=True,
29                     endpoint="http://localhost:4318/v1/traces",
30                     service_name="nemo-relay",
31                     service_namespace="agent",
32                     service_version="0.5.0",
33                     instrumentation_scope="nemo-relay-observability",
34                     resource_attributes={"deployment.environment": "dev"},
35                 ),
36                 openinference=OtlpConfig(
37                     enabled=True,
38                     endpoint="http://localhost:6006/v1/traces",
39                     service_name="nemo-relay",
40                     service_namespace="agent",
41                     service_version="0.5.0",
42                     instrumentation_scope="nemo-relay-openinference",
43                     resource_attributes={"deployment.environment": "dev"},
44                 ),
45             )
46         )
47     ]
48 )
49 
50 report = plugin.validate(config)
51 if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
52     raise RuntimeError(report["diagnostics"])
53 
54 async def arun():
55     async with plugin.plugin(config):
56         with scope.scope("agent", ScopeType.Agent):
57             pass
58 
59 asyncio.run(arun())

Validation and Teardown

Validate plugin configuration before activating it. The plugin reports unsupported transports, unsupported ATOF modes, invalid ATOF streaming endpoint URLs, non-string endpoint headers, non-positive endpoint timeouts, unsafe ATIF filename templates, unknown fields according to policy, and enabled exporters that are unavailable in the current build or target.

Call plugin.clear() or clear_plugin_configuration() during teardown. Clearing the plugin config deregisters inferred subscribers, flushes file exporters, drains and closes ATOF streaming endpoints, and shuts down owned OTLP subscribers.

Use manual subscriber/exporter APIs instead of the plugin when you need custom subscriber names, explicit per-run exporter objects, or direct control over the collection window.