Skip to navigation

Observability

View as MarkdownOpen in Claude

Use this reference for direct exporter and subscriber APIs. For configuration and plugin-managed export, start with Observability.

Agent Trajectory Interchange Format (ATIF)

Manual API

Use the manual AtifExporter API when you need explicit collection boundaries or one exporter object per run. export() and export_json() first wait for subscriber callbacks queued before the call, then take a snapshot of the collected event history. A separate subscriber flush immediately before export is unnecessary. The snapshot can also include events delivered after the internal flush returns but before the snapshot is taken. Events delivered after the snapshot are not included.

from nemo_relay import AtifExporter
exporter = AtifExporter("session-1", "agent", "1.0.0", model_name="demo-model")
exporter.register("atif-exporter")
# Run instrumented application work here.
trajectory = exporter.export()
exporter.deregister("atif-exporter")
exporter.clear()

Expected result: The exported trajectory contains the configured session and agent metadata. Its recorded steps depend on the instrumented work performed before export.

Agent Trajectory Observability Format (ATOF)

Manual API

Use the manual AtofExporter API when a test or script needs a custom subscriber name or explicit registration window. A manual exporter owns one sink. To send the same events to several destinations, create and register one exporter for each sink or use the observability plugin.

from nemo_relay import AtofExporter, AtofExporterConfig
config = AtofExporterConfig()
config.sink_type = "stream"
config.url = "http://localhost:8080/events"
config.transport = "http_post"
exporter = AtofExporter(config)
exporter.register("atof-exporter")
# Run instrumented application work here.
exporter.force_flush()
exporter.deregister("atof-exporter")
exporter.shutdown()

Expected outcome: The exporter registers, flushes, and shuts down without error when its destination is configured correctly. This example leaves the instrumented work to your application; it does not demonstrate event delivery.

Runtime Diagnostics

Read the snapshot from a manual exporter with runtime_diagnostics() in Rust and Python, runtimeDiagnostics() in Node.js, and RuntimeDiagnostics() in Go. C FFI nemo_relay_atof_exporter_runtime_diagnostics_json returns a caller-owned JSON array released with nemo_relay_string_free.

Use the exporter created in Manual API before shutting it down. The following examples flush the exporter and print diagnostics. The Rust example defines a helper; call print_diagnostics(&exporter)? to run it.

exporter.force_flush()
for diagnostic in exporter.runtime_diagnostics().entries:
print(diagnostic.code, diagnostic.count, diagnostic.message)

Expected output: Each recorded diagnostic prints its code, count, and message. An empty diagnostic snapshot prints no lines.

For diagnostic codes and plugin-report interpretation, refer to Runtime Diagnostics.

OpenTelemetry

Direct Subscribers

Set OTEL_AUTHORIZATION to the complete, nonblank authorization header value before constructing the subscriber. The following examples create a direct trace subscriber:

from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber
config = OpenTelemetryConfig(
"gen_ai",
"http://localhost:4318/v1/traces",
)
config.service_name = "agent-service"
config.header_env = {"authorization": "OTEL_AUTHORIZATION"}
subscriber = OpenTelemetrySubscriber(config)

Expected outcome: Construction succeeds when the required environment variables are valid. This example does not print output. Register the subscriber and perform instrumented work to produce telemetry.

Direct trace, log, and metric configs accept allow_insecure (allowInsecure in Node.js, with_allow_insecure in Rust), which allows plain HTTP to any host. It is not recommended. Refer to Insecure HTTP Hosts.

Direct trace, log, and metric configs resolve header_env when the subscriber is constructed and retain that value for the subscriber’s activation. Changing the process environment affects only a subsequently constructed subscriber. Static headers remain unchanged. A header name cannot appear in both maps, including names that differ only by ASCII case, and names within header_env must also be unique ignoring ASCII case.

Each header_env reference must be nonblank, have no surrounding whitespace, and contain neither = nor NUL. Its environment value must be set, nonblank, contain no leading or trailing whitespace, be valid Unicode, and be a valid HTTP header value. Validation errors name the header and environment variable but do not include the resolved value. Relay supplies resolved values only as outbound OTLP request headers; it does not copy them into Event data, OpenTelemetry payloads, resource attributes, or runtime diagnostics.

The log and metric equivalents are OpenTelemetryLogConfig with OpenTelemetryLogSubscriber, and OpenTelemetryMetricConfig with OpenTelemetryMetricSubscriber. Each config takes one required endpoint and exposes the signal settings documented in the OpenTelemetry guide. Bare OTLP/HTTP authorities gain the corresponding standard signal path. Rust, Python, Node.js, and Go expose the same three independently managed subscriber kinds. Go and the C FFI remain experimental and source-first.

Direct construction creates one independently managed exporter. Register the subscriber before instrumented work. During graceful teardown, deregister it, call the binding’s force-flush method (force_flush() or forceFlush()), and then call shutdown(). Force flush first crosses Relay’s subscriber barrier and then flushes the provider. Log shutdown drains the batch queue. Metric shutdown performs the reader’s final collection; it does not add a second metric flush. For direct trace and log subscribers, a successful force flush updates runtime_diagnostics() with any batch queue drops observed so far; the diagnostic count remains cumulative through later flushes and shutdown.

Log and Metric Subscriber Lifecycle

The following examples create and register direct log and metric subscribers, inspect runtime diagnostics, and perform graceful teardown.

from nemo_relay import (
OpenTelemetryLogConfig,
OpenTelemetryLogSubscriber,
OpenTelemetryMetricConfig,
OpenTelemetryMetricSubscriber,
)
logs = OpenTelemetryLogSubscriber(
OpenTelemetryLogConfig("http://localhost:4318/v1/logs")
)
metrics = OpenTelemetryMetricSubscriber(
OpenTelemetryMetricConfig("http://localhost:4318/v1/metrics")
)
logs.register("otlp-logs")
metrics.register("otlp-metrics")
try:
# Run instrumented work here.
pass
finally:
logs.deregister("otlp-logs")
logs.force_flush()
for diagnostic in logs.runtime_diagnostics().entries:
print(diagnostic.code, diagnostic.message)
logs.shutdown()
metrics.deregister("otlp-metrics")
metrics.force_flush()
metrics.shutdown()

Expected outcome: Both subscribers register, flush, and shut down. Diagnostic entries print when present; an empty snapshot prints no lines. Successful lifecycle calls alone do not confirm delivery to an OTLP collector.

Direct trace, log, and metric subscribers expose a bounded runtime diagnostics snapshot: runtime_diagnostics() in Rust and Python, runtimeDiagnostics() in Node.js, and RuntimeDiagnostics() in Go. It reports a stable code, count, and recorded messages per runtime condition. C FFI callers use nemo_relay_otel_subscriber_runtime_diagnostics_json, nemo_relay_otel_log_subscriber_runtime_diagnostics_json, or nemo_relay_otel_metric_subscriber_runtime_diagnostics_json. Each writes a caller-owned bounded JSON array of diagnostic entries that the caller must release with nemo_relay_string_free. Use diagnostics to monitor rejected metric marks, capacity limits, and delivery failures without configuring the observability plugin. The plugin continues to include the same conditions in its runtime report.

OpenInference

Direct Subscriber

Set OTEL_AUTHORIZATION to the complete, nonblank authorization header value before constructing OpenTelemetrySubscriber. Subscriber construction fails if the environment variable is missing or invalid.

Use OpenTelemetrySubscriber with the OpenInference discriminator:

from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber
config = OpenTelemetryConfig(
"openinference",
"http://localhost:6006/v1/traces",
)
config.service_name = "agent-service"
config.header_env = {"authorization": "OTEL_AUTHORIZATION"}
subscriber = OpenTelemetrySubscriber(config)

Expected outcome: Construction succeeds when OTEL_AUTHORIZATION is valid. This example does not print output. Register the subscriber and perform instrumented work to produce telemetry.

Direct OpenInference subscribers use the shared header_env construction, validation, and secret-handling behavior.

Register the subscriber before instrumented work. During graceful teardown, deregister it, flush it, and then shut it down.