Configuration

The observability plugin consumes the canonical NeMo Relay event stream and can install ATOF, ATIF, and OpenTelemetry exporters.

Section	Purpose
`atof`	Writes or streams raw ATOF lifecycle events.
`atif`	Writes completed ATIF agent trajectories.
`opentelemetry`	Configures typed OTLP traces and optional log and metric pipelines.

Observability configuration requires version = 4.

Complete Example

version = 1
[[components]]
kind = "observability"
enabled = true
[components.config]
version = 4
enable_full_payloads = false
[components.config.atof]
enabled = true
[[components.config.atof.sinks]]
type = "file"
output_directory = "./observability"
filename = "events.jsonl"
mode = "append"
[components.config.atif]
enabled = true
agent_name = "NeMo Relay"
model_name = "unknown"
output_directory = "./observability"
filename_template = "trajectory-{session_id}.json"
[components.config.opentelemetry]
enabled = true
[[components.config.opentelemetry.endpoints]]
type = "gen_ai"
endpoint = "http://localhost:4318/v1/traces"
service_name = "agent-service"
[components.config.opentelemetry.logs]
enabled = true
minimum_severity = "info"
[components.config.opentelemetry.metrics]
enabled = true
temporality = "cumulative"

When OpenTelemetry is enabled, configure at least one trace endpoint or an enabled log or metric section with explicit endpoints. In this example, Relay derives http://localhost:4318/v1/logs and http://localhost:4318/v1/metrics from the trace endpoint. An explicit nonempty signal endpoint list replaces derivation.

Trace endpoint types can be combined. Repeated endpoint types can use the same endpoint and transport. Two different trace endpoint types must not use the same endpoint and transport: Relay rejects that configuration because their deterministic trace and span IDs would collide at the receiver. For http_binary, this comparison uses the effective trace destination, so a bare URL and the same URL with /v1/traces also collide. A root trailing / is an explicit root-path destination and does not collide with /v1/traces. The comparison realizes HTTP port 80 and HTTPS port 443, collapses repeated path slashes, ignores a non-root trailing slash, and treats standardized loopback forms (localhost, names under .localhost, 127.0.0.0/8, and ::1) as the same host without resolving DNS. Query strings remain part of the destination. All endpoints are constructed before the plugin registers its fan-out subscriber.

Destination collision validation is signal-aware. Duplicate destinations within logs or within metrics are rejected, while different OTLP signals can share an authority.

Full LLM Payloads

By default, repeated LLM start events contain only the current user turn. Set enable_full_payloads = true at the top level of the observability component config to retain complete sanitized request input and annotations on every LLM start event. This disables turn filtering only; credential removal and sanitizers still apply.

Insecure HTTP Hosts

Not recommended. Plain HTTP and WebSocket send spans, logs, metrics, events, trajectories, and any configured headers, header_env, or header_file values unencrypted. Prefer HTTPS, or a collector reached over loopback.

OpenTelemetry trace endpoints require HTTPS unless the destination is localhost, a loopback IP address, or a host in insecure_http_hosts. This rule applies even when no headers are configured. Log and metric endpoints, ATOF stream sinks, and ATIF HTTP storage require HTTPS or WSS when they send configured headers. The header settings are headers, header_env, and header_file. Without configured headers, these other destinations can use plain HTTP or WebSocket without a host allowlist. Their telemetry still travels unencrypted.

By default, these transport checks allow plain HTTP or WebSocket for localhost and loopback IP addresses. To allow another host, list it at the top level of the observability component config. For example, a containerized agent can reach a collector by its private network name:

[components.config]
version = 4
insecure_http_hosts = ["host.docker.internal", "collector.observability.svc.cluster.local"]

The list applies to every destination in the component: OpenTelemetry trace, log, and metric endpoints, including environment-configured ones, ATOF stream sinks over HTTP or WebSocket, and ATIF HTTP storage.

Each entry is a bare hostname or IP address. Entries with a scheme, port, path, user information, wildcard, or leading dot fail validation. IPv4 addresses must use four decimal parts without leading zeros.
Matching is exact and case-insensitive, and ignores a trailing dot. A listed host can use any port and path. Subdomains of a listed host are not trusted.
Relay compares the host in the destination URL only. DNS resolution never grants trust, so a name that resolves to a listed IP address is still rejected.
Unlike other component lists, this list does not concatenate across configuration layers. The highest-precedence layer that sets it wins, so the system configuration can replace the hosts a user plugins.toml lists, or set insecure_http_hosts = [] to allow no additional hosts.

Programmatic plugin configs set the same top-level insecure_http_hosts field. Exporters constructed directly through the Manual API, outside the observability plugin, take an allow_insecure option instead (allowInsecure in Node.js, with_allow_insecure in Rust). It allows plain HTTP or WebSocket to any host for that exporter and defaults to false.

Multi-Endpoint Lifecycle

OpenTelemetry endpoint construction is isolated. An endpoint that cannot be constructed is skipped with an activation warning while valid endpoints activate and receive the event stream. Invalid batch settings also appear in the configuration report with their indexed opentelemetry.endpoints[N] field. Activation still fails when no trace, log, or metric endpoint can be registered. Structural configuration validation remains activation-blocking at the default policy. A previous active plugin configuration is preserved when possible, and the plugin host attempts to restore it if replacement fails after teardown begins.

After activation, trace projections consume the event stream, while log and metric subscribers consume only their routed marks. A runtime delivery failure in one exporter does not stop application work or delivery to the other exporters. During teardown, NeMo Relay attempts to shut down every endpoint and returns the first teardown error after all attempts finish. Close the activation with activation.close() during graceful shutdown so queued subscriber work can drain and every exporter receives a teardown attempt.

NeMo Relay shuts down OpenTelemetry endpoint providers sequentially. A slow or unreachable collector can delay activation.close() or process shutdown by approximately the OpenTelemetry SDK’s five-second shutdown bound plus one endpoint export timeout. Use each endpoint’s timeout_millis value to bound that export attempt. This timeout behavior does not make a full batch queue lossless; refer to OpenTelemetry for queue sizing and drop-warning behavior.

Top-level component config lists concatenate across configuration layers, with higher-precedence entries first. The observability destination lists atof.sinks, opentelemetry.endpoints, opentelemetry.logs.endpoints, opentelemetry.metrics.endpoints, and atif.storage follow the same rule for nonempty lists, so explicit-or-user, system, and programmatic layers can contribute destinations. An explicitly empty higher-precedence log or metric endpoints list instead clears inherited explicit endpoints. If that signal remains enabled, the resulting empty list fails validation; omit endpoints to derive destinations from trace endpoints. Arbitrary lists nested inside other structured values retain replacement semantics. List entries are not merged item by item.

When library initialization discovers plugins.toml files, Relay emits one plugin.configuration_inherited warning per file. Each warning is written to the operational log and included in the initialization result and active plugin report. The report also includes config_paths and resolved_config. resolved_config is the fully merged plugin configuration. Literal secrets are redacted, URL user info, query parameters, and fragments are removed, and every scalar in opaque dynamic-plugin configuration is redacted. Relay continues with the layered destination set; validation or activation errors in that effective configuration still fail normally. Except for the signal-list clearing behavior above, changing or removing an inherited endpoint requires editing the layer that declares it.

For complete layering rules, refer to Configure.

For exporter-specific fields and behavior, refer to:

Programmatic Configuration

Python

Node.js

from nemo_relay import plugin
from nemo_relay.observability import (
    ComponentSpec,
    ObservabilityConfig,
    OpenTelemetryLogSectionConfig,
    OpenTelemetryMetricSectionConfig,
    OpenTelemetrySectionConfig,
    OpenTelemetryEndpointConfig,
)
config = plugin.PluginConfig(
    components=[
        ComponentSpec(
            ObservabilityConfig(
                opentelemetry=OpenTelemetrySectionConfig(
                    enabled=True,
                    endpoints=[
                        OpenTelemetryEndpointConfig(
                            type="gen_ai",
                            endpoint="http://localhost:4318/v1/traces",
                            service_name="agent-service",
                        )
                    ],
                    logs=OpenTelemetryLogSectionConfig(enabled=True),
                    metrics=OpenTelemetryMetricSectionConfig(enabled=True),
                ),
                enable_full_payloads=True,
            )
        )
    ]
)

Validate programmatic configuration before initialization. Close the activation with activation.close() during graceful shutdown so every exporter gets a teardown attempt.