Configuration

The observability plugin consumes the canonical NeMo Relay event stream and can install ATOF, ATIF, and OpenTelemetry exporters.

Observability component configuration uses version = 3.

Complete Example

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 3
9
enable_full_payloads = false
10
11
[components.config.atof]
12
enabled = true
13
14
[[components.config.atof.sinks]]
15
type = "file"
16
output_directory = "./observability"
17
filename = "events.jsonl"
18
mode = "append"
19
20
[components.config.atif]
21
enabled = true
22
agent_name = "NeMo Relay"
23
model_name = "unknown"
24
output_directory = "./observability"
25
filename_template = "trajectory-{session_id}.json"
26
27
[components.config.opentelemetry]
28
enabled = true
29
30
[[components.config.opentelemetry.endpoints]]
31
type = "full"
32
endpoint = "http://localhost:4318/v1/traces"
33
service_name = "agent-service"
34
35
[[components.config.opentelemetry.endpoints]]
36
type = "gen_ai"
37
endpoint = "http://localhost:4319/v1/traces"
38
service_name = "agent-service"
39
40
[[components.config.opentelemetry.endpoints]]
41
type = "openinference"
42
endpoint = "http://localhost:6006/v1/traces"
43
service_name = "agent-service"

When OpenTelemetry is enabled, endpoints must contain at least one entry. Endpoint types can be combined. Repeated endpoint types can use the same endpoint and transport. Two different endpoint types must not use the same endpoint and transport: Relay rejects that configuration because their deterministic trace and span IDs would collide at the receiver. For http_binary, this comparison uses the effective trace destination, so a bare URL and the same URL with /v1/traces also collide. A root trailing / is an explicit root-path destination and does not collide with /v1/traces. The comparison realizes HTTP port 80 and HTTPS port 443, collapses repeated path slashes, ignores a non-root trailing slash, and treats standardized loopback forms (localhost, names under .localhost, 127.0.0.0/8, and ::1) as the same host without resolving DNS. Query strings remain part of the destination. All endpoints are constructed before the plugin registers its fan-out subscriber.

Full LLM Payloads

By default, repeated LLM start events contain only the current user turn. Set enable_full_payloads = true at the top level of the observability component config to retain complete sanitized request input and annotations on every LLM start event. This disables turn filtering only; credential removal and sanitizers still apply.

Multi-Endpoint Lifecycle

OpenTelemetry endpoint construction is transactional. If validation or construction fails for any endpoint, plugin activation fails before the fan-out subscriber is registered. A previous active plugin configuration is preserved when possible; the plugin host attempts to restore it if replacement fails after teardown begins.

After activation, each event is delivered to every endpoint. A runtime delivery failure in one exporter does not stop application work or delivery to the other exporters. During teardown, NeMo Relay attempts to shut down every endpoint and returns the first teardown error after all attempts finish. Clear the plugin during graceful shutdown so queued subscriber work can drain and every exporter receives a teardown attempt.

NeMo Relay shuts down OpenTelemetry endpoint providers sequentially. A slow or unreachable collector can delay plugin.clear() or process shutdown by approximately the OpenTelemetry SDK’s five-second shutdown bound plus one endpoint export timeout. Use each endpoint’s timeout_millis value to bound that export attempt. This timeout behavior does not make a full batch queue lossless; refer to OpenTelemetry for queue sizing and drop-warning behavior.

Top-level component config lists concatenate across configuration layers, with higher-precedence entries first. The observability destination lists atof.sinks, opentelemetry.endpoints, and atif.storage follow the same rule, so explicit-or-user, system, and programmatic layers can contribute destinations. Arbitrary lists nested inside structured values retain replacement semantics. List entries are not merged item by item.

When library initialization discovers plugins.toml files, Relay emits one plugin.configuration_inherited warning per file. Each warning is written to the operational log and included in the initialization result and active plugin report. It names only the source path, never destination values or credentials. Relay continues with the layered destination set; validation or activation errors in that effective configuration still fail normally. To change or remove an inherited endpoint, edit the layer that declares it.

For complete layering rules, refer to Plugin Configuration Files.

For exporter-specific fields and behavior, refer to:

• ATOF
• ATIF
• OpenTelemetry
• OpenInference

Programmatic Configuration

1
from nemo_relay import plugin
2
from nemo_relay.observability import (
3
    ComponentSpec,
4
    ObservabilityConfig,
5
    OpenTelemetrySectionConfig,
6
    OpenTelemetryEndpointConfig,
7
)
8
9
config = plugin.PluginConfig(
10
    components=[
11
        ComponentSpec(
12
            ObservabilityConfig(
13
                opentelemetry=OpenTelemetrySectionConfig(
14
                    enabled=True,
15
                    endpoints=[
16
                        OpenTelemetryEndpointConfig(
17
                            type="gen_ai",
18
                            endpoint="http://localhost:4318/v1/traces",
19
                            service_name="agent-service",
20
                        )
21
                    ],
22
                ),
23
                enable_full_payloads=True,
24
            )
25
        )
26
    ]
27
)

Validate programmatic configuration before initialization. Clear the plugin during graceful shutdown so every exporter gets a teardown attempt.

Migrating from Version 2

Version 2 used independent opentelemetry and openinference sections. Version 3 replaces them with typed endpoints:

• Convert the old opentelemetry section to a full endpoint.
• Convert the old openinference section to an openinference endpoint.
• Add a gen_ai endpoint for the standardized GenAI projection.

Move mark_projection, mark_exclude_names, and attribute_mappings into each full or openinference endpoint; their legacy behavior is preserved. The gen_ai projection ignores those controls. semantic_selector and capture_content are unsupported. Version-2 OTLP shapes are rejected under version 3.

The version change also makes type and endpoint required for every OpenTelemetry endpoint. The default service_name changes from nemo-relay to unknown_service, and the default instrumentation scope becomes opentelemetry. Move each old section’s headers and resource_attributes maps into its new endpoint. Use header_env when a header value comes from an environment variable.

For before-and-after TOML and Rust, Python, Node.js, Go, and C API changes, refer to Migration Guides.