> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Configuration

> Configure the built-in NeMo Relay observability plugin.

The observability plugin consumes the canonical NeMo Relay event stream and
can install ATOF, ATIF, and OpenTelemetry exporters.

| Section         | Purpose                                                                       |
| --------------- | ----------------------------------------------------------------------------- |
| `atof`          | Writes or streams raw ATOF lifecycle events.                                  |
| `atif`          | Writes completed ATIF agent trajectories.                                     |
| `opentelemetry` | Fans out events to typed `full`, `gen_ai`, or `openinference` OTLP endpoints. |

Observability component configuration uses `version = 3`.

## Complete Example

```toml
version = 1

[[components]]
kind = "observability"
enabled = true

[components.config]
version = 3
enable_full_payloads = false

[components.config.atof]
enabled = true

[[components.config.atof.sinks]]
type = "file"
output_directory = "./observability"
filename = "events.jsonl"
mode = "append"

[components.config.atif]
enabled = true
agent_name = "NeMo Relay"
model_name = "unknown"
output_directory = "./observability"
filename_template = "trajectory-{session_id}.json"

[components.config.opentelemetry]
enabled = true

[[components.config.opentelemetry.endpoints]]
type = "full"
endpoint = "http://localhost:4318/v1/traces"
service_name = "agent-service"

[[components.config.opentelemetry.endpoints]]
type = "gen_ai"
endpoint = "http://localhost:4319/v1/traces"
service_name = "agent-service"

[[components.config.opentelemetry.endpoints]]
type = "openinference"
endpoint = "http://localhost:6006/v1/traces"
service_name = "agent-service"
```

When OpenTelemetry is enabled, `endpoints` must contain at least one entry.
Endpoint types can be combined. Repeated endpoint types can use the same
endpoint and transport. Two different endpoint types must not use the same
endpoint and transport: Relay
rejects that configuration because their deterministic trace and span IDs would
collide at the receiver. For `http_binary`, this comparison uses the effective
trace destination, so a bare URL and the same URL with `/v1/traces` also
collide. The comparison realizes HTTP port `80` and HTTPS port `443`, collapses
repeated path slashes, ignores a non-root trailing slash, and treats standardized
loopback forms (`localhost`, names under `.localhost`, `127.0.0.0/8`, and `::1`)
as the same host without resolving DNS. Query strings remain part of the
destination. All endpoints are constructed before the plugin registers its
fan-out subscriber.

## Full LLM Payloads

By default, repeated LLM start events contain only the current user turn. Set
`enable_full_payloads = true` at the top level of the observability component
config to retain complete sanitized request input and annotations on every LLM
start event. This disables turn filtering only; credential removal and
sanitizers still apply.

## Multi-Endpoint Lifecycle

OpenTelemetry endpoint construction is transactional. If validation or
construction fails for any endpoint, plugin activation fails before the
fan-out subscriber is registered. A previous active plugin configuration is
preserved when possible; the plugin host attempts to restore it if replacement
fails after teardown begins.

After activation, each event is delivered to every endpoint. A runtime delivery
failure in one exporter does not stop application work or delivery to the other
exporters. During teardown, NeMo Relay attempts to shut down every endpoint and
returns the first teardown error after all attempts finish. Clear the plugin
during graceful shutdown so queued subscriber work can drain and every
exporter receives a teardown attempt.

NeMo Relay shuts down OpenTelemetry endpoint providers sequentially. A slow or
unreachable collector can delay `plugin.clear()` or process shutdown by
approximately the OpenTelemetry SDK's five-second shutdown bound plus one
endpoint export timeout. Use each endpoint's `timeout_millis` value to bound
that export attempt. This timeout behavior does not make a full batch queue
lossless; refer to [OpenTelemetry](/configure-plugins/observability/opentelemetry#batch-processor-environment-variables)
for queue sizing and drop-warning behavior.

Top-level component `config` lists concatenate across configuration layers,
with higher-precedence entries first. The observability destination lists
`atof.sinks`, `opentelemetry.endpoints`, and `atif.storage` follow the same
rule, so explicit-or-user, project, system, and programmatic layers can
contribute destinations. Arbitrary lists nested inside structured values
retain replacement semantics. List entries are not merged item by item.

When library initialization discovers `plugins.toml` files, Relay emits one
`plugin.configuration_inherited` warning per file. Each warning is written to
the operational log and included in the initialization result and active plugin
report. It names only the source path, never destination values or credentials.
Relay continues with the layered destination set; validation or activation
errors in that effective configuration still fail normally. To change or
remove an inherited endpoint, edit the layer that declares it.

For complete layering rules, refer to
[Plugin Configuration Files](/configure-plugins/plugin-configuration-files#precedence-and-merge-behavior).

For exporter-specific fields and behavior, refer to:

* [ATOF](/configure-plugins/observability/atof)
* [ATIF](/configure-plugins/observability/atif)
* [OpenTelemetry](/configure-plugins/observability/opentelemetry)
* [OpenInference](/configure-plugins/observability/openinference)

## Programmatic Configuration

#### Python

```python
from nemo_relay import plugin
from nemo_relay.observability import (
    ComponentSpec,
    ObservabilityConfig,
    OpenTelemetrySectionConfig,
    OpenTelemetryEndpointConfig,
)

config = plugin.PluginConfig(
    components=[
        ComponentSpec(
            ObservabilityConfig(
                opentelemetry=OpenTelemetrySectionConfig(
                    enabled=True,
                    endpoints=[
                        OpenTelemetryEndpointConfig(
                            type="gen_ai",
                            endpoint="http://localhost:4318/v1/traces",
                            service_name="agent-service",
                        )
                    ],
                ),
                enable_full_payloads=True,
            )
        )
    ]
)
```

#### Node.js

```javascript
const observability = require("nemo-relay-node/observability");

const component = observability.ComponentSpec({
  version: 3,
  opentelemetry: observability.openTelemetryConfig({
    enabled: true,
    endpoints: [
      observability.openTelemetryEndpoint({
        type: "gen_ai",
        endpoint: "http://localhost:4318/v1/traces",
        service_name: "agent-service",
      }),
    ],
  }),
  enable_full_payloads: true,
});
```

#### Go

```go
config := nemo_relay.NewObservabilityConfig()
otel := nemo_relay.NewObservabilityOpenTelemetryConfig()
otel.Enabled = true
otel.Endpoints = []nemo_relay.ObservabilityOpenTelemetryEndpointConfig{
    nemo_relay.NewObservabilityOpenTelemetryEndpointConfig(
        nemo_relay.OpenTelemetryTypeGenAI,
        "http://localhost:4318/v1/traces",
    ),
}
config.OpenTelemetry = &otel
config.EnableFullPayloads = true
component := nemo_relay.NewObservabilityComponentSpec(config)
```

Validate programmatic configuration before initialization. Clear the plugin
during graceful shutdown so every exporter gets a teardown attempt.

## Migrating from Version 2

Version 2 used independent `opentelemetry` and `openinference` sections.
Version 3 replaces them with typed endpoints:

* Convert the old `opentelemetry` section to a `full` endpoint.
* Convert the old `openinference` section to an `openinference` endpoint.
* Add a `gen_ai` endpoint for the standardized GenAI projection.

Move `mark_projection`, `mark_exclude_names`, and `attribute_mappings` into
each `full` or `openinference` endpoint; their legacy behavior is preserved.
The `gen_ai` projection ignores those controls. `semantic_selector` and
`capture_content` are unsupported. Version-2 OTLP shapes are rejected under
version 3.

The version change also makes `type` and `endpoint` required for every
OpenTelemetry endpoint. The default `service_name` changes from `nemo-relay`
to `unknown_service`, and the default instrumentation scope becomes
`opentelemetry`. Move each old section's `headers` and `resource_attributes`
maps into its new endpoint. Use `header_env` when a header value comes from an
environment variable.

For before-and-after TOML and Rust, Python, Node.js, Go, and C API changes,
refer to
[Migration Guides](/reference/migration-guides#observability-configuration-version-3).