> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/relay/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Observability Configuration

Use this page when an application should install standard observability
exporters from one plugin configuration document instead of manually registering
each subscriber.

The plugin kind is `observability`. It is registered by the core runtime, so
applications do not need to register a plugin implementation before validation
or initialization.

For plugin file discovery, precedence, merge behavior, editor controls, and
gateway conflict rules, see
[Plugin Configuration Files](/build-plugins/plugin-configuration-files).

Observability plugin configuration uses the generic NeMo Relay plugin document
shape, so field names are `snake_case` in every binding. This differs from
Node.js runtime classes such as `OpenTelemetrySubscriber`, which use
Node-native `camelCase` option names outside the plugin system.

## What It Installs

Every exporter section is optional and defaults to disabled. A section is active
only when it includes `enabled: true`.

| Section         | Runtime behavior                                                                                                                    |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `atof`          | Registers a global Agent Trajectory Observability Format (ATOF) JSONL exporter for raw lifecycle events.                            |
| `atif`          | Registers one Agent Trajectory Interchange Format (ATIF) dispatcher that writes one trajectory file for each top-level agent scope. |
| `opentelemetry` | Registers a global OpenTelemetry OTLP subscriber.                                                                                   |
| `openinference` | Registers a global OpenInference OTLP subscriber.                                                                                   |

`subscriber_name` is not part of this config. The runtime infers
component-local subscriber names and registers them under the observability
plugin namespace:

* Agent Trajectory Observability Format (ATOF): `__nemo_relay_plugin__observability__atof`
* Agent Trajectory Interchange Format (ATIF) dispatcher: `__nemo_relay_plugin__observability__atif`
* Per-agent ATIF scope subscriber: `__nemo_relay_plugin__observability__atif-{agent_scope_uuid}`
* OpenTelemetry: `__nemo_relay_plugin__observability__opentelemetry`
* OpenInference: `__nemo_relay_plugin__observability__openinference`

## `plugins.toml` Example

```toml
version = 1

[[components]]
kind = "observability"
enabled = true

[components.config]
version = 1

[components.config.atof]
enabled = true
output_directory = "logs"
filename = "events.jsonl"
mode = "overwrite"

[components.config.atif]
enabled = true
output_directory = "logs"
filename_template = "trajectory-{session_id}.json"

[components.config.opentelemetry]
enabled = true
transport = "http_binary"
endpoint = "http://localhost:4318/v1/traces"
service_name = "nemo-relay"
service_namespace = "agent"
service_version = "0.4.0"
instrumentation_scope = "nemo-relay-observability"
timeout_millis = 3000

[components.config.opentelemetry.headers]
authorization = "Bearer <token>"

[components.config.opentelemetry.resource_attributes]
"deployment.environment" = "dev"
"service.instance.id" = "local"

[components.config.openinference]
enabled = true
transport = "http_binary"
endpoint = "http://localhost:6006/v1/traces"
service_name = "nemo-relay"
service_namespace = "agent"
service_version = "0.4.0"
instrumentation_scope = "nemo-relay-openinference"
timeout_millis = 3000

[components.config.openinference.headers]
authorization = "Bearer <token>"

[components.config.openinference.resource_attributes]
"deployment.environment" = "dev"
"service.instance.id" = "local"

[components.config.policy]
unknown_component = "warn"
unknown_field = "warn"
unsupported_value = "error"
```

Include only the sections you want to configure. In layered `plugins.toml`
files, omission inherits lower-precedence values; write `enabled = false` to
disable an inherited section.

## Per-Language Plugin Configuration

```python
import asyncio

from nemo_relay import plugin, scope, ScopeType
from nemo_relay.observability import (
    AtifConfig,
    AtofConfig,
    ComponentSpec,
    ObservabilityConfig,
    OtlpConfig,
)

config = plugin.PluginConfig(
    components=[
        ComponentSpec(
            ObservabilityConfig(
                atof=AtofConfig(
                    enabled=True,
                    output_directory="logs",
                    filename="events.jsonl",
                    mode="overwrite",
                ),
                atif=AtifConfig(
                    enabled=True,
                    output_directory="logs",
                    filename_template="trajectory-{session_id}.json",
                ),
                opentelemetry=OtlpConfig(
                    enabled=True,
                    endpoint="http://localhost:4318/v1/traces",
                    service_name="nemo-relay",
                    service_namespace="agent",
                    service_version="0.4.0",
                    instrumentation_scope="nemo-relay-observability",
                    resource_attributes={"deployment.environment": "dev"},
                ),
                openinference=OtlpConfig(
                    enabled=True,
                    endpoint="http://localhost:6006/v1/traces",
                    service_name="nemo-relay",
                    service_namespace="agent",
                    service_version="0.4.0",
                    instrumentation_scope="nemo-relay-openinference",
                    resource_attributes={"deployment.environment": "dev"},
                ),
            )
        )
    ]
)

report = plugin.validate(config)
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
    raise RuntimeError(report["diagnostics"])

async def arun():
    async with plugin.plugin(config):
        with scope.scope("agent", ScopeType.Agent):
            pass

asyncio.run(arun())
```

```js
const plugin = require("nemo-relay-node/plugin");
const observability = require("nemo-relay-node/observability");

await plugin.initialize({
  version: 1,
  components: [
    observability.ComponentSpec({
      version: 1,
      atof: observability.atofConfig({
        enabled: true,
        output_directory: "logs",
        filename: "events.jsonl",
        mode: "overwrite",
      }),
      atif: observability.atifConfig({
        enabled: true,
        output_directory: "logs",
        filename_template: "trajectory-{session_id}.json",
      }),
      opentelemetry: observability.otlpConfig({
        enabled: true,
        endpoint: "http://localhost:4318/v1/traces",
        service_name: "nemo-relay",
        service_namespace: "agent",
        service_version: "0.4.0",
        instrumentation_scope: "nemo-relay-observability",
        resource_attributes: {
          "deployment.environment": "dev",
        },
      }),
      openinference: observability.otlpConfig({
        enabled: true,
        endpoint: "http://localhost:6006/v1/traces",
        service_name: "nemo-relay",
        service_namespace: "agent",
        service_version: "0.4.0",
        instrumentation_scope: "nemo-relay-openinference",
        resource_attributes: {
          "deployment.environment": "dev",
        },
      }),
    }),
  ],
});

try {
  // Run instrumented application work here.
} finally {
  plugin.clear();
}
```

```rust
use nemo_relay::observability::plugin_component::{
    AtifSectionConfig, AtofSectionConfig, ComponentSpec, ObservabilityConfig,
    OtlpSectionConfig,
};
use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig};

let component = ComponentSpec::new(ObservabilityConfig {
    atof: Some(AtofSectionConfig {
        enabled: true,
        output_directory: Some("logs".into()),
        filename: Some("events.jsonl".into()),
        mode: "overwrite".into(),
    }),
    atif: Some(AtifSectionConfig {
        enabled: true,
        output_directory: Some("logs".into()),
        filename_template: "trajectory-{session_id}.json".into(),
        ..AtifSectionConfig::default()
    }),
    opentelemetry: Some(OtlpSectionConfig {
        enabled: true,
        endpoint: Some("http://localhost:4318/v1/traces".into()),
        service_name: "nemo-relay".into(),
        service_namespace: Some("agent".into()),
        service_version: Some("0.4.0".into()),
        instrumentation_scope: Some("nemo-relay-observability".into()),
        resource_attributes: [("deployment.environment".into(), "dev".into())].into(),
        ..OtlpSectionConfig::default()
    }),
    openinference: Some(OtlpSectionConfig {
        enabled: true,
        endpoint: Some("http://localhost:6006/v1/traces".into()),
        service_name: "nemo-relay".into(),
        service_namespace: Some("agent".into()),
        service_version: Some("0.4.0".into()),
        instrumentation_scope: Some("nemo-relay-openinference".into()),
        resource_attributes: [("deployment.environment".into(), "dev".into())].into(),
        ..OtlpSectionConfig::default()
    }),
    ..ObservabilityConfig::default()
});

let config = PluginConfig {
    version: 1,
    components: vec![component.into()],
    policy: Default::default(),
};

let report = validate_plugin_config(&config);
assert!(!report.has_errors());

let active = initialize_plugins(config).await?;
```

## Validation And Teardown

Validate plugin configuration before activating it. The plugin reports
unsupported transports, unsupported ATOF modes, unsafe ATIF filename templates,
unknown fields according to policy, and enabled exporters that are unavailable
in the current build or target.

Call `plugin.clear()` or `clear_plugin_configuration()` during teardown.
Clearing the plugin config deregisters inferred subscribers, flushes file
exporters, and shuts down owned OTLP subscribers.

Use manual subscriber/exporter APIs instead of the plugin when you need custom
subscriber names, explicit per-run exporter objects, or direct control over the
collection window.