> This page is for NeMo Relay, version dev.
> For other versions, use one of these documentation indexes:
> - Latest (v0.11.2) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt
> - v0.11.2: https://docs.nvidia.com/nemo/relay/v0.11.2/llms.txt
> - v0.10.4: https://docs.nvidia.com/nemo/relay/v0.10.4/llms.txt
> - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt
> - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt
> - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.

# Observability

> Direct exporter and subscriber APIs for NeMo Relay observability.

Use this reference for direct exporter and subscriber APIs. For configuration
and plugin-managed export, start with
[Observability](/nemo/relay/dev/configure-plugins/observability/about).

## Agent Trajectory Interchange Format (ATIF)

### Manual API

Use the manual `AtifExporter` API when you need explicit collection boundaries
or one exporter object per run. `export()` and `export_json()` first wait for
subscriber callbacks queued before the call, then take a snapshot of the
collected event history. A separate subscriber flush immediately before export
is unnecessary. The snapshot can also include events delivered after the
internal flush returns but before the snapshot is taken. Events delivered after
the snapshot are not included.

#### Python

```python
from nemo_relay import AtifExporter

exporter = AtifExporter("session-1", "agent", "1.0.0", model_name="demo-model")
exporter.register("atif-exporter")

# Run instrumented application work here.

trajectory = exporter.export()
exporter.deregister("atif-exporter")
exporter.clear()
```

#### Node.js

```js
const { AtifExporter } = require("nemo-relay-node");

const exporter = new AtifExporter("session-1", "agent", "1.0.0", "demo-model");
exporter.register("atif-exporter");

try {
  // Run instrumented application work here.

  const trajectoryJson = exporter.exportJson();
  console.log(trajectoryJson);
} finally {
  exporter.deregister("atif-exporter");
  exporter.clear();
}
```

#### Rust

```rust title={"atif_exporter.rs"} maxLines={0}
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use nemo_relay::exporter::atif::{AtifExporter, AtifTrajectory};
use nemo_relay::subscriber::{deregister_subscriber, register_subscriber};
use nemo_relay_observability::atif::AtifAgentInfo;

pub fn run_example()
-> Result<nemo_relay::exporter::atif::AtifTrajectory, Box<dyn std::error::Error>> {
    let exporter = nemo_relay::exporter::atif(
        "session-1".to_string(),
        AtifAgentInfo {
            name: "agent".to_string(),
            version: "1.0.0".to_string(),
            model_name: Some("demo-model".to_string()),
            tool_definitions: None,
            extra: None,
        },
    );
    register_subscriber("atif-exporter", exporter.subscriber())?;

    // Run instrumented application work here.

    complete_export(&exporter, exporter.export())
}

fn complete_export(
    exporter: &AtifExporter,
    trajectory_result: Result<AtifTrajectory, nemo_relay::FlowError>,
) -> Result<AtifTrajectory, Box<dyn std::error::Error>> {
    let deregistration_result = deregister_subscriber("atif-exporter");
    exporter.clear();
    let trajectory = trajectory_result?;
    deregistration_result?;
    let trajectory_json = serde_json::to_string_pretty(&trajectory)?;
    println!("{trajectory_json}");
    Ok(trajectory)
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    run_example()?;
    Ok(())
}

```

**Expected result:** The exported trajectory contains the configured session
and agent metadata. Its recorded steps depend on the instrumented work performed
before export.

## Agent Trajectory Observability Format (ATOF)

### Manual API

Use the manual `AtofExporter` API when a test or script needs a custom
subscriber name or explicit registration window. A manual exporter owns one
sink. To send the same events to several destinations, create and register one
exporter for each sink or use the observability plugin.

#### Python

```python
from nemo_relay import AtofExporter, AtofExporterConfig

config = AtofExporterConfig()
config.sink_type = "stream"
config.url = "http://localhost:8080/events"
config.transport = "http_post"

exporter = AtofExporter(config)
exporter.register("atof-exporter")

# Run instrumented application work here.

exporter.force_flush()
exporter.deregister("atof-exporter")
exporter.shutdown()
```

#### Node.js

```js
const { AtofExporter } = require("nemo-relay-node");

const exporter = new AtofExporter({
  type: "stream",
  url: "http://localhost:8080/events",
  transport: "http_post",
});
exporter.register("atof-exporter");

try {
  // Run instrumented application work here.

  exporter.forceFlush();
} finally {
  exporter.deregister("atof-exporter");
  exporter.shutdown();
}
```

#### Rust

```rust title={"atof_exporter.rs"} maxLines={0}
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use nemo_relay_observability::atof::{
    AtofEndpointTransport, AtofExporterConfig, AtofStreamSinkConfig,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = AtofExporterConfig::new().with_stream_sink(AtofStreamSinkConfig::new(
        "http://localhost:8080/events",
        AtofEndpointTransport::HttpPost,
    ));
    let exporter = nemo_relay::exporter::atof(config)?;
    exporter.register("atof-exporter")?;

    // Run instrumented application work here.

    exporter.force_flush()?;
    let _ = exporter.deregister("atof-exporter")?;
    exporter.shutdown()?;
    Ok(())
}

```

**Expected outcome:** The exporter registers, flushes, and shuts down without
error when its destination is configured correctly. This example leaves the
instrumented work to your application; it does not demonstrate event delivery.

### Runtime Diagnostics

Read the snapshot from a manual exporter with `runtime_diagnostics()` in Rust
and Python, `runtimeDiagnostics()` in Node.js, and `RuntimeDiagnostics()` in
Go. C FFI `nemo_relay_atof_exporter_runtime_diagnostics_json` returns a
caller-owned JSON array released with `nemo_relay_string_free`.

Use the exporter created in [Manual API](#atof-manual-api) before shutting it
down. The following examples flush the exporter and print diagnostics. The Rust
example defines a helper; call `print_diagnostics(&exporter)?` to run it.

#### Python

```python
exporter.force_flush()
for diagnostic in exporter.runtime_diagnostics().entries:
    print(diagnostic.code, diagnostic.count, diagnostic.message)
```

#### Node.js

```js
exporter.forceFlush();
for (const diagnostic of exporter.runtimeDiagnostics()) {
  console.log(diagnostic.code, diagnostic.count, diagnostic.message);
}
```

#### Rust

```rust title={"atof_diagnostics.rs"} maxLines={0}
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

pub fn print_diagnostics(
    exporter: &nemo_relay::exporter::atof::AtofExporter,
) -> Result<(), Box<dyn std::error::Error>> {
    exporter.force_flush()?;
    for diagnostic in exporter.runtime_diagnostics().entries() {
        println!(
            "{} {} {}",
            diagnostic.code, diagnostic.count, diagnostic.message
        );
    }
    Ok(())
}

```

**Expected output:** Each recorded diagnostic prints its code, count, and
message. An empty diagnostic snapshot prints no lines.

For diagnostic codes and plugin-report interpretation, refer to
[Runtime Diagnostics](/nemo/relay/dev/configure-plugins/observability/atof#runtime-diagnostics).

## OpenTelemetry

### Direct Subscribers

Set `OTEL_AUTHORIZATION` to the complete, nonblank authorization header value
before constructing the subscriber. The following examples create a direct
trace subscriber:

#### Python

```python
from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber

config = OpenTelemetryConfig(
    "gen_ai",
    "http://localhost:4318/v1/traces",
)
config.service_name = "agent-service"
config.header_env = {"authorization": "OTEL_AUTHORIZATION"}
subscriber = OpenTelemetrySubscriber(config)
```

#### Node.js

```javascript
const { OpenTelemetrySubscriber } = require("nemo-relay-node");

const subscriber = new OpenTelemetrySubscriber({
  type: "gen_ai",
  endpoint: "http://localhost:4318/v1/traces",
  serviceName: "agent-service",
  headerEnv: { authorization: "OTEL_AUTHORIZATION" },
});
```

#### Rust

```rust title={"opentelemetry_subscriber.rs"} maxLines={0}
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use nemo_relay::exporter::otel::{OpenTelemetrySubscriber, Result};

pub fn create_subscriber() -> Result<OpenTelemetrySubscriber> {
    use nemo_relay_observability::OpenTelemetryType;
    use nemo_relay_observability::otel::OpenTelemetryConfig;

    let config =
        OpenTelemetryConfig::new(OpenTelemetryType::GenAi, "http://localhost:4318/v1/traces")
            .with_service_name("agent-service")
            .with_header_env("authorization", "OTEL_AUTHORIZATION");
    nemo_relay::exporter::opentelemetry(config)
}

```

**Expected outcome:** Construction succeeds when the required environment
variables are valid. This example does not print output. Register the subscriber
and perform instrumented work to produce telemetry.

Direct trace, log, and metric configs accept `allow_insecure` (`allowInsecure`
in Node.js, `with_allow_insecure` in Rust), which allows plain HTTP to any host.
It is not recommended. Refer to [Insecure HTTP Hosts](/nemo/relay/dev/configure-plugins/observability/configuration#insecure-http-hosts).

Direct trace, log, and metric configs resolve `header_env` when the subscriber
is constructed and retain that value for the subscriber's activation. Changing
the process environment affects only a subsequently constructed subscriber.
Static `headers` remain unchanged. A header name cannot appear in both maps,
including names that differ only by ASCII case, and names within `header_env`
must also be unique ignoring ASCII case.

Each `header_env` reference must be nonblank, have no surrounding whitespace,
and contain neither `=` nor NUL. Its environment value must be set, nonblank,
contain no leading or trailing whitespace, be valid Unicode, and be a valid HTTP
header value. Validation errors name the header and environment variable but do
not include the resolved value.
Relay supplies resolved values only as outbound OTLP request headers; it does
not copy them into Event data, OpenTelemetry payloads, resource attributes, or
runtime diagnostics.

The log and metric equivalents are `OpenTelemetryLogConfig` with
`OpenTelemetryLogSubscriber`, and `OpenTelemetryMetricConfig` with
`OpenTelemetryMetricSubscriber`. Each config takes one required endpoint and
exposes the signal settings documented in the
[OpenTelemetry guide](/nemo/relay/dev/configure-plugins/observability/opentelemetry). Bare OTLP/HTTP authorities gain
the corresponding standard signal path. Rust, Python, Node.js, and Go expose
the same three independently managed subscriber kinds. Go and the C FFI remain
experimental and source-first.

Direct construction creates one independently managed exporter. Register the
subscriber before instrumented work. During graceful teardown, deregister it,
call the binding's force-flush method (`force_flush()` or `forceFlush()`), and
then call `shutdown()`. Force flush first crosses Relay's subscriber barrier
and then flushes the provider. Log shutdown drains the batch queue. Metric
shutdown performs the reader's final collection; it does not add a second
metric flush. For direct trace and log subscribers, a successful force flush
updates `runtime_diagnostics()` with any batch queue drops observed so far; the
diagnostic count remains cumulative through later flushes and shutdown.

#### Log and Metric Subscriber Lifecycle

The following examples create and register direct log and metric subscribers,
inspect runtime diagnostics, and perform graceful teardown.

#### Python

```python
from nemo_relay import (
    OpenTelemetryLogConfig,
    OpenTelemetryLogSubscriber,
    OpenTelemetryMetricConfig,
    OpenTelemetryMetricSubscriber,
)

logs = OpenTelemetryLogSubscriber(
    OpenTelemetryLogConfig("http://localhost:4318/v1/logs")
)
metrics = OpenTelemetryMetricSubscriber(
    OpenTelemetryMetricConfig("http://localhost:4318/v1/metrics")
)
logs.register("otlp-logs")
metrics.register("otlp-metrics")
try:
    # Run instrumented work here.
    pass
finally:
    logs.deregister("otlp-logs")
    logs.force_flush()
    for diagnostic in logs.runtime_diagnostics().entries:
        print(diagnostic.code, diagnostic.message)
    logs.shutdown()
    metrics.deregister("otlp-metrics")
    metrics.force_flush()
    metrics.shutdown()
```

#### Node.js

```javascript
const {
  OpenTelemetryLogSubscriber,
  OpenTelemetryMetricSubscriber,
} = require("nemo-relay-node");

const logs = new OpenTelemetryLogSubscriber({
  endpoint: "http://localhost:4318/v1/logs",
});
const metrics = new OpenTelemetryMetricSubscriber({
  endpoint: "http://localhost:4318/v1/metrics",
});
logs.register("otlp-logs");
metrics.register("otlp-metrics");
try {
  // Run instrumented work here.
} finally {
  logs.deregister("otlp-logs");
  logs.forceFlush();
  for (const diagnostic of logs.runtimeDiagnostics()) {
    console.error(diagnostic.code, diagnostic.message);
  }
  logs.shutdown();
  metrics.deregister("otlp-metrics");
  metrics.forceFlush();
  metrics.shutdown();
}
```

#### Rust

```rust title={"opentelemetry_signals.rs"} maxLines={0}
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use nemo_relay::exporter::{
    otel_logs::{OpenTelemetryLogConfig, OpenTelemetryLogSubscriber},
    otel_metrics::{OpenTelemetryMetricConfig, OpenTelemetryMetricSubscriber},
};

pub fn run_example() -> Result<(), Box<dyn std::error::Error>> {
    let logs = nemo_relay::exporter::opentelemetry_logs(OpenTelemetryLogConfig::new(
        "http://localhost:4318/v1/logs",
    ))?;
    let metrics = nemo_relay::exporter::opentelemetry_metrics(OpenTelemetryMetricConfig::new(
        "http://localhost:4318/v1/metrics",
    ))?;
    logs.register("otlp-logs")?;
    metrics.register("otlp-metrics")?;

    // Run instrumented work here.

    complete_subscribers(&logs, &metrics)
}

fn complete_subscribers(
    logs: &OpenTelemetryLogSubscriber,
    metrics: &OpenTelemetryMetricSubscriber,
) -> Result<(), Box<dyn std::error::Error>> {
    let deregistration_results = [
        logs.deregister("otlp-logs"),
        metrics.deregister("otlp-metrics"),
    ];
    let flush_results = [logs.force_flush(), metrics.force_flush()];
    for diagnostic in logs.runtime_diagnostics().entries() {
        eprintln!("{}: {}", diagnostic.code, diagnostic.message);
    }
    let shutdown_results = [logs.shutdown(), metrics.shutdown()];
    for result in deregistration_results {
        result?;
    }
    for result in flush_results.into_iter().chain(shutdown_results) {
        result?;
    }
    Ok(())
}

```

**Expected outcome:** Both subscribers register, flush, and shut down.
Diagnostic entries print when present; an empty snapshot prints no lines.
Successful lifecycle calls alone do not confirm delivery to an OTLP collector.

Direct trace, log, and metric subscribers expose a bounded runtime
diagnostics snapshot: `runtime_diagnostics()` in Rust and Python,
`runtimeDiagnostics()` in Node.js, and `RuntimeDiagnostics()` in Go. It reports
a stable code, count, and recorded messages per runtime condition. C FFI
callers use `nemo_relay_otel_subscriber_runtime_diagnostics_json`,
`nemo_relay_otel_log_subscriber_runtime_diagnostics_json`, or
`nemo_relay_otel_metric_subscriber_runtime_diagnostics_json`. Each writes a
caller-owned bounded JSON array of diagnostic entries that the caller must
release with `nemo_relay_string_free`. Use diagnostics to monitor rejected
metric marks, capacity limits, and delivery failures without configuring the
observability plugin. The plugin continues to include the same conditions in
its runtime report.

## OpenInference

### Direct Subscriber

Set `OTEL_AUTHORIZATION` to the complete, nonblank authorization header value
before constructing `OpenTelemetrySubscriber`. Subscriber construction fails
if the environment variable is missing or invalid.

Use `OpenTelemetrySubscriber` with the OpenInference discriminator:

#### Python

```python
from nemo_relay import OpenTelemetryConfig, OpenTelemetrySubscriber

config = OpenTelemetryConfig(
    "openinference",
    "http://localhost:6006/v1/traces",
)
config.service_name = "agent-service"
config.header_env = {"authorization": "OTEL_AUTHORIZATION"}
subscriber = OpenTelemetrySubscriber(config)
```

#### Node.js

```javascript
const { OpenTelemetrySubscriber } = require("nemo-relay-node");

const subscriber = new OpenTelemetrySubscriber({
  type: "openinference",
  endpoint: "http://localhost:6006/v1/traces",
  serviceName: "agent-service",
  headerEnv: { authorization: "OTEL_AUTHORIZATION" },
});
```

#### Rust

```rust title={"openinference_subscriber.rs"} maxLines={0}
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use nemo_relay::exporter::otel::{OpenTelemetrySubscriber, Result};

pub fn create_subscriber() -> Result<OpenTelemetrySubscriber> {
    use nemo_relay_observability::OpenTelemetryType;
    use nemo_relay_observability::otel::OpenTelemetryConfig;

    let config = OpenTelemetryConfig::new(
        OpenTelemetryType::OpenInference,
        "http://localhost:6006/v1/traces",
    )
    .with_service_name("agent-service")
    .with_header_env("authorization", "OTEL_AUTHORIZATION");
    nemo_relay::exporter::opentelemetry(config)
}

```

**Expected outcome:** Construction succeeds when `OTEL_AUTHORIZATION` is
valid. This example does not print output. Register the subscriber and perform
instrumented work to produce telemetry.

Direct OpenInference subscribers use the shared
[`header_env` construction, validation, and secret-handling behavior](/nemo/relay/dev/reference/api/observability#direct-subscribers).

Register the subscriber before instrumented work. During graceful teardown,
deregister it, flush it, and then shut it down.