Agent Trajectory Observability Format (ATOF)

Use the atof section to write raw Agent Trajectory Observability Format (ATOF) 0.1 events to JSONL files, send them to raw-event collectors, or do both.

File sinks are useful for local debugging, offline inspection, and preserving the canonical event stream before another exporter translates it. Stream sinks send the same events to collectors in near real time. They can also transform dotted object keys before delivery.

plugins.toml Example

Add the following ATOF exporter configuration to plugins.toml:

1
version = 1
2
3
[[components]]
4
kind = "observability"
5
enabled = true
6
7
[components.config]
8
version = 2
9
10
[components.config.atof]
11
enabled = true
12
13
[[components.config.atof.sinks]]
14
type = "file"
15
output_directory = "logs"
16
filename = "events.jsonl"
17
mode = "overwrite"
18
19
[[components.config.atof.sinks]]
20
type = "stream"
21
name = "archive"
22
url = "http://localhost:8080/events"
23
transport = "http_post"
24
timeout_millis = 3000
25
field_name_policy = "replace_dots"
26
27
[components.config.atof.sinks.header_env]
28
authorization = "NEMO_RELAY_ATOF_AUTH_HEADER"

Before you validate or activate this example, set NEMO_RELAY_ATOF_AUTH_HEADER to the complete authorization header value, such as Bearer <token>. NeMo Relay rejects a missing or blank value.

This configuration registers the plugin-managed ATOF exporter and writes one JSON object per lifecycle event to logs/events.jsonl. It also applies the replace_dots field-name policy and sends each raw event to the named archive stream sink.

Fields

The following table describes the top-level ATOF settings:

File Sinks

Each file sink writes the raw event stream to one JSONL file. The following table describes each file sink:

Stream Sinks

Each stream sink receives the same canonical ATOF event that file sinks write. Before sending it, NeMo Relay applies the configured field-name policy and then uses the transport-specific encoding. WebSocket sends a JSON text message, while the HTTP transports send a JSONL record. Each stream runs independently, so a failed connection does not block file output or other streams.

The following table describes each stream sink:

Use header_env for secrets so the configuration stores the environment variable name instead of the secret value.

preserve sends canonical ATOF field names unchanged. replace_dots replaces dots in JSON object keys with underscores recursively. If that creates a duplicate key, NeMo Relay keeps both values and appends _2, _3, and later suffixes in a deterministic order. Validation rejects any other sinks[i].field_name_policy value for a stream sink.

The following transports are supported:

• http_post sends each event as one JSONL record in an HTTP POST request with Content-Type: application/x-ndjson. Any 2xx response is treated as success.
• websocket opens one connection and sends each event as one JSON text message.
• ndjson opens one long-lived HTTP upload and writes each event as one newline-delimited JSON record.

For a file sink, the shared subscriber flush completes the synchronous file callback. For a stream sink, it only confirms that the ATOF callback queued the stream work. A manual exporter’s force_flush() first waits for queued subscriber callbacks, then flushes file output or asks each stream worker to drain its queued events while keeping the exporter open. It waits for each stream worker up to that sink’s timeout_millis value. If a worker does not acknowledge in time, NeMo Relay logs a warning and force_flush() can still return successfully; the worker can still have queued work.

shutdown() follows the same delivery barrier, flushes file output, and asks each stream worker to drain and close within the configured timeout. It ignores events delivered after the exporter closes. A timeout is logged and does not by itself cause shutdown() to return an error.

During graceful shutdown, plugin.clear() or clear_plugin_configuration() uses that same terminal stream-sink operation. It does not guarantee that a worker which timed out drained or closed.

Migrate From Version 1

Observability component configuration now requires version = 2. NeMo Relay does not translate the removed ATOF fields from version 1.

Make the following changes when you migrate a plugin configuration:

• Move output_directory, filename, and mode from [components.config.atof] into a sink with type = "file".
• Replace each [[components.config.atof.endpoints]] entry with a [[components.config.atof.sinks]] entry that has type = "stream".
• Keep static request headers in headers, or use header_env to read complete header values from environment variables at activation time.

The plugin fans out each event delivered to its ATOF subscriber to every configured sink. The manual AtofExporter API is different: each exporter owns one file or stream sink.

Expected Output

Each file sink writes a scope, tool, LLM, middleware, or mark event delivered to the exporter as one ATOF JSON object per line. Stream sinks encode the delivered event as described in Stream Sinks. For event field semantics, refer to Events.

Coding-agent sessions emit a session.start mark with the logical metadata.session_id, a Relay-owned UUIDv7 metadata.session_instance_id, and optional session metadata such as metadata.user_id. Later trace exporters use these identities for correlation, but ATOF preserves them as raw event metadata.

ATOF is the raw-event export path. It preserves the canonical event shape; it does not add semantic extraction, replay policy, scope-owner resolution, or exporter projection rules. Each owning agent scope starts fresh, and a compaction mark refreshes it. The first subsequent LLM start annotation retains complete history. Later starts retain system instructions, the latest user message, and every following assistant or tool message. With a request codec annotation, the emitted event input uses the same projection. If that event-only encoding fails, Relay retains the complete event input and annotation. The request used for provider execution remains unchanged. Refer to Events for the complete freshness contract.

Register the plugin before instrumented work starts. Clear it during shutdown so every sink flushes pending events and stream connections close cleanly. Terminating the process before subscriber delivery or plugin teardown completes can leave the final records missing.

Plugin Configuration

Use plugin configuration when you want NeMo Relay to manage the ATOF exporter lifecycle. The following examples configure and activate the exporter through each supported language binding.

validate() checks only the supplied in-memory object. initialize() also layers discovered plugins.toml configuration. For effective file-backed validation, refer to Plugin Configuration Files and run the gateway with the same configuration path that production uses.

1
import asyncio
2
3
from nemo_relay import plugin
4
from nemo_relay.observability import (
5
    AtofConfig,
6
    AtofFileSinkConfig,
7
    AtofStreamSinkConfig,
8
    ComponentSpec,
9
    ObservabilityConfig,
10
)
11
12
config = plugin.PluginConfig(
13
    components=[
14
        ComponentSpec(
15
            ObservabilityConfig(
16
                atof=AtofConfig(
17
                    enabled=True,
18
                    sinks=[
19
                        AtofFileSinkConfig(
20
                            output_directory="logs",
21
                            filename="events.jsonl",
22
                            mode="overwrite",
23
                        ),
24
                        AtofStreamSinkConfig(
25
                            url="http://localhost:8080/events",
26
                            transport="http_post",
27
                        ),
28
                    ],
29
                )
30
            )
31
        )
32
    ]
33
)
34
35
report = plugin.validate(config)
36
if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]):
37
    raise RuntimeError(report["diagnostics"])
38
39
async def main():
40
    await plugin.initialize(config)
41
    try:
42
        # Run instrumented application work here.
43
        pass
44
    finally:
45
        plugin.clear()
46
47
asyncio.run(main())

Manual API

Use the manual AtofExporter API when a test or script needs a custom subscriber name or explicit registration window. A manual exporter owns one sink. To send the same events to several destinations, create and register one exporter for each sink or use the observability plugin.

1
from nemo_relay import AtofExporter, AtofExporterConfig
2
3
config = AtofExporterConfig()
4
config.sink_type = "stream"
5
config.url = "http://localhost:8080/events"
6
config.transport = "http_post"
7
8
exporter = AtofExporter(config)
9
exporter.register("atof-exporter")
10
11
# Run instrumented application work here.
12
13
exporter.force_flush()
14
exporter.deregister("atof-exporter")
15
exporter.shutdown()

Common Configuration and Runtime Issues

Common configuration and runtime issues include:

• ATOF is enabled without at least one plugin-configured sink.
• A file sink’s mode is not append or overwrite.
• A stream sink has an empty url, an unsupported transport, or a timeout_millis value of 0.
• The output directory is not writable at runtime.
• nemo-relay doctor cannot deliver its synthetic ATOF mark probe to a configured stream sink.
• A file sink is configured in a target that cannot access the native filesystem.
• The process terminates before a subscriber or exporter barrier, so the file or stream destination is missing queued tail records.