> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Agent Trajectory Observability Format (ATOF) > Configure raw ATOF event export to JSONL files and stream sinks. Use the `atof` section to write raw Agent Trajectory Observability Format (ATOF) `0.1` events to JSONL files, send them to raw-event collectors, or do both. File sinks are useful for local debugging, offline inspection, and preserving the canonical event stream before another exporter translates it. Stream sinks send the same events to collectors in near real time. They can also transform dotted object keys before delivery. ## `plugins.toml` Example Add the following ATOF exporter configuration to `plugins.toml`: ```toml version = 1 [[components]] kind = "observability" enabled = true [components.config] version = 2 [components.config.atof] enabled = true [[components.config.atof.sinks]] type = "file" output_directory = "logs" filename = "events.jsonl" mode = "overwrite" [[components.config.atof.sinks]] type = "stream" name = "archive" url = "http://localhost:8080/events" transport = "http_post" timeout_millis = 3000 field_name_policy = "replace_dots" [components.config.atof.sinks.header_env] authorization = "NEMO_RELAY_ATOF_AUTH_HEADER" ``` Before you validate or activate this example, set `NEMO_RELAY_ATOF_AUTH_HEADER` to the complete authorization header value, such as `Bearer `. NeMo Relay rejects a missing or blank value. This configuration registers the plugin-managed ATOF exporter and writes one JSON object per lifecycle event to `logs/events.jsonl`. It also applies the `replace_dots` field-name policy and sends each raw event to the named `archive` stream sink. ## Fields The following table describes the top-level ATOF settings: | Field | Default | Notes | | --------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `false` | Must be `true` to export events. | | `sinks` | `[]` | File and stream destinations. An enabled ATOF section requires at least one sink. Each event delivered to the ATOF subscriber is sent to every configured sink. | ## File Sinks Each file sink writes the raw event stream to one JSONL file. The following table describes each file sink: | Field | Default | Notes | | ------------------ | --------------------------------------- | ------------------------------------ | | `type` | Required | Sink variant. Must be `file`. | | `output_directory` | Current working directory | Directory containing the JSONL file. | | `filename` | Timestamped `nemo-relay-events-*.jsonl` | Output filename. | | `mode` | `append` | `append` or `overwrite`. | Native event-producing APIs return before the ATOF subscriber necessarily runs. After it receives an event, a file sink writes one complete JSONL record and flushes the process writer. A subscriber flush waits for the file callback to complete its write attempt for every event queued before the barrier. A manual exporter reports any stored file errors during `force_flush()` or `shutdown()`. Plugin teardown also reports stored file errors. Flushing the process writer does not call `fsync` or guarantee durability after a machine crash. If the process terminates before the subscriber receives an event, the corresponding record can still be lost. ## Stream Sinks Each stream sink receives the same canonical ATOF event that file sinks write. Before sending it, NeMo Relay applies the configured field-name policy and then uses the transport-specific encoding. WebSocket sends a JSON text message, while the HTTP transports send a JSONL record. Each stream runs independently, so a failed connection does not block file output or other streams. The following table describes each stream sink: | Field | Default | Notes | | ------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Required | Sink variant. Must be `stream`. | | `name` | Omitted | Optional stable name for cross-component references. Stream sink names must be unique, non-empty, and have no leading or trailing whitespace. | | `url` | Required | Stream URL. Use `http` or `https` for `http_post` and `ndjson`, and `ws` or `wss` for `websocket`. | | `transport` | `http_post` | `http_post`, `websocket`, or `ndjson`. | | `headers` | `{}` | String-to-string headers for requests or handshakes. | | `header_env` | `{}` | Maps each header name to an environment variable that contains the full header value. Each variable must be set and nonblank, and the same header cannot also appear in `headers`. | | `timeout_millis` | `3000` | Per-sink timeout for stream I/O and flush or close acknowledgement. Must be greater than `0`. | | `field_name_policy` | `preserve` | Field-name handling before Relay sends stream events. Accepted values are `preserve` and `replace_dots`. | Use `header_env` for secrets so the configuration stores the environment variable name instead of the secret value. `preserve` sends canonical ATOF field names unchanged. `replace_dots` replaces dots in JSON object keys with underscores recursively. If that creates a duplicate key, NeMo Relay keeps both values and appends `_2`, `_3`, and later suffixes in a deterministic order. Validation rejects any other `sinks[i].field_name_policy` value for a stream sink. The following transports are supported: * `http_post` sends each event as one JSONL record in an HTTP `POST` request with `Content-Type: application/x-ndjson`. Any `2xx` response is treated as success. * `websocket` opens one connection and sends each event as one JSON text message. * `ndjson` opens one long-lived HTTP upload and writes each event as one newline-delimited JSON record. For a file sink, the shared subscriber flush completes the synchronous file callback. For a stream sink, it only confirms that the ATOF callback queued the stream work. A manual exporter's `force_flush()` first waits for queued subscriber callbacks, then flushes file output or asks each stream worker to drain its queued events while keeping the exporter open. It waits for each stream worker up to that sink's `timeout_millis` value. If a worker does not acknowledge in time, NeMo Relay logs a warning and `force_flush()` can still return successfully; the worker can still have queued work. `shutdown()` follows the same delivery barrier, flushes file output, and asks each stream worker to drain and close within the configured timeout. It ignores events delivered after the exporter closes. A timeout is logged and does not by itself cause `shutdown()` to return an error. During graceful shutdown, `plugin.clear()` or `clear_plugin_configuration()` uses that same terminal stream-sink operation. It does not guarantee that a worker which timed out drained or closed. ## Migrate From Version 1 Observability component configuration now requires `version = 2`. NeMo Relay does not translate the removed ATOF fields from version 1. Make the following changes when you migrate a plugin configuration: * Move `output_directory`, `filename`, and `mode` from `[components.config.atof]` into a sink with `type = "file"`. * Replace each `[[components.config.atof.endpoints]]` entry with a `[[components.config.atof.sinks]]` entry that has `type = "stream"`. * Keep static request headers in `headers`, or use `header_env` to read complete header values from environment variables at activation time. The plugin fans out each event delivered to its ATOF subscriber to every configured sink. The manual `AtofExporter` API is different: each exporter owns one file or stream sink. ## Expected Output Each file sink writes a scope, tool, LLM, middleware, or mark event delivered to the exporter as one ATOF JSON object per line. Stream sinks encode the delivered event as described in [Stream Sinks](#stream-sinks). For event field semantics, refer to [Events](/about-nemo-relay/concepts/events). Coding-agent sessions emit a `session.start` mark with the logical `metadata.session_id`, a Relay-owned UUIDv7 `metadata.session_instance_id`, and optional session metadata such as `metadata.user_id`. Later trace exporters use these identities for correlation, but ATOF preserves them as raw event metadata. ATOF is the raw-event export path. It preserves the canonical event shape; it does not add semantic extraction, replay policy, scope-owner resolution, or exporter projection rules. Each owning agent scope starts fresh, and a `compaction` mark refreshes it. The first subsequent LLM start annotation retains complete history. Later starts retain system instructions, the latest user message, and every following assistant or tool message. With a request codec annotation, the emitted event input uses the same projection. If that event-only encoding fails, Relay retains the complete event input and annotation. The request used for provider execution remains unchanged. Refer to [Events](/about-nemo-relay/concepts/events) for the complete freshness contract. Register the plugin before instrumented work starts. Clear it during shutdown so every sink flushes pending events and stream connections close cleanly. Terminating the process before subscriber delivery or plugin teardown completes can leave the final records missing. ## Plugin Configuration Use plugin configuration when you want NeMo Relay to manage the ATOF exporter lifecycle. The following examples configure and activate the exporter through each supported language binding. `validate()` checks only the supplied in-memory object. `initialize()` also layers discovered `plugins.toml` configuration. For effective file-backed validation, refer to [Plugin Configuration Files](/configure-plugins/plugin-configuration-files) and run the gateway with the same configuration path that production uses. #### Python ```python import asyncio from nemo_relay import plugin from nemo_relay.observability import ( AtofConfig, AtofFileSinkConfig, AtofStreamSinkConfig, ComponentSpec, ObservabilityConfig, ) config = plugin.PluginConfig( components=[ ComponentSpec( ObservabilityConfig( atof=AtofConfig( enabled=True, sinks=[ AtofFileSinkConfig( output_directory="logs", filename="events.jsonl", mode="overwrite", ), AtofStreamSinkConfig( url="http://localhost:8080/events", transport="http_post", ), ], ) ) ) ] ) report = plugin.validate(config) if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): raise RuntimeError(report["diagnostics"]) async def main(): await plugin.initialize(config) try: # Run instrumented application work here. pass finally: plugin.clear() asyncio.run(main()) ``` #### Node.js ```js const plugin = require("nemo-relay-node/plugin"); const observability = require("nemo-relay-node/observability"); void (async () => { await plugin.initialize({ version: 1, components: [ observability.ComponentSpec({ version: 2, atof: observability.atofConfig({ enabled: true, sinks: [ { type: "file", output_directory: "logs", filename: "events.jsonl", mode: "overwrite", }, { type: "stream", url: "http://localhost:8080/events", transport: "http_post", }, ], }), }), ], }); try { // Run instrumented application work here. } finally { plugin.clear(); } })().catch((error) => { console.error(error); process.exitCode = 1; }); ``` #### Rust ```rust use nemo_relay::observability::plugin_component::{ AtofFileSinkSectionConfig, AtofSectionConfig, AtofSinkSectionConfig, AtofStreamSinkSectionConfig, ComponentSpec, ObservabilityConfig, }; use nemo_relay::plugin::{ clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, }; #[tokio::main] async fn main() -> Result<(), Box> { let component = ComponentSpec::new(ObservabilityConfig { atof: Some(AtofSectionConfig { enabled: true, sinks: vec![ AtofSinkSectionConfig::File(AtofFileSinkSectionConfig { output_directory: Some("logs".into()), filename: Some("events.jsonl".into()), mode: "overwrite".into(), }), AtofSinkSectionConfig::Stream(AtofStreamSinkSectionConfig { url: "http://localhost:8080/events".into(), transport: "http_post".into(), headers: Default::default(), header_env: Default::default(), timeout_millis: 3000, field_name_policy: "replace_dots".into(), name: Some("archive".into()), }), ], }), ..ObservabilityConfig::default() }); let config = PluginConfig { version: 1, components: vec![component.into()], policy: Default::default(), }; let report = validate_plugin_config(&config); assert!(!report.has_errors()); let _active = initialize_plugins(config).await?; // Run instrumented application work here. clear_plugin_configuration()?; Ok(()) } ``` ## Manual API Use the manual `AtofExporter` API when a test or script needs a custom subscriber name or explicit registration window. A manual exporter owns one sink. To send the same events to several destinations, create and register one exporter for each sink or use the observability plugin. #### Python ```python from nemo_relay import AtofExporter, AtofExporterConfig config = AtofExporterConfig() config.sink_type = "stream" config.url = "http://localhost:8080/events" config.transport = "http_post" exporter = AtofExporter(config) exporter.register("atof-exporter") # Run instrumented application work here. exporter.force_flush() exporter.deregister("atof-exporter") exporter.shutdown() ``` #### Node.js ```js const { AtofExporter } = require("nemo-relay-node"); const exporter = new AtofExporter({ type: "stream", url: "http://localhost:8080/events", transport: "http_post", }); exporter.register("atof-exporter"); try { // Run instrumented application work here. exporter.forceFlush(); } finally { exporter.deregister("atof-exporter"); exporter.shutdown(); } ``` #### Rust ```rust use nemo_relay::observability::atof::{ AtofEndpointTransport, AtofExporter, AtofExporterConfig, AtofStreamSinkConfig, }; fn main() -> Result<(), Box> { let config = AtofExporterConfig::new().with_stream_sink(AtofStreamSinkConfig::new( "http://localhost:8080/events", AtofEndpointTransport::HttpPost, )); let exporter = AtofExporter::new(config)?; exporter.register("atof-exporter")?; // Run instrumented application work here. exporter.force_flush()?; let _ = exporter.deregister("atof-exporter")?; exporter.shutdown()?; Ok(()) } ``` ## Common Configuration and Runtime Issues Common configuration and runtime issues include: * ATOF is enabled without at least one plugin-configured sink. * A file sink's `mode` is not `append` or `overwrite`. * A stream sink has an empty `url`, an unsupported `transport`, or a `timeout_millis` value of `0`. * The output directory is not writable at runtime. * `nemo-relay doctor` cannot deliver its synthetic ATOF mark probe to a configured stream sink. * A file sink is configured in a target that cannot access the native filesystem. * The process terminates before a subscriber or exporter barrier, so the file or stream destination is missing queued tail records.