Operational Logging

Operational logging records diagnostics about the Relay process, including startup, configuration, plugins, gateway behavior, and runtime failures. It is separate from agent observability through ATOF, ATIF, OpenTelemetry, or OpenInference.

Relay writes operational logs to stderr by default. Set stderr_enabled to false to disable that sink; configured file sinks continue receiving records. Each record includes a root Relay ID for correlation.

Rust hosts can create a diagnostic context before logging initialization and pass it to LoggingRuntime::configure_with_diagnostics. Sink failures remain inspectable through that context even if logging cannot initialize. This opt-in API also projects typed diagnostic events through the configured logging destinations. File sinks are required by default. Set failure_policy = "optional" on an individual file sink to let startup continue when its file cannot be opened.

MCP failure records include error_kind. The mcp_session_failed record and the matching command_failed record for command = "mcp" also include failure_reason. This field is a short, fixed code that names the MCP step that failed. It never includes the source error text. Use the shared root Relay ID to match the two records. Read stderr for the full error message.

Gateway startup failures use the same pattern. gateway_acquisition_failed and gateway_recovery_failed include failure_reason as well as the existing failure_kind. For example, gateway_recovered_then_unhealthy means the gateway became unhealthy after Relay had already tried to recover it once.

Daemon worker activation failures include failure_reason on both worker_launch_failed and the matching worker_activation_failed record. The code shows whether Relay could not find or start the worker, send its activation permission, or wait for it to become ready. It never includes activation data or the source error text.

CLI Hook and Upstream Records

At debug level, the CLI writes lifecycle records for hooks and for gateway calls to an upstream LLM. This includes local hooks, persistent hooks, daemon hooks, and managed-worker hooks. A hook forwarder and the Relay handler use the same random operation_id. Each gateway call gets a new ID. If Relay selects a session, the record also includes session_id. This is a stable, derived tag, not the native session ID.

Normal lifecycle records are hook_started, hook_completed, upstream_started, upstream_headers_received, upstream_first_event, upstream_completed, and upstream_cancelled. They carry only operation_id, optional session_id, boundary, outcome, and elapsed_millis.

Warnings show slow or failed external work. They do not change request behavior:

• upstream_headers_delayed after 10 seconds without upstream response headers.
• upstream_first_event_delayed after 10 seconds without a decoded stream event.
• upstream_stream_stalled every 60 seconds while no decoded stream event arrives. Network heartbeats do not reset these timers.
• upstream_non_success when the upstream HTTP status is not a success.
• upstream_stream_read_failed when Relay can recover from a stream read failure.
• hook_failed for a hook that is allowed to fail open.

These timing records only report what happened. They do not cancel a request or a stream. Local gateway calls retain their 30-second connect limit. Provider waits have no deadline by default; [upstream] response_timeout_secs can set a limit. error records cover rejected payload and hook-response limits, hooks that must fail closed, upstream transport failures, and Relay adapter, session, or runtime failures. These records use safe fields such as error_kind, status_code, threshold_millis, and elapsed_millis. A limit_exceeded record also includes limit_name and the configured limit_bytes.

These records never include tool names or call IDs, hook event names, arguments, results, prompts, model names, provider names, route names, upstream URLs, headers, bodies, body sizes, or credentials. Relay removes its internal correlation header when it receives the request and never sends it upstream.

Defaults

Without configuration, Relay uses:

• error as the minimum log level
• Human-readable stderr output
• No file sinks

Choose a Configuration Source

Use the source that matches how Relay is launched:

For CLI commands, Relay applies logging settings in this order:

1. Built-in defaults
2. [logging] in the resolved Relay config.toml
3. NEMO_RELAY_LOG, NEMO_RELAY_LOG_STDERR, and NEMO_RELAY_LOG_STDERR_FORMAT
4. --log-level and --log-stderr-format

Each step changes only the fields it sets. If two steps set the same field, Relay uses the later value.

For example, NEMO_RELAY_LOG=info changes the minimum log level. It keeps the file sinks, rotation settings, and flush interval from TOML.

For the primary nemo-relay daemon service, step 2 reads [daemon.logging]. The daemon does not use [logging].

You can also choose a logging file directly. Relay then skips its usual search for config.toml:

• NEMO_RELAY_LOG_CONFIG_PATH loads the file you name. You can override its log level and stderr format with CLI options. Do not combine this variable with other logging environment variables.
• --log-config-path loads the file you name and ignores logging environment variables.

Python, Node.js, and Go bindings use environment settings or built-in defaults when they start LoggingRuntime. The logger stays active until the process exits. Rust applications choose how to set up LoggingRuntime. The CLI rules do not apply to these bindings or to Rust applications.

CLI Options

Configure the minimum level and stderr format directly:

nemo-relay --log-level debug --log-stderr-format jsonl

Use an absolute TOML path when file sinks or other logging settings are needed:

nemo-relay --log-config-path /absolute/path/to/logging.toml

Do not combine --log-config-path with --log-level or --log-stderr-format.

Environment Variables

Set these variables for a Python, Node.js, or Go process, the CLI, or a Rust application that initializes logging with LoggingRuntime::configure_from_environment():

export NEMO_RELAY_LOG=debug
export NEMO_RELAY_LOG_STDERR=false
export NEMO_RELAY_LOG_STDERR_FORMAT=jsonl

Supported values are:

• NEMO_RELAY_LOG: error, warn, info, debug, or trace
• NEMO_RELAY_LOG_STDERR: true (default) or false
• NEMO_RELAY_LOG_STDERR_FORMAT: human or jsonl

Alternatively, select an absolute TOML file:

export NEMO_RELAY_LOG_CONFIG_PATH=/absolute/path/to/logging.toml

NEMO_RELAY_LOG_CONFIG_PATH cannot be combined with the other logging environment variables.

When none of these variables are set, Python, Node.js, and Go install Relay’s built-in default logger unless the host already owns Rust’s process-global log facade. They also preserve an existing Relay logger rather than replacing it. Records emitted before any logger is installed are discarded, and a host cannot install its own logger after Relay has claimed the facade. Set one of these variables when Relay must configure its own logging.

This binding behavior differs from LoggingRuntime::configure_from_environment(), which always attempts to install Relay’s built-in defaults when no variables are set.

Python and Node.js drain pending file-sink records during normal runtime teardown. Go applications that configure file sinks must call nemo_relay.ShutdownLogging before main returns; defer it near the start of main so it runs after other Relay cleanup.

Emit records from language bindings

Python and Node.js can emit application records into Relay’s configured operational sinks. Pass a level, a message, optional JSON-object fields, and an optional subsystem target. The level takes the same names as logging configuration: error, warn (or warning), info, debug, or trace, in any case.

The target is not a full Relay target: use a subsystem such as payments.client. Relay prefixes it with the binding namespace. Omitting it uses nemo_relay.python or nemo_relay.node.

log::trace!(target: "payments.client"; "checking cache");
log::debug!(target: "payments.client", key = key; "cache key prepared");
log::info!(target: "payments.client", event = "cache_miss", attempt = 2; "cache lookup missed");
log::warn!(target: "payments.client", retry_after_ms = 100; "cache temporarily unavailable");
log::error!(target: "payments.client", error = %error; "cache request failed");

Relay preserves nested values in JSONL. These APIs do not replace Python’s root logger or Node’s console.

Emit records from plugins

Plugin records are sent to the Relay host, which applies its configured level filter and sinks. Plugin targets are subsystem names: Relay adds a native or worker-plugin namespace. Use structured fields rather than encoding data into the message.

use nemo_relay_plugin::{relay_debug, relay_error, relay_info, relay_trace, relay_warn};
use serde_json::json;
relay_trace!(runtime, target: "payments.client", "checking cache")?;
relay_debug!(runtime, target: "payments.client", fields: json!({"key": key}), "cache key prepared")?;
relay_info!(runtime, target: "payments.client", fields: json!({"event": "cache_miss", "attempt": 2}), "cache lookup missed")?;
relay_warn!(runtime, target: "payments.client", fields: json!({"retry_after_ms": 100}), "cache temporarily unavailable")?;
relay_error!(runtime, target: "payments.client", fields: json!({"error": error.to_string()}), "cache request failed")?;

Rust native plugins require a host with native ABI v6 or newer. The worker Log operation is additive to grpc-v1; a worker must use a host that exposes that operation.

TOML Configuration

Logging settings use a [logging] table:

[logging]
level = "info"
stderr_enabled = false
stderr_format = "human"
flush_interval_millis = 1000
[[logging.sinks]]
path = ".nemo-relay/logs/relay.log.jsonl"
format = "jsonl"
level = "debug"
queue_capacity = 1024
failure_policy = "required" # Default; use "optional" for best-effort startup.
max_file_size_bytes = 10485760
retained_files = 5

File sink paths are resolved relative to the process working directory. File sinks use asynchronous queues, and queue_capacity cannot exceed 8,192 entries per sink. Size-based rotation is optional; when enabled, max_file_size_bytes and retained_files must be configured together. retained_files counts backup files in addition to the active file and cannot exceed 9. A record larger than max_file_size_bytes remains intact rather than being split. File sinks remain append-only when rotation settings are omitted.

When Relay layers config.toml files, the resolved destination path is the file sink identity. Distinct paths are emitted in highest-to-lowest precedence order: system, then explicit-or-user. For matching paths, higher-layer fields recursively overlay lower-layer fields, producing one effective sink. Path aliases such as relay.log and ./relay.log resolve to one sink, using the higher-layer spelling and settings.

The primary daemon uses the identical schema under [daemon.logging]. Its configuration follows the normal explicit-or-user then system layering, with the system layer taking precedence:

[daemon.logging]
level = "info"
stderr_format = "jsonl"
[[daemon.logging.sinks]]
path = "/var/log/nemo-relay/daemon.jsonl"
format = "jsonl"

File Sink Startup Failure Policy

Each entry in [[logging.sinks]] or [[daemon.logging.sinks]] accepts failure_policy:

For example, make the general operational file optional while requiring the error-only file:

[logging]
level = "info"
[[logging.sinks]]
path = "/var/log/nemo-relay/relay.jsonl"
failure_policy = "optional"
[[logging.sinks]]
path = "/var/log/nemo-relay/errors.jsonl"
level = "error"
failure_policy = "required"

The policy applies to filesystem failures during initialization of both append-only and rotating files. Invalid settings, duplicate or colliding destinations, and failures to create logging delivery infrastructure still fail initialization. Relay validates all sink settings and reserves their destinations before opening any files. An unavailable optional sink does not conceal invalid configuration. The interactive configuration editor exposes this setting as Startup failure policy. Configuration layering merges it by destination path like other sink fields.

An unavailable optional sink produces one direct stderr warning with its stable logging.file.<index> identity and a fixed filesystem reason. This warning is emitted even when stderr logging or diagnostics is disabled. It excludes the file path and backend error text. With diagnostics enabled, the sink also records logging.sink.initialize_failed with failure class io. If the embedded service starts after logging, attaching it observes the sink’s current unavailable state once; it does not replay earlier log records. This observation does not change the daemon health probe or classify application readiness.

Disabled sinks have no delivery queue, retry buffer, or replay. Copies of records that would have gone to that sink are lost. Repairing the filesystem does not activate the sink: restart Relay, or shut down and reinitialize the owned Rust logging runtime, to retry. Other sinks keep receiving their copies. Successful sinks retain their existing queue, rotation, and runtime write-error handling. This policy controls initialization only.

Sinks are shared destinations. Diagnostic components do not each create a sink. The same sink policy applies to ordinary operational records and eligible diagnostic summaries delivered to that destination; diagnostic retention has its own budget and remains independent of delivery. The policy does not affect ATOF, ATIF, or OpenTelemetry exporters.

Python, Node.js, and Go use the same setting through the TOML file selected by NEMO_RELAY_LOG_CONFIG_PATH. No binding-specific startup option is needed.

Rust Library API

Initialize operational logging once during application startup by choosing one of these public APIs:

• LoggingRuntime::configure(config) for a constructed LoggingConfig
• LoggingRuntime::configure_from_environment() for environment configuration
• LoggingRuntime::configure_from_file_path(path) for an absolute TOML path

For example:

let _logging_runtime =
    nemo_relay::logging::LoggingRuntime::configure_from_environment()?;

Keep the returned runtime alive until application shutdown so pending file records can be flushed. For programmatic configuration, set FileLogSinkConfig.failure_policy to FileLogSinkFailurePolicy::Optional or FileLogSinkFailurePolicy::Required.