> This page is for NeMo Relay, version v0.10.0.
> For other versions, use one of these documentation indexes:
> - Latest (v0.10.0) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt
> - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt
> - v0.10.0: https://docs.nvidia.com/nemo/relay/v0.10.0/llms.txt
> - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt
> - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt
> - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt
> - v0.6.0: https://docs.nvidia.com/nemo/relay/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/nemo/relay/v0.5.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server.

# Operational Logging

> Configure NeMo Relay operational logs through CLI options, environment variables, TOML, or Rust APIs.

Operational logging records diagnostics about the Relay process, including
startup, configuration, plugins, gateway behavior, and runtime failures. It is
separate from agent observability through ATOF, ATIF, OpenTelemetry, or
OpenInference.

Relay writes operational logs to stderr by default. Set `stderr_enabled` to
`false` to disable that sink; optional file sinks continue receiving records.
Each record includes a root Relay ID for correlation.

MCP failure records include `error_kind`. The `mcp_session_failed` record and
the matching `command_failed` record for `command = "mcp"` also include
`failure_reason`. This field is a short, fixed code that names the MCP step that
failed. It never includes the source error text. Use the shared root Relay ID to
match the two records. Read stderr for the full error message.

Gateway startup failures use the same pattern. `gateway_acquisition_failed` and
`gateway_recovery_failed` include `failure_reason` as well as the existing
`failure_kind`. For example, `gateway_recovered_then_unhealthy` means the
gateway became unhealthy after Relay had already tried to recover it once.

Daemon worker activation failures include `failure_reason` on both
`worker_launch_failed` and the matching `worker_activation_failed` record. The
code shows whether Relay could not find or start the worker, send its activation
permission, or wait for it to become ready. It never includes activation data or the
source error text.

## CLI Hook and Upstream Records

At `debug` level, the CLI writes lifecycle records for hooks and for gateway calls
to an upstream LLM. This includes local hooks, persistent hooks, daemon hooks, and
managed-worker hooks. A hook forwarder and the Relay handler use the same random
`operation_id`. Each gateway call gets a new ID. If Relay selects a session, the
record also includes `session_id`. This is a stable, derived tag, not the native
session ID.

Normal lifecycle records are `hook_started`, `hook_completed`,
`upstream_started`, `upstream_headers_received`, `upstream_first_event`,
`upstream_completed`, and `upstream_cancelled`. They carry only `operation_id`,
optional `session_id`, `boundary`, `outcome`, and `elapsed_millis`.

Warnings show slow or failed external work. They do not change request behavior:

* `upstream_headers_delayed` after 10 seconds without upstream response headers.
* `upstream_first_event_delayed` after 10 seconds without a decoded stream event.
* `upstream_stream_stalled` every 60 seconds while no decoded stream event arrives.
  Network heartbeats do not reset these timers.
* `upstream_non_success` when the upstream HTTP status is not a success.
* `upstream_stream_read_failed` when Relay can recover from a stream read failure.
* `hook_failed` for a hook that is allowed to fail open.

These timing records only report what happened. They do not cancel a request or a
stream. Local gateway calls retain their 30-second connect limit. Provider waits
have no deadline by default; `[upstream] response_timeout_secs` can set a limit.
`error` records cover rejected payload and hook-response limits, hooks that
must fail closed, upstream transport failures, and Relay adapter, session, or
runtime failures. These records use safe fields such as `error_kind`, `status_code`,
`threshold_millis`, and `elapsed_millis`. A `limit_exceeded` record also includes
`limit_name` and the configured `limit_bytes`.

These records never include tool names or call IDs, hook event names, arguments,
results, prompts, model names, provider names, route names, upstream URLs, headers,
bodies, body sizes, or credentials. Relay removes its internal correlation header
when it receives the request and never sends it upstream.

## Defaults

Without configuration, Relay uses:

* `error` as the minimum log level
* Human-readable stderr output
* No file sinks

## Choose a Configuration Source

Use the source that matches how Relay is launched:

| Use Case                                    | Configuration Source                    |
| ------------------------------------------- | --------------------------------------- |
| Run the Relay CLI with temporary settings   | `--log-*` options                       |
| Configure a language binding or CLI process | `NEMO_RELAY_LOG*` environment variables |
| Reuse logging settings across runs          | `[logging]` in TOML                     |
| Configure the primary daemon service        | `[daemon.logging]` in TOML              |
| Embed Relay in a Rust application           | `LoggingConfig` and `LoggingRuntime`    |

For CLI processes, Relay selects one source in this order:

1. `--log-*` options or `--log-config-path`
2. `NEMO_RELAY_LOG*` environment variables
3. `[logging]` in the resolved Relay `config.toml`
4. Built-in defaults

For the primary `nemo-relay daemon` service, step 3 instead reads
`[daemon.logging]`. It does not inherit `[logging]`; explicit CLI and
environment sources retain the same precedence.

Sources are selected rather than merged. Python, Node.js, and Go install a
process-lifetime `LoggingRuntime` when the binding loads, using environment
configuration or built-in defaults. Rust applications explicitly choose which
`LoggingRuntime` initialization method to use and do not apply the CLI
precedence rules.

## CLI Options

Configure the minimum level and stderr format directly:

```bash
nemo-relay --log-level debug --log-stderr-format jsonl
```

Use an absolute TOML path when file sinks or other logging settings are needed:

```bash
nemo-relay --log-config-path /absolute/path/to/logging.toml
```

Do not combine `--log-config-path` with `--log-level` or
`--log-stderr-format`.

## Environment Variables

Set these variables for a Python, Node.js, or Go process, the CLI, or a Rust
application that initializes logging with
`LoggingRuntime::configure_from_environment()`:

```bash
export NEMO_RELAY_LOG=debug
export NEMO_RELAY_LOG_STDERR=false
export NEMO_RELAY_LOG_STDERR_FORMAT=jsonl
```

Supported values are:

* `NEMO_RELAY_LOG`: `error`, `warn`, `info`, `debug`, or `trace`
* `NEMO_RELAY_LOG_STDERR`: `true` (default) or `false`
* `NEMO_RELAY_LOG_STDERR_FORMAT`: `human` or `jsonl`

Alternatively, select an absolute TOML file:

```bash
export NEMO_RELAY_LOG_CONFIG_PATH=/absolute/path/to/logging.toml
```

`NEMO_RELAY_LOG_CONFIG_PATH` cannot be combined with the other logging
environment variables.

When none of these variables are set, Python, Node.js, and Go install Relay's
built-in default logger unless the host already owns Rust's process-global
`log` facade. They also preserve an existing Relay logger rather than replacing
it. Records emitted before any logger is installed are discarded, and a host
cannot install its own logger after Relay has claimed the facade. Set one of
these variables when Relay must configure its own logging.

This binding behavior differs from
`LoggingRuntime::configure_from_environment()`, which always attempts to
install Relay's built-in defaults when no variables are set.

Python and Node.js drain pending file-sink records during normal runtime
teardown. Go applications that configure file sinks must call
`nemo_relay.ShutdownLogging` before `main` returns; defer it near the start of
`main` so it runs after other Relay cleanup.

## Emit records from language bindings

Python and Node.js can emit application records into Relay's configured
operational sinks. Pass a level (`error`, `warn`, `info`, `debug`, or `trace`),
a message, optional JSON-object fields, and an optional subsystem target.

The target is not a full Relay target: use a subsystem such as
`payments.client`. Relay prefixes it with the binding namespace. Omitting it
uses `nemo_relay.python` or `nemo_relay.node`.

#### Rust

```rust
log::trace!(target: "payments.client"; "checking cache");
log::debug!(target: "payments.client", key = key; "cache key prepared");
log::info!(target: "payments.client", event = "cache_miss", attempt = 2; "cache lookup missed");
log::warn!(target: "payments.client", retry_after_ms = 100; "cache temporarily unavailable");
log::error!(target: "payments.client", error = %error; "cache request failed");
```

#### Python

```python
nemo_relay.trace("checking cache", target="payments.client")
nemo_relay.debug("cache key prepared", target="payments.client", fields={"key": key})
nemo_relay.info("cache lookup missed", target="payments.client", fields={"event": "cache_miss", "attempt": 2})
nemo_relay.warn("cache temporarily unavailable", target="payments.client", fields={"retry_after_ms": 100})
nemo_relay.error("cache request failed", target="payments.client", fields={"error": str(error)})
```

#### Node.js

```javascript
relay.trace("checking cache", "payments.client");
relay.debug("cache key prepared", "payments.client", { key });
relay.info("cache lookup missed", "payments.client", { event: "cache_miss", attempt: 2 });
relay.warn("cache temporarily unavailable", "payments.client", { retry_after_ms: 100 });
relay.error("cache request failed", "payments.client", { error: String(error) });
```

Relay preserves nested values in JSONL. These APIs do not replace Python's root
logger or Node's `console`.

## Emit records from plugins

Plugin records are sent to the Relay host, which applies its configured level
filter and sinks. Plugin targets are subsystem names: Relay adds a native or
worker-plugin namespace. Use structured fields rather than encoding data into
the message.

#### Rust Native

```rust
use nemo_relay_plugin::{relay_debug, relay_error, relay_info, relay_trace, relay_warn};
use serde_json::json;

relay_trace!(runtime, target: "payments.client", "checking cache")?;
relay_debug!(runtime, target: "payments.client", fields: json!({"key": key}), "cache key prepared")?;
relay_info!(runtime, target: "payments.client", fields: json!({"event": "cache_miss", "attempt": 2}), "cache lookup missed")?;
relay_warn!(runtime, target: "payments.client", fields: json!({"retry_after_ms": 100}), "cache temporarily unavailable")?;
relay_error!(runtime, target: "payments.client", fields: json!({"error": error.to_string()}), "cache request failed")?;
```

#### Rust gRPC

```rust
use serde_json::{Map, json};

let fields = Map::from_iter([("event".into(), json!("cache_miss")), ("attempt".into(), json!(2))]);
runtime.trace("payments.client", "checking cache", None).await?;
runtime.debug("payments.client", "cache key prepared", Some(&fields)).await?;
runtime.info("payments.client", "cache lookup missed", Some(&fields)).await?;
runtime.warn("payments.client", "cache temporarily unavailable", Some(&fields)).await?;
runtime.error("payments.client", "cache request failed", Some(&fields)).await?;
```

#### Python gRPC

```python
await runtime.trace("checking cache", target="payments.client")
await runtime.debug("cache key prepared", target="payments.client", fields={"key": key})
await runtime.info("cache lookup missed", target="payments.client", fields={"event": "cache_miss", "attempt": 2})
await runtime.warn("cache temporarily unavailable", target="payments.client", fields={"retry_after_ms": 100})
await runtime.error("cache request failed", target="payments.client", fields={"error": str(error)})
```

Rust native plugins require a host with native ABI v6 or newer. The worker
`Log` operation is additive to `grpc-v1`; a worker must use a host that exposes
that operation.

## TOML Configuration

Logging settings use a `[logging]` table:

```toml
[logging]
level = "info"
stderr_enabled = false
stderr_format = "human"
flush_interval_millis = 1000

[[logging.sinks]]
path = ".nemo-relay/logs/relay.log.jsonl"
format = "jsonl"
level = "debug"
queue_capacity = 1024
max_file_size_bytes = 10485760
retained_files = 5
```

File sink paths are resolved relative to the process working directory. File
sinks use asynchronous queues, and `queue_capacity` cannot exceed 8,192 entries
per sink. Size-based rotation is optional; when enabled,
`max_file_size_bytes` and `retained_files` must be configured together.
`retained_files` counts backup files in addition to the active file and cannot
exceed 9. A record larger than `max_file_size_bytes` remains intact rather
than being split. File sinks remain append-only when rotation settings are
omitted.

When Relay layers `config.toml` files, the resolved destination `path` is the
file sink identity. Distinct paths are emitted in highest-to-lowest precedence
order: system, then explicit-or-user. For matching paths, higher-layer
fields recursively overlay lower-layer fields, producing one effective sink.
Path aliases such as `relay.log` and `./relay.log` resolve to one sink, using
the higher-layer spelling and settings.

The primary daemon uses the identical schema under `[daemon.logging]`. Its
configuration follows the normal explicit-or-user then system layering, with
the system layer taking precedence:

```toml
[daemon.logging]
level = "info"
stderr_format = "jsonl"

[[daemon.logging.sinks]]
path = "/var/log/nemo-relay/daemon.jsonl"
format = "jsonl"
```

## Rust Library API

Initialize operational logging once during application startup by choosing one
of these public APIs:

* `LoggingRuntime::configure(config)` for a constructed `LoggingConfig`
* `LoggingRuntime::configure_from_environment()` for environment configuration
* `LoggingRuntime::configure_from_file_path(path)` for an absolute TOML path

For example:

```rust
let _logging_runtime =
    nemo_relay::logging::LoggingRuntime::configure_from_environment()?;
```

Keep the returned runtime alive until application shutdown so pending file
records can be flushed.