Diagnostics API

Use the diagnostics API to inspect component activity, total counts, and recent events in a running Relay context. Reads do not reset counters or change execution. An empty event list does not prove that the runtime is healthy or has recovered. Check the snapshot’s coverage and retention fields for missing evidence.

Diagnostics is disabled by default. Enable it at startup through your deployment’s diagnostics configuration. Reading does not start a gateway, initialize plugins, or enable collection. A reused gateway keeps its existing settings until explicitly restarted.

Choose a Reader

All readers return the same snapshot contract. Choose the interface for the runtime you want to inspect:

Refer to CLI and Daemon Diagnostics or Embedded Diagnostics for startup configuration and language examples. A dynamic-plugin worker submits observations to its host. Its recording permission does not grant access to host snapshots. The plugin host’s reader remains available after a plugin closes or fails. Native SDK readers are activation-scoped and require a separate whole-context policy grant. Refer to Native Plugin Diagnostics for that permission and its lifecycle.

Discover Capabilities and Limits

Before choosing page sizes, inspect the running context’s capabilities. To inspect an existing gateway from the CLI, run this command:

nemo-relay diagnostics --capabilities

For a language-hosted runtime, choose your binding below. These examples assume that the service is already configured. The Rust example uses a core DiagnosticsReader. An embedded Rust reader exposes discovery through reader.info()?.discovery.

let discovery = reader.discovery();

Check these fields before requesting a snapshot:

Discovery remains available when max_queries = 0, although HTTP still requires an available connection and response slot. HTTP adds the process’s CLI/daemon producers, inspection_policy, limits, and response budget. Embedded metadata adds logging coverage and memory/copy budgets. Missing capability fields in an older response mean that the capability is unadvertised.

Read a Snapshot

For example, request up to ten recent events and omit component rows:

nemo-relay diagnostics --query '{"max_components":0,"max_events":10}'

This form works in Bash and PowerShell 7 with modern native argument passing. For Windows PowerShell 5.1, escape the embedded JSON quotes:

nemo-relay diagnostics --query '{\"max_components\":0,\"max_events\":10}'

See JSON Queries in Windows Shells for PowerShell’s legacy argument-passing mode.

Use a max_events value within the limits returned by discovery. Add --gateway-url http://127.0.0.1:4040 when selecting an existing gateway explicitly. The command uses Relay’s authenticated, pinned TLS tunnel. That URL is not a plain-HTTP read credential.

HTTP readers use these routes:

Pass a query as one percent-encoded JSON query parameter. Canonical JSON uses snake_case. Node.js native helpers use camelCase. An omitted or null schema_version selects the runtime’s current snapshot version.

Queries support these selections and page controls:

Requests above configured row or byte limits return invalid_query. Limits are not silently clamped. Core and binding JSON query input is limited to 4,096 UTF-8 bytes. CLI and HTTP requests share a 4,096-byte URI query budget, including query= and percent encoding. This is not a 4,096-byte allowance for raw JSON. The original selectors and any continuation must fit together within that budget. Duplicate or unknown fields, including nested selectors, are rejected. An accepted response-byte cap that cannot fit the required envelope or a row returns snapshot_capacity.

Interpret the Page

A snapshot identifies its source, capture time, and source update sequence. It contains these fields:

• lifetime holds totals for the whole context.
• components holds selected component rows.
• events holds selected recent observations.
• coverage describes missing or incomplete evidence.
• continuation holds an optional cursor for the next page.

Filters select rows. They do not filter the context-wide totals. Component lifetime summaries cover the whole registration instance, while since_refresh describes its current refresh period. Inspect coverage for unavailable evidence, truncated detail, evictions, and saturated counts. Evicted events do not erase accumulated lifetime counts. A saturated count is a lower bound.

Distinguish Calls, Occurrences, and Retained Evidence

One recording call can describe several occurrences. For example, two accepted failure observations with occurrence deltas of two and three produce these totals:

{
  "recorded_events_total": {"value": "2", "saturated": false},
  "occurrences_total": {"value": "5", "saturated": false},
  "failures_total": {"value": "5", "saturated": false},
  "history_evictions_total": {"value": "0", "saturated": false}
}

This is an excerpt from lifetime.totals, not a complete snapshot. If history later evicts one of those events, the first three totals remain unchanged and history_evictions_total increases. Event rows describe recording calls. They are not one row per occurrence.

A window’s coverage.partial_start and partial_end identify partial edge buckets. incomplete identifies known loss. Use the actual covered interval when interpreting rates. Do not treat a partial bucket as a full interval or assume that an unavailable window is a zero-error window.

Distinguish Retention Loss from Read Limits

Inspect coverage.loss and coverage.retention separately:

Loss counters are cumulative source information, not losses incurred by this particular filtered read. Reclamation of a closed component can remove its component aggregates, while context lifetime totals survive. Neither retained row counts nor a page’s event list can reconstruct all lifetime occurrences.

Continue or Restart a Read

If continuation is non-null, copy that object unchanged into the next query, keeping the original selectors and page limits. Authorize every request. Stop when the response has a null continuation.

A cursor binds the source instance, selection, and source revision. New observations or eviction can invalidate it. On invalid_cursor, discard the cursor and restart the query. Pages are not a frozen transaction: do not add repeated context-wide lifetime totals together across pages or polling attempts.

The daemon’s cached-worker wrapper has separate receipt and freshness metadata. It preserves each worker’s source identity and accepts no query selectors or continuations. Query an authorized source reader when you need filtered pages.

Handle Read Errors

Python and Node.js errors expose .reason. Rust maps DiagnosticError through read_reason() to DiagnosticReadReason. Include a fallback arm when matching that non-exhaustive Rust enum. Native SDK readers expose stable labels through nemo_relay_plugin::diagnostics::DiagnosticError::reason(). A denied native read returns permission_denied. A revoked activation returns closed. These are separate from the HTTP status mapping below.

The HTTP error body has this form:

{"error":{"reason":"invalid_query"}}

An unsupported_schema response also includes supported_schema_version. Startup can fail with invalid_limits or allocation_failed. Binding lifecycle errors such as not_initialized, configuration_conflict, and fork_requires_exec are separate from snapshot failures.

HTTP authorization failures return 401 or 403. Plain HTTP returns 426 and requires Relay’s authenticated TLS upgrade. Errors contain no raw query values, credentials, or retained diagnostic detail. These failures describe the inspection operation. They do not classify runtime health.

Preserve Numeric Precision and Compatibility

Canonical JSON uses unsigned decimal strings for these 64-bit values:

• Counts.
• Generations.
• Sequences.
• Durations.
• Discovery capacities.

This keeps values exact above 2^53, up to the unsigned 64-bit maximum. Schema and catalogue versions, process IDs, and small row caps remain JSON numbers.

Missing or null data means the value is unavailable. It does not mean zero or recovery. Readers preserve unknown catalogue and plugin-context names within a supported snapshot version. Snapshots allow unknown object fields so later versions can add fields. Queries reject unknown fields to catch invalid requests.

The Rust decoder rejects unsupported snapshot versions. Python and Node.js read pages from their matching runtime. They do not decode arbitrary remote snapshots.

Protect Diagnostic Data

Authorize readers for the whole context: filters and cursors are not access controls. HTTP responses use Cache-Control: no-store. Names, paths, detail, and caller correlation can contain unsanitized data. Truncation does not sanitize it.

Snapshot reads never automatically feed logs, legacy reports, subscribers, or exporters. Logging projections receive only the approved built-in summary. State-only recording invokes no projection. An application that exports a reader result must apply its own authorization and data-handling policy.

Diagnostic storage and delivery have separate budgets. Caller-retained JSON and decoded objects are outside the service’s storage limits. Memory gauges describe core allocation and occupancy, not a whole-process RSS ceiling or a guaranteed polling rate. Refer to Runtime Diagnostics for the deployment-specific budgets.

These existing interfaces retain their own contracts:

• Health and status endpoints.
• Configuration validation.
• Plugin reports.
• Native legacy callbacks.
• Worker GetRuntimeDiagnostics. Snapshot readers do not fall back to those reports. OTel diagnostics integration, CPU attribution, and durable restart history are not supported.

Detailed Memory Inspection

Use the same authorized reader to request a separate bounded memory page. Choose the operation for your reader:

• In Rust, call reader.inspect_memory(&query).
• In the CLI, run nemo-relay diagnostics --memory.
• Over the existing authorized HTTP transport, use GET /_nemo_relay/v1/diagnostics/memory.

The memory inspection guide explains the binding APIs, memory accounting, and resource availability. It also describes memory pages’ separate cursors and schema.