> This page is for NeMo Relay, version dev. > For other versions, use one of these documentation indexes: > - Latest (v0.10.4) (default): https://docs.nvidia.com/nemo/relay/latest/llms.txt > - dev: https://docs.nvidia.com/nemo/relay/dev/llms.txt > - v0.11.0: https://docs.nvidia.com/nemo/relay/v0.11.0/llms.txt > - v0.10.4: https://docs.nvidia.com/nemo/relay/v0.10.4/llms.txt > - v0.9.4: https://docs.nvidia.com/nemo/relay/v0.9.4/llms.txt > - v0.8.4: https://docs.nvidia.com/nemo/relay/v0.8.4/llms.txt > - v0.7.3: https://docs.nvidia.com/nemo/relay/v0.7.3/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/relay/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/relay/_mcp/server. # Diagnostics API > Discover diagnostic capabilities, read bounded snapshots, and handle pagination and errors. Use the diagnostics API to inspect component activity, total counts, and recent events in a running Relay context. Reads do not reset counters or change execution. An empty event list does not prove that the runtime is healthy or has recovered. Check the snapshot's coverage and retention fields for missing evidence. Diagnostics is disabled by default. Enable it at startup through your deployment's [diagnostics configuration](/nemo/relay/dev/reference/diagnostics). Reading does not start a gateway, initialize plugins, or enable collection. A reused gateway keeps its existing settings until explicitly restarted. ## Choose a Reader All readers return the same snapshot contract. Choose the interface for the runtime you want to inspect: | Runtime | Reader | | ------------------------------------------- | ------------------------------------------------------------------------------------ | | Rust-owned context | `DiagnosticsReader` | | Rust native plugin with a host-policy grant | `runtime.diagnostics().reader()` | | Existing CLI gateway | `nemo-relay diagnostics` or authorized diagnostics HTTP routes | | Daemon broker or managed Relay worker | Use authorized diagnostics HTTP routes. The broker also exposes cached worker pages. | | Embedded Python or Node.js | Optional diagnostics reader with native objects or canonical JSON | Refer to [CLI and Daemon Diagnostics](/nemo/relay/dev/reference/diagnostics/cli) or [Embedded Diagnostics](/nemo/relay/dev/reference/diagnostics/bindings) for startup configuration and language examples. A dynamic-plugin worker submits observations to its host. Its recording permission does not grant access to host snapshots. The plugin host's reader remains available after a plugin closes or fails. Native SDK readers are activation-scoped and require a separate whole-context policy grant. Refer to [Native Plugin Diagnostics](/nemo/relay/dev/build-plugins/native/diagnostics#read-host-evidence) for that permission and its lifecycle. ## Discover Capabilities and Limits Before choosing page sizes, inspect the running context's capabilities. To inspect an existing gateway from the CLI, run this command: **`CLI`** ```bash title="CLI" showLineNumbers={false} nemo-relay diagnostics --capabilities ``` For a language-hosted runtime, choose your binding below. These examples assume that the service is already configured. The Rust example uses a core `DiagnosticsReader`. An embedded Rust reader exposes discovery through `reader.info()?.discovery`. #### Rust ```rust title={"Read capabilities"} showLineNumbers={false} let discovery = reader.discovery(); ``` #### Python ```python title={"Read capabilities"} showLineNumbers={false} from nemo_relay import diagnostics reader = diagnostics.reader() discovery = reader.info()["discovery"] ``` #### Node.js ```javascript title={"Read capabilities"} showLineNumbers={false} const discovery = reader.info().discovery; ``` Check these fields before requesting a snapshot: | Field | How to Use It | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `discovery_version`, `snapshot_version`, `catalogue_version` | Check the independent contract versions, currently 1. | | `source` | Identify the context instance and process that owns the evidence. | | `collection_enabled`, `queries_enabled` | Distinguish disabled collection from collection with no snapshot workspaces. | | `max_query_components`, `max_query_events`, `max_snapshot_bytes` | Keep requested row counts and response bytes within these page limits. | | `producer_support` | Check which producer integrations the build supports. This does not confirm that a particular plugin is configured or has emitted evidence. | | `memory_inspection` | Indicates support for aggregate core allocation and occupancy inspection. | | `detailed_memory_inspection`, `resource_correlation` | Indicate compiled per-entry and component accounting and latest resource-sample correlation, independently of collection/query enablement. | | `restricted_detail` | Indicates that inspection can expose bounded, unsanitized evidence. | Discovery remains available when `max_queries = 0`, although HTTP still requires an available connection and response slot. HTTP adds the process's CLI/daemon `producers`, `inspection_policy`, limits, and response budget. Embedded metadata adds logging coverage and memory/copy budgets. Missing capability fields in an older response mean that the capability is unadvertised. ## Read a Snapshot For example, request up to ten recent events and omit component rows: ```bash nemo-relay diagnostics --query '{"max_components":0,"max_events":10}' ``` This form works in Bash and PowerShell 7 with modern native argument passing. For Windows PowerShell 5.1, escape the embedded JSON quotes: ```powershell nemo-relay diagnostics --query '{\"max_components\":0,\"max_events\":10}' ``` See [JSON Queries in Windows Shells](/nemo/relay/dev/reference/diagnostics/cli#json-queries-in-windows-shells) for PowerShell's legacy argument-passing mode. Use a `max_events` value within the limits returned by discovery. Add `--gateway-url http://127.0.0.1:4040` when selecting an existing gateway explicitly. The command uses Relay's authenticated, pinned TLS tunnel. That URL is not a plain-HTTP read credential. HTTP readers use these routes: | GET Route | Result | | ------------------------------------------ | -------------------------------------------- | | `/_nemo_relay/v1/diagnostics/capabilities` | Discovery metadata | | `/_nemo_relay/v1/diagnostics` | One snapshot page | | `/_nemo_relay/v1/diagnostics/components` | A snapshot page with event rows disabled | | `/_nemo_relay/v1/diagnostics/events` | A snapshot page with component rows disabled | Pass a query as one percent-encoded JSON `query` parameter. Canonical JSON uses `snake_case`. Node.js native helpers use `camelCase`. An omitted or null `schema_version` selects the runtime's current snapshot version. Queries support these selections and page controls: | Fields | Selection | | ---------------------------------------- | ------------------------------------------------------------------------------------------------- | | `component_key`, `component_instance_id` | A logical component or one registration instance | | `code`, `domain`, `plugin_family` | Exact code or catalogue ownership | | `failure_class` | Selects events by cause. Code counters do not retain a cause breakdown. | | `ownership` | Component rows by current ownership and events by their original ownership | | `origin_generation`, `after_sequence` | Selects recent events from a refresh generation or after a sequence. Summaries remain cumulative. | | `max_components`, `max_events` | Sets the requested row caps. Zero omits that row kind. | | `max_response_bytes` | Requested encoded page cap | | `continuation` | The cursor returned by the previous page | Requests above configured row or byte limits return `invalid_query`. Limits are not silently clamped. Core and binding JSON query input is limited to 4,096 UTF-8 bytes. CLI and HTTP requests share a 4,096-byte URI query budget, including `query=` and percent encoding. This is not a 4,096-byte allowance for raw JSON. The original selectors and any continuation must fit together within that budget. Duplicate or unknown fields, including nested selectors, are rejected. An accepted response-byte cap that cannot fit the required envelope or a row returns `snapshot_capacity`. ### Interpret the Page A snapshot identifies its `source`, capture time, and source update `sequence`. It contains these fields: * `lifetime` holds totals for the whole context. * `components` holds selected component rows. * `events` holds selected recent observations. * `coverage` describes missing or incomplete evidence. * `continuation` holds an optional cursor for the next page. Filters select rows. They do not filter the context-wide totals. Component lifetime summaries cover the whole registration instance, while `since_refresh` describes its current refresh period. Inspect `coverage` for unavailable evidence, truncated detail, evictions, and saturated counts. Evicted events do not erase accumulated lifetime counts. A saturated count is a lower bound. ### Distinguish Calls, Occurrences, and Retained Evidence One recording call can describe several occurrences. For example, two accepted failure observations with occurrence deltas of two and three produce these totals: ```json { "recorded_events_total": {"value": "2", "saturated": false}, "occurrences_total": {"value": "5", "saturated": false}, "failures_total": {"value": "5", "saturated": false}, "history_evictions_total": {"value": "0", "saturated": false} } ``` This is an excerpt from `lifetime.totals`, not a complete snapshot. If history later evicts one of those events, the first three totals remain unchanged and `history_evictions_total` increases. Event rows describe recording calls. They are not one row per occurrence. | Field or View | Interpretation | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lifetime.totals.recorded_events_total` | Accepted recording calls, independent of each call's occurrence delta. | | `lifetime.totals.occurrences_total` | Sum of accepted deltas across success, failure, and unspecified outcomes. | | `lifetime.totals.failures_total` | Counts explicit failed-operation occurrences. Different producers can observe the same request, so this is not a unique failed-request count. | | Component `since_refresh` | The current counting period for that registration. Refresh resets this period, not its lifetime totals or rolling windows. | | `lifetime.elapsed_millis` | Monotonic elapsed context or component lifetime, not CPU time. | | `lifetime.since_last_failure_millis` | Elapsed time since an observed failure. A missing value means no failure has established this value. It does not prove health or recovery. | | Component `windows` | Fixed-resolution outcome buckets with their own covered interval. Batched deltas belong to their recording interval. Individual occurrence times cannot be reconstructed. | A window's `coverage.partial_start` and `partial_end` identify partial edge buckets. `incomplete` identifies known loss. Use the actual covered interval when interpreting rates. Do not treat a partial bucket as a full interval or assume that an unavailable window is a zero-error window. ### Distinguish Retention Loss from Read Limits Inspect `coverage.loss` and `coverage.retention` separately: | Field | Meaning and Consequence | | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `loss.events_evicted` | Previously retained evidence was reclaimed. Accepted lifetime totals remain. | | `loss.events_omitted` | An observation was counted but its incoming event evidence was not retained. | | `loss.observations_rejected` | An observation was rejected before canonical counting. Its occurrences are absent from accepted totals. | | `loss.messages_truncated`, `details_omitted` | Detail was shortened or omitted. The retained event can lack the original message. | | `loss.correlations_truncated`, `correlations_omitted` | Correlation was shortened or omitted independently of the event's numeric accounting. | | `retention.query_truncated` | The page omitted matching retained rows because of read limits. This does not mean storage evicted those rows. Inspect `continuation`. | Loss counters are cumulative source information, not losses incurred by this particular filtered read. Reclamation of a closed component can remove its component aggregates, while context lifetime totals survive. Neither retained row counts nor a page's event list can reconstruct all lifetime occurrences. ### Continue or Restart a Read If `continuation` is non-null, copy that object unchanged into the next query, keeping the original selectors and page limits. Authorize every request. Stop when the response has a null continuation. A cursor binds the source instance, selection, and source revision. New observations or eviction can invalidate it. On `invalid_cursor`, discard the cursor and restart the query. Pages are not a frozen transaction: do not add repeated context-wide lifetime totals together across pages or polling attempts. The daemon's cached-worker wrapper has separate receipt and freshness metadata. It preserves each worker's source identity and accepts no query selectors or continuations. Query an authorized source reader when you need filtered pages. ## Handle Read Errors Python and Node.js errors expose `.reason`. Rust maps `DiagnosticError` through `read_reason()` to `DiagnosticReadReason`. Include a fallback arm when matching that non-exhaustive Rust enum. Native SDK readers expose stable labels through `nemo_relay_plugin::diagnostics::DiagnosticError::reason()`. A denied native read returns `permission_denied`. A revoked activation returns `closed`. These are separate from the HTTP status mapping below. | Reason | HTTP Status | Action | | -------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `disabled` | 503 | Enable diagnostics through startup configuration. | | `query_disabled` | 503 | Configure a nonzero query pool at startup. Collection can still be enabled. | | `unsupported_schema` | 406 | Use the advertised supported snapshot version. | | `invalid_query` | 400 | Check fields, numeric encodings, and discovered limits. | | `invalid_cursor` | 409 | Restart the query without a continuation. | | `busy` | 429 | Retry later. HTTP supplies `Retry-After: 1`. | | `snapshot_capacity` | 422 | Increase the requested byte cap within the discovered limit. If a required row still cannot fit, adjust startup limits or narrow the selection. | | `internal` | 500 | Treat the read as failed. No raw source error is exposed. | The HTTP error body has this form: ```json {"error":{"reason":"invalid_query"}} ``` An `unsupported_schema` response also includes `supported_schema_version`. Startup can fail with `invalid_limits` or `allocation_failed`. Binding lifecycle errors such as `not_initialized`, `configuration_conflict`, and `fork_requires_exec` are separate from snapshot failures. HTTP authorization failures return 401 or 403. Plain HTTP returns 426 and requires Relay's authenticated TLS upgrade. Errors contain no raw query values, credentials, or retained diagnostic detail. These failures describe the inspection operation. They do not classify runtime health. ## Preserve Numeric Precision and Compatibility Canonical JSON uses unsigned decimal strings for these 64-bit values: * Counts. * Generations. * Sequences. * Durations. * Discovery capacities. This keeps values exact above 2^53, up to the unsigned 64-bit maximum. Schema and catalogue versions, process IDs, and small row caps remain JSON numbers. | Reader | Diagnostic 64-Bit Values | | -------------- | ------------------------------------------------------------------------------------------------------- | | Rust | `DiagnosticNumber` with checked `u64` access | | Python native | `int` | | Node.js native | Uses `bigint`. Call `snapshotJson()` for portable JSON because `JSON.stringify` cannot encode `bigint`. | | Canonical JSON | Unsigned decimal strings | Missing or null data means the value is unavailable. It does not mean zero or recovery. Readers preserve unknown catalogue and plugin-context names within a supported snapshot version. Snapshots allow unknown object fields so later versions can add fields. Queries reject unknown fields to catch invalid requests. The Rust decoder rejects unsupported snapshot versions. Python and Node.js read pages from their matching runtime. They do not decode arbitrary remote snapshots. ## Protect Diagnostic Data Authorize readers for the whole context: filters and cursors are not access controls. HTTP responses use `Cache-Control: no-store`. Names, paths, detail, and caller correlation can contain unsanitized data. Truncation does not sanitize it. Snapshot reads never automatically feed logs, legacy reports, subscribers, or exporters. Logging projections receive only the approved built-in summary. State-only recording invokes no projection. An application that exports a reader result must apply its own authorization and data-handling policy. Diagnostic storage and delivery have separate budgets. Caller-retained JSON and decoded objects are outside the service's storage limits. Memory gauges describe core allocation and occupancy, not a whole-process RSS ceiling or a guaranteed polling rate. Refer to [Runtime Diagnostics](/nemo/relay/dev/reference/diagnostics) for the deployment-specific budgets. These existing interfaces retain their own contracts: * Health and status endpoints. * Configuration validation. * Plugin reports. * Native legacy callbacks. * Worker `GetRuntimeDiagnostics`. Snapshot readers do not fall back to those reports. OTel diagnostics integration, CPU attribution, and durable restart history are not supported. ## Detailed Memory Inspection Use the same authorized reader to request a separate bounded memory page. Choose the operation for your reader: * In Rust, call `reader.inspect_memory(&query)`. * In the CLI, run `nemo-relay diagnostics --memory`. * Over the existing authorized HTTP transport, use `GET /_nemo_relay/v1/diagnostics/memory`. The [memory inspection guide](/nemo/relay/dev/reference/diagnostics/memory) explains the binding APIs, memory accounting, and resource availability. It also describes memory pages' separate cursors and schema.