> This page is for NVSentinel, version v1.27.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt
> - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt
> - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server.

# NVSentinel Metrics

This document outlines all Prometheus metrics exposed by NVSentinel components.

## Table of Contents

- [Fault Quarantine Module](#fault-quarantine)
- [Node Drainer Module](#node-drainer)
- [Fault Remediation Module](#fault-remediation)
- [Labeler Module](#labeler)
- [Janitor](#janitor)
- [Platform Connectors](#platform-connectors)
- [Health Monitors](#health-monitors)
  - [Health Event Publisher](#health-event-publisher)
  - [GPU Health Monitor](#gpu-health-monitor)
  - [NIC Health Monitor](#nic-health-monitor)
  - [Syslog Health Monitor](#syslog-health-monitor)
  - [CSP Health Monitor](#csp-health-monitor)
- [Change Stream Metrics](#change-stream-metrics)

---

## Fault Quarantine Module

### Event Processing Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_quarantine_events_received_total` | Counter | - | Total number of events received from the watcher |
| `fault_quarantine_events_successfully_processed_total` | Counter | - | Total number of events successfully processed |
| `fault_quarantine_processing_errors_total` | Counter | `error_type` | Total number of errors encountered during event processing |
| `fault_quarantine_cold_start_events_total` | Counter | `result` | Health events examined during startup recovery, labeled `processed`, `skipped`, `superseded`, `invalid`, or `failed` |
| `fault_quarantine_cold_start_duration_seconds` | Histogram | - | Startup recovery duration. Buckets: ExponentialBuckets(start=0.1s, factor=2, count=18), max ~3.6 hours |
| `fault_quarantine_event_backlog_count` | Gauge | - | Number of health events which fault quarantine is yet to process |
| `fault_quarantine_event_handling_duration_seconds` | Histogram | - | Histogram of event handling durations |

### Node Quarantine Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_quarantine_nodes_quarantined_total` | Counter | `node` | Total number of nodes quarantined |
| `fault_quarantine_nodes_unquarantined_total` | Counter | `node` | Total number of nodes unquarantined |
| `fault_quarantine_nodes_manually_uncordoned_total` | Counter | `node` | Total number of manually uncordons for nodes |
| `fault_quarantine_nodes_manually_untainted_total` | Counter | `node` | Total number of manual taint removals for nodes |
| `fault_quarantine_current_quarantined_nodes` | Gauge | `node` | Current number of quarantined nodes |

### Taint and Cordon Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_quarantine_taints_applied_total` | Counter | `taint_key`, `taint_effect` | Total number of taints applied to nodes |
| `fault_quarantine_taints_removed_total` | Counter | `taint_key`, `taint_effect` | Total number of taints removed from nodes |
| `fault_quarantine_cordons_applied_total` | Counter | - | Total number of cordons applied to nodes |
| `fault_quarantine_cordons_removed_total` | Counter | - | Total number of cordons removed from nodes |
| `fault_quarantine_node_quarantine_duration_seconds` | Histogram | - | Time from health event generation to node quarantine completion. Buckets: Prometheus DefBuckets |
| `fault_quarantine_node_remediation_duration_seconds` | Histogram | `recommended_action` | End-to-end node remediation time: `generatedTimestamp` (from original unhealthy event) to node unquarantine. Emitted on both auto unquarantine (via healthy event) and manual uncordon. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=27), max ~4.4 days |
| `fault_quarantine_node_remediation_duration_excluding_drain_seconds` | Histogram | `recommended_action` | Remediation time excluding node-drainer duration: `(unquarantineTime - generatedTimestamp) - (drainFinishTimestamp - quarantineFinishTimestamp)`. Emitted only when both `quarantineFinishTimestamp` and `drainFinishTimestamp` are present in the original event document. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=19), max ~4.1 hours |

### Ruleset Evaluation Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_quarantine_ruleset_evaluations_total` | Counter | `ruleset`, `status` | Total number of ruleset evaluations. Status values: `passed`, `failed` |

### Circuit Breaker Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_quarantine_breaker_state` | Gauge | `state` | State of the fault quarantine breaker |
| `fault_quarantine_breaker_utilization` | Gauge | - | Fraction of GPU nodes cordoned within the circuit breaker's sliding window |
| `fault_quarantine_breaker_threshold_nodes` | Gauge | `bound` | Cordoned-node count that trips the circuit breaker, and which configured bound produced it (`percentage`, `maxNodes`, or `fleetSize` when clamped) |
| `fault_quarantine_get_total_nodes_duration_seconds` | Histogram | `result` | Duration of getTotalNodesWithRetry calls in seconds |
| `fault_quarantine_get_total_nodes_errors_total` | Counter | `error_type` | Total number of errors from getTotalNodesWithRetry |
| `fault_quarantine_get_total_nodes_retry_attempts` | Histogram | - | Number of retry attempts needed for getTotalNodesWithRetry (buckets: 0, 1, 2, 3, 5, 10) |

---

## Node Drainer Module

### Event Processing Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `node_drainer_events_received_total` | Counter | - | Total number of events received from the watcher |
| `node_drainer_events_replayed_total` | Counter | - | Total number of in-progress events replayed at startup |
| `node_drainer_events_processed_total` | Counter | `drain_status`, `node` | Total number of events processed by drain status outcome. Drain status values: `drained`, `cancelled`, `skipped` |
| `node_drainer_cancelled_event_total` | Counter | `node`, `check_name` | Total number of cancelled drain events (due to manual uncordon or healthy recovery) |
| `node_drainer_processing_errors_total` | Counter | `error_type`, `node` | Total number of errors encountered during event processing and node draining |
| `node_drainer_event_handling_duration_seconds` | Histogram | - | Histogram of event handling durations |
| `node_drainer_queue_depth` | Gauge | - | Total number of pending events in the queue |
| `node_drainer_pod_eviction_duration_seconds` | Histogram | - | Time from event receipt by node-drainer to successful pod eviction completion. Buckets: Exponential (0.1s, factor 2, 23 buckets, up to ~3 days) |

### Node Draining Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `node_drainer_waiting_for_timeout` | Gauge | `node` | Shows if node drainer operation is waiting for timeout before force deletion (1=waiting, 0=not waiting) |
| `node_drainer_force_delete_pods_after_timeout` | Counter | `node`, `namespace` | Total number of node drainer operations that reached timeout and force deleted pods |

---

## Fault Remediation Module

### Event Processing Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_remediation_events_received_total` | Counter | - | Total number of events received from the watcher |
| `fault_remediation_events_processed_total` | Counter | `cr_status`, `node_name` | Total number of remediation events processed by CR creation status. CR status values: `created`, `skipped` |
| `fault_remediation_processing_errors_total` | Counter | `error_type`, `node_name` | Total number of errors encountered during event processing |
| `fault_remediation_unsupported_actions_total` | Counter | `action`, `node_name` | Total number of health events with currently unsupported remediation actions |
| `fault_remediation_event_handling_duration_seconds` | Histogram | - | Histogram of event handling durations |
| `fault_remediation_cr_generate_duration_seconds` | Histogram | - | Time from drain completion (or quarantine completion if drain timestamp unavailable) to maintenance CR creation. Buckets: Prometheus DefBuckets |

### Log Collector Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fault_remediation_log_collector_jobs_total` | Counter | `node_name`, `status` | Total number of log collector jobs. Status values: `success`, `failure`, `timeout` |
| `fault_remediation_log_collector_job_duration_seconds` | Histogram | `node_name`, `status` | Duration of log collector jobs in seconds |
| `fault_remediation_log_collector_errors_total` | Counter | `error_type`, `node_name` | Total number of errors encountered in log collector operations |

### File Server Metrics

#### HTTP Request Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `http_response_count_total` | Counter | `method`, `status`, `app` | Total HTTP responses by method and status code |
| `http_request_duration_seconds` | Histogram | `method`, `status` | HTTP request duration in seconds |

#### Log Rotation Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `fileserver_log_rotation_successful_total` | Counter | - | Total successful log cleanup operations |
| `fileserver_log_rotation_failed_total` | Counter | - | Total failed log cleanup operations |
| `fileserver_disk_space_free_bytes` | Gauge | - | Free disk space in bytes |

---

## Labeler Module

### Event Processing Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `labeler_events_processed_total` | Counter | `status` | Total number of pod events processed. Status values: `success`, `failed` |
| `labeler_node_update_failures_total` | Counter | - | Total number of node update failures during reconciliation |
| `labeler_event_handling_duration_seconds` | Histogram | - | Histogram of event handling durations |

---

## Janitor

### Action Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `janitor_actions_count` | Counter | `action_type`, `status`, `node` | Total number of janitor actions by type and status. Action types: `reboot`, `terminate`. Status values: `started`, `succeeded`, `failed` |
| `janitor_action_mttr_seconds` | Histogram | `action_type` | Time from CR creation to action completion (Mean Time To Repair). Uses exponential buckets (10, 2, 10) for log-scale MTTR measurement |

---

## Platform Connectors

### Health Event Authentication Metrics

These track the node-binding interceptor that stops a caller on one node from
submitting health events naming another node. See
[Health Event Authentication](/nvsentinel/v1.27.0/components/platform-connectors#health-event-node-binding).

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `platform_connector_auth_decisions_total` | Counter | `decision` | Health event batches by the node scope granted to the caller. Values: `node_local`, `cross_node` |
| `platform_connector_auth_node_claim_total` | Counter | `result` | Whether an authenticated caller's token carried a node claim. `verified`: it carried one and it named this node (on the deployment platform connector, which has no node of its own, the claim is taken as the caller's node). `absent`: it carried none, so the caller was pinned to this node, the same scope a tokenless caller gets; on the deployment platform connector such a caller is rejected instead (`node_claim_absent` or `unbound_token`; `cross_node_claim_absent` or `unbound_cross_node_token` when the identity is on the cross-node list). |
| `platform_connector_auth_violations_total` | Counter | `reason` | Health event batches that violated the node-binding rule. Under `global.platformConnectorAuth.mode: enforce` (default) these are rejected; under `mode: audit` they are recorded here but let through. Reasons: `node_mismatch` (a node-local caller named a different node), `node_claim_mismatch` (the token was issued on another node; applies to every caller, allowlisted or not), `unbound_cross_node_token` (an allowlisted caller presented a token bound to no pod, e.g. from `kubectl create token`), `cross_node_claim_absent` (pod-bound, but the pod was never scheduled), `missing_node_name` (no node name on the event and none could be stamped), `token_invalid` (TokenReview rejected the token), `malformed_credentials` (the authorization header was duplicated or did not use the Bearer scheme). Three reasons occur only on the deployment platform connector, which has no local node to fall back to: `token_missing` (no token at all), `unbound_token` (a token bound to no pod), `node_claim_absent` (a token bound to a pod that never scheduled, so no node to pin its events to). Three further reasons mean **no verdict could be reached**, not that the caller was rejected: `validator_unavailable` (the API server was unreachable), `validator_timeout` (the caller gave up or its deadline passed) and `validator_error` (an unexpected failure, typically this component's own RBAC to create TokenReviews). With `failOpenOnUnavailable: true`, `validator_unavailable` and `validator_timeout` still increment this counter but fall back to a degraded node-local scope instead of rejecting the request, in `mode: enforce` too, because an outage says nothing about the caller's credential. That scope is a guess, not a verified identity: an event with a blank or matching node name is accepted and stamped as usual in either mode, but one naming a different node depends on `mode`: under `enforce` it is refused as retryable `Unavailable` rather than counted as `node_mismatch`, since the caller might really be an allowlisted cross-node publisher the outage prevented from being verified; under `audit` that same `Unavailable` is recorded and then let through like any other audited violation, so the event is forwarded with whatever node name it carried. `missing_node_name` is the one violation `mode: audit` does not let through: a blank node name from a caller resolved to **verified** cross-node scope is always rejected, in both modes, because nothing downstream can handle it. This does not cover a caller that fell back to a degraded node-local scope under `failOpenOnUnavailable`: its identity was never verified as cross-node, so a blank name from it is stamped like any other node-local caller's. Exclude those three when alerting on suspected credential abuse, because a control-plane outage increments them for every in-flight request and would otherwise look identical to an attack. The selector to alert on is below this table. |

The alerting selector for suspected credential abuse, with the three no-verdict reasons left out:

```promql
sum(rate(platform_connector_auth_violations_total{reason=~"node_mismatch|token_invalid|node_claim_mismatch|unbound_cross_node_token|malformed_credentials|cross_node_claim_absent|missing_node_name|token_missing|unbound_token|node_claim_absent"}[5m]))
```

### Kubernetes Connector Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `k8s_platform_connector_node_condition_update_total` | Counter | `status` | Total number of node condition updates by status. Status values: `success`, `failed`, `skipped` (an update that would leave the node's conditions as they are, such as a monitor's repeat of a fault the node already shows, is skipped) |
| `k8s_platform_connector_node_event_operations_total` | Counter | `operation`, `status` | Total number of node event operations by type and status. Operation values: `create`, `update`. Status values: `success`, `failed`, `skipped` (a repeat of a fault whose Event was written less than 10 minutes ago is skipped; later repeats refresh the Event) |
| `k8s_platform_connector_node_condition_update_duration_milliseconds` | Histogram | - | Duration of node condition updates in milliseconds. Uses linear buckets (0, 10, 500) |
| `k8s_platform_connector_node_event_update_create_duration_milliseconds` | Histogram | - | Duration of node event updates/creations in milliseconds. Uses linear buckets (0, 10, 500) |
| `k8s_platform_connector_dropped_writes_total` | Counter | `operation`, `reason`, `is_healthy` | Kubernetes writes discarded, including unattempted writes at deadline or shutdown. `is_healthy` separates a lost clear, which can leave a fault standing with nothing left to clear it, from a lost set, which the next change re-reports. For `operation="node_condition"` one write carries every event grouped for that node, so `is_healthy="true"` means the write contained at least one recovery, not that all of it was recoveries |
| `k8s_platform_connector_dropped_batches_total` | Counter | `reason` | Batches containing discarded Kubernetes writes, counted once per terminal reason. Deliberately not labelled by health direction: one batch can contain both |

### Prometheus Connector Metrics

Every health monitor publishes through the platform connector, so this one metric covers
`gpu-health-monitor`, `syslog-health-monitor`, `nic-health-monitor`, `csp-health-monitor`,
`kubernetes-object-monitor` and `health-events-analyzer` without per-monitor instrumentation.
Enabled with `platformConnector.promConnector.enabled` (default `false`, like the other optional connectors).

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `health_events_total` | Counter | `node`, `agent`, `check_name`, `recommended_action`, `is_fatal`, `is_healthy` | Health events received by the platform connector. `recommended_action` is the enum name (`NONE`, `CONTACT_SUPPORT`, `RESTART_VM`, …); a `CUSTOM` action reports as `CUSTOM` rather than expanding `customRecommendedAction`, which is operator-defined and unbounded |

`node` carries the **event's own** `nodeName`, not the node of the connector pod that
received it. That distinction matters: the DaemonSet monitors publish over the node-local
socket so the two coincide, but `health-events-analyzer` is a Deployment whose derived
events describe other nodes, and an allowlisted cross-node publisher can name any node. An
explicit label makes every agent correct without a `kube_pod_info` join.

`errorCode` is excluded because it is unbounded: suffixed XIDs such as
`145.RLW_SRC_TRACK` would grow the series set without limit. `node` is bounded by the fleet.

`is_fatal` is kept even though producers derive it as `recommendedAction != NONE`. The
redundancy is the point: a producer that disagrees with that derivation becomes visible
rather than silent.

Useful queries:

```promql
# Actionable event rate by action, fleet-wide
sum by (recommended_action) (rate(health_events_total{recommended_action!="NONE"}[5m]))

# Which agent and check is producing the actionable events
sum by (agent, check_name) (rate(health_events_total{recommended_action!="NONE",is_healthy="false"}[1h]))

# Per node, correct for every agent including health-events-analyzer
sum by (node) (rate(health_events_total{recommended_action!="NONE"}[1h]))
```

### Platform Connector Request Metrics

Both roles of the platform connector, the node-local DaemonSet and the deployment platform connector, expose these for the batches that reach the request handler. Alert on `failed` outcomes and on best-effort failures. The deployment platform connector (`--mode=deployment`) also exposes the metrics above for its Kubernetes connector and for the node-binding interceptor (its rejections are counted in `platform_connector_auth_violations_total`, by the reasons listed there), and the refusals below, which it answers before the handler sees the batch.

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `platform_connector_request_duration_seconds` | Histogram | `outcome` | Duration of health event batch requests that reached the handler, by outcome: `ok` (acknowledged), `rejected` (the batch is invalid), `failed` (the connector returned an error while processing the batch; the caller retries). Both roles expose it: on the node-local DaemonSet the connector is the set of ring buffers, which always accept, so a batch is `ok` once queued unless it is invalid; on the deployment platform connector the store connector's write decides when config.json enables one, while the node update and the sink are best effort and never fail a batch, so without a store every batch that passed validation is `ok`. |
| `platform_connector_deployment_refusals_total` | Counter | `reason` | Batches refused before the handler saw them: `index_unverified` (this replica has a store connector and has not verified the idempotency index; the caller retries) or `idempotency_key` (the idempotency-key header is missing or malformed) |
| `platform_connector_store_batches_total` | Counter | `outcome` | Batches written to the datastore by either role: `stored`, or `duplicate` (a resend, or a retried batch, whose events already existed; treated like a store) |
| `platform_connector_best_effort_failures_total` | Counter | `connector`, `reason` | Deployment platform connector only: work a best-effort connector (`kubernetes` node conditions and Events, `grpcsink` forwards) did not finish although the batch was acknowledged: `failed` or `timeout`; the node shows the old state until the monitor next reports |

### Workqueue Metrics

These metrics track the internal ring buffer workqueue performance:

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `platform_connector_workqueue_depth_{name}` | Gauge | `workqueue` | Current depth of Platform connector workqueue |
| `platform_connector_workqueue_adds_total_{name}` | Counter | `workqueue` | Total number of adds handled by Platform connector workqueue |
| `platform_connector_workqueue_latency_seconds_{name}` | Histogram | `workqueue` | How long an item stays in Platform connector workqueue before being requested. Uses linear buckets (0, 10, 500) |
| `platform_connector_workqueue_work_duration_seconds_{name}` | Histogram | `workqueue` | How long processing an item from Platform connector workqueue takes. Uses linear buckets (0, 10, 500) |
| `platform_connector_workqueue_retries_total_{name}` | Counter | `workqueue` | Total number of retries handled by Platform connector workqueue |
| `platform_connector_workqueue_longest_running_processor_seconds_{name}` | Gauge | `workqueue` | How many seconds the longest running processor for Platform connector workqueue has been running |
| `platform_connector_workqueue_unfinished_work_seconds_{name}` | Gauge | `workqueue` | The total time in seconds of work in progress in Platform connector workqueue |

**Note:** `{name}` in the metric names is replaced with the actual workqueue name at runtime.

---

## Health Monitors

### Health Event Publisher

The Go health monitors send their events through the shared publishing client (`commons/pkg/healthpub`). The `monitor` label names the monitor. The socket rows apply to the node-local path; the direct-mode rows apply only when a monitor publishes to the deployment platform connector (`HEALTH_PUBLISH_TARGET` set, which the chart's `publishTo: deployment` does once the chart ships).

| Metric Name                                                       | Type    | Labels              | Description                                                                                                                                                                       |
|-------------------------------------------------------------------|---------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `nvsentinel_health_events_publisher_skipped_pc_unavailable_total` | Counter | `monitor`           | Socket-path sends skipped because the node-local platform connector socket was missing                                                                                            |
| `nvsentinel_health_events_publisher_sends_success_total`          | Counter | `monitor`           | Batches the platform connector acknowledged, on either path                                                                                                                       |
| `nvsentinel_health_events_publisher_sends_error_total`            | Counter | `monitor`, `code`   | Socket-path sends that failed after retries, by gRPC status code                                                                                                                  |
| `nvsentinel_health_events_publisher_dropped_total`                | Counter | `monitor`, `reason` | Direct-mode batches given up on: `rejected` (the server refuses the batch for good, or it exceeds the 4 MiB message limit), `retry_window_exhausted`, `shutdown`, `withdrawn` (the caller stopped waiting: its context ended or its timeout expired, and the batch was taken back)               |
| `nvsentinel_health_events_publisher_retries_total`                | Counter | `monitor`           | Direct-mode send attempts that failed and were retried inside the batch's retry window                                                                                            |

---

### GPU Health Monitor

These metrics track GPU health events detected via DCGM (Data Center GPU Manager):

| Metric Name                                       | Type      | Labels                             | Description                                                                                               |
|---------------------------------------------------|-----------|------------------------------------|-----------------------------------------------------------------------------------------------------------|
| `dcgm_health_events_publish_time_to_grpc_channel` | Histogram | `operation_name`                   | Amount of time spent in publishing DCGM health events on the gRPC channel                                 |
| `health_events_insertion_to_uds_succeed`          | Counter   | -                                  | Total number of successful insertions of health events to UDS                                             |
| `health_events_insertion_to_uds_error`            | Counter   | -                                  | Total number of failed insertions of health events to UDS                                                 |
| `health_events_insertion_skipped_pc_unavailable`  | Counter   | -                                  | Sends skipped because the node-local platform connector socket was missing                                |
| `health_events_direct_publish_succeed`            | Counter   | -                                  | Batches the deployment platform connector acknowledged (direct mode)                          |
| `health_events_direct_publish_dropped`            | Counter   | `reason`                           | Direct-mode batches given up on; the same reasons as the Go publisher's `dropped_total`                   |
| `health_events_direct_publish_retries`            | Counter   | -                                  | Direct-mode send attempts that failed and were retried                                                    |
| `dcgm_health_active_events`                       | Gauge     | `event_type`, `gpu_id`, `error_code`, `is_fatal`, `processing_strategy` | Total number of active GPU health events at any given time. `is_fatal` is `true` or `false`, the same as the `isFatal` that gpu-health-monitor sends (`recommendedAction` is not `NONE`). Use it to exclude informational events from alerts. The value comes from the gpu-health-monitor error-code mapping. `processing_strategy` is the processing strategy that gpu-health-monitor sends with the event, for example `EXECUTE_REMEDIATION` or `STORE_ONLY`. The global `processingStrategy` sets it, and a per-check setting can change it to `STORE_ONLY`, for example for `GpuThermalMarginWatch`, `GpuPowerBrakeWatch` and `GpuDcgmUnresponsive`. When the value is `STORE_ONLY`, no NVSentinel module acts on the event. When the value is `EXECUTE_REMEDIATION`, gpu-health-monitor requests action, but platform-connectors can still change the strategy. For example, it sets `STORE_ONLY` for a node with the `nvsentinel.dgxc.nvidia.com/managed=false` label. Like `is_fatal`, this label shows what gpu-health-monitor sends, not the final decision. |
| `dcgm_health_active_switch_events`                | Gauge     | `event_type`, `switch_id`, `error_code` | Total number of active NVSwitch health events at any given time. Separate from `dcgm_health_active_events` because switch and GPU ids are distinct sequences that both start at 0 |
| `dcgm_api_latency`                                | Histogram | `operation_name`                   | Amount of time spent calling DCGM APIs                                                                    |
| `dcgm_reconcile_time`                             | Histogram | -                                  | Amount of time spent running a single DCGM reconcile loop                                                 |
| `number_of_health_watches`                        | Gauge     | -                                  | Number of DCGM health watches available                                                                   |
| `number_of_fields`                                | Gauge     | -                                  | Number of available DCGM fields to monitor                                                                |
| `callback_failures`                               | Counter   | `class_name`, `func_name`          | Number of times a callback function has thrown an exception                                               |
| `callback_success`                                | Counter   | `class_name`, `func_name`          | Number of times a callback function has successfully completed                                            |
| `dcgm_api_failures`                               | Counter   | `error_name`                       | Number of DCGM API errors                                                                                 |
| `dcgm_health_check_unknown_system_skipped`        | Counter   | -                                  | Number of DCGM health check incidents skipped due to unrecognized system value                            |
| `dcgm_probe_hangs`                                | Counter   | `operation_name`                   | Number of DCGM probes that exceeded the watchdog deadline without returning                                |

---

### NIC Health Monitor

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `nic_health_monitor_poll_cycle_last_completed_timestamp_seconds` | Gauge | `node`, `category` | Unix timestamp initialized at monitor startup and advanced after each completed poll cycle (`state` or `counter`). Use `time() - metric` to measure how long the category has gone without completing a poll, including a stall in the first poll. |

---

### Syslog Health Monitor

The syslog health monitor tracks GPU-related errors detected from system logs.

#### XID Error Metrics

XID (GPU Error ID) errors are NVIDIA GPU driver errors:

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `syslog_health_monitor_xid_errors` | Counter | `node`, `err_code` | Total number of XID errors found |
| `syslog_health_monitor_xid_processing_errors` | Counter | `error_type`, `node` | Total number of errors encountered during XID processing |
| `syslog_health_monitor_xid_processing_latency_seconds` | Histogram | - | Histogram of XID processing latency |

#### SXID Error Metrics

SXID errors are NVSwitch-related errors:

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `syslog_health_monitor_sxid_errors` | Counter | `node`, `err_code`, `link`, `nvswitch` | Total number of SXID errors found |

#### GPU Fallen Off Bus Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `syslog_health_monitor_gpu_fallen_errors` | Counter | `node` | Total number of GPU fallen off bus errors detected |

---

### CSP Health Monitor

The CSP health monitor tracks cloud provider maintenance events and node health issues.

#### CSP Client Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `csp_health_monitor_csp_events_received_total` | Counter | `csp` | Total number of raw events received from CSP API/source |
| `csp_health_monitor_csp_polling_duration_seconds` | Histogram | `csp` | Duration of CSP polling cycles |
| `csp_health_monitor_csp_api_errors_total` | Counter | `csp`, `error_type` | Total number of errors encountered during CSP API calls |
| `csp_health_monitor_csp_api_polling_duration_seconds` | Histogram | `csp`, `api` | Duration of CSP API polling cycles |
| `csp_health_monitor_csp_monitor_errors_total` | Counter | `csp`, `error_type` | Total number of errors initializing or starting CSP monitors |
| `csp_health_monitor_csp_events_by_type_unsupported_total` | Counter | `csp`, `event_type` | Total number of raw CSP events received, partitioned by event type code |

#### Event Processing Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `csp_health_monitor_main_events_to_normalize_total` | Counter | `csp` | Total number of events passed to the normalizer |
| `csp_health_monitor_main_normalization_errors_total` | Counter | `csp` | Total number of errors during event normalization |
| `csp_health_monitor_main_events_received_total` | Counter | `csp` | Total number of normalized events received by the main processor |
| `csp_health_monitor_main_events_processed_success_total` | Counter | `csp` | Total number of events successfully processed |
| `csp_health_monitor_main_processing_errors_total` | Counter | `csp`, `error_type` | Total number of errors during event processing |
| `csp_health_monitor_main_event_processing_duration_seconds` | Histogram | `csp` | Duration of processing a single event |

#### Datastore Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `csp_health_monitor_main_datastore_upsert_attempts_total` | Counter | `csp` | Total number of attempts to upsert maintenance events |
| `csp_health_monitor_main_datastore_upsert_total` | Counter | `csp`, `status` | Total number of maintenance event upserts by status. Status values: `success`, `failed` |

#### Trigger Engine Metrics

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `csp_health_monitor_trigger_poll_cycles_total` | Counter | - | Total number of polling cycles executed by the trigger engine |
| `csp_health_monitor_trigger_poll_errors_total` | Counter | - | Total number of errors during a trigger engine poll cycle |
| `csp_health_monitor_trigger_events_found_total` | Counter | `trigger_type` | Total number of events found potentially needing a trigger |
| `csp_health_monitor_trigger_attempts_total` | Counter | `trigger_type` | Total number of trigger attempts made |
| `csp_health_monitor_trigger_success_total` | Counter | `trigger_type` | Total number of successful triggers |
| `csp_health_monitor_trigger_failures_total` | Counter | `trigger_type`, `failure_reason` | Total number of failed trigger attempts |
| `csp_health_monitor_trigger_datastore_query_duration_seconds` | Histogram | `query_type` | Duration of datastore queries performed by the trigger engine |
| `csp_health_monitor_trigger_datastore_query_errors_total` | Counter | `query_type` | Total number of errors during datastore queries |
| `csp_health_monitor_trigger_datastore_update_errors_total` | Counter | `trigger_type` | Total number of errors updating event status after trigger |
| `csp_health_monitor_trigger_uds_send_duration_seconds` | Histogram | - | Duration of sending health events via UDS |
| `csp_health_monitor_trigger_uds_send_errors_total` | Counter | - | Total number of errors encountered when sending events via UDS |
| `csp_health_monitor_node_not_ready_timeout_total` | Counter | `node_name` | Total number of nodes that remained not ready after the timeout period |
| `csp_health_monitor_node_readiness_monitoring_started_total` | Counter | `node_name` | Total number of times background node readiness monitoring was started |

---

## Change Stream Metrics

Emitted by `store-client`, so they appear on every module that reads the change stream:
`event-exporter`, `fault-quarantine`, `health-events-analyzer`, `node-drainer`, and
`fault-remediation`, on both the MongoDB and PostgreSQL providers. The `client` label is the
consumer's name.

`fault-remediation` serves only controller-runtime's registry, so it passes that registry
explicitly; the others use the default registry.

| Metric Name | Type | Labels | Description |
|------------|------|--------|-------------|
| `change_stream_lag_seconds` | Gauge | `client` | Seconds since this consumer last had evidence it was caught up with its own change stream, from either an empty batch or the server-side timestamp of the last event it read. Absent until one of those has been observed |
| `change_stream_lag_known` | Gauge | `client` | 1 once `change_stream_lag_seconds` can be computed for this consumer, 0 before then |
| `change_stream_resume_token_recoveries_total` | Counter | `client`, `phase` | Times a stale or invalid resume token was deleted, restarting the stream from the current position |

Lag is measured against the consumer's **own filtered stream**, so a module whose pipeline
admits nothing still reports zero lag while it is caught up. See
[ADR-054](designs/054-changestream-lag-metrics.md).

### Suggested alerts

```yaml
# A consumer is behind. Tune the threshold per fleet; no lag_known qualifier is needed,
# because the series is absent rather than zero while lag is unknown.
- alert: ChangeStreamConsumerBehind
  expr: change_stream_lag_seconds > 900
  for: 10m

# A watcher that never started, or is wedged before its first read. The grace period covers
# normal startup, which closes in milliseconds on a live stream.
- alert: ChangeStreamLagUnknown
  expr: change_stream_lag_known == 0
  for: 10m
```

### What zero lag does not tell you

Each of these is a way a consumer can be behind while the gauge reads zero.

1. **Replication lag.** The MongoDB stream is opened `SecondaryPreferred` with no max staleness,
   so an empty batch means "caught up with the secondary I am reading", not with the primary.
2. **Data skipped after a resume-token recovery.** When a stored token is too old for the oplog
   the watcher reopens from now, so lag reads near zero precisely when the most was skipped.
   Read `change_stream_lag_seconds` together with `change_stream_resume_token_recoveries_total`:
   a lag of zero is only reassuring if the recoveries counter has not moved.
3. **Durable position.** This measures the watcher's progress against its own stream, not
   whether its position was persisted. A consumer that reads an event and dies before
   `MarkProcessed` succeeds looks healthy here, because the read did happen.

---

## Metrics Configuration

### Scraping Metrics

All NVSentinel components expose Prometheus metrics on a metrics endpoint (typically `:2112/metrics`). The metrics can be scraped by Prometheus using standard scrape configurations.

### Helm Chart Configuration

The NVSentinel Helm chart automatically creates a `PodMonitor` resource for Prometheus Operator integration:

```bash
helm install nvsentinel ./distros/kubernetes/nvsentinel \
  --namespace nvsentinel --create-namespace
```

The PodMonitor is configured to scrape all NVSentinel component pods on their metrics endpoints (`/metrics` on port `metrics`).

### Annotation-based Discovery

Components can be configured to include Prometheus scrape annotations:

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "2112"
  prometheus.io/path: "/metrics"
```

---

## Metric Types Reference

- **Counter**: A cumulative metric that only increases or resets to zero on restart
- **Gauge**: A metric that can arbitrarily go up and down
- **Histogram**: Samples observations and counts them in configurable buckets
- **Summary**: Similar to histogram but calculates configurable quantiles over a sliding time window

---

## Common Label Values

### Status Labels
- `success` / `failed` - Operation outcome
- `started` / `succeeded` / `failed` - Action lifecycle status

### Action Types
- `reboot` - Node reboot action
- `terminate` - Node termination action

### CSP Labels
- `gcp` - Google Cloud Platform
- `aws` - Amazon Web Services

### Trigger Types
- `quarantine` - Node quarantine trigger
- `healthy` - Node healthy trigger