Observability

NVSentinel Metrics

View as Markdown

This document outlines all Prometheus metrics exposed by NVSentinel components.

Table of Contents

Fault Quarantine Module

Event Processing Metrics

Metric NameTypeLabelsDescription
fault_quarantine_events_received_totalCounter-Total number of events received from the watcher
fault_quarantine_events_successfully_processed_totalCounter-Total number of events successfully processed
fault_quarantine_processing_errors_totalCountererror_typeTotal number of errors encountered during event processing
fault_quarantine_cold_start_events_totalCounterresultHealth events examined during startup recovery, labeled processed, skipped, superseded, invalid, or failed
fault_quarantine_cold_start_duration_secondsHistogram-Startup recovery duration. Buckets: ExponentialBuckets(start=0.1s, factor=2, count=18), max ~3.6 hours
fault_quarantine_event_backlog_countGauge-Number of health events which fault quarantine is yet to process
fault_quarantine_event_handling_duration_secondsHistogram-Histogram of event handling durations

Node Quarantine Metrics

Metric NameTypeLabelsDescription
fault_quarantine_nodes_quarantined_totalCounternodeTotal number of nodes quarantined
fault_quarantine_nodes_unquarantined_totalCounternodeTotal number of nodes unquarantined
fault_quarantine_nodes_manually_uncordoned_totalCounternodeTotal number of manually uncordons for nodes
fault_quarantine_nodes_manually_untainted_totalCounternodeTotal number of manual taint removals for nodes
fault_quarantine_current_quarantined_nodesGaugenodeCurrent number of quarantined nodes

Taint and Cordon Metrics

Metric NameTypeLabelsDescription
fault_quarantine_taints_applied_totalCountertaint_key, taint_effectTotal number of taints applied to nodes
fault_quarantine_taints_removed_totalCountertaint_key, taint_effectTotal number of taints removed from nodes
fault_quarantine_cordons_applied_totalCounter-Total number of cordons applied to nodes
fault_quarantine_cordons_removed_totalCounter-Total number of cordons removed from nodes
fault_quarantine_node_quarantine_duration_secondsHistogram-Time from health event generation to node quarantine completion. Buckets: Prometheus DefBuckets
fault_quarantine_node_remediation_duration_secondsHistogramrecommended_actionEnd-to-end node remediation time: generatedTimestamp (from original unhealthy event) to node unquarantine. Emitted on both auto unquarantine (via healthy event) and manual uncordon. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=27), max ~4.4 days
fault_quarantine_node_remediation_duration_excluding_drain_secondsHistogramrecommended_actionRemediation time excluding node-drainer duration: (unquarantineTime - generatedTimestamp) - (drainFinishTimestamp - quarantineFinishTimestamp). Emitted only when both quarantineFinishTimestamp and drainFinishTimestamp are present in the original event document. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=19), max ~4.1 hours

Ruleset Evaluation Metrics

Metric NameTypeLabelsDescription
fault_quarantine_ruleset_evaluations_totalCounterruleset, statusTotal number of ruleset evaluations. Status values: passed, failed

Circuit Breaker Metrics

Metric NameTypeLabelsDescription
fault_quarantine_breaker_stateGaugestateState of the fault quarantine breaker
fault_quarantine_breaker_utilizationGauge-Fraction of GPU nodes cordoned within the circuit breaker’s sliding window
fault_quarantine_breaker_threshold_nodesGaugeboundCordoned-node count that trips the circuit breaker, and which configured bound produced it (percentage, maxNodes, or fleetSize when clamped)
fault_quarantine_get_total_nodes_duration_secondsHistogramresultDuration of getTotalNodesWithRetry calls in seconds
fault_quarantine_get_total_nodes_errors_totalCountererror_typeTotal number of errors from getTotalNodesWithRetry
fault_quarantine_get_total_nodes_retry_attemptsHistogram-Number of retry attempts needed for getTotalNodesWithRetry (buckets: 0, 1, 2, 3, 5, 10)

Node Drainer Module

Event Processing Metrics

Metric NameTypeLabelsDescription
node_drainer_events_received_totalCounter-Total number of events received from the watcher
node_drainer_events_replayed_totalCounter-Total number of in-progress events replayed at startup
node_drainer_events_processed_totalCounterdrain_status, nodeTotal number of events processed by drain status outcome. Drain status values: drained, cancelled, skipped
node_drainer_cancelled_event_totalCounternode, check_nameTotal number of cancelled drain events (due to manual uncordon or healthy recovery)
node_drainer_processing_errors_totalCountererror_type, nodeTotal number of errors encountered during event processing and node draining
node_drainer_event_handling_duration_secondsHistogram-Histogram of event handling durations
node_drainer_queue_depthGauge-Total number of pending events in the queue
node_drainer_pod_eviction_duration_secondsHistogram-Time from event receipt by node-drainer to successful pod eviction completion. Buckets: Exponential (0.1s, factor 2, 23 buckets, up to ~3 days)

Node Draining Metrics

Metric NameTypeLabelsDescription
node_drainer_waiting_for_timeoutGaugenodeShows if node drainer operation is waiting for timeout before force deletion (1=waiting, 0=not waiting)
node_drainer_force_delete_pods_after_timeoutCounternode, namespaceTotal number of node drainer operations that reached timeout and force deleted pods

Fault Remediation Module

Event Processing Metrics

Metric NameTypeLabelsDescription
fault_remediation_events_received_totalCounter-Total number of events received from the watcher
fault_remediation_events_processed_totalCountercr_status, node_nameTotal number of remediation events processed by CR creation status. CR status values: created, skipped
fault_remediation_processing_errors_totalCountererror_type, node_nameTotal number of errors encountered during event processing
fault_remediation_unsupported_actions_totalCounteraction, node_nameTotal number of health events with currently unsupported remediation actions
fault_remediation_event_handling_duration_secondsHistogram-Histogram of event handling durations
fault_remediation_cr_generate_duration_secondsHistogram-Time from drain completion (or quarantine completion if drain timestamp unavailable) to maintenance CR creation. Buckets: Prometheus DefBuckets

Log Collector Metrics

Metric NameTypeLabelsDescription
fault_remediation_log_collector_jobs_totalCounternode_name, statusTotal number of log collector jobs. Status values: success, failure, timeout
fault_remediation_log_collector_job_duration_secondsHistogramnode_name, statusDuration of log collector jobs in seconds
fault_remediation_log_collector_errors_totalCountererror_type, node_nameTotal number of errors encountered in log collector operations

File Server Metrics

HTTP Request Metrics

Metric NameTypeLabelsDescription
http_response_count_totalCountermethod, status, appTotal HTTP responses by method and status code
http_request_duration_secondsHistogrammethod, statusHTTP request duration in seconds

Log Rotation Metrics

Metric NameTypeLabelsDescription
fileserver_log_rotation_successful_totalCounter-Total successful log cleanup operations
fileserver_log_rotation_failed_totalCounter-Total failed log cleanup operations
fileserver_disk_space_free_bytesGauge-Free disk space in bytes

Labeler Module

Event Processing Metrics

Metric NameTypeLabelsDescription
labeler_events_processed_totalCounterstatusTotal number of pod events processed. Status values: success, failed
labeler_node_update_failures_totalCounter-Total number of node update failures during reconciliation
labeler_event_handling_duration_secondsHistogram-Histogram of event handling durations

Janitor

Action Metrics

Metric NameTypeLabelsDescription
janitor_actions_countCounteraction_type, status, nodeTotal number of janitor actions by type and status. Action types: reboot, terminate. Status values: started, succeeded, failed
janitor_action_mttr_secondsHistogramaction_typeTime from CR creation to action completion (Mean Time To Repair). Uses exponential buckets (10, 2, 10) for log-scale MTTR measurement

Platform Connectors

Health Event Authentication Metrics

These track the node-binding interceptor that stops a caller on one node from submitting health events naming another node. See Health Event Authentication.

Metric NameTypeLabelsDescription
platform_connector_auth_decisions_totalCounterdecisionHealth event batches by the node scope granted to the caller. Values: node_local, cross_node
platform_connector_auth_node_claim_totalCounterresultWhether an authenticated caller’s token carried a node claim. verified: it carried one and it named this node. absent: it carried none, so the caller was pinned to this node — the same scope a tokenless caller gets.
platform_connector_auth_violations_totalCounterreasonHealth event batches that violated the node-binding rule. Under global.platformConnectorAuth.mode: enforce (default) these are rejected; under mode: audit they are recorded here but let through. Reasons: node_mismatch (a node-local caller named a different node), node_claim_mismatch (the token was issued on another node — applies to every caller, allowlisted or not), unbound_cross_node_token (an allowlisted caller presented a token bound to no pod, e.g. from kubectl create token), cross_node_claim_absent (pod-bound, but the pod was never scheduled), missing_node_name (no node name on the event and none could be stamped), token_invalid (TokenReview rejected the token), malformed_credentials (the authorization header was duplicated or did not use the Bearer scheme). Three further reasons mean no verdict could be reached, not that the caller was rejected: validator_unavailable (the API server was unreachable), validator_timeout (the caller gave up or its deadline passed) and validator_error (an unexpected failure, typically this component’s own RBAC to create TokenReviews). With failOpenOnUnavailable: true, validator_unavailable and validator_timeout still increment this counter but fall back to a degraded node-local scope instead of rejecting the request, in mode: enforce too — an outage says nothing about the caller’s credential. That scope is a guess, not a verified identity: an event with a blank or matching node name is accepted and stamped as usual in either mode, but one naming a different node depends on mode — under enforce it is refused as retryable Unavailable rather than counted as node_mismatch, since the caller might really be an allowlisted cross-node publisher the outage prevented from being verified; under audit that same Unavailable is recorded and then let through like any other audited violation, so the event is forwarded with whatever node name it carried. missing_node_name is the one violation mode: audit does not let through — a blank node name from a caller resolved to verified cross-node scope is always rejected, in both modes, because nothing downstream can handle it. This does not cover a caller that fell back to a degraded node-local scope under failOpenOnUnavailable: its identity was never verified as cross-node, so a blank name from it is stamped like any other node-local caller’s. Exclude those three when alerting on suspected credential abuse — a control-plane outage increments them for every in-flight request and would otherwise look identical to an attack: sum(rate(platform_connector_auth_violations_total{reason=~"node_mismatch|token_invalid|node_claim_mismatch|unbound_cross_node_token|malformed_credentials|cross_node_claim_absent|missing_node_name"}[5m]))

Kubernetes Connector Metrics

Metric NameTypeLabelsDescription
k8s_platform_connector_node_condition_update_totalCounterstatusTotal number of node condition updates by status. Status values: success, failed, skipped (an update that would leave the node’s conditions as they are, such as a monitor’s repeat of a fault the node already shows, is skipped)
k8s_platform_connector_node_event_operations_totalCounteroperation, statusTotal number of node event operations by type and status. Operation values: create, update. Status values: success, failed, skipped (a repeat of a fault whose Event was written less than 10 minutes ago is skipped; later repeats refresh the Event)
k8s_platform_connector_node_condition_update_duration_millisecondsHistogram-Duration of node condition updates in milliseconds. Uses linear buckets (0, 10, 500)
k8s_platform_connector_node_event_update_create_duration_millisecondsHistogram-Duration of node event updates/creations in milliseconds. Uses linear buckets (0, 10, 500)

Prometheus Connector Metrics

Every health monitor publishes through the platform connector, so this one metric covers gpu-health-monitor, syslog-health-monitor, nic-health-monitor, csp-health-monitor, kubernetes-object-monitor and health-events-analyzer without per-monitor instrumentation. Enabled with platformConnector.promConnector.enabled (default false, like the other optional connectors).

Metric NameTypeLabelsDescription
health_events_totalCounternode, agent, check_name, recommended_action, is_fatal, is_healthyHealth events received by the platform connector. recommended_action is the enum name (NONE, CONTACT_SUPPORT, RESTART_VM, …); a CUSTOM action reports as CUSTOM rather than expanding customRecommendedAction, which is operator-defined and unbounded

node carries the event’s own nodeName, not the node of the connector pod that received it. That distinction matters: the DaemonSet monitors publish over the node-local socket so the two coincide, but health-events-analyzer is a Deployment whose derived events describe other nodes, and an allowlisted cross-node publisher can name any node. An explicit label makes every agent correct without a kube_pod_info join.

errorCode is excluded because it is unbounded: suffixed XIDs such as 145.RLW_SRC_TRACK would grow the series set without limit. node is bounded by the fleet.

is_fatal is kept even though producers derive it as recommendedAction != NONE. The redundancy is the point: a producer that disagrees with that derivation becomes visible rather than silent.

Useful queries:

# Actionable event rate by action, fleet-wide
sum by (recommended_action) (rate(health_events_total{recommended_action!="NONE"}[5m]))
# Which agent and check is producing the actionable events
sum by (agent, check_name) (rate(health_events_total{recommended_action!="NONE",is_healthy="false"}[1h]))
# Per node, correct for every agent including health-events-analyzer
sum by (node) (rate(health_events_total{recommended_action!="NONE"}[1h]))

Platform Connector Request Metrics

The platform connector exposes these for the batches that reach its request handler.

Metric NameTypeLabelsDescription
platform_connector_request_duration_secondsHistogramoutcomeDuration of health event batch requests that reached the handler, by outcome: ok (acknowledged), rejected (the batch is invalid), failed (the connector returned an error while processing the batch; the caller retries). On the node-local DaemonSet the connector is the set of ring buffers, which always accept, so a batch is ok once queued unless it is invalid.
platform_connector_store_batches_totalCounteroutcomeBatches written to the datastore by the store connector: stored, or duplicate (a resend, or a retried batch, whose events already existed; treated like a store)

Workqueue Metrics

These metrics track the internal ring buffer workqueue performance:

Metric NameTypeLabelsDescription
platform_connector_workqueue_depth_{name}GaugeworkqueueCurrent depth of Platform connector workqueue
platform_connector_workqueue_adds_total_{name}CounterworkqueueTotal number of adds handled by Platform connector workqueue
platform_connector_workqueue_latency_seconds_{name}HistogramworkqueueHow long an item stays in Platform connector workqueue before being requested. Uses linear buckets (0, 10, 500)
platform_connector_workqueue_work_duration_seconds_{name}HistogramworkqueueHow long processing an item from Platform connector workqueue takes. Uses linear buckets (0, 10, 500)
platform_connector_workqueue_retries_total_{name}CounterworkqueueTotal number of retries handled by Platform connector workqueue
platform_connector_workqueue_longest_running_processor_seconds_{name}GaugeworkqueueHow many seconds the longest running processor for Platform connector workqueue has been running
platform_connector_workqueue_unfinished_work_seconds_{name}GaugeworkqueueThe total time in seconds of work in progress in Platform connector workqueue

Note: {name} in the metric names is replaced with the actual workqueue name at runtime.

Health Monitors

GPU Health Monitor

These metrics track GPU health events detected via DCGM (Data Center GPU Manager):

Metric NameTypeLabelsDescription
dcgm_health_events_publish_time_to_grpc_channelHistogramoperation_nameAmount of time spent in publishing DCGM health events on the gRPC channel
health_events_insertion_to_uds_succeedCounter-Total number of successful insertions of health events to UDS
health_events_insertion_to_uds_errorCounter-Total number of failed insertions of health events to UDS
dcgm_health_active_eventsGaugeevent_type, gpu_id, error_codeTotal number of active health events at any given time
dcgm_api_latencyHistogramoperation_nameAmount of time spent calling DCGM APIs
dcgm_reconcile_timeHistogram-Amount of time spent running a single DCGM reconcile loop
number_of_health_watchesGauge-Number of DCGM health watches available
number_of_fieldsGauge-Number of available DCGM fields to monitor
callback_failuresCounterclass_name, func_nameNumber of times a callback function has thrown an exception
callback_successCounterclass_name, func_nameNumber of times a callback function has successfully completed
dcgm_api_failuresCountererror_nameNumber of DCGM API errors
dcgm_health_check_unknown_system_skippedCounter-Number of DCGM health check incidents skipped due to unrecognized system value
dcgm_probe_hangsCounteroperation_nameNumber of DCGM probes that exceeded the watchdog deadline without returning

Syslog Health Monitor

The syslog health monitor tracks GPU-related errors detected from system logs.

XID Error Metrics

XID (GPU Error ID) errors are NVIDIA GPU driver errors:

Metric NameTypeLabelsDescription
syslog_health_monitor_xid_errorsCounternode, err_codeTotal number of XID errors found
syslog_health_monitor_xid_processing_errorsCountererror_type, nodeTotal number of errors encountered during XID processing
syslog_health_monitor_xid_processing_latency_secondsHistogram-Histogram of XID processing latency

SXID Error Metrics

SXID errors are NVSwitch-related errors:

Metric NameTypeLabelsDescription
syslog_health_monitor_sxid_errorsCounternode, err_code, link, nvswitchTotal number of SXID errors found

GPU Fallen Off Bus Metrics

Metric NameTypeLabelsDescription
syslog_health_monitor_gpu_fallen_errorsCounternodeTotal number of GPU fallen off bus errors detected

CSP Health Monitor

The CSP health monitor tracks cloud provider maintenance events and node health issues.

CSP Client Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_csp_events_received_totalCountercspTotal number of raw events received from CSP API/source
csp_health_monitor_csp_polling_duration_secondsHistogramcspDuration of CSP polling cycles
csp_health_monitor_csp_api_errors_totalCountercsp, error_typeTotal number of errors encountered during CSP API calls
csp_health_monitor_csp_api_polling_duration_secondsHistogramcsp, apiDuration of CSP API polling cycles
csp_health_monitor_csp_monitor_errors_totalCountercsp, error_typeTotal number of errors initializing or starting CSP monitors
csp_health_monitor_csp_events_by_type_unsupported_totalCountercsp, event_typeTotal number of raw CSP events received, partitioned by event type code

Event Processing Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_main_events_to_normalize_totalCountercspTotal number of events passed to the normalizer
csp_health_monitor_main_normalization_errors_totalCountercspTotal number of errors during event normalization
csp_health_monitor_main_events_received_totalCountercspTotal number of normalized events received by the main processor
csp_health_monitor_main_events_processed_success_totalCountercspTotal number of events successfully processed
csp_health_monitor_main_processing_errors_totalCountercsp, error_typeTotal number of errors during event processing
csp_health_monitor_main_event_processing_duration_secondsHistogramcspDuration of processing a single event

Datastore Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_main_datastore_upsert_attempts_totalCountercspTotal number of attempts to upsert maintenance events
csp_health_monitor_main_datastore_upsert_totalCountercsp, statusTotal number of maintenance event upserts by status. Status values: success, failed

Trigger Engine Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_trigger_poll_cycles_totalCounter-Total number of polling cycles executed by the trigger engine
csp_health_monitor_trigger_poll_errors_totalCounter-Total number of errors during a trigger engine poll cycle
csp_health_monitor_trigger_events_found_totalCountertrigger_typeTotal number of events found potentially needing a trigger
csp_health_monitor_trigger_attempts_totalCountertrigger_typeTotal number of trigger attempts made
csp_health_monitor_trigger_success_totalCountertrigger_typeTotal number of successful triggers
csp_health_monitor_trigger_failures_totalCountertrigger_type, failure_reasonTotal number of failed trigger attempts
csp_health_monitor_trigger_datastore_query_duration_secondsHistogramquery_typeDuration of datastore queries performed by the trigger engine
csp_health_monitor_trigger_datastore_query_errors_totalCounterquery_typeTotal number of errors during datastore queries
csp_health_monitor_trigger_datastore_update_errors_totalCountertrigger_typeTotal number of errors updating event status after trigger
csp_health_monitor_trigger_uds_send_duration_secondsHistogram-Duration of sending health events via UDS
csp_health_monitor_trigger_uds_send_errors_totalCounter-Total number of errors encountered when sending events via UDS
csp_health_monitor_node_not_ready_timeout_totalCounternode_nameTotal number of nodes that remained not ready after the timeout period
csp_health_monitor_node_readiness_monitoring_started_totalCounternode_nameTotal number of times background node readiness monitoring was started

Change Stream Metrics

Emitted by store-client, so they appear on every module that reads the change stream: event-exporter, fault-quarantine, health-events-analyzer, node-drainer, and fault-remediation, on both the MongoDB and PostgreSQL providers. The client label is the consumer’s name.

fault-remediation serves only controller-runtime’s registry, so it passes that registry explicitly; the others use the default registry.

Metric NameTypeLabelsDescription
change_stream_lag_secondsGaugeclientSeconds since this consumer last had evidence it was caught up with its own change stream, from either an empty batch or the server-side timestamp of the last event it read. Absent until one of those has been observed
change_stream_lag_knownGaugeclient1 once change_stream_lag_seconds can be computed for this consumer, 0 before then
change_stream_resume_token_recoveries_totalCounterclient, phaseTimes a stale or invalid resume token was deleted, restarting the stream from the current position

Lag is measured against the consumer’s own filtered stream, so a module whose pipeline admits nothing still reports zero lag while it is caught up. See ADR-054.

Suggested alerts

# A consumer is behind. Tune the threshold per fleet; no lag_known qualifier is needed,
# because the series is absent rather than zero while lag is unknown.
- alert: ChangeStreamConsumerBehind
  expr: change_stream_lag_seconds > 900
  for: 10m
# A watcher that never started, or is wedged before its first read. The grace period covers
# normal startup, which closes in milliseconds on a live stream.
- alert: ChangeStreamLagUnknown
  expr: change_stream_lag_known == 0
  for: 10m

What zero lag does not tell you

Each of these is a way a consumer can be behind while the gauge reads zero.

  1. Replication lag. The MongoDB stream is opened SecondaryPreferred with no max staleness, so an empty batch means “caught up with the secondary I am reading”, not with the primary.
  2. Data skipped after a resume-token recovery. When a stored token is too old for the oplog the watcher reopens from now, so lag reads near zero precisely when the most was skipped. Read change_stream_lag_seconds together with change_stream_resume_token_recoveries_total: a lag of zero is only reassuring if the recoveries counter has not moved.
  3. Durable position. This measures the watcher’s progress against its own stream, not whether its position was persisted. A consumer that reads an event and dies before MarkProcessed succeeds looks healthy here, because the read did happen.

Metrics Configuration

Scraping Metrics

All NVSentinel components expose Prometheus metrics on a metrics endpoint (typically :2112/metrics). The metrics can be scraped by Prometheus using standard scrape configurations.

Helm Chart Configuration

The NVSentinel Helm chart automatically creates a PodMonitor resource for Prometheus Operator integration:

helm install nvsentinel ./distros/kubernetes/nvsentinel \
  --namespace nvsentinel --create-namespace

The PodMonitor is configured to scrape all NVSentinel component pods on their metrics endpoints (/metrics on port metrics).

Annotation-based Discovery

Components can be configured to include Prometheus scrape annotations:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "2112"
  prometheus.io/path: "/metrics"

Metric Types Reference

  • Counter: A cumulative metric that only increases or resets to zero on restart
  • Gauge: A metric that can arbitrarily go up and down
  • Histogram: Samples observations and counts them in configurable buckets
  • Summary: Similar to histogram but calculates configurable quantiles over a sliding time window

Common Label Values

Status Labels

  • success / failed - Operation outcome
  • started / succeeded / failed - Action lifecycle status

Action Types

  • reboot - Node reboot action
  • terminate - Node termination action

CSP Labels

  • gcp - Google Cloud Platform
  • aws - Amazon Web Services

Trigger Types

  • quarantine - Node quarantine trigger
  • healthy - Node healthy trigger