Skip to navigation
Observability

NVSentinel Metrics

View as Markdown

This document outlines all Prometheus metrics exposed by NVSentinel components.

Table of Contents

Fault Quarantine Module

Event Processing Metrics

Metric NameTypeLabelsDescription
fault_quarantine_events_received_totalCounter-Total number of events received from the watcher
fault_quarantine_events_successfully_processed_totalCounter-Total number of events successfully processed
fault_quarantine_processing_errors_totalCountererror_typeTotal number of errors encountered during event processing
fault_quarantine_cold_start_events_totalCounterresultHealth events examined during startup recovery, labeled processed, skipped, superseded, invalid, or failed
fault_quarantine_cold_start_duration_secondsHistogram-Startup recovery duration. Buckets: ExponentialBuckets(start=0.1s, factor=2, count=18), max ~3.6 hours
fault_quarantine_event_backlog_countGauge-Number of health events which fault quarantine is yet to process
fault_quarantine_event_handling_duration_secondsHistogram-Histogram of event handling durations

Node Quarantine Metrics

Metric NameTypeLabelsDescription
fault_quarantine_nodes_quarantined_totalCounternodeTotal number of nodes quarantined
fault_quarantine_nodes_unquarantined_totalCounternodeTotal number of nodes unquarantined
fault_quarantine_nodes_manually_uncordoned_totalCounternodeTotal number of manually uncordons for nodes
fault_quarantine_nodes_manually_untainted_totalCounternodeTotal number of manual taint removals for nodes
fault_quarantine_current_quarantined_nodesGaugenodeCurrent number of quarantined nodes
fault_quarantine_dry_run_actions_totalCounteractionTotal number of quarantine and unquarantine actions skipped by dry run

Taint and Cordon Metrics

Metric NameTypeLabelsDescription
fault_quarantine_taints_applied_totalCountertaint_key, taint_effectTotal number of taints applied to nodes
fault_quarantine_taints_removed_totalCountertaint_key, taint_effectTotal number of taints removed from nodes
fault_quarantine_labels_applied_totalCounterlabel_keyTotal number of quarantine labels applied to nodes
fault_quarantine_labels_removed_totalCounterlabel_keyTotal number of quarantine labels removed from nodes
fault_quarantine_cordons_applied_totalCounter-Total number of cordons applied to nodes
fault_quarantine_cordons_removed_totalCounter-Total number of cordons removed from nodes
fault_quarantine_node_quarantine_duration_secondsHistogram-Time from health event generation to node quarantine completion. Buckets: Prometheus DefBuckets
fault_quarantine_node_remediation_duration_secondsHistogramrecommended_actionEnd-to-end node remediation time: generatedTimestamp (from original unhealthy event) to node unquarantine. Emitted on both auto unquarantine (via healthy event) and manual uncordon. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=27), max ~4.4 days
fault_quarantine_node_remediation_duration_excluding_drain_secondsHistogramrecommended_actionRemediation time excluding node-drainer duration: (unquarantineTime - generatedTimestamp) - (drainFinishTimestamp - quarantineFinishTimestamp). Emitted only when both quarantineFinishTimestamp and drainFinishTimestamp are present in the original event document. Buckets: ExponentialBuckets(start=10s, factor=1.5, count=19), max ~4.1 hours

Ruleset Evaluation Metrics

Metric NameTypeLabelsDescription
fault_quarantine_ruleset_evaluations_totalCounterruleset, statusTotal number of ruleset evaluations. Status values: passed, failed

Circuit Breaker Metrics

Metric NameTypeLabelsDescription
fault_quarantine_breaker_stateGaugestateState of the fault quarantine breaker
fault_quarantine_breaker_utilizationGauge-Fraction of GPU nodes cordoned within the circuit breaker’s sliding window
fault_quarantine_breaker_threshold_nodesGaugeboundCordoned-node count that trips the circuit breaker, and which configured bound produced it (percentage, maxNodes, fleetSize when clamped, or noGPUNodes when no GPU nodes are counted and any recent cordon trips it)
fault_quarantine_get_total_nodes_duration_secondsHistogramresultDuration of getTotalNodes calls in seconds
fault_quarantine_get_total_nodes_errors_totalCountererror_typeTotal number of errors from getTotalNodes

Node Drainer Module

Event Processing Metrics

Metric NameTypeLabelsDescription
node_drainer_events_received_totalCounter-Total number of events received from the watcher
node_drainer_events_replayed_totalCounter-Total number of in-progress events replayed at startup
node_drainer_events_processed_totalCounterdrain_status, node, drain_scopeTotal number of events processed by drain status outcome and drain scope. Drain status values: drained, cancelled, skipped
node_drainer_cancelled_event_totalCounternode, check_nameTotal number of cancelled drain events (due to manual uncordon or healthy recovery)
node_drainer_processing_errors_totalCountererror_type, nodeTotal number of errors encountered during event processing and node draining
node_drainer_queue_items_assigned_totalCounterpriority, reasonTotal number of ready queue items assigned by priority and reason
node_drainer_custom_drain_crd_not_found_totalCounternodeTotal number of custom drain operations that failed because the CRD was not found in the cluster
node_drainer_event_handling_duration_secondsHistogram-Histogram of event handling durations
node_drainer_queue_depthGauge-Total number of pending events in the queue
node_drainer_pod_eviction_duration_secondsHistogram-Time from event receipt by node-drainer to successful pod eviction completion. Buckets: Exponential (0.1s, factor 2, 23 buckets, up to ~3 days)

Node Draining Metrics

Metric NameTypeLabelsDescription
node_drainer_waiting_for_timeoutGaugenodeShows if node drainer operation is waiting for timeout before force deletion (1=waiting, 0=not waiting)
node_drainer_force_delete_pods_after_timeoutCounternode, namespaceTotal number of node drainer operations that reached timeout and force deleted pods
node_drainer_partial_drains_totalCounternode, entity_type, entity_valueTotal number of partial drains by the entity they targeted. Disabled by default; enable via node-drainer.partialDrainEntityMetricEnabled

Fault Remediation Module

Event Processing Metrics

Metric NameTypeLabelsDescription
fault_remediation_events_received_totalCounter-Total number of events received from the watcher
fault_remediation_events_processed_totalCountercr_status, node_nameTotal number of remediation events processed by CR creation status. CR status values: created, skipped
fault_remediation_processing_errors_totalCountererror_type, node_nameTotal number of errors encountered during event processing
fault_remediation_unsupported_actions_totalCounteraction, node_nameTotal number of health events with currently unsupported remediation actions
fault_remediation_event_handling_duration_secondsHistogram-Histogram of event handling durations
fault_remediation_cr_generate_duration_secondsHistogram-Time from drain completion (or quarantine completion if drain timestamp unavailable) to maintenance CR creation. Buckets: Prometheus DefBuckets

Log Collector Metrics

Metric NameTypeLabelsDescription
fault_remediation_log_collector_jobs_totalCounternode_name, statusTotal number of log collector jobs. Status values: success, failure, timeout
fault_remediation_log_collector_job_duration_secondsHistogramnode_name, statusDuration of log collector jobs in seconds
fault_remediation_log_collector_errors_totalCountererror_type, node_nameTotal number of errors encountered in log collector operations

File Server Metrics

HTTP Request Metrics

Metric NameTypeLabelsDescription
http_response_count_totalCountermethod, status, appTotal HTTP responses by method and status code
http_request_duration_secondsHistogrammethod, statusHTTP request duration in seconds

Log Rotation Metrics

Metric NameTypeLabelsDescription
fileserver_log_rotation_successful_totalCounter-Total successful log cleanup operations
fileserver_log_rotation_failed_totalCounter-Total failed log cleanup operations
fileserver_disk_space_free_bytesGauge-Free disk space in bytes

Labeler Module

Event Processing Metrics

Metric NameTypeLabelsDescription
labeler_events_processed_totalCounterstatusTotal number of pod events processed. Status values: success, failed
labeler_node_update_failures_totalCounter-Total number of node update failures during reconciliation
labeler_event_handling_duration_secondsHistogram-Histogram of event handling durations
labeler_device_count_currentGaugenode, classCurrent device count observed for a node and device class
labeler_device_count_expectedGaugeclass, partitionExpected device count for a device class and hardware partition
labeler_device_count_sourceGaugenode, class, sourceSource that counted a node’s devices for a device class (1 for the selected source)
labeler_device_count_skipped_updates_totalCounterclass, reasonTotal number of skipped device-count label updates
labeler_node_labels_skipped_managed_totalCounter-Node label reconcile attempts skipped because the node is opted out of NVSentinel management

Janitor

Action Metrics

Metric NameTypeLabelsDescription
janitor_actions_countCounteraction_type, status, nodeTotal number of janitor actions by type and status. Action types: reboot, terminate. Status values: started, succeeded, failed
janitor_action_mttr_secondsHistogramaction_typeTime from CR creation to action completion (Mean Time To Repair). Uses exponential buckets (10, 2, 10) for log-scale MTTR measurement

GPU Reset Metrics

Metric NameTypeLabelsDescription
gpu_reset_requests_totalCounternodeTotal number of GPU reset requests initiated
gpu_reset_requests_completed_totalCounternode, statusTotal number of completed GPU reset requests, labeled by their final status
gpu_reset_duration_secondsHistogramnode, statusThe end-to-end duration of GPU reset workflows, from controller acknowledgment to completion, labeled by their final status
gpu_reset_active_requestsGaugenodeThe number of GPU reset requests currently in progress
gpu_reset_failure_reasons_totalCounternode, reasonTotal number of GPU reset failures, labeled by the specific reason
janitor_ttl_deletions_totalCounterkindTotal number of CRs deleted by the TTL reconciler, labeled by kind

External Remediation Metrics

Metric NameTypeLabelsDescription
nvsentinel_external_remediation_totalCounterphase, resultLifecycle transitions of ExternalRemediationRequest objects, labeled by phase and outcome
nvsentinel_external_remediation_openGaugenode, recommended_action, stateCurrently-open ExternalRemediationRequest objects by node, action, and substate
nvsentinel_external_remediation_age_secondsHistogramrecommended_action, resultAge of an ExternalRemediationRequest at close time

Platform Connectors

Health Event Authentication Metrics

These track the node-binding interceptor that stops a caller on one node from submitting health events naming another node. See Health Event Authentication.

Metric NameTypeLabelsDescription
platform_connector_auth_decisions_totalCounterdecisionHealth event batches by the node scope granted to the caller. Values: node_local, cross_node
platform_connector_auth_node_claim_totalCounterresultWhether an authenticated caller’s token carried a node claim. verified: it carried one and it named this node (on the deployment platform connector, which has no node of its own, the claim is taken as the caller’s node). absent: it carried none, so the caller was pinned to this node, the same scope a tokenless caller gets; on the deployment platform connector such a caller is rejected instead (node_claim_absent or unbound_token; cross_node_claim_absent or unbound_cross_node_token when the identity is on the cross-node list).
platform_connector_auth_violations_totalCounterreasonHealth event batches that violated the node-binding rule. Under global.platformConnectorAuth.mode: enforce (default) these are rejected; under mode: audit they are recorded here but let through. Reasons: node_mismatch (a node-local caller named a different node), node_claim_mismatch (the token was issued on another node; applies to every caller, allowlisted or not), unbound_cross_node_token (an allowlisted caller presented a token bound to no pod, e.g. from kubectl create token), cross_node_claim_absent (pod-bound, but the pod was never scheduled), missing_node_name (no node name on the event and none could be stamped), token_invalid (TokenReview rejected the token), malformed_credentials (the authorization header was duplicated or did not use the Bearer scheme). Three reasons occur only on the deployment platform connector, which has no local node to fall back to: token_missing (no token at all), unbound_token (a token bound to no pod), node_claim_absent (a token bound to a pod that never scheduled, so no node to pin its events to). Three further reasons mean no verdict could be reached, not that the caller was rejected: validator_unavailable (the API server was unreachable), validator_timeout (the caller gave up or its deadline passed) and validator_error (an unexpected failure, typically this component’s own RBAC to create TokenReviews). With failOpenOnUnavailable: true, validator_unavailable and validator_timeout still increment this counter but fall back to a degraded node-local scope instead of rejecting the request, in mode: enforce too, because an outage says nothing about the caller’s credential. That scope is a guess, not a verified identity: an event with a blank or matching node name is accepted and stamped as usual in either mode, but one naming a different node depends on mode: under enforce it is refused as retryable Unavailable rather than counted as node_mismatch, since the caller might really be an allowlisted cross-node publisher the outage prevented from being verified; under audit that same Unavailable is recorded and then let through like any other audited violation, so the event is forwarded with whatever node name it carried. missing_node_name is the one violation mode: audit does not let through: a blank node name from a caller resolved to verified cross-node scope is always rejected, in both modes, because nothing downstream can handle it. This does not cover a caller that fell back to a degraded node-local scope under failOpenOnUnavailable: its identity was never verified as cross-node, so a blank name from it is stamped like any other node-local caller’s. Exclude those three when alerting on suspected credential abuse, because a control-plane outage increments them for every in-flight request and would otherwise look identical to an attack. The selector to alert on is below this table.

The alerting selector for suspected credential abuse, with the three no-verdict reasons left out:

sum(rate(platform_connector_auth_violations_total{reason=~"node_mismatch|token_invalid|node_claim_mismatch|unbound_cross_node_token|malformed_credentials|cross_node_claim_absent|missing_node_name|token_missing|unbound_token|node_claim_absent"}[5m]))

Kubernetes Connector Metrics

Metric NameTypeLabelsDescription
k8s_platform_connector_node_condition_update_totalCounterstatusTotal number of node condition updates by status. Status values: success, failed, skipped (an update that would leave the node’s conditions as they are, such as a monitor’s repeat of a fault the node already shows, is skipped)
k8s_platform_connector_node_event_operations_totalCounteroperation, statusTotal number of node event operations by type and status. Operation values: create, update. Status values: success, failed, skipped (a repeat of a fault whose Event was written less than 10 minutes ago is skipped; later repeats refresh the Event)
k8s_platform_connector_node_condition_update_duration_millisecondsHistogram-Duration of node condition updates in milliseconds. Uses linear buckets (0, 10, 500)
k8s_platform_connector_node_event_update_create_duration_millisecondsHistogram-Duration of node event updates/creations in milliseconds. Uses linear buckets (0, 10, 500)
k8s_platform_connector_dropped_writes_totalCounteroperation, reason, is_healthyKubernetes writes discarded, including unattempted writes at deadline or shutdown. is_healthy separates a lost clear, which can leave a fault standing with nothing left to clear it, from a lost set, which the next change re-reports. For operation="node_condition" one write carries every event grouped for that node, so is_healthy="true" means the write contained at least one recovery, not that all of it was recoveries
k8s_platform_connector_dropped_batches_totalCounterreasonBatches containing discarded Kubernetes writes, counted once per terminal reason. Deliberately not labelled by health direction: one batch can contain both

Prometheus Connector Metrics

Every health monitor publishes through the platform connector, so this one metric covers gpu-health-monitor, syslog-health-monitor, nic-health-monitor, csp-health-monitor, kubernetes-object-monitor and health-events-analyzer without per-monitor instrumentation. Enabled with platformConnector.promConnector.enabled (default false, like the other optional connectors).

Metric NameTypeLabelsDescription
health_events_totalCounternode, agent, check_name, recommended_action, is_fatal, is_healthyHealth events received by the platform connector. recommended_action is the enum name (NONE, CONTACT_SUPPORT, RESTART_VM, …); a CUSTOM action reports as CUSTOM rather than expanding customRecommendedAction, which is operator-defined and unbounded

node carries the event’s own nodeName, not the node of the connector pod that received it. That distinction matters: the DaemonSet monitors publish over the node-local socket so the two coincide, but health-events-analyzer is a Deployment whose derived events describe other nodes, and an allowlisted cross-node publisher can name any node. An explicit label makes every agent correct without a kube_pod_info join.

errorCode is excluded because it is unbounded: suffixed XIDs such as 145.RLW_SRC_TRACK would grow the series set without limit. node is bounded by the fleet.

is_fatal is kept even though producers derive it as recommendedAction != NONE. The redundancy is the point: a producer that disagrees with that derivation becomes visible rather than silent.

Useful queries:

# Actionable event rate by action, fleet-wide
sum by (recommended_action) (rate(health_events_total{recommended_action!="NONE"}[5m]))
# Which agent and check is producing the actionable events
sum by (agent, check_name) (rate(health_events_total{recommended_action!="NONE",is_healthy="false"}[1h]))
# Per node, correct for every agent including health-events-analyzer
sum by (node) (rate(health_events_total{recommended_action!="NONE"}[1h]))

Platform Connector Request Metrics

Both roles of the platform connector, the node-local DaemonSet and the deployment platform connector, expose these for the batches that reach the request handler. Alert on failed outcomes and on best-effort failures. The deployment platform connector (--mode=deployment) also exposes the metrics above for its Kubernetes connector and for the node-binding interceptor (its rejections are counted in platform_connector_auth_violations_total, by the reasons listed there), and the refusals below, which it answers before the handler sees the batch.

Metric NameTypeLabelsDescription
platform_connector_request_duration_secondsHistogramoutcomeDuration of health event batch requests that reached the handler, by outcome: ok (acknowledged), rejected (the batch is invalid), failed (the connector returned an error while processing the batch; the caller retries). Both roles expose it: on the node-local DaemonSet the connector is the set of ring buffers, which always accept, so a batch is ok once queued unless it is invalid; on the deployment platform connector the store connector’s write decides when config.json enables one, while the node update and the sink are best effort and never fail a batch, so without a store every batch that passed validation is ok.
platform_connector_health_events_received_totalCounterprocessing_strategy, is_healthyThe total number of health events that the platform connector has received, by processing strategy and event health
platform_connector_deployment_refusals_totalCounterreasonBatches refused before the handler saw them: index_unverified (this replica has a store connector and has not verified the idempotency index; the caller retries) or idempotency_key (the idempotency-key header is missing or malformed)
platform_connector_store_batches_totalCounteroutcomeBatches written to the datastore by either role: stored, or duplicate (a resend, or a retried batch, whose events already existed; treated like a store)
platform_connector_best_effort_failures_totalCounterconnector, reasonDeployment platform connector only: work a best-effort connector (kubernetes node conditions and Events, grpcsink forwards) did not finish although the batch was acknowledged: failed or timeout; the node shows the old state until the monitor next reports

Workqueue Metrics

These metrics track the internal ring buffer workqueue performance:

Metric NameTypeLabelsDescription
platform_connector_workqueue_depth_{name}GaugeworkqueueCurrent depth of Platform connector workqueue
platform_connector_workqueue_adds_total_{name}CounterworkqueueTotal number of adds handled by Platform connector workqueue
platform_connector_workqueue_latency_seconds_{name}HistogramworkqueueHow long an item stays in Platform connector workqueue before being requested. Uses linear buckets (0, 10, 500)
platform_connector_workqueue_work_duration_seconds_{name}HistogramworkqueueHow long processing an item from Platform connector workqueue takes. Uses linear buckets (0, 10, 500)
platform_connector_workqueue_retries_total_{name}CounterworkqueueTotal number of retries handled by Platform connector workqueue
platform_connector_workqueue_longest_running_processor_seconds_{name}GaugeworkqueueHow many seconds the longest running processor for Platform connector workqueue has been running
platform_connector_workqueue_unfinished_work_seconds_{name}GaugeworkqueueThe total time in seconds of work in progress in Platform connector workqueue

Note: {name} in the metric names is replaced with the actual workqueue name at runtime.

gRPC Sink Connector Metrics

Metric NameTypeLabelsDescription
grpc_sink_platform_connector_send_totalCounterstatusThe total number of health event sends to the gRPC sink by status
grpc_sink_platform_connector_send_duration_millisecondsHistogram-Duration of gRPC sink sends in milliseconds
grpc_sink_platform_connector_retry_totalCounter-The total number of retried health event sends to the gRPC sink

Transformer Metrics

Metric NameTypeLabelsDescription
nvsentinel_overrides_applied_totalCounterrule_name, fieldTotal number of overrides applied by rule and field
nvsentinel_override_evaluation_errors_totalCounter-Total number of override rule evaluation errors
nvsentinel_platform_connector_dedup_store_and_analyse_totalCountercheck, err_codeTotal number of duplicate health events marked STORE_AND_ANALYSE by deduplication
nvsentinel_platform_connector_metadata_skip_label_gated_totalCounter-Total health events gated to STORE_ONLY because the target node carries the configured skip label
nvsentinel_platform_connector_metadata_lookup_failures_totalCounter-Total metadata lookup failures where the event proceeded ungated (fail-open)

Health Events Analyzer

Event Processing Metrics

Metric NameTypeLabelsDescription
health_event_analyzer_events_received_totalCounternode_nameTotal number of events received from the watcher
health_event_analyzer_events_successfully_processed_totalCounter-Total number of events successfully processed
health_event_analyzer_event_processing_errorsCountererror_typeTotal number of errors encountered during event processing
fatal_event_publishing_errorsCountererror_typeTotal number of errors that occurred while publishing events to UDS
fatal_events_published_totalCounterentity_valueTotal number of times a fatal event is published for an entity
recovery_events_published_totalCounterrule_name, node_nameTotal number of recovery health events accepted by the platform connector, including republishes awaiting storage
rule_matched_totalCounterrule_name, node_nameTotal number of times a rule matched for a node
rule_matched_entity_totalCounterrule_name, node_name, entity_type, entity_valueTotal number of times a rule matched, labeled by the entity it selected on (recorded when ruleMatchedEntityMetricEnabled is set)
health_event_analyzer_rule_skipped_totalCounterrule_nameTotal number of rule evaluations skipped because the rule’s when expression was false
health_event_analyzer_rule_when_errors_totalCounterrule_nameTotal number of times a rule’s when expression failed to evaluate. The rule was evaluated anyway
mongo_query_execution_duration_secondsHistogramrule_nameHistogram of MongoDB pipeline execution durations
health_event_analyzer_event_handling_duration_secondsHistogram-Histogram of event handling durations

Event Exporter

Event Export Metrics

Metric NameTypeLabelsDescription
health_events_exporter_events_received_totalCounter-Total number of health events received from the change stream
health_events_exporter_events_published_totalCounterstatusTotal number of events published to the sink. Status values: success, failure
health_events_exporter_publish_duration_secondsHistogram-Histogram of event publish latency in seconds
health_events_exporter_transform_errors_totalCounter-Total number of errors transforming events to CloudEvents format
health_events_exporter_publish_errors_totalCountererror_typeTotal number of errors publishing events to the sink
health_events_exporter_token_refresh_errors_totalCounter-Total number of OIDC token refresh errors
health_events_exporter_events_filtered_totalCounter-Total number of events not published because the filter expression evaluated false
health_events_exporter_filter_errors_totalCounter-Total number of filter evaluation errors. The event is exported anyway (counts events that bypassed the filter)
health_events_exporter_resume_token_update_timestampGauge-Timestamp of the last resume token update (Unix seconds)
health_events_exporter_backfill_in_progressGauge-1 during backfill, 0 otherwise
health_events_exporter_backfill_events_processed_totalCounter-Total number of events processed during backfill
health_events_exporter_backfill_duration_secondsHistogram-Histogram of backfill operation duration in seconds

Lifecycle Manager

Validation Request Metrics

Metric NameTypeLabelsDescription
validation_requests_totalCounter-Total number of ValidationRequests initiated
validation_requests_completed_totalCounterstatusTotal number of completed ValidationRequests, labeled by their final status. Status values: success, failure
validation_requests_duration_secondsHistogramstatusThe end-to-end duration of ValidationRequests, labeled by their final status
new_node_validation_batches_totalCounterstatusTotal number of new node validation batch flush attempts, labeled by status
new_node_validation_batch_sizeHistogram-Number of nodes included in each new node validation batch that succeeded

Metadata Collector

Collection Metrics

Metric NameTypeLabelsDescription
metadata_collector_collection_completed_timestamp_secondsGauge-Unix time GPU metadata collection completed, or 0 while it is still running. Collection can block on a wedged GPU driver, so 0 long after process start means it has stalled

Pod Mapper Metrics

Metric NameTypeLabelsDescription
metadata_collector_pod_mapper_failures_totalCounter-Pod device mapper poll cycles that failed
metadata_collector_pod_mapper_consecutive_failuresGauge-Consecutive failed pod device mapper poll cycles, reset to 0 by any success. The container exits when this reaches --pod-mapper-max-consecutive-failures

Commons

Feature Flag and State Metrics

Metric NameTypeLabelsDescription
nvsentinel_feature_flag_enabledGaugeservice, flagReports whether a feature flag is enabled (1) or disabled (0)
nvsentinel_state_transition_unexpected_totalCounterfrom_state, to_state, nodeTotal number of unexpected state transitions detected (race conditions or bugs)

Health Monitors

Health Event Publisher

The Go health monitors send their events through the shared publishing client (commons/pkg/healthpub). The monitor label names the monitor. The socket rows apply to the node-local path; the direct-mode rows apply only when a monitor publishes to the deployment platform connector (HEALTH_PUBLISH_TARGET set, which the chart’s publishTo: deployment does once the chart ships).

Metric NameTypeLabelsDescription
nvsentinel_health_events_publisher_skipped_pc_unavailable_totalCountermonitorSocket-path sends skipped because the node-local platform connector socket was missing
nvsentinel_health_events_publisher_sends_success_totalCountermonitorBatches the platform connector acknowledged, on either path
nvsentinel_health_events_publisher_sends_error_totalCountermonitor, codeSocket-path sends that failed after retries, by gRPC status code
nvsentinel_health_events_publisher_dropped_totalCountermonitor, reasonDirect-mode batches given up on: rejected (the server refuses the batch for good, or it exceeds the 4 MiB message limit), retry_window_exhausted, shutdown, withdrawn (the caller stopped waiting: its context ended or its timeout expired, and the batch was taken back)
nvsentinel_health_events_publisher_retries_totalCountermonitorDirect-mode send attempts that failed and were retried inside the batch’s retry window

GPU Health Monitor

These metrics track GPU health events detected via DCGM (Data Center GPU Manager):

Metric NameTypeLabelsDescription
dcgm_health_events_publish_time_to_grpc_channelHistogramoperation_nameAmount of time spent in publishing DCGM health events on the gRPC channel
health_events_insertion_to_uds_succeed_totalCounter-Total number of successful insertions of health events to UDS
health_events_insertion_to_uds_error_totalCounter-Total number of failed insertions of health events to UDS
health_events_insertion_skipped_pc_unavailable_totalCounter-Sends skipped because the node-local platform connector socket was missing
health_events_direct_publish_succeed_totalCounter-Batches the deployment platform connector acknowledged (direct mode)
health_events_direct_publish_dropped_totalCounterreasonDirect-mode batches given up on; the same reasons as the Go publisher’s dropped_total
health_events_direct_publish_retries_totalCounter-Direct-mode send attempts that failed and were retried
dcgm_connectivity_consecutive_observationsGaugeresultConsecutive DCGM connectivity observations by result
dcgm_health_active_eventsGaugeevent_type, gpu_id, error_code, is_fatal, processing_strategyTotal number of active GPU health events at any given time. is_fatal is true or false, the same as the isFatal that gpu-health-monitor sends (recommendedAction is not NONE). Use it to exclude informational events from alerts. The value comes from the gpu-health-monitor error-code mapping. processing_strategy is the processing strategy that gpu-health-monitor sends with the event, for example EXECUTE_REMEDIATION or STORE_ONLY. The global processingStrategy sets it, and a per-check setting can change it to STORE_ONLY, for example for GpuThermalMarginWatch, GpuPowerBrakeWatch and GpuDcgmUnresponsive. When the value is STORE_ONLY, no NVSentinel module acts on the event. When the value is EXECUTE_REMEDIATION, gpu-health-monitor requests action, but platform-connectors can still change the strategy. For example, it sets STORE_ONLY for a node with the nvsentinel.dgxc.nvidia.com/managed=false label. Like is_fatal, this label shows what gpu-health-monitor sends, not the final decision.
dcgm_health_active_switch_eventsGaugeevent_type, switch_id, error_codeTotal number of active NVSwitch health events at any given time. Separate from dcgm_health_active_events because switch and GPU ids are distinct sequences that both start at 0
dcgm_health_check_suppressed_incidents_totalCountererror_codeNumber of DCGM health check incidents suppressed due to a non-actionable error code
dcgm_health_check_debounced_incidents_totalCountererror_code, gpu_idNumber of DCGM health check polls where an error code was withheld for a GPU because it had not yet persisted for its configured minimum consecutive polls
dcgm_api_latencyHistogramoperation_nameAmount of time spent calling DCGM APIs
dcgm_reconcile_timeHistogram-Amount of time spent running a single DCGM reconcile loop
number_of_health_watchesGauge-Number of DCGM health watches available
number_of_fieldsGauge-Number of available DCGM fields to monitor
callback_failures_totalCounterclass_name, func_nameNumber of times a callback function has thrown an exception
callback_success_totalCounterclass_name, func_nameNumber of times a callback function has successfully completed
dcgm_api_failures_totalCountererror_nameNumber of DCGM API errors
dcgm_health_check_unknown_system_skipped_totalCounter-Number of DCGM health check incidents skipped due to unrecognized system value
dcgm_probe_hangs_totalCounteroperation_nameNumber of DCGM probes that exceeded the watchdog deadline without returning
gpu_temp_limit_margin_blank_totalCounter-Number of poll cycles where DCGM field 153 (T.Limit) returned a blank value
gpu_temp_limit_slowdown_threshold_missing_totalCounter-Number of poll cycles where slowdown T.Limit metadata from NVML field 194 is missing
gpu_power_brake_reasons_blank_totalCounter-Number of GPU polls where DCGM clocks-event-reasons was missing or blank, so GpuPowerBrakeWatch could not be evaluated

NIC Health Monitor

Metric NameTypeLabelsDescription
nic_health_monitor_devices_discoveredGaugenode, checkNumber of NIC devices discovered in sysfs
nic_health_monitor_health_events_sent_totalCounternode, check, is_fatalTotal number of health events sent to the platform connector
nic_health_monitor_poll_cycle_duration_secondsHistogramnode, categoryDuration of each poll cycle in seconds
nic_health_monitor_poll_cycle_last_completed_timestamp_secondsGaugenode, categoryUnix timestamp initialized at monitor startup and advanced after each completed poll cycle (state or counter). Use time() - metric to measure how long the category has gone without completing a poll, including a stall in the first poll.
nic_health_monitor_state_check_errors_totalCounternode, check, device, portTotal number of state check error events
nic_health_monitor_port_state_blips_totalCounternode, check, device, portPorts that turned unhealthy and recovered before the port-state hold-down reported them
nic_health_monitor_counter_threshold_breaches_totalCounternode, counter, device, port, is_fatalTotal number of counter threshold breaches detected
nic_health_monitor_first_poll_deferred_totalCounternode, checkPolls deferred before the first evaluation because devices were unreadable

Syslog Health Monitor

The syslog health monitor tracks GPU-related errors detected from system logs.

XID Error Metrics

XID (GPU Error ID) errors are NVIDIA GPU driver errors:

Metric NameTypeLabelsDescription
syslog_health_monitor_xid_errorsCounternode, err_codeTotal number of XID errors found
syslog_health_monitor_xid_processing_errorsCountererror_type, nodeTotal number of errors encountered during XID processing
syslog_health_monitor_xid_processing_latency_secondsHistogram-Histogram of XID processing latency
syslog_health_monitor_cancellations_emitted_totalCountercheck, source_error_code, target_error_codeTotal number of synthetic cancellation health events emitted by configured rules
syslog_health_monitor_xid_empty_driver_version_totalCounternodeTotal number of XID decode requests issued with an empty driver version

SXID Error Metrics

SXID errors are NVSwitch-related errors:

Metric NameTypeLabelsDescription
syslog_health_monitor_sxid_errorsCounternode, err_code, link, nvswitchTotal number of SXID errors found

GPU Fallen Off Bus Metrics

Metric NameTypeLabelsDescription
syslog_health_monitor_gpu_fallen_errorsCounternodeTotal number of GPU fallen off bus errors detected

NIC Driver Error Metrics

Metric NameTypeLabelsDescription
syslog_health_monitor_nic_driver_errorsCounternode, event, severityTotal number of NIC driver/firmware errors detected from kernel logs

CSP Health Monitor

The CSP health monitor tracks cloud provider maintenance events and node health issues.

CSP Client Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_csp_events_received_totalCountercspTotal number of raw events received from CSP API/source
csp_health_monitor_csp_polling_duration_secondsHistogramcspDuration of CSP polling cycles
csp_health_monitor_csp_api_errors_totalCountercsp, error_typeTotal number of errors encountered during CSP API calls
csp_health_monitor_csp_api_polling_duration_secondsHistogramcsp, apiDuration of CSP API polling cycles
csp_health_monitor_csp_monitor_errors_totalCountercsp, error_typeTotal number of errors initializing or starting CSP monitors
csp_health_monitor_csp_events_by_type_unsupported_totalCountercsp, event_typeTotal number of raw CSP events received, partitioned by event type code

Event Processing Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_main_events_to_normalize_totalCountercspTotal number of events passed to the normalizer
csp_health_monitor_main_normalization_errors_totalCountercspTotal number of errors during event normalization
csp_health_monitor_main_events_received_totalCountercspTotal number of normalized events received by the main processor
csp_health_monitor_main_events_processed_success_totalCountercspTotal number of events successfully processed
csp_health_monitor_main_processing_errors_totalCountercsp, error_typeTotal number of errors during event processing
csp_health_monitor_main_event_processing_duration_secondsHistogramcspDuration of processing a single event

Datastore Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_main_datastore_upsert_attempts_totalCountercspTotal number of attempts to upsert maintenance events
csp_health_monitor_main_datastore_upsert_totalCountercsp, statusTotal number of maintenance event upserts by status. Status values: success, failed

Trigger Engine Metrics

Metric NameTypeLabelsDescription
csp_health_monitor_trigger_poll_cycles_totalCounter-Total number of polling cycles executed by the trigger engine
csp_health_monitor_trigger_poll_errors_totalCounter-Total number of errors during a trigger engine poll cycle
csp_health_monitor_trigger_events_found_totalCountertrigger_typeTotal number of events found potentially needing a trigger
csp_health_monitor_trigger_attempts_totalCountertrigger_typeTotal number of trigger attempts made
csp_health_monitor_trigger_success_totalCountertrigger_typeTotal number of successful triggers
csp_health_monitor_trigger_failures_totalCountertrigger_type, failure_reasonTotal number of failed trigger attempts
csp_health_monitor_trigger_datastore_query_duration_secondsHistogramquery_typeDuration of datastore queries performed by the trigger engine
csp_health_monitor_trigger_datastore_query_errors_totalCounterquery_typeTotal number of errors during datastore queries
csp_health_monitor_trigger_datastore_update_errors_totalCountertrigger_typeTotal number of errors updating event status after trigger
csp_health_monitor_trigger_uds_send_duration_secondsHistogram-Duration of sending health events via UDS
csp_health_monitor_trigger_uds_send_errors_totalCounter-Total number of errors encountered when sending events via UDS
csp_health_monitor_node_not_ready_timeout_totalCounternode_nameTotal number of nodes that remained not ready after the timeout period
csp_health_monitor_node_readiness_monitoring_started_totalCounternode_nameTotal number of times background node readiness monitoring was started

NVCRE Certification Monitor

Metric NameTypeLabelsDescription
nvcre_certification_monitor_sweep_errors_totalCountererror_typeTotal number of errors hit during reconciliation sweeps over Certification CRs, by error type
nvcre_certification_monitor_sweep_duration_secondsHistogram-Duration of a reconciliation sweep over Certification CRs
nvcre_certification_monitor_health_events_published_totalCounternode, is_healthyTotal number of health events published to platform-connectors, by node and health state
nvcre_certification_monitor_health_event_publish_errors_totalCounternode, is_healthyTotal number of health events that could not be published to platform-connectors, by node and health state
nvcre_certification_monitor_active_failuresGauge-Number of (node, variant, reason) certification failures asserted by terminal Certification CRs
nvcre_certification_monitor_malformed_node_annotationsGauge-Number of nodes skipped in the last sweep because their cert-failures annotation is unparsable

Kubernetes Object Monitor

Metric NameTypeLabelsDescription
k8s_object_monitor_policy_matches_totalCounterpolicy_name, node, resource_kindTotal number of times a policy matched (predicate=true), indicating detected issues by node
k8s_object_monitor_policy_evaluation_errors_totalCounterpolicy_name, error_typePolicy evaluation errors
k8s_object_monitor_health_events_publish_errors_totalCounterpolicy_name, error_typeErrors publishing health events to Platform Connector via gRPC
k8s_object_monitor_reconciliation_errors_totalCounterresource_kind, error_typeController reconciliation errors

Slurm Drain Monitor

Metric NameTypeLabelsDescription
slurm_drain_monitor_external_drains_detected_totalCounterpattern_nameTotal number of external Slurm drains detected by pattern
slurm_drain_monitor_parse_matches_totalCounterpattern_nameTotal number of parse matches by pattern name
slurm_drain_monitor_reconciliation_errors_totalCountererror_typeController reconciliation errors
slurm_drain_monitor_health_events_publish_errors_totalCountererror_typeErrors publishing health events to Platform Connector via gRPC

Change Stream Metrics

Emitted by store-client, so they appear on every module that reads the change stream: event-exporter, fault-quarantine, health-events-analyzer, node-drainer, and fault-remediation, on both the MongoDB and PostgreSQL providers. The client label is the consumer’s name.

fault-remediation serves only controller-runtime’s registry, so it passes that registry explicitly; the others use the default registry.

datastore_connected is the one exception in this table: it comes from commons/pkg/server rather than store-client, is exported only by fault-quarantine, node-drainer and fault-remediation, and carries no client label.

Metric NameTypeLabelsDescription
change_stream_lag_secondsGaugeclientSeconds since this consumer last had evidence it was caught up with its own change stream, from either an empty batch or the server-side timestamp of the last event it read. Absent until one of those has been observed
change_stream_lag_knownGaugeclient1 once change_stream_lag_seconds can be computed for this consumer, 0 before then
change_stream_resume_token_recoveries_totalCounterclient, phaseTimes a stale or invalid resume token was deleted, restarting the stream from the current position
change_stream_reopens_totalCounterclient, resultAttempts to reopen the stream after a runtime failure. success is a disconnect that would previously have ended the stream permanently; failure is one attempt that did not get through; exhausted is the recovery budget running out, after which the watcher stops and the process is expected to exit so the pod restarts — pair it with kube_pod_container_status_restarts_total
datastore_connectedGaugenone1 once a change stream watcher has been handed to the readiness checker — after datastore initialisation, but before the watcher’s event loop starts — and 0 before that. It latches: it does not return to 0 if the datastore is lost later, because that stops the component and the pod restarts instead — pair it with kube_pod_container_status_restarts_total. The name predates this behaviour and reads as live connectivity, which it is not. It is also the whole of these components’ readiness check, so kube_pod_status_ready carries the same answer with pod labels and without depending on this endpoint being scrapable; prefer that unless you need a signal from the component’s own registry

Lag is measured against the consumer’s own filtered stream, so a module whose pipeline admits nothing still reports zero lag while it is caught up. See ADR-054.

Suggested alerts

# A consumer is behind. Tune the threshold per fleet; no lag_known qualifier is needed,
# because the series is absent rather than zero while lag is unknown.
- alert: ChangeStreamConsumerBehind
  expr: change_stream_lag_seconds > 900
  for: 10m
# A watcher that never started, or is wedged before its first read. The grace period covers
# normal startup, which closes in milliseconds on a live stream.
- alert: ChangeStreamLagUnknown
  expr: change_stream_lag_known == 0
  for: 10m

What zero lag does not tell you

Each of these is a way a consumer can be behind while the gauge reads zero.

  1. Replication lag. The MongoDB stream is opened SecondaryPreferred with no max staleness, so an empty batch means “caught up with the secondary I am reading”, not with the primary.
  2. Data skipped after a resume-token recovery. When a stored token is too old for the oplog the watcher reopens from now, so lag reads near zero precisely when the most was skipped. Read change_stream_lag_seconds together with change_stream_resume_token_recoveries_total: a lag of zero is only reassuring if the recoveries counter has not moved.
  3. Durable position. This measures the watcher’s progress against its own stream, not whether its position was persisted. A consumer that reads an event and dies before MarkProcessed succeeds looks healthy here, because the read did happen.

Metrics Configuration

Scraping Metrics

All NVSentinel components expose Prometheus metrics on a metrics endpoint (typically :2112/metrics). The metrics can be scraped by Prometheus using standard scrape configurations.

Helm Chart Configuration

The NVSentinel Helm chart automatically creates a PodMonitor resource for Prometheus Operator integration:

helm install nvsentinel ./distros/kubernetes/nvsentinel \
  --namespace nvsentinel --create-namespace

The PodMonitor is configured to scrape all NVSentinel component pods on their metrics endpoints (/metrics on port metrics).

Annotation-based Discovery

Components can be configured to include Prometheus scrape annotations:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "2112"
  prometheus.io/path: "/metrics"

Metric Types Reference

  • Counter: A cumulative metric that only increases or resets to zero on restart
  • Gauge: A metric that can arbitrarily go up and down
  • Histogram: Samples observations and counts them in configurable buckets
  • Summary: Similar to histogram but calculates configurable quantiles over a sliding time window

Common Label Values

Status Labels

  • success / failed - Operation outcome
  • started / succeeded / failed - Action lifecycle status

Action Types

  • reboot - Node reboot action
  • terminate - Node termination action

CSP Labels

  • gcp - Google Cloud Platform
  • aws - Amazon Web Services

Trigger Types

  • quarantine - Node quarantine trigger
  • healthy - Node healthy trigger