GPU Health Monitor Configuration

Overview

The GPU Health Monitor module watches GPU health using NVIDIA DCGM (Data Center GPU Manager) and reports hardware failures. This document covers all Helm configuration options for system administrators.

DCGM Deployment Modes

The GPU Health Monitor supports three DCGM source modes, selected with global.dcgm.mode.

Operator Service

The GPU Operator runs DCGM as a DaemonSet and exposes it through a Kubernetes service. GPU Health Monitor pods connect to the service endpoint.

Characteristics:

• DCGM runs as a DaemonSet (one pod per GPU node)
• Kubernetes service provides DNS endpoint for DCGM
• GPU Health Monitor connects via service DNS name

External Hostengine

An externally managed hostengine runs on each GPU node. GPU Health Monitor pods use host networking and connect to the configured endpoint, which defaults to localhost:5555.

Characteristics:

• The hostengine lifecycle is managed outside NVSentinel
• No Kubernetes service needed
• GPU Health Monitor enables host networking automatically
• You label each GPU node with its DCGM version before you install

Embedded Mode

GPU Health Monitor starts an in-process DCGM hostengine and exposes it to pod-local clients on a loopback endpoint.

Characteristics:

• No separate DCGM DaemonSet or service is needed
• gpu-health-monitor.runtimeClassName must name the cluster’s NVIDIA RuntimeClass
• The chart automatically sets privileged: true on the GPU Health Monitor container
• The endpoint must be localhost, 127.0.0.1, or ::1
• You label each GPU node with its DCGM version before you install

DCGM Version Selection

Use a GPU Health Monitor image from the same DCGM major version as its hostengine. The node nvsentinel.dgxc.nvidia.com/dcgm.version label selects the 3.x or 4.x monitor image.

In operator-service mode, labeler derives this label from the DCGM pod image. In external-hostengine and embedded-mode, the cluster operator supplies the label.

Configuration Reference

Module Enable/Disable

Controls whether the gpu-health-monitor module is deployed in the cluster.

global:
  gpuHealthMonitor:
    enabled: true

Resources

Defines CPU and memory resource requests and limits for the gpu-health-monitor pod.

gpu-health-monitor:
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi

Logging

Controls verbosity of gpu-health-monitor logs.

gpu-health-monitor:
  verbose: "False"  # Options: "True", "False"

DCGM Configuration

Operator Service Mode

This is the default mode.

global:
  dcgm:
    mode: operator-service
    enabled: true
    service:
      endpoint: "nvidia-dcgm.gpu-operator.svc"
      port: 5555

To use a service in another namespace, override its endpoint:

global:
  dcgm:
    mode: operator-service
    service:
      endpoint: "dcgm-service.custom-namespace.svc.cluster.local"
      port: 5555

External Hostengine Mode

NVSentinel does not deploy the hostengine in this mode. The configured hostengine must already be running and reachable on every selected GPU node.

global:
  dcgm:
    mode: external-hostengine
    externalHostengine:
      endpoint: localhost
      port: 5555

GPU Health Monitor enables host networking automatically in this mode.

This mode does not set the DCGM version node label for you. Label every GPU node before you install, or no monitor pod schedules.

Embedded Mode

global:
  dcgm:
    mode: embedded-mode
    embedded:
      endpoint: localhost
      port: 5555
gpu-health-monitor:
  runtimeClassName: nvidia

runtimeClassName is required and must match an NVIDIA RuntimeClass installed in the cluster. The chart automatically sets the GPU Health Monitor container to privileged in embedded mode so the NVIDIA Container Toolkit can provide GPU and driver access; no separate security-context value is required.

This mode does not set the DCGM version node label for you. Label every GPU node before you install, or no monitor pod schedules.

DCGM Version Node Label

GPU Health Monitor ships one DaemonSet per DCGM major version. Each DaemonSet selects nodes with the nvsentinel.dgxc.nvidia.com/dcgm.version label, so every node gets the monitor image that matches its DCGM. All three modes render this nodeSelector, but only operator-service supplies the label for you.

In external-hostengine and embedded-mode there is no DCGM pod for labeler to read, so labeler cannot derive the version. Selecting either mode automatically configures labeler to keep a valid label that already exists. Labeler never creates it.

One case still removes the label: a node labelled nvsentinel.dgxc.nvidia.com/managed=false is opted out of NVSentinel management, and labeler strips all of its detection labels from that node, dcgm.version included. This applies in every source mode. Clear the opt-out before you label the node.

An unlabeled node runs no GPU Health Monitor pod, and nothing reports an error. The DaemonSet stays healthy because Kubernetes never schedules a pod it can reject. GPU health monitoring is silently absent on that node.

Label every GPU node with its DCGM major version. The only accepted values are 3.x and 4.x; labeler treats any other value as absent.

# One node
kubectl label node <node-name> nvsentinel.dgxc.nvidia.com/dcgm.version=4.x
# Many nodes at once. Label each DCGM version separately, using a selector that
# matches only the nodes running that version.
kubectl label node -l <your-4.x-selector> nvsentinel.dgxc.nvidia.com/dcgm.version=4.x
kubectl label node -l <your-3.x-selector> nvsentinel.dgxc.nvidia.com/dcgm.version=3.x

Do not label every GPU node with one version. A node labelled 4.x while running DCGM 3.x gets the 4.x monitor image, which then fails against the hostengine it finds. If your fleet is genuinely uniform, nvidia.com/gpu.present=true is a usable selector — it comes from Node Feature Discovery, which the GPU Operator installs. Substitute your own selector if you run neither, which is common in embedded-mode.

Confirm a monitor pod now runs on each labeled node:

kubectl get pods -n nvsentinel -l app.kubernetes.io/name=gpu-health-monitor -o wide

Compare that count against the nodes that carry the label:

kubectl get nodes -L nvsentinel.dgxc.nvidia.com/dcgm.version

Host Networking Override

external-hostengine enables host networking automatically. For other modes, it can be enabled explicitly when required by a custom deployment:

gpu-health-monitor:
  useHostNetworking: true

DCGM Health Check Incident Suppression

Drops DCGM health check incidents matching specific error codes before they generate a health event, so they are never persisted or acted on. Useful for high-frequency, non-actionable flaps (e.g. normal power-cap boost-clock behavior).

gpu-health-monitor:
  dcgmHealthCheck:
    imexMonitoringEnabled: false
    suppressedErrorCodes:
      - DCGM_FR_CLOCK_THROTTLE_POWER
      - DCGM_FR_CLOCKS_EVENT_POWER
      - DCGM_FR_CLOCK_THROTTLE_THERMAL
      - DCGM_FR_CLOCKS_EVENT_THERMAL

imexMonitoringEnabled

Enables DCGM IMEX health monitoring. It defaults to false. In DRA deployments, IMEX can be created only while a workload requests IMEX channels, so an idle node can legitimately have no IMEX daemon. Reporting that state as a DCGM health failure is not actionable.

With DCGM 4.7 or newer hostengine, false removes IMEX from the requested health-watch mask while retaining NVLink monitoring. With an older hostengine, the monitor filters the legacy DCGM_FR_IMEX_UNHEALTHY incident before it is combined with GPU incidents. The hostengine version controls which behavior is available.

Set this to true only where the IMEX daemon is expected to be continuously available. This restores the previous monitoring behavior. IMEX incidents are global DCGM entities, but GPU-health-monitor’s existing routing can attribute them to GPU 0. Dedicated node-level IMEX event handling is deferred to a separate change.

suppressedErrorCodes

List of DCGM error code names (as reported by DCGM, e.g. DCGM_FR_CLOCK_THROTTLE_POWER) to suppress. Suppression is scoped to the listed error codes only — other incidents on the same health watch (e.g. other GpuPowerWatch error codes) are still reported.

The defaults include throttling: of the errors these two watches raise, it is the only one that tracks load rather than a fault, and it maps to NONE, so it only ever produced node events.

Four names, two codes: DCGM_FR_CLOCK_THROTTLE_* is a deprecated alias of DCGM_FR_CLOCKS_EVENT_* with the same number, and either name can be reported, so both are listed.

DCGM_FR_NVLINK_ERROR_CRITICAL (71) temporarily maps to NONE as clarity is not there around supported recovery and counter-clearing procedure. Both WARN and FAIL incidents produce non-fatal unhealthy events. The default quarantine policy does not cordon nodes for these events. This applies to every incident with code 71, including other NVLink errors that share this code. Other error codes retain their configured actions.

Genuine power and cooling faults are unaffected, arriving as GPU_HW_POWER_BRAKE_VIOLATION via GpuPowerBrakeWatch and GPU_TEMP_HW_SLOWDOWN_VIOLATION via GpuThermalMarginWatch, both CONTACT_SUPPORT. DCGM_FR_THROTTLING_VIOLATION is not suppressed either: it comes only from dcgmi diag, not these watches.

Example: Disable suppression

Use this to report all error codes again, including throttling.

gpu-health-monitor:
  dcgmHealthCheck:
    suppressedErrorCodes: []

Hardware Power Brake Detection

GpuPowerBrakeWatch fails a GPU whose clocks-event-reasons mask has the hardware power brake bit (0x80) set, meaning the power delivery path is forcing clocks down. It reads DCGM_FI_DEV_CLOCKS_EVENT_REASONS directly, in the same way GpuThermalMarginWatch reads field 153, because DCGM’s POWER health watch does not report the brake.

That distinction matters. The POWER watch’s dominant code, DCGM_FR_CLOCK_THROTTLE_POWER, tracks power-capped clock throttling, maps to NONE, and is suppressed by default under DCGM Health Check Incident Suppression above. On a 288-node GB200 cluster it was observed active on 69% of nodes while exactly 50% had a brake asserted, correlating with neither the brake bit nor the SW power cap bit (0x04). A sustained brake is a power delivery fault, so it needs its own signal rather than sharing a non-actionable one.

GPU_HW_POWER_BRAKE_VIOLATION maps to CONTACT_SUPPORT in dcgmerrorsmapping.csv, matching the thermal precedent: an asserted brake is a facility or power delivery problem, not something a node reboot resolves.

gpu-health-monitor:
  dcgmFieldsMonitoring:
    gpuPowerBrakeMonitoringEnabled: false
    gpuPowerBrakeStoreOnly: true
    gpuPowerBrakeMinConsecutivePolls: 3

gpuPowerBrakeMonitoringEnabled

Enables the watch. Off by default so existing deployments see no behaviour change. If neither DCGM_FI_DEV_CLOCKS_EVENT_REASONS nor the older DCGM_FI_DEV_CLOCK_THROTTLE_REASONS is available in the running DCGM build, the monitor logs a warning and disables itself.

gpuPowerBrakeStoreOnly

Dry run. When true, this check’s events are emitted with processingStrategy=STORE_ONLY, so they are persisted and exported as metrics but excluded from the remediation pipeline: no node condition and no cordon. Defaults to true, so enabling the watch is observable before it can act.

gpuPowerBrakeMinConsecutivePolls

Consecutive polls with the bit set before the GPU is failed. A brake asserted for a single poll can be a load transient; a sustained assertion is the actionable case. A clear resets the counter, so a flapping brake never accumulates to a failure. 1 fails on first observation. A GPU with no usable sample is skipped and keeps its counter, so a gap in DCGM data neither raises nor clears a finding. This includes DCGM’s int64 “no data” sentinels, whose low byte has the brake bit set and which would otherwise read as an assertion.

DCGM Startup Gate

For remote DCGM modes, the GPU health monitor can wait for a functional DCGM API before its main container starts:

gpu-health-monitor:
  dcgmConnectivity:
    startupGate:
      enabled: true
      retryIntervalSeconds: 5
      connectTimeoutSeconds: 10

When enabled, the chart adds a wait-for-dcgm init container. Each attempt creates a DCGM handle and performs supported-GPU discovery; opening the TCP port alone is not considered ready. Failed attempts are retried indefinitely at retryIntervalSeconds. Each attempt runs in a child process bounded by connectTimeoutSeconds, so a blocked native DCGM call can be terminated without restarting the init container. The init container does not publish health events, so an unavailable DCGM endpoint during installation or restart cannot create node conditions, quarantine nodes, or trigger remediation before the monitor has established its first connection.

The default 10-second hard timeout leaves time for DCGM’s own 5-second connection timeout to return and log a specific connection error first; the parent timeout remains a backstop for a genuinely stuck native call.

The gate is disabled by default for backward compatibility and supports operator-service and external-hostengine. It cannot be enabled in embedded-mode, because the main GPU health monitor container starts the embedded hostengine; Helm rejects that combination rather than creating a pod that can never leave init.

While DCGM remains unavailable, the pod stays in Init:0/1; ordinary readiness failures do not restart the init container. This state should be monitored with the deployment’s existing pod-state alerts (for example, kube-state-metrics). An Init:CrashLoopBackOff instead indicates an image, configuration, or implementation failure. After startup, runtime DCGM failures are still handled by the GPU health monitor’s connectivity checks; the startup gate does not replace runtime failure handling or debouncing.

For example, alert when the startup gate has been waiting for more than five minutes:

- alert: NVSentinelGPUHealthMonitorWaitingForDCGM
  expr: |
    kube_pod_init_container_status_running{
      container="wait-for-dcgm"
    } == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "gpu-health-monitor is waiting for DCGM"
    description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been waiting for DCGM for more than 5 minutes."

Alert separately when the init program itself is repeatedly failing:

- alert: NVSentinelGPUHealthMonitorDCGMInitFailed
  expr: |
    kube_pod_init_container_status_waiting_reason{
      container="wait-for-dcgm",
      reason="CrashLoopBackOff"
    } == 1
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "gpu-health-monitor DCGM startup gate is failing"
    description: "The wait-for-dcgm init container in {{ $labels.namespace }}/{{ $labels.pod }} is repeatedly failing."

Runtime DCGM Connectivity Debounce

After the main container has started, transient DCGM timeouts or connection errors can be debounced before they become GpuDcgmConnectivityFailure transitions. For example, the following configuration requires three consecutive failures:

gpu-health-monitor:
  dcgmConnectivity:
    runtimeDebounce:
      failureThreshold: 3
      successThreshold: 1

failureThreshold is the number of consecutive failed DCGM connectivity cycles required before the unhealthy event is published. A successful cycle before the threshold resets the failure streak, so intermittent failures do not accumulate across healthy polls.

successThreshold is the number of consecutive successful cycles required to clear an already-published connectivity failure. A failed cycle during recovery resets the success streak and keeps the existing unhealthy event active. The threshold does not delay the initial healthy baseline event emitted for a node with no prior connectivity state.

Both values must be integers greater than or equal to 1. They default to 1, preserving the existing fail-once and recover-once behavior. Raising failureThreshold filters shorter connectivity interruptions at the cost of up to failureThreshold - 1 additional poll intervals of detection latency.

The debounce state belongs to the running monitor process and is not persisted. Restarting the process resets both observation streaks. The replacement process emits its normal initial healthy baseline as soon as a complete health check succeeds, allowing platform state left by the previous process to converge without waiting for successThreshold.

The following metric exposes the current streaks:

dcgm_connectivity_consecutive_observations{result="failure"}
dcgm_connectivity_consecutive_observations{result="success"}

Runtime debounce and the startup gate address different phases:

• startupGate prevents the monitor from starting and publishing connectivity failures before DCGM has been functional once.
• runtimeDebounce filters short connectivity transitions after the monitor is running.
• The probe watchdog is not delayed by failureThreshold: a DCGM call that has stopped returning may have no later poll from which to build a failure streak.

dcgmHealthCheck.connectivityFailureEscalationThreshold continues to count consecutive failed observations. If its threshold is lower than the debounce failure threshold, the first published unhealthy event may already recommend RESTART_BM; configure escalation at or above failureThreshold when a preliminary CONTACT_SUPPORT event is desired.

Example behavior for failureThreshold: 4 and successThreshold: 2:

failure 1-3  -> no unhealthy event
failure 4    -> publish GpuDcgmConnectivityFailure
success 1    -> keep the unhealthy event active
failure      -> reset the recovery streak
success 1    -> keep the unhealthy event active
success 2    -> publish the healthy recovery event

Unresponsive DCGM Detection

A DCGM call that stops answering never returns an error — callers park and the probe blocks forever rather than raising DCGMError_Timeout. Meanwhile the node can still report Ready with every GPU allocatable and no taint, so no other signal in the stack registers a fault. In embedded-mode that hang is node-local, but it is not yet proof of a kernel-driver wedge: DCGM userspace deadlock or lock contention can look the same until an independent NVML/nvidia-smi probe confirms the driver itself.

The poll loop cannot report this itself: it is blocked before the point where it would publish anything, and /healthz only observes that the loop is frozen, so kubelet restarts the container and the replacement blocks in the same place. The settings below close that gap.

gpu-health-monitor:
  dcgm:
    pollIntervalSeconds: 15
    # Omit (or null) to default to pollIntervalSeconds * 3; set 0 to disable.
    probeStoreOnly: true
  dcgmHealthCheck:
    connectivityFailureEscalationThreshold: 0

probeStoreOnly

Ships the check in dry-run. While true (the default) GpuDcgmUnresponsive is emitted with processingStrategy=STORE_ONLY, so it is persisted and exported as metrics but excluded from the remediation pipeline — no node condition, no cordon, no reboot. The event still carries RESTART_BM so the record shows what the node needs.

Watch dcgm_probe_hangs and the stored events for a release or two, confirm the detections match real on-node hangs on your fleet, then set probeStoreOnly: false to let remediation act on them. Both the unhealthy and the clearing event use the same strategy, so fault-quarantine always sees a consistent pair.

probeDeadlineSeconds

Seconds a single DCGM probe may run before a watchdog thread — which the blocked probe cannot stop — reports the stalled operation. In embedded-mode the call is in-process and node-local, so it publishes GpuDcgmUnresponsive with error code DCGM_PROBE_HANG and recommended action RESTART_BM. In operator-service and external-hostengine modes, the same symptom can come from the endpoint, DNS, or network; those modes publish GpuDcgmConnectivityFailure with CONTACT_SUPPORT instead. Defaults to PollIntervalSeconds * 3 when unset. Set to 0 to disable the watchdog.

The default equals the /healthz staleness window (PollIntervalSeconds * 3), so the monitor reports when the poll loop is officially considered stalled. Critical event delivery is capped at 15 seconds, leaving the liveness probe’s remaining failure budget to persist the finding before kubelet restarts the container.

DCGM exposes timeout errors but does not document a fixed timeout for every RPC. Treat any deadline you configure as a fleet-specific value, not proof that every slower operation is a hard hang. The chart templates PollIntervalSeconds from dcgm.pollIntervalSeconds and, when probeDeadlineSeconds is null/omitted, sets the deadline to pollIntervalSeconds * 3 so the two stay coupled. Leave probeStoreOnly enabled while measuring normal embedded-mode probe latencies. If you substantially raise probeDeadlineSeconds, verify the resulting deadline still precedes the configured liveness restart; the chart exposes livenessProbe.periodSeconds and livenessProbe.failureThreshold for that adjustment.

The event reports once per hang episode. After delivery, a marker under the monitor’s persistent /var/run/nvsentinel state survives liveness restarts; it prevents the same hang from being republished and lets the first successful probe emit the healthy clearing event. Every DCGM call in the poll loop is tracked, including connect, health check, thermal margin evaluation, and the cleanup that follows a connectivity failure. Cleanup during intentional shutdown is not tracked, so a slow teardown while DCGM is restarting cannot be reported as a hang. dcgm_probe_hangs increments when the deadline is crossed even if event delivery must be retried.

connectivityFailureEscalationThreshold

Number of consecutive GpuDcgmConnectivityFailure cycles after which the recommended action escalates from CONTACT_SUPPORT to RESTART_BM.

Enable this only when the configured DCGM endpoint is node-local and repeated unreachability has been validated as a driver wedge. With a shared service, service failure, DNS issue, or network policy, rebooting the node is not a valid remediation.

Defaults to 0, which disables escalation and keeps every connectivity failure at CONTACT_SUPPORT. The counter resets once connectivity is restored, and the escalated event is published once rather than on every subsequent cycle.

Note: Both settings recommend RESTART_BM, which fault-remediation maps to a RebootNode CR. A reboot is the practical recovery when an on-node DCGM probe will not return — whether the underlying cause is a wedged driver or DCGM userspace holding driver locks. Nodes are drained before the reboot by node-drainer. Note that probeStoreOnly gates this for GpuDcgmUnresponsive, while connectivityFailureEscalationThreshold is opt-in by being 0 by default.

minConsecutivePolls

Number of consecutive polls a health check incident must persist, per error code and GPU, before it is published. Codes absent from the map publish on their first observation, which is the default behaviour.

gpu-health-monitor:
  dcgmHealthCheck:
    minConsecutivePolls:
      DCGM_FR_NVLINK_DOWN: 2

Use this for a code whose single-poll occurrences are transients rather than faults. DCGM_FR_NVLINK_DOWN is the motivating case: on SXM systems the NVLink links are briefly down after every node boot while they train, so a routine maintenance reboot otherwise produces a FATAL event carrying RESTART_VM. _is_nvlink_down_false_positive cannot cover this, because it is metadata-based and deliberately fails closed on SXM where an untrained link is indistinguishable from a dead one. A time-based threshold separates them: an untrained link trains, a dead one does not.

The streak is keyed on (error code, GPU) and resets whenever the code is absent for that GPU in a poll, so a code that appears on alternate polls for a GPU never accumulates and is never published. Note the key is the code and GPU, not the individual link: a GPU reporting one down link on one poll and a different one on the next keeps its streak, because that GPU has had a link down continuously. A streak also survives a failed poll, since a DCGM timeout observes nothing and must not clear it.

Tradeoff: a threshold of N delays a genuinely sustained fault by up to N-1 poll intervals. At the default pollIntervalSeconds: 15, a threshold of 2 costs at most 15 seconds of detection latency. Prefer the smallest threshold that suppresses the transient; permanently suppressing the code via suppressedErrorCodes would also mask genuine failures, which this avoids.

Withheld incidents are counted by dcgm_health_check_debounced_incidents{error_code, gpu_id}, so suppression stays observable per GPU. It increments once per poll per (error code, GPU), not once per incident record.

Additional Volumes

Extension point for mounting additional host paths required by DCGM in specific environments.

Configuration Structure

gpu-health-monitor:
  additionalVolumeMounts: []
  additionalHostVolumes: []

Parameters

additionalVolumeMounts

List of volume mounts to add to the GPU Health Monitor container. Each mount specifies where a volume should be mounted inside the container.

additionalHostVolumes

List of host path volumes to make available to the pod. Each volume references a path on the host node.

When to Use Additional Volumes

Additional volumes are required in environments where DCGM needs access to GPU drivers or libraries installed in non-standard host locations.

Common scenarios:

• GCP GKE nodes with GPU drivers in /home/kubernetes/bin/nvidia
• Custom driver installation paths

Volume Mount Examples

Example 1: GCP GKE Configuration

GCP GKE installs NVIDIA drivers and Vulkan ICD files in custom locations that the DCGM SDK needs to access.

gpu-health-monitor:
  additionalVolumeMounts:
    - mountPath: /usr/local/nvidia
      name: nvidia-install-dir-host
      readOnly: true
    - mountPath: /etc/vulkan/icd.d
      name: vulkan-icd-mount
      readOnly: true
  
  additionalHostVolumes:
    - name: nvidia-install-dir-host
      hostPath:
        path: /home/kubernetes/bin/nvidia
        type: Directory
    - name: vulkan-icd-mount
      hostPath:
        path: /home/kubernetes/bin/nvidia/vulkan/icd.d
        type: Directory