> This page is for NVSentinel, version v1.26.0. > For other versions, use one of these documentation indexes: > - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt > - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt > - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server. # GPU Health Monitor Configuration ## Overview The GPU Health Monitor module watches GPU health using NVIDIA DCGM (Data Center GPU Manager) and reports hardware failures. This document covers all Helm configuration options for system administrators. ## DCGM Deployment Modes The GPU Health Monitor supports three DCGM source modes, selected with `global.dcgm.mode`. ### Operator Service The GPU Operator runs DCGM as a DaemonSet and exposes it through a Kubernetes service. GPU Health Monitor pods connect to the service endpoint. **Characteristics:** - DCGM runs as a DaemonSet (one pod per GPU node) - Kubernetes service provides DNS endpoint for DCGM - GPU Health Monitor connects via service DNS name ### External Hostengine An externally managed hostengine runs on each GPU node. GPU Health Monitor pods use host networking and connect to the configured endpoint, which defaults to `localhost:5555`. **Characteristics:** - The hostengine lifecycle is managed outside NVSentinel - No Kubernetes service needed - GPU Health Monitor enables host networking automatically - You label each GPU node with its [DCGM version](#dcgm-version-node-label) before you install ### Embedded Mode GPU Health Monitor starts an in-process DCGM hostengine and exposes it to pod-local clients on a loopback endpoint. **Characteristics:** - No separate DCGM DaemonSet or service is needed - `gpu-health-monitor.runtimeClassName` must name the cluster's NVIDIA RuntimeClass - The chart automatically sets `privileged: true` on the GPU Health Monitor container - The endpoint must be `localhost`, `127.0.0.1`, or `::1` - You label each GPU node with its [DCGM version](#dcgm-version-node-label) before you install ## DCGM Version Selection Use a GPU Health Monitor image from the same DCGM major version as its hostengine. The node `nvsentinel.dgxc.nvidia.com/dcgm.version` label selects the 3.x or 4.x monitor image. In `operator-service` mode, labeler derives this label from the DCGM pod image. In `external-hostengine` and `embedded-mode`, the cluster operator supplies the label. ## Configuration Reference ### Module Enable/Disable Controls whether the gpu-health-monitor module is deployed in the cluster. ```yaml global: gpuHealthMonitor: enabled: true ``` ### Resources Defines CPU and memory resource requests and limits for the gpu-health-monitor pod. ```yaml gpu-health-monitor: resources: limits: cpu: 500m memory: 512Mi requests: cpu: 100m memory: 128Mi ``` ### Logging Controls verbosity of gpu-health-monitor logs. ```yaml gpu-health-monitor: verbose: "False" # Options: "True", "False" ``` ## DCGM Configuration ### Operator Service Mode This is the default mode. ```yaml global: dcgm: mode: operator-service enabled: true service: endpoint: "nvidia-dcgm-dra.gpu-operator.svc,nvidia-dcgm.gpu-operator.svc" port: 5555 ``` The endpoint may be a comma-separated list of hosts. The monitor tries them in order on every connect. Only one DCGM Service exists per cluster: `nvidia-dcgm-dra` in GPU Operator GPUCluster (DRA) mode and `nvidia-dcgm` otherwise, so the default works in both modes without per mode configuration. To use a service in another namespace, override its endpoint: ```yaml global: dcgm: mode: operator-service service: endpoint: "dcgm-service.custom-namespace.svc.cluster.local" port: 5555 ``` ### External Hostengine Mode NVSentinel does not deploy the hostengine in this mode. The configured hostengine must already be running and reachable on every selected GPU node. ```yaml global: dcgm: mode: external-hostengine externalHostengine: endpoint: localhost port: 5555 ``` GPU Health Monitor enables host networking automatically in this mode. This mode does not set the [DCGM version node label](#dcgm-version-node-label) for you. Label every GPU node before you install, or no monitor pod schedules. ### Embedded Mode ```yaml global: dcgm: mode: embedded-mode embedded: endpoint: localhost port: 5555 gpu-health-monitor: runtimeClassName: nvidia ``` `runtimeClassName` is required and must match an NVIDIA RuntimeClass installed in the cluster. The chart automatically sets the GPU Health Monitor container to privileged in embedded mode so the NVIDIA Container Toolkit can provide GPU and driver access; no separate security-context value is required. This mode does not set the [DCGM version node label](#dcgm-version-node-label) for you. Label every GPU node before you install, or no monitor pod schedules. ### DCGM Version Node Label GPU Health Monitor ships one DaemonSet per DCGM major version. Each DaemonSet selects nodes with the `nvsentinel.dgxc.nvidia.com/dcgm.version` label, so every node gets the monitor image that matches its DCGM. All three modes render this `nodeSelector`, but only `operator-service` supplies the label for you. | Mode | Who sets the label | |---|---| | `operator-service` | Labeler, from the GPU Operator DCGM pod image | | `external-hostengine` | You, before you install NVSentinel | | `embedded-mode` | You, before you install NVSentinel | In `external-hostengine` and `embedded-mode` there is no DCGM pod for labeler to read, so labeler cannot derive the version. Selecting either mode automatically configures labeler to keep a valid label that already exists. Labeler never creates it. One case still removes the label: a node labelled `nvsentinel.dgxc.nvidia.com/managed=false` is opted out of NVSentinel management, and labeler strips all of its detection labels from that node, `dcgm.version` included. This applies in every source mode. Clear the opt-out before you label the node. An unlabeled node runs no GPU Health Monitor pod, and nothing reports an error. The DaemonSet stays healthy because Kubernetes never schedules a pod it can reject. GPU health monitoring is silently absent on that node. Label every GPU node with its DCGM major version. The only accepted values are `3.x` and `4.x`; labeler treats any other value as absent. ```bash # One node kubectl label node nvsentinel.dgxc.nvidia.com/dcgm.version=4.x # Many nodes at once. Label each DCGM version separately, using a selector that # matches only the nodes running that version. kubectl label node -l nvsentinel.dgxc.nvidia.com/dcgm.version=4.x kubectl label node -l nvsentinel.dgxc.nvidia.com/dcgm.version=3.x ``` Do not label every GPU node with one version. A node labelled `4.x` while running DCGM 3.x gets the 4.x monitor image, which then fails against the hostengine it finds. If your fleet is genuinely uniform, `nvidia.com/gpu.present=true` is a usable selector — it comes from Node Feature Discovery, which the GPU Operator installs. Substitute your own selector if you run neither, which is common in `embedded-mode`. Confirm a monitor pod now runs on each labeled node: ```bash kubectl get pods -n nvsentinel -l app.kubernetes.io/name=gpu-health-monitor -o wide ``` Compare that count against the nodes that carry the label: ```bash kubectl get nodes -L nvsentinel.dgxc.nvidia.com/dcgm.version ``` ### Host Networking Override `external-hostengine` enables host networking automatically. For other modes, it can be enabled explicitly when required by a custom deployment: ```yaml gpu-health-monitor: useHostNetworking: true ``` ## DCGM Health Check Incident Suppression Drops DCGM health check incidents matching specific error codes before they generate a health event, so they are never persisted or acted on. Useful for high-frequency, non-actionable flaps (e.g. normal power-cap boost-clock behavior). ```yaml gpu-health-monitor: dcgmHealthCheck: imexMonitoringEnabled: false suppressedErrorCodes: - DCGM_FR_CLOCK_THROTTLE_POWER - DCGM_FR_CLOCKS_EVENT_POWER - DCGM_FR_CLOCK_THROTTLE_THERMAL - DCGM_FR_CLOCKS_EVENT_THERMAL ``` ### imexMonitoringEnabled Enables DCGM IMEX health monitoring. It defaults to `false`. In DRA deployments, IMEX can be created only while a workload requests IMEX channels, so an idle node can legitimately have no IMEX daemon. Reporting that state as a DCGM health failure is not actionable. With DCGM 4.7 or newer hostengine, `false` removes IMEX from the requested health-watch mask while retaining NVLink monitoring. With an older hostengine, the monitor filters the legacy `DCGM_FR_IMEX_UNHEALTHY` incident before it is combined with GPU incidents. The hostengine version controls which behavior is available. Set this to `true` only where the IMEX daemon is expected to be continuously available. This restores the previous monitoring behavior. IMEX incidents are global DCGM entities, but GPU-health-monitor's existing routing can attribute them to GPU 0. Dedicated node-level IMEX event handling is deferred to a separate change. ### suppressedErrorCodes List of DCGM error code names (as reported by DCGM, e.g. `DCGM_FR_CLOCK_THROTTLE_POWER`) to suppress. Suppression is scoped to the listed error codes only — other incidents on the same health watch (e.g. other `GpuPowerWatch` error codes) are still reported. The defaults include throttling: of the errors these two watches raise, it is the only one that tracks load rather than a fault, and it maps to `NONE`, so it only ever produced node events. | Watch | Suppressed | Still reported | | --- | --- | --- | | `GpuPowerWatch` | `DCGM_FR_CLOCKS_EVENT_POWER` (12) | `DCGM_FR_POWER_UNREADABLE`, `DCGM_FR_XID_ERROR` (XIDs 54, 56, 58, 78) | | `GpuThermalWatch` | `DCGM_FR_CLOCKS_EVENT_THERMAL` (10) | `DCGM_FR_THERMAL_VIOLATIONS`, `DCGM_FR_XID_ERROR` (XID 61) | Four names, two codes: `DCGM_FR_CLOCK_THROTTLE_*` is a deprecated alias of `DCGM_FR_CLOCKS_EVENT_*` with the same number, and either name can be reported, so both are listed. `DCGM_FR_NVLINK_ERROR_CRITICAL` (71) temporarily maps to `NONE` as clarity is not there around supported recovery and counter-clearing procedure. Both WARN and FAIL incidents produce non-fatal unhealthy events. The default quarantine policy does not cordon nodes for these events. This applies to every incident with code 71, including other NVLink errors that share this code. Other error codes retain their configured actions. Genuine power and cooling faults are unaffected, arriving as `GPU_HW_POWER_BRAKE_VIOLATION` via [`GpuPowerBrakeWatch`](#hardware-power-brake-detection) and `GPU_TEMP_HW_SLOWDOWN_VIOLATION` via `GpuThermalMarginWatch`, both `CONTACT_SUPPORT`. `DCGM_FR_THROTTLING_VIOLATION` is not suppressed either: it comes only from `dcgmi diag`, not these watches. ### Example: Disable suppression Use this to report all error codes again, including throttling. ```yaml gpu-health-monitor: dcgmHealthCheck: suppressedErrorCodes: [] ``` ## GPU Thermal Margin Detection `GpuThermalMarginWatch` reads DCGM field 153, the margin in degrees to the GPU's slowdown temperature limit, and fails a GPU whose margin reaches the hardware slowdown threshold. It reads the field directly because DCGM's own thermal health watch does not report the margin. ```yaml gpu-health-monitor: dcgmFieldsMonitoring: gpuTempLimitMonitoringEnabled: true gpuTempLimitStoreOnly: true ``` ### gpuTempLimitMonitoringEnabled Enables the watch. On by default. ### gpuTempLimitStoreOnly Dry run. When true, this check's events are emitted with `processingStrategy=STORE_ONLY`, so they are persisted and exported as metrics but excluded from the remediation pipeline: no node condition and no cordon. Defaults to true, so the watch is observable before it can act. Set it to `false` once you have confirmed the thresholds suit your hardware and cooling. To interpret a firing check, see the [GPU Thermal Margin runbook](/nvsentinel/v1.26.0/runbooks/gpu-thermal-margin). ## NVLink Suppression on Unbridged PCIe Cards Suppresses the NVLink-down fault on PCIe cards that have NVLink silicon but no bridge fitted. DCGM otherwise reports those permanently inactive links as a fatal fault with `RESTART_VM`. ```yaml gpu-health-monitor: suppressNvlinkDownOnUnbridgedPcie: "False" ``` The value is a quoted string, not a boolean. Leave it `"False"` if any GPU pool uses NVLink bridges. An unbridged card and a card whose bridge was already dead when metadata was collected look identical, so enabling suppression on a bridged pool could hide a real bridge failure that was present at boot. GPUs with no NVLink silicon at all, such as L40 and A40, are suppressed regardless of this value. ## Hardware Power Brake Detection `GpuPowerBrakeWatch` fails a GPU whose clocks-event-reasons mask has the hardware power brake bit (`0x80`) set, meaning the power delivery path is forcing clocks down. It reads `DCGM_FI_DEV_CLOCKS_EVENT_REASONS` directly, in the same way `GpuThermalMarginWatch` reads field 153, because DCGM's POWER health watch does not report the brake. That distinction matters. The POWER watch's dominant code, `DCGM_FR_CLOCK_THROTTLE_POWER`, tracks power-capped clock throttling, maps to `NONE`, and is suppressed by default under [DCGM Health Check Incident Suppression](#dcgm-health-check-incident-suppression) above. On a 288-node GB200 cluster it was observed active on 69% of nodes while exactly 50% had a brake asserted, correlating with neither the brake bit nor the SW power cap bit (`0x04`). A sustained brake is a power delivery fault, so it needs its own signal rather than sharing a non-actionable one. `GPU_HW_POWER_BRAKE_VIOLATION` maps to `CONTACT_SUPPORT` in `dcgmerrorsmapping.csv`, matching the thermal precedent: an asserted brake is a facility or power delivery problem, not something a node reboot resolves. ```yaml gpu-health-monitor: dcgmFieldsMonitoring: gpuPowerBrakeMonitoringEnabled: false gpuPowerBrakeStoreOnly: true gpuPowerBrakeMinConsecutivePolls: 3 ``` ### gpuPowerBrakeMonitoringEnabled Enables the watch. Off by default so existing deployments see no behaviour change. If neither `DCGM_FI_DEV_CLOCKS_EVENT_REASONS` nor the older `DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` is available in the running DCGM build, the monitor logs a warning and disables itself. ### gpuPowerBrakeStoreOnly Dry run. When true, this check's events are emitted with `processingStrategy=STORE_ONLY`, so they are persisted and exported as metrics but excluded from the remediation pipeline: no node condition and no cordon. Defaults to true, so enabling the watch is observable before it can act. ### gpuPowerBrakeMinConsecutivePolls Consecutive polls with the bit set before the GPU is failed. A brake asserted for a single poll can be a load transient; a sustained assertion is the actionable case. A clear resets the counter, so a flapping brake never accumulates to a failure. `1` fails on first observation. A GPU with no usable sample is skipped and keeps its counter, so a gap in DCGM data neither raises nor clears a finding. This includes DCGM's int64 "no data" sentinels, whose low byte has the brake bit set and which would otherwise read as an assertion. ## DCGM Startup Gate For remote DCGM modes, the GPU health monitor can wait for a functional DCGM API before its main container starts: ```yaml gpu-health-monitor: dcgmConnectivity: startupGate: enabled: true retryIntervalSeconds: 5 connectTimeoutSeconds: 10 ``` When enabled, the chart adds a `wait-for-dcgm` init container. Each attempt creates a DCGM handle and performs supported-GPU discovery; opening the TCP port alone is not considered ready. Failed attempts are retried indefinitely at `retryIntervalSeconds`. Each attempt runs in a child process bounded by `connectTimeoutSeconds`, so a blocked native DCGM call can be terminated without restarting the init container. The init container does not publish health events, so an unavailable DCGM endpoint during installation or restart cannot create node conditions, quarantine nodes, or trigger remediation before the monitor has established its first connection. The default 10-second hard timeout leaves time for DCGM's own 5-second connection timeout to return and log a specific connection error first; the parent timeout remains a backstop for a genuinely stuck native call. The gate is disabled by default for backward compatibility and supports `operator-service` and `external-hostengine`. It cannot be enabled in `embedded-mode`, because the main GPU health monitor container starts the embedded hostengine; Helm rejects that combination rather than creating a pod that can never leave init. While DCGM remains unavailable, the pod stays in `Init:0/1`; ordinary readiness failures do not restart the init container. This state should be monitored with the deployment's existing pod-state alerts (for example, kube-state-metrics). An `Init:CrashLoopBackOff` instead indicates an image, configuration, or implementation failure. After startup, runtime DCGM failures are still handled by the GPU health monitor's connectivity checks; the startup gate does not replace runtime failure handling or debouncing. For example, alert when the startup gate has been waiting for more than five minutes: ```yaml - alert: NVSentinelGPUHealthMonitorWaitingForDCGM expr: | kube_pod_init_container_status_running{ container="wait-for-dcgm" } == 1 for: 5m labels: severity: warning annotations: summary: "gpu-health-monitor is waiting for DCGM" description: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has been waiting for DCGM for more than 5 minutes." ``` Alert separately when the init program itself is repeatedly failing: ```yaml - alert: NVSentinelGPUHealthMonitorDCGMInitFailed expr: | kube_pod_init_container_status_waiting_reason{ container="wait-for-dcgm", reason="CrashLoopBackOff" } == 1 for: 2m labels: severity: critical annotations: summary: "gpu-health-monitor DCGM startup gate is failing" description: "The wait-for-dcgm init container in {{ $labels.namespace }}/{{ $labels.pod }} is repeatedly failing." ``` ## Runtime DCGM Connectivity Debounce After the main container has started, transient DCGM timeouts or connection errors can be debounced before they become `GpuDcgmConnectivityFailure` transitions. For example, the following configuration requires three consecutive failures: ```yaml gpu-health-monitor: dcgmConnectivity: runtimeDebounce: failureThreshold: 3 successThreshold: 1 ``` `failureThreshold` is the number of consecutive failed DCGM connectivity cycles required before the unhealthy event is published. A successful cycle before the threshold resets the failure streak, so intermittent failures do not accumulate across healthy polls. `successThreshold` is the number of consecutive successful cycles required to clear an already-published connectivity failure. A failed cycle during recovery resets the success streak and keeps the existing unhealthy event active. The threshold does not delay the initial healthy baseline event emitted for a node with no prior connectivity state. Both values must be integers greater than or equal to `1`. They default to `1`, preserving the existing fail-once and recover-once behavior. Raising `failureThreshold` filters shorter connectivity interruptions at the cost of up to `failureThreshold - 1` additional poll intervals of detection latency. The debounce state belongs to the running monitor process and is not persisted. Restarting the process resets both observation streaks. The replacement process emits its normal initial healthy baseline as soon as a complete health check succeeds, allowing platform state left by the previous process to converge without waiting for `successThreshold`. The following metric exposes the current streaks: ```text dcgm_connectivity_consecutive_observations{result="failure"} dcgm_connectivity_consecutive_observations{result="success"} ``` Runtime debounce and the startup gate address different phases: - `startupGate` prevents the monitor from starting and publishing connectivity failures before DCGM has been functional once. - `runtimeDebounce` filters short connectivity transitions after the monitor is running. - The probe watchdog is not delayed by `failureThreshold`: a DCGM call that has stopped returning may have no later poll from which to build a failure streak. `dcgmHealthCheck.connectivityFailureEscalationThreshold` continues to count consecutive failed observations. If its threshold is lower than the debounce failure threshold, the first published unhealthy event may already recommend `RESTART_BM`; configure escalation at or above `failureThreshold` when a preliminary `CONTACT_SUPPORT` event is desired. Example behavior for `failureThreshold: 4` and `successThreshold: 2`: ```text failure 1-3 -> no unhealthy event failure 4 -> publish GpuDcgmConnectivityFailure success 1 -> keep the unhealthy event active failure -> reset the recovery streak success 1 -> keep the unhealthy event active success 2 -> publish the healthy recovery event ``` ## Unresponsive DCGM Detection A DCGM call that stops answering never returns an error — callers park and the probe blocks forever rather than raising `DCGMError_Timeout`. Meanwhile the node can still report `Ready` with every GPU allocatable and no taint, so no other signal in the stack registers a fault. In `embedded-mode` that hang is node-local, but it is not yet proof of a kernel-driver wedge: DCGM userspace deadlock or lock contention can look the same until an independent NVML/`nvidia-smi` probe confirms the driver itself. The poll loop cannot report this itself: it is blocked before the point where it would publish anything, and `/healthz` only observes that the loop is frozen, so kubelet restarts the container and the replacement blocks in the same place. The settings below close that gap. ```yaml gpu-health-monitor: dcgm: pollIntervalSeconds: 15 # Omit (or null) to default to pollIntervalSeconds * 3; set 0 to disable. probeStoreOnly: true dcgmHealthCheck: connectivityFailureEscalationThreshold: 0 ``` ### probeStoreOnly Ships the check in dry-run. While `true` (the default) `GpuDcgmUnresponsive` is emitted with `processingStrategy=STORE_ONLY`, so it is persisted and exported as metrics but excluded from the remediation pipeline — no node condition, no cordon, no reboot. The event still carries `RESTART_BM` so the record shows what the node needs. Watch `dcgm_probe_hangs` and the stored events for a release or two, confirm the detections match real on-node hangs on your fleet, then set `probeStoreOnly: false` to let remediation act on them. Both the unhealthy and the clearing event use the same strategy, so fault-quarantine always sees a consistent pair. ### probeDeadlineSeconds Seconds a single DCGM probe may run before a watchdog thread — which the blocked probe cannot stop — reports the stalled operation. In `embedded-mode` the call is in-process and node-local, so it publishes `GpuDcgmUnresponsive` with error code `DCGM_PROBE_HANG` and recommended action `RESTART_BM`. In `operator-service` and `external-hostengine` modes, the same symptom can come from the endpoint, DNS, or network; those modes publish `GpuDcgmConnectivityFailure` with `CONTACT_SUPPORT` instead. Defaults to `PollIntervalSeconds * 3` when unset. Set to `0` to disable the watchdog. The default equals the `/healthz` staleness window (`PollIntervalSeconds * 3`), so the monitor reports when the poll loop is officially considered stalled. Critical event delivery is capped at 15 seconds, leaving the liveness probe's remaining failure budget to persist the finding before kubelet restarts the container. DCGM exposes timeout errors but does not document a fixed timeout for every RPC. Treat any deadline you configure as a fleet-specific value, not proof that every slower operation is a hard hang. The chart templates `PollIntervalSeconds` from `dcgm.pollIntervalSeconds` and, when `probeDeadlineSeconds` is null/omitted, sets the deadline to `pollIntervalSeconds * 3` so the two stay coupled. Leave `probeStoreOnly` enabled while measuring normal embedded-mode probe latencies. If you substantially raise `probeDeadlineSeconds`, verify the resulting deadline still precedes the configured liveness restart; the chart exposes `livenessProbe.periodSeconds` and `livenessProbe.failureThreshold` for that adjustment. The event reports once per hang episode. After delivery, a marker under the monitor's persistent `/var/run/nvsentinel` state survives liveness restarts; it prevents the same hang from being republished and lets the first successful probe emit the healthy clearing event. Every DCGM call in the poll loop is tracked, including connect, health check, thermal margin evaluation, and the cleanup that follows a connectivity failure. Cleanup during intentional shutdown is not tracked, so a slow teardown while DCGM is restarting cannot be reported as a hang. `dcgm_probe_hangs` increments when the deadline is crossed even if event delivery must be retried. ### connectivityFailureEscalationThreshold Number of consecutive `GpuDcgmConnectivityFailure` cycles after which the recommended action escalates from `CONTACT_SUPPORT` to `RESTART_BM`. Enable this only when the configured DCGM endpoint is node-local and repeated unreachability has been validated as a driver wedge. With a shared service, service failure, DNS issue, or network policy, rebooting the node is not a valid remediation. Defaults to `0`, which disables escalation and keeps every connectivity failure at `CONTACT_SUPPORT`. The counter resets once connectivity is restored, and the escalated event is published once rather than on every subsequent cycle. > **Note**: Both settings recommend `RESTART_BM`, which fault-remediation maps to a `RebootNode` CR. A reboot is the practical recovery when an on-node DCGM probe will not return — whether the underlying cause is a wedged driver or DCGM userspace holding driver locks. Nodes are drained before the reboot by node-drainer. Note that `probeStoreOnly` gates this for `GpuDcgmUnresponsive`, while `connectivityFailureEscalationThreshold` is opt-in by being `0` by default. ### minConsecutivePolls Number of consecutive polls a health check incident must persist, per error code and GPU, before it is published. Codes absent from the map publish on their first observation, which is the default behaviour. ```yaml gpu-health-monitor: dcgmHealthCheck: minConsecutivePolls: DCGM_FR_NVLINK_DOWN: 2 ``` Use this for a code whose single-poll occurrences are transients rather than faults. `DCGM_FR_NVLINK_DOWN` is the motivating case: on SXM systems the NVLink links are briefly down after every node boot while they train, so a routine maintenance reboot otherwise produces a FATAL event carrying `RESTART_VM`. `_is_nvlink_down_false_positive` cannot cover this, because it is metadata-based and deliberately fails closed on SXM where an untrained link is indistinguishable from a dead one. A time-based threshold separates them: an untrained link trains, a dead one does not. The streak is keyed on `(error code, GPU)` and resets whenever the code is absent for that GPU in a poll, so a code that appears on alternate polls for a GPU never accumulates and is never published. Note the key is the code and GPU, not the individual link: a GPU reporting one down link on one poll and a different one on the next keeps its streak, because that GPU has had a link down continuously. A streak also survives a failed poll, since a DCGM timeout observes nothing and must not clear it. **Tradeoff**: a threshold of `N` delays a genuinely sustained fault by up to `N-1` poll intervals. At the default `pollIntervalSeconds: 15`, a threshold of `2` costs at most 15 seconds of detection latency. Prefer the smallest threshold that suppresses the transient; permanently suppressing the code via `suppressedErrorCodes` would also mask genuine failures, which this avoids. Withheld incidents are counted by `dcgm_health_check_debounced_incidents{error_code, gpu_id}`, so suppression stays observable per GPU. It increments once per poll per `(error code, GPU)`, not once per incident record. ## Additional Volumes Extension point for mounting additional host paths required by DCGM in specific environments. ### Configuration Structure ```yaml gpu-health-monitor: additionalVolumeMounts: [] additionalHostVolumes: [] ``` ### Parameters #### additionalVolumeMounts List of volume mounts to add to the GPU Health Monitor container. Each mount specifies where a volume should be mounted inside the container. #### additionalHostVolumes List of host path volumes to make available to the pod. Each volume references a path on the host node. ### When to Use Additional Volumes Additional volumes are required in environments where DCGM needs access to GPU drivers or libraries installed in non-standard host locations. **Common scenarios:** - GCP GKE nodes with GPU drivers in `/home/kubernetes/bin/nvidia` - Custom driver installation paths ### Volume Mount Examples #### Example 1: GCP GKE Configuration GCP GKE installs NVIDIA drivers and Vulkan ICD files in custom locations that the DCGM SDK needs to access. ```yaml gpu-health-monitor: additionalVolumeMounts: - mountPath: /usr/local/nvidia name: nvidia-install-dir-host readOnly: true - mountPath: /etc/vulkan/icd.d name: vulkan-icd-mount readOnly: true additionalHostVolumes: - name: nvidia-install-dir-host hostPath: path: /home/kubernetes/bin/nvidia type: Directory - name: vulkan-icd-mount hostPath: path: /home/kubernetes/bin/nvidia/vulkan/icd.d type: Directory ```