Health Events Analyzer Configuration

Overview

The Health Events Analyzer evaluates MongoDB aggregation pipeline rules against incoming health events and emits derived events with recommended actions (for example, repeated XID 48 errors on the same GPU within 24 hours triggers a CONTACT_SUPPORT recommendation). This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the health-events-analyzer module is deployed in the cluster.

1
global:
2
  healthEventsAnalyzer:
3
    enabled: false

Replica Count

Number of health-events-analyzer pod replicas to run.

1
health-events-analyzer:
2
  replicaCount: 1

Resources

Defines CPU and memory resource requests and limits for the health-events-analyzer pod.

1
health-events-analyzer:
2
  resources:
3
    limits:
4
      cpu: "1"
5
      memory: "1Gi"
6
    requests:
7
      cpu: "1"
8
      memory: "1Gi"

Logging

Sets the verbosity level for health-events-analyzer logs.

1
health-events-analyzer:
2
  logLevel: info  # Options: debug, info, warn, error

Processing Strategy

Controls whether derived events trigger downstream remediation actions or are recorded only.

1
health-events-analyzer:
2
  processingStrategy: EXECUTE_REMEDIATION  # Options: EXECUTE_REMEDIATION, STORE_AND_ANALYSE, STORE_ONLY

Options:

EXECUTE_REMEDIATION

Normal operating mode. When a rule fires, downstream modules may update cluster state — applying node conditions, quarantining nodes, draining workloads, or triggering remediations.

STORE_AND_ANALYSE

Events are persisted and ingested by the Health Events Analyzer for rule evaluation. The individual event does not directly trigger quarantine or remediation, but HEA rules can still match on it and emit new EXECUTE_REMEDIATION synthetic events as a result — for example, a burst of deduplicated STORE_AND_ANALYSE events may collectively satisfy a repeated-failure rule. Use this when you want the raw event suppressed from immediate action but still want HEA correlation to fire.

STORE_ONLY

Observability-only mode. Derived events are persisted and exported but do not modify any cluster resources. Use this mode to shadow-test new or customised rules in production before enabling full remediation.

Client Certificate Mount Path

Path inside the container where TLS client certificates are mounted for authenticated MongoDB connections. Certificates are typically provisioned by cert-manager and mounted via a Kubernetes secret volume.

1
health-events-analyzer:
2
  clientCertMountPath: /etc/ssl/client-certs

Rule Enable/Disable Flags

Each built-in rule can be independently enabled or disabled. All rules are enabled by default. Set a flag to false to suppress a rule without removing it from the configuration.

1
health-events-analyzer:
2
  enableMultipleRemediationsRule: true
3
  enableRepeatedXIDErrorOnSameGPURule: true
4
  enableRepeatedXID31OnSameGPURule: true
5
  enableRepeatedXID31OnDifferentGPURule: true
6
  enableRepeatedXID13OnSameGPCAndTPCRule: true
7
  enableRepeatedXID13OnDifferentGPCAndTPCRule: true
8
  enableXIDErrorSoloNoBurstRule: true
9
  enableXID74Reg0SoloNVLinkErrorRule: true
10
  enableXID74Reg0ECCParityErrorRule: true
11
  enableRepeatedXID74Reg0HardwareIssueRule: true
12
  enableXID74Reg0SignalIntegrityErrorRule: true
13
  enableXID74Reg0Bit27Or29SetRule: true
14
  enableRepeatedXID74Reg2HardwareIssueRule: true
15
  enableXID74Reg2Bit13SetRule: true
16
  enableRepeatedXID74Reg2Bit16Or19SetRule: true
17
  enableRepeatedXID74Reg2Bit17Or18SetRule: true
18
  enableXID74Reg3UnexpectedErrorRule: true
19
  enableXID74Reg3Bit18SetRule: true
20
  enableXID74Reg4HardwareIssueRule: true
21
  enableXID74Reg4ECCErrorRule: true
22
  enableRepeatedNICDriverErrorRule: true
23
  enableRepeatedNICDegradationRule: true

The following table summarises every flag, the XID or event type it covers, and the recommended action emitted when the rule fires.

Rule Configuration

Rule Structure

Rules are defined as TOML in the config: block of values.yaml. Each rule entry has the following fields:

1
[[rules]]
2
name        = "RuleName"
3
description = "Human-readable description"
4
recommended_action = "CONTACT_SUPPORT"   # or RUN_DCGMEUD, NONE, etc.
5
evaluate_rule = true                     # Helm template expression; maps to the enable flag
6
stage = [
7
  '\{ "$match": \{ ... \} \}',              # MongoDB aggregation pipeline stages as JSON strings
8
  '\{ "$count": "count" \}',
9
  '\{ "$match": \{ "count": \{ "$gte": N \} \} \}'
10
]

The full default ruleset — including all aggregation pipeline stage definitions — is in the chart’s values.yaml at distros/kubernetes/nvsentinel/charts/health-events-analyzer/values.yaml. Refer to that file when writing or reviewing custom rules.

MultipleRemediations Rule

The MultipleRemediations rule fires when five or more remediations have been performed on the same node within the preceding 7 days. Unlike other rules, it applies a node condition that NVSentinel does not automatically clear, because the rule does not emit healthy events.

After the underlying hardware issue is resolved, remove the condition manually:

$
kubectl get node \{NODE_NAME\} -o json \
>
  | jq '.status.conditions |= map(select(.type != "MultipleRemediations"))' \
>
  | kubectl replace -f - --subresource=status

Replace MultipleRemediations with the actual condition type name if a custom rule applied a different condition.

XID 74 Register Rules

XID 74 (NVLink error) carries register fields (REG0–REG6) that encode the specific failure mode. The XID 74 rules inspect these register bit patterns to distinguish three categories of failure:

Signal integrity issues — REG0 bits 21 or 22 set (XID74Reg0SignalIntegrityErrorRule). Suggests marginal signal levels; check link mechanical connections and run field diagnostics if the issue persists.

ECC/parity errors — REG0 bits 4 or 5 set (XID74Reg0ECCParityErrorRule), or REG2 bits 0, 1, 2, or 6 set (RepeatedXID74Reg2HardwareIssueRule). Repeated occurrences on the same NVLink and GPU indicate a hardware fault.

Unexpected or unexplained errors — REG0 bits 1 or 20 set, or REG3 or REG4 anomalies: escalated immediately to CONTACT_SUPPORT. REG0 bits 27 or 29 set: escalated to CONTACT_SUPPORT after two or more occurrences within 24 hours (XID74Reg0Bit27Or29SetRule).

Rules that check REG0 bits 1, 20, 21, or 22 additionally verify that no other XID errors are active on the same GPU before firing, to avoid false positives during multi-error storms.

XID 31 and XID 13 Rules

Both XID 31 (GPU memory error) and XID 13 (graphics engine fault) have two rule variants:

Same GPU / same GPC+TPC — repeated occurrences on the same hardware unit suggest a hardware defect. The recommended action is RUN_DCGMEUD (run DCGM End-User Diagnostics), followed by field diagnostics if tests pass.

Different GPU / different GPC+TPC combinations — the same error appearing on different hardware units within a short window suggests a software or workload problem. The recommended action is NONE; investigate the application process identified in the error PID.

Both variants use a burst-window algorithm (3-minute window for XID 31, 15-second window for XID 13) to avoid counting rapid re-occurrences of the same event as separate bursts.

NIC Rules

Two NIC rules complement the NIC Health Monitor module:

enableRepeatedNICDriverErrorRule — correlates SysLogsNICDriverError events from the Syslog Health Monitor. It fires when the same kernel-log error pattern (for example, mlx5_tx_timeout_detected or netdev_watchdog) occurs three or more times on the same node within one hour, indicating that automatic driver recovery is failing.

enableRepeatedNICDegradationRule — correlates InfiniBandDegradationCheck and EthernetDegradationCheck events from the NIC Health Monitor. It fires when a non-fatal counter degradation occurs three or more times on the same NIC port within one hour, suggesting a persistent physical-layer problem.

See the NIC Health Monitor configuration for the corresponding check and detection configuration.