> This page is for NVSentinel, version v1.26.0. > For other versions, use one of these documentation indexes: > - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt > - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt > - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server. # Health Events Analyzer Configuration ## Overview The Health Events Analyzer evaluates MongoDB aggregation pipeline rules against incoming health events and emits derived events with recommended actions (for example, repeated XID 48 errors on the same GPU within 24 hours triggers a `CONTACT_SUPPORT` recommendation). This document covers all Helm configuration options for system administrators. ## Configuration Reference ### Module Enable/Disable Controls whether the health-events-analyzer module is deployed in the cluster. ```yaml global: healthEventsAnalyzer: enabled: false ``` ### Replica Count Number of health-events-analyzer pod replicas to run. ```yaml health-events-analyzer: replicaCount: 1 ``` ### Resources Defines CPU and memory resource requests and limits for the health-events-analyzer pod. ```yaml health-events-analyzer: resources: limits: cpu: "1" memory: "1Gi" requests: cpu: "1" memory: "1Gi" ``` ### Logging Sets the verbosity level for health-events-analyzer logs. ```yaml health-events-analyzer: logLevel: info # Options: debug, info, warn, error ``` ### Processing Strategy Controls whether derived events trigger downstream remediation actions or are recorded only. ```yaml health-events-analyzer: processingStrategy: EXECUTE_REMEDIATION # Options: EXECUTE_REMEDIATION, STORE_AND_ANALYSE, STORE_ONLY ``` **Options:** #### EXECUTE_REMEDIATION Normal operating mode. When a rule fires, downstream modules may update cluster state — applying node conditions, quarantining nodes, draining workloads, or triggering remediations. #### STORE_AND_ANALYSE Events are persisted and ingested by the Health Events Analyzer for rule evaluation. The individual event does not directly trigger quarantine or remediation, but HEA rules can still match on it and emit new `EXECUTE_REMEDIATION` synthetic events as a result — for example, a burst of deduplicated `STORE_AND_ANALYSE` events may collectively satisfy a repeated-failure rule. Use this when you want the raw event suppressed from immediate action but still want HEA correlation to fire. #### STORE_ONLY Observability-only mode. Derived events are persisted and exported but do not modify any cluster resources. Use this mode to shadow-test new or customised rules in production before enabling full remediation. ### Concurrent Event Processing The Health Events Analyzer partitions incoming events across a concurrent worker pool by node name. Events for distinct nodes are evaluated concurrently, while events for the same node are processed in strict chronological order. Checkpoints advance using a low-water mark tracker to guarantee at-least-once delivery without head-of-line blocking. ```yaml health-events-analyzer: workers: 1 # Number of concurrent workers (default: 1) maxInFlight: 1000 # Maximum uncheckpointed in-flight events before backpressure (default: 1000) ruleConcurrency: 1 # Maximum rule queries that run at the same time for one event (default: 1) ``` #### Scaling Workers by Datastore Event Rate Each event evaluation executes rule aggregation queries against the datastore, averaging approximately 17.5 ms of I/O round-trip latency. A single worker achieves a processing ceiling of approximately 57 events/second. Throughput scales linearly with the number of workers ($\approx \text\{Workers\} \times 57\text\{ events/s\}$). Use the following reference table to configure `workers` and `maxInFlight` based on cluster size and expected event rate: | Offered Event Rate in DB | Recommended `workers` | Recommended `maxInFlight` | Estimated Throughput Capacity | Recommended Cluster Scale | |---|---|---|---|---| | $< 50$ events/s | `1` (default) | `1000` | ~57 events/s | Up to ~500 nodes | | $50 - 200$ events/s | `4` | `1000` | ~220 events/s | 500 – 2,000 nodes | | $200 - 400$ events/s | `8` | `2000` | ~440 events/s | 2,000 – 4,000 nodes | | $400 - 800$ events/s | `16` | `2000` | ~860 events/s | 4,000 – 8,000 nodes | | $800 - 1,500$ events/s | `32` | `4000` | ~1,680 events/s | 8,000 – 15,000 nodes | | $> 1,500$ events/s | `64` | `8000` | ~3,500 events/s | 15,000+ nodes | `maxInFlight` bounds uncheckpointed in-flight events in memory. When in-flight events reach this limit, stream ingestion pauses until workers resolve earlier events. Increase `maxInFlight` proportionally for larger worker counts to absorb bursty event traffic without stalling ingestion. #### Concurrent Rule Evaluation The analyzer runs one datastore query for each enabled rule when it evaluates an event. By default, it runs these queries one at a time. `ruleConcurrency` sets the maximum number of rule queries that run at the same time for one event. `workers` and `ruleConcurrency` work at different levels: - `workers` evaluates events from different nodes at the same time. - `ruleConcurrency` evaluates the rules for one event at the same time. The analyzer keeps the events from one node in order. Thus, more workers cannot process a burst of events from one node faster. A higher `ruleConcurrency` can. The rule results do not change. The analyzer publishes the matched events in rule order, as it does when it runs the queries one at a time. The analyzer runs the same queries, and each query examines the same documents. But in measurements on MongoDB, the datastore used more CPU for each event when the queries ran at the same time. Use these guidelines: - Keep the default of `1` if the datastore CPU is near its limit. More CPU for each event can decrease throughput. - If the datastore has spare CPU, set `ruleConcurrency` to `4` or `8`. The analyzer can run up to `workers` × `ruleConcurrency` rule queries at the same time. ### Matched-entity metric `rule_matched_total` is labeled `{rule_name, node_name}` only. Rules that select on a GPU, GPC, TPC, or NIC therefore fire without saying which unit they selected. The optional counter below adds that identity; it is off by default because the extra labels raise cardinality. ```yaml health-events-analyzer: ruleMatchedEntityMetricEnabled: false ``` When enabled, a match on an entity-keyed rule increments: ```text rule_matched_entity_total{rule_name, node_name, entity_type, entity_value} ``` `entity_type` / `entity_value` come from the triggering event's impacted entities. Only PCI, GPU, GPC, TPC, SM, NVLINK, NIC, NICPort, and NVSwitch are exported, using the producer spelling. GPU UUID is never exported; PCI or GPU index is used instead. Node-scoped rules (for example `MultipleRemediations`) do not emit this series. `RepeatedXidError` does. ### Client Certificate Mount Path Path inside the container where TLS client certificates are mounted for authenticated MongoDB connections. Certificates are typically provisioned by cert-manager and mounted via a Kubernetes secret volume. ```yaml health-events-analyzer: clientCertMountPath: /etc/ssl/client-certs ``` ### Rule Enable/Disable Flags Each built-in rule can be independently enabled or disabled. All rules are enabled by default. Set a flag to `false` to suppress a rule without removing it from the configuration. ```yaml health-events-analyzer: enableMultipleRemediationsRule: true enableRepeatedXIDErrorOnSameGPURule: true enableRepeatedXID31OnSameGPURule: true enableRepeatedXID31OnDifferentGPURule: true enableRepeatedXID13OnSameGPCAndTPCRule: true enableRepeatedXID13OnDifferentGPCAndTPCRule: true enableXIDErrorSoloNoBurstRule: true enableXID74Reg0SoloNVLinkErrorRule: true enableXID74Reg0ECCParityErrorRule: true enableRepeatedXID74Reg0HardwareIssueRule: true enableXID74Reg0SignalIntegrityErrorRule: true enableXID74Reg0Bit27Or29SetRule: true enableRepeatedXID74Reg2HardwareIssueRule: true enableXID74Reg2Bit13SetRule: true enableRepeatedXID74Reg2Bit16Or19SetRule: true enableRepeatedXID74Reg2Bit17Or18SetRule: true enableXID74Reg3UnexpectedErrorRule: true enableXID74Reg3Bit18SetRule: true enableXID74Reg4HardwareIssueRule: true enableXID74Reg4ECCErrorRule: true enableRepeatedNICDriverErrorRule: true enableRepeatedNICDegradationRule: true ``` The following table summarises every flag, the XID or event type it covers, and the recommended action emitted when the rule fires. | Flag | Rule | Recommended Action | Description | |------|------|--------------------|-------------| | `enableMultipleRemediationsRule` | MultipleRemediations | `CONTACT_SUPPORT` | 5 or more remediations on the same node within 7 days | | `enableRepeatedXIDErrorOnSameGPURule` | RepeatedXIDErrorOnSameGPU | `CONTACT_SUPPORT` | Any XID except 31 and 45 five or more times within 24 hours on the same GPU (burst window 3 min, sticky XID window 3 h) | | `enableRepeatedXID31OnSameGPURule` | RepeatedXID31OnSameGPU | `RUN_DCGMEUD` | XID 31 two or more times on the same GPU within 24 hours | | `enableRepeatedXID31OnDifferentGPURule` | RepeatedXID31OnDifferentGPU | `NONE` | XID 31 on two or more different GPUs within 24 hours | | `enableRepeatedXID13OnSameGPCAndTPCRule` | RepeatedXID13OnSameGPCAndTPC | `RUN_DCGMEUD` | XID 13 two or more times on the same GPC and TPC within 24 hours | | `enableRepeatedXID13OnDifferentGPCAndTPCRule` | RepeatedXID13OnDifferentGPCAndTPC | `NONE` | XID 13 on two or more different GPC/TPC combinations within 24 hours | | `enableXIDErrorSoloNoBurstRule` | XIDErrorSoloNoBurst | `NONE` | XID 13 or 31 appeared exactly once in the most recent burst within 24 hours | | `enableXID74Reg0SoloNVLinkErrorRule` | XID74Reg0SoloNVLinkError | `CONTACT_SUPPORT` | XID 74 with REG0 bit 1 or 20 set, no other active errors on the same GPU | | `enableXID74Reg0ECCParityErrorRule` | XID74Reg0ECCParityError | `CONTACT_SUPPORT` | XID 74 with REG0 bit 4 or 5 set, two or more times on the same NVLink and GPU within 24 hours | | `enableRepeatedXID74Reg0HardwareIssueRule` | RepeatedXID74Reg0HardwareIssue | `CONTACT_SUPPORT` | XID 74 with REG0 bits 8, 9, 12, 16, 17, 24, or 28 set, two or more times on same GPU within 24 hours | | `enableXID74Reg0SignalIntegrityErrorRule` | XID74Reg0SignalIntegrityError | `CONTACT_SUPPORT` | XID 74 with REG0 bit 21 or 22 set, no other active errors on the same GPU | | `enableXID74Reg0Bit27Or29SetRule` | XID74Reg0Bit27Or29Set | `CONTACT_SUPPORT` | XID 74 with REG0 bit 27 or 29 set, two or more times on same GPU within 24 hours | | `enableRepeatedXID74Reg2HardwareIssueRule` | RepeatedXID74Reg2HardwareIssue | `CONTACT_SUPPORT` | XID 74 with REG2 bits 0, 1, 2, or 6 set, two or more times on same NVLink and GPU within 24 hours | | `enableXID74Reg2Bit13SetRule` | XID74Reg2Bit13Set | `CONTACT_SUPPORT` | XID 74 with REG2 bit 13 set | | `enableRepeatedXID74Reg2Bit16Or19SetRule` | RepeatedXID74Reg2Bit16Or19Set | `CONTACT_SUPPORT` | XID 74 with REG2 bit 16 or 19 set, two or more times within 24 hours | | `enableRepeatedXID74Reg2Bit17Or18SetRule` | RepeatedXID74Reg2Bit17Or18Set | `CONTACT_SUPPORT` | XID 74 with REG2 bit 17 or 18 set, two or more times within 24 hours | | `enableXID74Reg3UnexpectedErrorRule` | XID74Reg3UnexpectedError | `CONTACT_SUPPORT` | XID 74 with an unexpected REG3 value | | `enableXID74Reg3Bit18SetRule` | XID74Reg3Bit18Set | `CONTACT_SUPPORT` | XID 74 with REG3 bit 18 set | | `enableXID74Reg4HardwareIssueRule` | XID74Reg4HardwareIssue | `CONTACT_SUPPORT` | XID 74 with a hardware-indicating REG4 value | | `enableXID74Reg4ECCErrorRule` | XID74Reg4ECCError | `CONTACT_SUPPORT` | XID 74 with an ECC-indicating REG4 value | | `enableRepeatedNICDriverErrorRule` | RepeatedNICDriverError | `CONTACT_SUPPORT` | Non-fatal NIC driver kernel-log pattern (e.g. TX/RX timeout, NAPI lockup) three or more times on the same node within 1 hour | | `enableRepeatedNICDegradationRule` | RepeatedNICDegradation | `CONTACT_SUPPORT` | Non-fatal NIC counter degradation three or more times on the same NIC port within 1 hour | ## Rule Configuration ### Rule Structure Rules are defined as TOML in the `config:` block of `values.yaml`. Each rule entry has the following fields: ```toml [[rules]] name = "RuleName" description = "Human-readable description" recommended_action = "CONTACT_SUPPORT" # or RUN_DCGMEUD, NONE, etc. evaluate_rule = true # Helm template expression; maps to the enable flag when = "size(event.errorCode) > 0 && event.errorCode[0] == '74'" # Optional CEL; see below stage = [ '{ "$match": { ... } }', # MongoDB aggregation pipeline stages as JSON strings '{ "$count": "count" }', '{ "$match": { "count": { "$gte": N } } }' ] ``` The full default ruleset — including all aggregation pipeline stage definitions — is in the chart's `values.yaml` at `distros/kubernetes/nvsentinel/charts/health-events-analyzer/values.yaml`. Refer to that file when writing or reviewing custom rules. ### Skip Rules That Cannot Match Each enabled rule sends one query to the datastore for each event. Most rules apply to only one XID or one check. For all other events the query cannot match, but the datastore still reads the events of the node to answer it. The optional `when` field prevents this. It is a CEL expression over the incoming event. The analyzer runs the stages of the rule only when `when` is true. When it is false, the analyzer sends no query and the rule does not match. ```toml [[rules]] name = "XID74Reg0ECCParityError" evaluate_rule = true when = "size(event.errorCode) > 0 && event.errorCode[0] == '74'" stage = [ ... ] ``` The expression can read these fields of the incoming event: `agent`, `checkName`, `componentClass`, `errorCode`, `isFatal`, `isHealthy`, `recommendedAction`, `nodeName`, `metadata`, and `message`. `errorCode` is a list and `metadata` is a map. The platform connector overrides and the event exporter filter use the same fields. **The expression must be true for every event that the stages can match.** If it is false for such an event, the analyzer misses that match. To write it safely, copy the stage of the rule that compares `this.` fields to constants. For example, the stage `{"$match": {"$expr": {"$eq": ["this.healthevent.errorcode.0", "74"]}}}` becomes the `when` in the example above. The analyzer resolves `this.healthevent.errorcode.0` to null when the event has no error code, so check `size(event.errorCode) > 0` before you read `event.errorCode[0]`. The analyzer handles `when` as follows: - If `when` is empty or not set, the analyzer runs the rule for every event. - If `when` does not compile or does not return a boolean, the analyzer does not start. The error names the rule. - If `when` fails for an event, the analyzer runs the query of the rule and logs a warning. A failure costs a query, but it does not hide a match. ### MultipleRemediations Rule The `MultipleRemediations` rule fires when five or more remediations have been performed on the same node within the preceding 7 days. Unlike other rules, **it applies a node condition that NVSentinel does not automatically clear**, because the rule does not emit healthy events. After the underlying hardware issue is resolved, remove the condition manually: ```bash kubectl get node {NODE_NAME} -o json \ | jq '.status.conditions |= map(select(.type != "MultipleRemediations"))' \ | kubectl replace -f - --subresource=status ``` Replace `MultipleRemediations` with the actual condition type name if a custom rule applied a different condition. ### XID 74 Register Rules XID 74 (NVLink error) carries register fields (`REG0`–`REG6`) that encode the specific failure mode. The XID 74 rules inspect these register bit patterns to distinguish three categories of failure: **Signal integrity issues** — REG0 bits 21 or 22 set (`XID74Reg0SignalIntegrityErrorRule`). Suggests marginal signal levels; check link mechanical connections and run field diagnostics if the issue persists. **ECC/parity errors** — REG0 bits 4 or 5 set (`XID74Reg0ECCParityErrorRule`), or REG2 bits 0, 1, 2, or 6 set (`RepeatedXID74Reg2HardwareIssueRule`). Repeated occurrences on the same NVLink and GPU indicate a hardware fault. **Unexpected or unexplained errors** — REG0 bits 1 or 20 set, or REG3 or REG4 anomalies: escalated immediately to `CONTACT_SUPPORT`. REG0 bits 27 or 29 set: escalated to `CONTACT_SUPPORT` after two or more occurrences within 24 hours (`XID74Reg0Bit27Or29SetRule`). Rules that check REG0 bits 1, 20, 21, or 22 additionally verify that no other XID errors are active on the same GPU before firing, to avoid false positives during multi-error storms. ### XID 31 and XID 13 Rules Both XID 31 (GPU memory error) and XID 13 (graphics engine fault) have two rule variants: **Same GPU / same GPC+TPC** — repeated occurrences on the same hardware unit suggest a hardware defect. The recommended action is `RUN_DCGMEUD` (run DCGM End-User Diagnostics), followed by field diagnostics if tests pass. **Different GPU / different GPC+TPC combinations** — the same error appearing on different hardware units within a short window suggests a software or workload problem. The recommended action is `NONE`; investigate the application process identified in the error PID. Both variants use a burst-window algorithm (3-minute window for XID 31, 15-second window for XID 13) to avoid counting rapid re-occurrences of the same event as separate bursts. ### NIC Rules Two NIC rules complement the NIC Health Monitor module: `enableRepeatedNICDriverErrorRule` — correlates `SysLogsNICDriverError` events from the Syslog Health Monitor. It fires when the same kernel-log error pattern (for example, `mlx5_tx_timeout_detected` or `netdev_watchdog`) occurs three or more times on the same node within one hour, indicating that automatic driver recovery is failing. `enableRepeatedNICDegradationRule` — correlates `InfiniBandDegradationCheck` and `EthernetDegradationCheck` events from the NIC Health Monitor. It fires when a non-fatal counter degradation occurs three or more times on the same NIC port within one hour, suggesting a persistent physical-layer problem. See the [NIC Health Monitor configuration](/nvsentinel/v1.26.0/configuration/nic-health-monitor) for the corresponding check and detection configuration.