NIC Health Monitor Configuration

Overview

The NIC Health Monitor module detects InfiniBand and RoCE NIC failures in GPU clusters using three detection layers: link state polling, error counter thresholds, and kernel log monitoring (via the Syslog Health Monitor). This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the nic-health-monitor DaemonSet is deployed in the cluster.

1
global:
2
  nicHealthMonitor:
3
    enabled: false

Note: The NIC Health Monitor requires the metadata collector to be running on every GPU node and to have published /var/lib/nvsentinel/gpu_metadata.json. The monitor fails to start if that file is missing or does not contain GPU NUMA and NIC topology data. Exception: when nicInclusionRegexOverride is set, automatic NIC discovery and NUMA-based management exclusion are bypassed, so the metadata file is not required.

Resources

Defines CPU and memory resource requests and limits for the nic-health-monitor pod.

1
nic-health-monitor:
2
  resources:
3
    limits:
4
      cpu: 500m
5
      memory: 512Mi
6
    requests:
7
      cpu: 100m
8
      memory: 128Mi

Logging

Sets the verbosity level for nic-health-monitor logs.

1
nic-health-monitor:
2
  logLevel: info  # Options: debug, info, warn, error

Processing Strategy

Controls whether detected health events trigger downstream remediation or are stored for observability only.

1
nic-health-monitor:
2
  processingStrategy: EXECUTE_REMEDIATION

Use STORE_ONLY when rolling out new detection patterns (especially mlx5_napi_soft_lockup in the Syslog Health Monitor) to observe coverage before enabling remediation.

Enabled Checks

Selects which detection checks are active on each node.

1
nic-health-monitor:
2
  enabledChecks:
3
    - InfiniBandStateCheck
4
    - InfiniBandDegradationCheck
5
    - InfiniBandCharDeviceCheck
6
    - EthernetStateCheck
7
    - EthernetDegradationCheck

Check Types

InfiniBandStateCheck

Polls /sys/class/infiniband/*/ports/*/state and phys_state every statePollingInterval. Emits fatal events on port DOWN, device disappearance (after 3 consecutive missed enumerations), and uncabled port anomalies (card has fewer active ports than peer cards of the same role). Auto-excludes management NICs by NUMA locality and default-route detection; auto-filters SR-IOV Virtual Functions.

InfiniBandDegradationCheck

Polls InfiniBand hardware counters every 1 second. Emits fatal events when counters like link_downed, rnr_nak_retry_err, or excessive_buffer_overrun_errors increment. Rate-based threshold breaches emit events at the severity configured for each counter — symbol_error > 10/sec is non-fatal, symbol_error_fatal > 120/hour is fatal. Counter breach state is persisted across pod restarts.

InfiniBandCharDeviceCheck

Verifies that each InfiniBand device’s character-device nodes exist: one uverbs per device plus one umad and one issm per InfiniBand-mode port, read from /sys/class/infiniband_verbs and /sys/class/infiniband_mad (udev creates the /dev/infiniband/* nodes from these class entries, so a missing entry means workloads fail with errors like lstat /dev/infiniband/issm9: no such file or directory even while the port reads ACTIVE/LinkUp). The expectation is derived per device from its own discovered ports — never from an absolute device count — and issm is only expected on InfiniBand-mode ports (RoCE ports legitimately have none). A node missing for 3 consecutive polls emits a fatal REPLACE_VM event; the fault latches (persisted across pod restarts and reboots) and clears only when the node is positively observed again. An entirely absent class directory (e.g. ib_umad module not loaded) is treated as an uncertain observation and held rather than reported.

EthernetStateCheck

Same as InfiniBandStateCheck but for Ethernet/RoCE devices (reads link_layer = Ethernet). Monitors operstate in addition to state and phys_state.

EthernetDegradationCheck

Same as InfiniBandDegradationCheck but for Ethernet/RoCE devices. Tracks the same counter set where available on RoCE adapters; additionally monitors /sys/class/net/\{iface\}/statistics/carrier_changes.

State Polling Interval

Controls how frequently the state checks poll sysfs for link state changes.

1
nic-health-monitor:
2
  statePollingInterval: "1s"

Counter checks always run on a fixed 1-second cadence regardless of this setting — they need fresh data for velocity window calculations and cannot share the state check interval.

NIC Discovery Filtering

nicExclusionRegex

Comma-separated regex patterns for device names to exclude from discovery. Applied after vendor detection and VF filtering, before NUMA-based management NIC exclusion.

1
nic-health-monitor:
2
  nicExclusionRegex: "^veth.*,^docker.*,^br-.*,^lo$"

Use this to suppress virtual interfaces, bridge devices, or loopback from appearing as potential NICs.

nicInclusionRegexOverride

When non-empty, bypasses all automatic NIC discovery and monitors only devices whose names match these comma-separated regex patterns. All automatic filters — vendor detection, VF filtering, NUMA-based management exclusion, and nicExclusionRegex — are skipped. Pinned devices are reported unconditionally: the first-poll peer-evidence gate and the card homogeneity check do not apply.

1
nic-health-monitor:
2
  nicInclusionRegexOverride: ""  # Default: empty (auto-discovery enabled)

Warning: Changing nicInclusionRegexOverride or nicExclusionRegex resets the monitor’s persisted port and device state on the next pod start (healthy baselines are re-emitted). Counter snapshots and latched fatal counter breaches are preserved across scope changes.

When to use: Emergency override when automatic discovery misclassifies a NIC on an unusual platform. In normal deployments, leave empty.

Counter Detection

Enable/Disable

Enables or disables counter-based degradation monitoring globally.

1
nic-health-monitor:
2
  counterDetection:
3
    enabled: true

Counter Profiles

Counter names are validated against a hardcoded allowlist in the monitor. Operators can enable or disable individual counters, tune thresholds, and change velocity windows. Sysfs paths, severity, recommended action, and event descriptions are owned by code and are not configurable here.

1
nic-health-monitor:
2
  counterDetection:
3
    enabled: true
4
    counters:
5
      - name: link_downed
6
        enabled: true
7
        thresholdType: delta
8
        threshold: 0

Counter Profile Fields

Default Counter Configuration

1
nic-health-monitor:
2
  counterDetection:
3
    enabled: true
4
    counters:
5
      # Fatal counters — any increment triggers REPLACE_VM
6
      - name: link_downed                    # Port training failure
7
        enabled: true
8
        thresholdType: delta
9
        threshold: 0
10
11
      - name: excessive_buffer_overrun_errors  # Lossless contract violated
12
        enabled: true
13
        thresholdType: delta
14
        threshold: 0
15
16
      - name: local_link_integrity_errors    # Physical errors exceed hardware cap
17
        enabled: true
18
        thresholdType: delta
19
        threshold: 0
20
21
      - name: rnr_nak_retry_err              # Connection severed by retry exhaustion
22
        enabled: true
23
        thresholdType: delta
24
        threshold: 0
25
26
      # Fatal PHY threshold
27
      - name: symbol_error_fatal             # IBTA BER spec violation (> 120/hour = fatal)
28
        enabled: true
29
        thresholdType: velocity
30
        threshold: 120.0
31
        velocityUnit: hour
32
33
      # PHY degradation — non-fatal
34
      - name: symbol_error                   # PHY bit errors (> 10/sec = degradation)
35
        enabled: true
36
        thresholdType: velocity
37
        threshold: 10.0
38
        velocityUnit: second
39
40
      - name: link_error_recovery            # Link retraining / micro-flapping
41
        enabled: true
42
        thresholdType: velocity
43
        threshold: 5.0
44
        velocityUnit: minute
45
46
      # Transport — non-fatal
47
      - name: port_rcv_errors                # Malformed packets / CRC errors
48
        enabled: true
49
        thresholdType: velocity
50
        threshold: 10.0
51
        velocityUnit: second
52
53
      - name: local_ack_timeout_err          # ACK timeout (fabric path issue or remote crash)
54
        enabled: true
55
        thresholdType: velocity
56
        threshold: 1.0
57
        velocityUnit: second
58
59
      # Congestion — non-fatal
60
      - name: port_xmit_discards             # TX discards due to flow control breakdown
61
        enabled: true
62
        thresholdType: velocity
63
        threshold: 100.0
64
        velocityUnit: second
65
66
      # RoCE-specific — non-fatal
67
      - name: roce_slow_restart              # Victim flow oscillation (grey failure indicator)
68
        enabled: true
69
        thresholdType: velocity
70
        threshold: 10.0
71
        velocityUnit: second
72
73
      # Interface level — non-fatal
74
      - name: carrier_changes                # OS-visible link instability
75
        enabled: true
76
        thresholdType: delta
77
        threshold: 2

Allowed Counter Names

Counters outside this list are rejected at startup with a validation error.

Standard IB counters (counters/): excessive_buffer_overrun_errors, link_downed, link_error_recovery, local_link_integrity_errors, port_rcv_discards, port_rcv_errors, port_rcv_remote_physical_errors, port_rcv_switch_relay_errors, port_xmit_discards, port_xmit_wait, symbol_error, symbol_error_fatal

Extended counters (hw_counters/): implied_nak_seq_err, local_ack_timeout_err, out_of_sequence, packet_seq_err, req_transport_retries_exceeded, rnr_nak_retry_err, roce_slow_restart

Ethernet statistics (statistics/): carrier_changes, rx_crc_errors, rx_errors, rx_missed_errors, tx_carrier_errors, tx_errors

Counter Customization Examples

Stricter BER threshold

1
nic-health-monitor:
2
  counterDetection:
3
    counters:
4
      - name: symbol_error_fatal
5
        enabled: true
6
        thresholdType: velocity
7
        threshold: 60.0       # Stricter than the default 120/hour
8
        velocityUnit: hour

Add an allowed counter not enabled by default

1
nic-health-monitor:
2
  counterDetection:
3
    counters:
4
      - name: rx_missed_errors
5
        enabled: true
6
        thresholdType: velocity
7
        threshold: 10.0
8
        velocityUnit: second

Disable a noisy counter

1
nic-health-monitor:
2
  counterDetection:
3
    counters:
4
      - name: port_xmit_wait
5
        enabled: false

Note: Specifying a counter list in your values overrides the full default list. If you want to add or modify one counter, include the complete default list alongside your changes.

NIC Driver Error Detection (Syslog Health Monitor)

NIC driver and firmware errors are monitored by the Syslog Health Monitor DaemonSet, not the NIC Health Monitor. Enable the SysLogsNICDriverError check and configure pattern selection in the Syslog Health Monitor’s Helm values:

1
syslog-health-monitor:
2
  enabledChecks:
3
    - SysLogsXIDError
4
    - SysLogsSXIDError
5
    - SysLogsGPUFallenOff
6
    - SysLogsNICDriverError  # Add this to enable NIC driver log monitoring

NIC Driver Pattern Configuration

Pattern regexes, severity, and recommended actions are owned by code. The following Helm/YAML configuration selects which patterns are active and optionally overrides the per-pattern processing strategy:

1
syslog-health-monitor:
2
  nicDriverDetection:
3
    patterns:
4
      - name: cmd_exec_timeout        # Fatal: firmware hung, driver cannot issue commands
5
        enabled: true
6
        processingStrategy: EXECUTE_REMEDIATION
7
8
      - name: health_poll_failed      # Fatal: firmware heartbeat lost
9
        enabled: true
10
        processingStrategy: EXECUTE_REMEDIATION
11
12
      - name: unrecoverable_err       # Fatal: hardware admission of failure
13
        enabled: true
14
        processingStrategy: EXECUTE_REMEDIATION
15
16
      - name: mlx5_napi_soft_lockup   # Fatal: CPU wedged in NIC NAPI poll loop
17
        enabled: true
18
        processingStrategy: STORE_ONLY  # Start in shadow mode; graduate to EXECUTE_REMEDIATION after validation
19
20
      - name: netdev_watchdog         # Non-fatal: TX queue stall with auto-recovery
21
        enabled: true
22
23
      - name: mlx5_tx_timeout_detected  # Non-fatal: TX timeout (driver-reported)
24
        enabled: true
25
26
      - name: mlx5_rx_timeout_detected  # Non-fatal: RX timeout (driver-reported)
27
        enabled: true
28
29
      - name: port_module_high_temp   # Non-fatal: thermal warning
30
        enabled: true
31
32
      - name: pci_power_insufficient  # Non-fatal: PCIe power negotiation
33
        enabled: true
34
35
      - name: module_unplugged        # Non-fatal: SFP/transceiver removed
36
        enabled: true
37
38
      - name: access_reg_failed       # Non-fatal: monitoring tool conflict noise
39
        enabled: true

See Syslog Health Monitor Configuration for complete syslog-health-monitor options.

Health Events Analyzer Rules

Repeated non-fatal NIC events are escalated to CONTACT_SUPPORT by the Health Events Analyzer. Enable these rules in the health-events-analyzer Helm values:

1
health-events-analyzer:
2
  enableRepeatedNICDegradationRule: true   # 3 non-fatal counter events on same NIC+port in 1 hour
3
  enableRepeatedNICDriverErrorRule: true   # 3 non-fatal syslog events of same pattern on same node in 1 hour

Scheduling

Configure pod placement for the NIC Health Monitor DaemonSet.

1
nic-health-monitor:
2
  nodeSelector: \{\}
3
  tolerations: []
4
  affinity: \{\}

The DaemonSet runs on all nodes by default. Use nodeSelector or affinity to limit it to nodes with monitored NICs.