NIC Health Monitor Configuration

Overview

The NIC Health Monitor module detects InfiniBand and RoCE NIC failures in GPU clusters using three detection layers: link state polling, error counter thresholds, and kernel log monitoring (via the Syslog Health Monitor). This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the nic-health-monitor DaemonSet is deployed in the cluster.

global:
  nicHealthMonitor:
    enabled: false

Note: The NIC Health Monitor requires the metadata collector to be running on every GPU node and to have published /var/lib/nvsentinel/gpu_metadata.json. The monitor fails to start if that file is missing or does not contain GPU NUMA and NIC topology data. Exception: when nicInclusionRegexOverride is set, automatic NIC discovery and NUMA-based management exclusion are bypassed, so the metadata file is not required.

Resources

Defines CPU and memory resource requests and limits for the nic-health-monitor pod.

nic-health-monitor:
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi

Logging

Sets the verbosity level for nic-health-monitor logs.

nic-health-monitor:
  logLevel: info  # Options: debug, info, warn, error

Processing Strategy

Controls whether detected health events trigger downstream remediation or are stored for observability only.

nic-health-monitor:
  processingStrategy: EXECUTE_REMEDIATION

Use STORE_ONLY when rolling out new detection patterns (especially mlx5_napi_soft_lockup in the Syslog Health Monitor) to observe coverage before enabling remediation.

Enabled Checks

Selects which detection checks are active on each node.

nic-health-monitor:
  enabledChecks:
    - InfiniBandStateCheck
    - InfiniBandDegradationCheck
    - InfiniBandCharDeviceCheck
    - EthernetStateCheck
    - EthernetDegradationCheck

Check Types

InfiniBandStateCheck

Polls /sys/class/infiniband/*/ports/*/state and phys_state every statePollingInterval. Emits fatal events on port DOWN, device disappearance (after 3 consecutive missed enumerations), and uncabled port anomalies (card has fewer active ports than peer cards of the same role). Auto-excludes management NICs by NUMA locality and default-route detection; auto-filters SR-IOV Virtual Functions.

InfiniBandDegradationCheck

Polls InfiniBand hardware counters every 1 second. Emits fatal events when counters like link_downed, rnr_nak_retry_err, or excessive_buffer_overrun_errors increment. Rate-based threshold breaches emit events at the severity configured for each counter — symbol_error > 10/sec is non-fatal, symbol_error_fatal > 120/hour is fatal. Counter breach state is persisted across pod restarts.

InfiniBandCharDeviceCheck

Verifies that each InfiniBand device’s character-device nodes exist: one uverbs per device and one umad per InfiniBand-mode port (plus, only when charDeviceCheck.issm is always, one issm per InfiniBand-mode port), read from /sys/class/infiniband_verbs and /sys/class/infiniband_mad (udev creates the /dev/infiniband/* nodes from these class entries, so a missing entry means workloads fail with errors like lstat /dev/infiniband/issm9: no such file or directory even while the port reads ACTIVE/LinkUp). The expectation is derived per device from its own discovered ports — never from an absolute device count — and umad/issm are only expected on InfiniBand-mode ports (RoCE ports legitimately have none). A node missing for 3 consecutive polls emits a fatal REPLACE_VM event; the fault latches (persisted across pod restarts and reboots) and clears only when the node is positively observed again. An entirely absent class directory (e.g. ib_umad module not loaded) is treated as an uncertain observation and held rather than reported.

The per-port issm expectation is opt-in via charDeviceCheck.issm and off by default, because whether issm nodes are created is architecture- and provider-dependent (some platforms, e.g. GB300, legitimately create none). umad and uverbs are always required.

EthernetStateCheck

Same as InfiniBandStateCheck but for Ethernet/RoCE devices (reads link_layer = Ethernet). Monitors operstate in addition to state and phys_state.

EthernetDegradationCheck

Same as InfiniBandDegradationCheck but for Ethernet/RoCE devices. Tracks the same counter set where available on RoCE adapters; additionally monitors /sys/class/net/{iface}/statistics/carrier_changes.

State Polling Interval

Controls how frequently the state checks poll sysfs for link state changes.

nic-health-monitor:
  statePollingInterval: "1s"

Counter checks always run on a fixed 1-second cadence regardless of this setting — they need fresh data for velocity window calculations and cannot share the state check interval.

Port State Hold-Down

A port that drops for a few seconds and comes back is reported at once by default, and a DOWN port is fatal, so a short fabric blip can cordon a node. portStateHoldDown makes a port stay unhealthy for that long before InfiniBandStateCheck or EthernetStateCheck reports it.

nic-health-monitor:
  portStateHoldDown: "15s"

• Off by default. "0s" reports on the first unhealthy poll.
• Port and card events alike. While a port is held, its card’s homogeneity check sees it as up too, so the card event waits as well.
• Blips are counted, not reported. A port that recovers before it is reported publishes no event and increments nic_health_monitor_port_state_blips_total{node,check,device,port}.
• Recovery is immediate. The healthy event is never delayed.
• A restart restarts the hold. The hold-down is not persisted.

The hold-down applies only to port state. The link_downed counter (InfiniBandDegradationCheck) is fatal on any increase and stays latched until the counter resets or the host reboots, so a blip that bumps it still produces a fatal event. If you see blips there, you can disable that counter (see Disable a noisy counter).

Reporting Ports Suppressed on the First Poll

On the first poll, an unhealthy port is reported only with peer evidence that it should be up: its card has fewer active ports than the decisive mode of its role group. Otherwise the port is suppressed, because it may be uncabled or disabled by design. That happens when:

• the card is at or above its group’s mode, like the uncabled second port of a dual-port card. This is the most common case.
• the card shares a PCI card with a management NIC, so it is left out of peer comparison.
• the card is alone in its role group.
• the group’s vote is tied.
• the group’s mode is 0, because most of its cards are down.

A suppressed port that never comes up is otherwise never reported. A node with half its compute links down from boot gives a tie and stays silent.

suppressedPortReportAfter reports such a port once it has stayed unhealthy that long, but only on a compute card the peers gave no verdict on (a singleton group, a tie, or a mode of 0):

nic-health-monitor:
  suppressedPortReportAfter: "10m"

• Off by default. "0s" keeps suppressing the port. Leave it off if some compute ports are down by design, or exclude those devices with nicExclusionRegex.
• Compute cards only. A card qualifies when at least one monitored device on it is classified as compute and has a port on the same link layer. Then every port on the card qualifies, also the ports of a device classified as storage. For example, a GB300 SMI device has no topology matrix entry, but it shares its card with a compute device. A port on a card with no compute device is never reported. Storage and frontend ports can be down by design with no peer to compare against, like the unused frontend aux port on OCI BM.GPU.H100.8.
• Healthy cards are never reported. A port on a card at or above its group’s mode, or on a card that shares a PCI card with a management NIC, is not reported however long it stays down. If a later poll finds the card at or above the mode, for example once its other ports finish training, the port is dropped.
• Unreadable devices. While a device is unreadable the peer comparison does not run, so a port on a card with an active port waits for a complete poll: its card may be at its group’s mode. A port on a card with no active port is reported, since no peer vote could find that card healthy, unless one of the card’s own functions is unreadable. If the monitor cannot find the PCI card of an unreadable device, every card waits. A card with an unreadable function also keeps its ports even when no compute device was read on it, because the unreadable function can be its compute device.
• Severity. An InfiniBand port still DOWN is fatal with REPLACE_VM, Polling included, the same result the homogeneity check gives a card below its mode. INIT and ARMED stay non-fatal, so a subnet manager outage at boot does not mark every node fatal. A RoCE port is reported only when DOWN, which is fatal.
• Ports still training at boot are not reported. A port that comes up before the delay is dropped.
• Survives restarts. The suppression time is kept in the state file, so a pod restart does not restart the wait. This includes a restart while the reboot baseline is still owed because a device is unreadable. A reboot or discovery-scope change resets it, and the next first poll decides again.
• Recorded while off. The monitor saves the suppression time even when the option is off. If you turn the option on later, a port that has stayed down since its suppression is reported at the first poll after the delay, without a reboot.
• Upgrades. Versions before this option did not save a suppression time. After an upgrade, a port suppressed by the older version is picked up only after the next reboot or discovery-scope change.

Poll Stall Detection

Reports when the monitor itself cannot observe NIC state. If a state or counter poll has been in flight for pollStallDeadline, the monitor publishes an unhealthy, non-fatal NICPollStallCheck event (recommended action NONE). Once no poll is stalled it publishes the healthy event. No check runs while a publish is waiting for the deployment platform connector, so a connector outage is not reported as a NIC stall. For the same condition as a metric, use time() - nic_health_monitor_poll_cycle_last_completed_timestamp_seconds. A stall event that fails to publish is retried until delivered, even if the poll has completed in the meantime, so the episode is still recorded. The first poll to complete after the monitor starts also publishes a healthy event, which closes a stall left open when the liveness probe restarted the container.

A stalled poll usually means the host is stalling the sysfs reads, for example through lock contention in the network stack. Restarting the container does not fix that, and without this check the only trace is the restart count.

nic-health-monitor:
  pollStallDeadline: "10s"
  pollStallStoreOnly: true

Keep pollStallDeadline well below the liveness restart window, which is about 60 to 90 seconds into a stall, or the stall is never reported. "0s" disables detection. pollStallStoreOnly emits the events as STORE_ONLY whatever processingStrategy is, because the cause is the host rather than a NIC.

Character-Device Check Tuning

charDeviceCheck.issm

Controls whether InfiniBandCharDeviceCheck expects the per-port issm character device. umad and uverbs are always required and are not configurable; only issm is, because whether issm nodes are created is architecture- and provider-dependent.

nic-health-monitor:
  charDeviceCheck:
    issm: never  # never (default) | always

Why this exists: on newer hardware such as GB300 the platform does not create issm device nodes, so the previous unconditional check falsely marked every such GPU node REPLACE_VM. issm presence cannot be inferred from any port attribute — on a fabric run by an external subnet manager every compute port reads SM_DISABLED (cap_mask bit 0x400) whether or not issm exists — so the check is an explicit opt-in rather than an inferred default.

Changing this value forces a one-time baseline reconciliation on the next pod start (when there are outstanding issm conditions to clear), so stale issm conditions are cleared immediately rather than being held until the next reboot.

NIC Discovery Filtering

nicExclusionRegex

Comma-separated regex patterns for device names to exclude from discovery. Applied after vendor detection and VF filtering, before NUMA-based management NIC exclusion.

nic-health-monitor:
  nicExclusionRegex: "^veth.*,^docker.*,^br-.*,^lo$"

Use this to suppress virtual interfaces, bridge devices, or loopback from appearing as potential NICs.

nicInclusionRegexOverride

When non-empty, bypasses all automatic NIC discovery and monitors only devices whose names match these comma-separated regex patterns. All automatic filters — vendor detection, VF filtering, NUMA-based management exclusion, and nicExclusionRegex — are skipped. Pinned devices are reported unconditionally: the first-poll peer-evidence gate and the card homogeneity check do not apply.

nic-health-monitor:
  nicInclusionRegexOverride: ""  # Default: empty (auto-discovery enabled)

Warning: Changing nicInclusionRegexOverride or nicExclusionRegex resets the monitor’s persisted port and device state on the next pod start (healthy baselines are re-emitted). Counter snapshots and latched fatal counter breaches are preserved across scope changes.

When to use: Emergency override when automatic discovery misclassifies a NIC on an unusual platform. In normal deployments, leave empty.

Sysfs Paths

Where the monitor reads network and InfiniBand device state. The DaemonSet mounts the host’s /sys under these paths, so the values must match the host mounts in the pod spec.

nic-health-monitor:
  sysClassNetPath: "/nvsentinel/sys/class/net"
  sysClassInfinibandPath: "/nvsentinel/sys/class/infiniband"

Change these only when you alter where the chart mounts host /sys. A path that does not exist leaves the monitor with no devices to discover.

Counter Detection

Enable/Disable

Enables or disables counter-based degradation monitoring globally.

nic-health-monitor:
  counterDetection:
    enabled: true

Counter Profiles

Counter names are validated against a hardcoded allowlist in the monitor. Operators can enable or disable individual counters, tune thresholds, and change velocity windows. Sysfs paths, severity, recommended action, and event descriptions are owned by code and are not configurable here.

nic-health-monitor:
  counterDetection:
    enabled: true
    counters:
      - name: link_downed
        enabled: true
        thresholdType: delta
        threshold: 0

Counter Profile Fields

Default Counter Configuration

nic-health-monitor:
  counterDetection:
    enabled: true
    counters:
      # Fatal counters — any increment triggers REPLACE_VM
      - name: link_downed                    # Port training failure
        enabled: true
        thresholdType: delta
        threshold: 0
      - name: excessive_buffer_overrun_errors  # Lossless contract violated
        enabled: true
        thresholdType: delta
        threshold: 0
      - name: local_link_integrity_errors    # Physical errors exceed hardware cap
        enabled: true
        thresholdType: delta
        threshold: 0
      - name: rnr_nak_retry_err              # Connection severed by retry exhaustion
        enabled: true
        thresholdType: delta
        threshold: 0
      # Fatal PHY threshold
      - name: symbol_error_fatal             # IBTA BER spec violation (> 120/hour = fatal)
        enabled: true
        thresholdType: velocity
        threshold: 120.0
        velocityUnit: hour
      # PHY degradation — non-fatal
      - name: symbol_error                   # PHY bit errors (> 10/sec = degradation)
        enabled: true
        thresholdType: velocity
        threshold: 10.0
        velocityUnit: second
      - name: link_error_recovery            # Link retraining / micro-flapping
        enabled: true
        thresholdType: velocity
        threshold: 5.0
        velocityUnit: minute
      # Transport — non-fatal
      - name: port_rcv_errors                # Malformed packets / CRC errors
        enabled: true
        thresholdType: velocity
        threshold: 10.0
        velocityUnit: second
      - name: local_ack_timeout_err          # ACK timeout (fabric path issue or remote crash)
        enabled: true
        thresholdType: velocity
        threshold: 1.0
        velocityUnit: second
      # Congestion — non-fatal
      - name: port_xmit_discards             # TX discards due to flow control breakdown
        enabled: true
        thresholdType: velocity
        threshold: 100.0
        velocityUnit: second
      # RoCE-specific — non-fatal
      - name: roce_slow_restart              # Victim flow oscillation (grey failure indicator)
        enabled: true
        thresholdType: velocity
        threshold: 10.0
        velocityUnit: second
      # Interface level — non-fatal
      - name: carrier_changes                # OS-visible link instability
        enabled: true
        thresholdType: delta
        threshold: 2

Allowed Counter Names

Counters outside this list are rejected at startup with a validation error.

Standard IB counters (counters/): excessive_buffer_overrun_errors, link_downed, link_error_recovery, local_link_integrity_errors, port_rcv_discards, port_rcv_errors, port_rcv_remote_physical_errors, port_rcv_switch_relay_errors, port_xmit_discards, port_xmit_wait, symbol_error, symbol_error_fatal

Extended counters (hw_counters/): implied_nak_seq_err, local_ack_timeout_err, out_of_sequence, packet_seq_err, req_transport_retries_exceeded, rnr_nak_retry_err, roce_slow_restart

Ethernet statistics (statistics/): carrier_changes, rx_crc_errors, rx_errors, rx_missed_errors, tx_carrier_errors, tx_errors

Counter Customization Examples

Stricter BER threshold

nic-health-monitor:
  counterDetection:
    counters:
      - name: symbol_error_fatal
        enabled: true
        thresholdType: velocity
        threshold: 60.0       # Stricter than the default 120/hour
        velocityUnit: hour

Add an allowed counter not enabled by default

nic-health-monitor:
  counterDetection:
    counters:
      - name: rx_missed_errors
        enabled: true
        thresholdType: velocity
        threshold: 10.0
        velocityUnit: second

Disable a noisy counter

nic-health-monitor:
  counterDetection:
    counters:
      - name: port_xmit_wait
        enabled: false

Note: Specifying a counter list in your values overrides the full default list. If you want to add or modify one counter, include the complete default list alongside your changes.

NIC Driver Error Detection (Syslog Health Monitor)

NIC driver and firmware errors are monitored by the Syslog Health Monitor DaemonSet, not the NIC Health Monitor. Enable the SysLogsNICDriverError check and configure pattern selection in the Syslog Health Monitor’s Helm values:

syslog-health-monitor:
  enabledChecks:
    - SysLogsXIDError
    - SysLogsSXIDError
    - SysLogsGPUFallenOff
    - SysLogsNICDriverError  # Add this to enable NIC driver log monitoring

NIC Driver Pattern Configuration

Pattern regexes, severity, and recommended actions are owned by code. The following Helm/YAML configuration selects which patterns are active and optionally overrides the per-pattern processing strategy:

syslog-health-monitor:
  nicDriverDetection:
    patterns:
      - name: cmd_exec_timeout        # Fatal: firmware hung, driver cannot issue commands
        enabled: true
        processingStrategy: EXECUTE_REMEDIATION
      - name: health_poll_failed      # Fatal: firmware heartbeat lost
        enabled: true
        processingStrategy: EXECUTE_REMEDIATION
      - name: unrecoverable_err       # Fatal: hardware admission of failure
        enabled: true
        processingStrategy: EXECUTE_REMEDIATION
      - name: mlx5_napi_soft_lockup   # Fatal: CPU wedged in NIC NAPI poll loop
        enabled: true
        processingStrategy: STORE_ONLY  # Start in shadow mode; graduate to EXECUTE_REMEDIATION after validation
      - name: netdev_watchdog         # Non-fatal: TX queue stall with auto-recovery
        enabled: true
      - name: mlx5_tx_timeout_detected  # Non-fatal: TX timeout (driver-reported)
        enabled: true
      - name: mlx5_rx_timeout_detected  # Non-fatal: RX timeout (driver-reported)
        enabled: true
      - name: port_module_high_temp   # Non-fatal: thermal warning
        enabled: true
      - name: pci_power_insufficient  # Non-fatal: PCIe power negotiation
        enabled: true
      - name: module_unplugged        # Non-fatal: SFP/transceiver removed
        enabled: true
      - name: access_reg_failed       # Non-fatal: monitoring tool conflict noise
        enabled: true

See Syslog Health Monitor Configuration for complete syslog-health-monitor options.

Health Events Analyzer Rules

Repeated non-fatal NIC events are escalated to CONTACT_SUPPORT by the Health Events Analyzer. Enable these rules in the health-events-analyzer Helm values:

health-events-analyzer:
  enableRepeatedNICDegradationRule: true   # 3 non-fatal counter events on same NIC+port in 1 hour
  enableRepeatedNICDriverErrorRule: true   # 3 non-fatal syslog events of same pattern on same node in 1 hour

Scheduling

Configure pod placement for the NIC Health Monitor DaemonSet.

nic-health-monitor:
  nodeSelector: {}
  tolerations: []
  affinity: {}

The DaemonSet runs on all nodes by default. Use nodeSelector or affinity to limit it to nodes with monitored NICs.