> This page is for NVSentinel, version v1.30.0. > For other versions, use one of these documentation indexes: > - Latest · v1.31.0 (default): https://docs.nvidia.com/nvsentinel/v1.31.0/llms.txt > - v1.30.0: https://docs.nvidia.com/nvsentinel/v1.30.0/llms.txt > - v1.29.0: https://docs.nvidia.com/nvsentinel/v1.29.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server. # NIC Health Monitor Configuration ## Overview The NIC Health Monitor module detects InfiniBand and RoCE NIC failures in GPU clusters using three detection layers: link state polling, error counter thresholds, and kernel log monitoring (via the Syslog Health Monitor). This document covers all Helm configuration options for system administrators. ## Configuration Reference ### Module Enable/Disable Controls whether the nic-health-monitor DaemonSet is deployed in the cluster. ```yaml global: nicHealthMonitor: enabled: false ``` > **Note**: The NIC Health Monitor requires the metadata collector to be running on every GPU node and to have published `/var/lib/nvsentinel/gpu_metadata.json`. The monitor fails to start if that file is missing or does not contain GPU NUMA and NIC topology data. Exception: when `nicInclusionRegexOverride` is set, automatic NIC discovery and NUMA-based management exclusion are bypassed, so the metadata file is not required. ### Resources Defines CPU and memory resource requests and limits for the nic-health-monitor pod. ```yaml nic-health-monitor: resources: limits: cpu: 500m memory: 512Mi requests: cpu: 100m memory: 128Mi ``` ### Logging Sets the verbosity level for nic-health-monitor logs. ```yaml nic-health-monitor: logLevel: info # Options: debug, info, warn, error ``` ### Processing Strategy Controls whether detected health events trigger downstream remediation or are stored for observability only. ```yaml nic-health-monitor: processingStrategy: EXECUTE_REMEDIATION ``` | Value | Behavior | |-------|----------| | `EXECUTE_REMEDIATION` | Normal operation; fault-quarantine and node-drainer act on events | | `STORE_ONLY` | Events are persisted and exported but do not modify cluster resources | Use `STORE_ONLY` when rolling out new detection patterns (especially `mlx5_napi_soft_lockup` in the Syslog Health Monitor) to observe coverage before enabling remediation. ## Enabled Checks Selects which detection checks are active on each node. ```yaml nic-health-monitor: enabledChecks: - InfiniBandStateCheck - InfiniBandDegradationCheck - InfiniBandCharDeviceCheck - EthernetStateCheck - EthernetDegradationCheck ``` ### Check Types #### InfiniBandStateCheck Polls `/sys/class/infiniband/*/ports/*/state` and `phys_state` every `statePollingInterval`. Emits fatal events on port DOWN, device disappearance (after 3 consecutive missed enumerations), and uncabled port anomalies (card has fewer active ports than peer cards of the same role). Auto-excludes management NICs by NUMA locality and default-route detection; auto-filters SR-IOV Virtual Functions. #### InfiniBandDegradationCheck Polls InfiniBand hardware counters every 1 second. Emits fatal events when counters like `link_downed`, `rnr_nak_retry_err`, or `excessive_buffer_overrun_errors` increment. Rate-based threshold breaches emit events at the severity configured for each counter — `symbol_error > 10/sec` is non-fatal, `symbol_error_fatal > 120/hour` is fatal. Counter breach state is persisted across pod restarts. #### InfiniBandCharDeviceCheck Verifies that each InfiniBand device's character-device nodes exist: one `uverbs` per device and one `umad` per InfiniBand-mode port (plus, only when [`charDeviceCheck.issm`](#chardevicecheckissm) is `always`, one `issm` per InfiniBand-mode port), read from `/sys/class/infiniband_verbs` and `/sys/class/infiniband_mad` (udev creates the `/dev/infiniband/*` nodes from these class entries, so a missing entry means workloads fail with errors like `lstat /dev/infiniband/issm9: no such file or directory` even while the port reads ACTIVE/LinkUp). The expectation is derived per device from its own discovered ports — never from an absolute device count — and `umad`/`issm` are only expected on InfiniBand-mode ports (RoCE ports legitimately have none). A node missing for 3 consecutive polls emits a fatal `REPLACE_VM` event; the fault latches (persisted across pod restarts and reboots) and clears only when the node is positively observed again. An entirely absent class directory (e.g. `ib_umad` module not loaded) is treated as an uncertain observation and held rather than reported. The per-port `issm` expectation is opt-in via [`charDeviceCheck.issm`](#chardevicecheckissm) and **off by default**, because whether `issm` nodes are created is architecture- and provider-dependent (some platforms, e.g. GB300, legitimately create none). `umad` and `uverbs` are always required. #### EthernetStateCheck Same as `InfiniBandStateCheck` but for Ethernet/RoCE devices (reads `link_layer = Ethernet`). Monitors `operstate` in addition to `state` and `phys_state`. #### EthernetDegradationCheck Same as `InfiniBandDegradationCheck` but for Ethernet/RoCE devices. Tracks the same counter set where available on RoCE adapters; additionally monitors `/sys/class/net/{iface}/statistics/carrier_changes`. ## State Polling Interval Controls how frequently the state checks poll sysfs for link state changes. ```yaml nic-health-monitor: statePollingInterval: "1s" ``` Counter checks always run on a fixed 1-second cadence regardless of this setting — they need fresh data for velocity window calculations and cannot share the state check interval. ## Port State Hold-Down A port that drops for a few seconds and comes back is reported at once by default, and a `DOWN` port is fatal, so a short fabric blip can cordon a node. `portStateHoldDown` makes a port stay unhealthy for that long before `InfiniBandStateCheck` or `EthernetStateCheck` reports it. ```yaml nic-health-monitor: portStateHoldDown: "15s" ``` - **Off by default.** `"0s"` reports on the first unhealthy poll. - **Port and card events alike.** While a port is held, its card's homogeneity check sees it as up too, so the card event waits as well. - **Blips are counted, not reported.** A port that recovers before it is reported publishes no event and increments `nic_health_monitor_port_state_blips_total{node,check,device,port}`. - **Recovery is immediate.** The healthy event is never delayed. - **A restart restarts the hold.** The hold-down is not persisted. The hold-down applies only to port state. The `link_downed` counter (InfiniBandDegradationCheck) is fatal on any increase and stays latched until the counter resets or the host reboots, so a blip that bumps it still produces a fatal event. If you see blips there, you can disable that counter (see [Disable a noisy counter](#disable-a-noisy-counter)). ## Reporting Ports Suppressed on the First Poll On the first poll, an unhealthy port is reported only with peer evidence that it should be up: its card has fewer active ports than the decisive mode of its role group. Otherwise the port is suppressed, because it may be uncabled or disabled by design. That happens when: - the card is at or above its group's mode, like the uncabled second port of a dual-port card. This is the most common case. - the card shares a PCI card with a management NIC, so it is left out of peer comparison. - the card is alone in its role group. - the group's vote is tied. - the group's mode is 0, because most of its cards are down. A suppressed port that never comes up is otherwise never reported. A node with half its compute links down from boot gives a tie and stays silent. `suppressedPortReportAfter` reports such a port once it has stayed unhealthy that long, but only on a compute card the peers gave no verdict on (a singleton group, a tie, or a mode of 0): ```yaml nic-health-monitor: suppressedPortReportAfter: "10m" ``` - **Off by default.** `"0s"` keeps suppressing the port. Leave it off if some compute ports are down by design, or exclude those devices with `nicExclusionRegex`. - **Compute cards only.** A card qualifies when at least one monitored device on it is classified as compute and has a port on the same link layer. Then every port on the card qualifies, also the ports of a device classified as storage. For example, a GB300 SMI device has no topology matrix entry, but it shares its card with a compute device. A port on a card with no compute device is never reported. Storage and frontend ports can be down by design with no peer to compare against, like the unused frontend aux port on OCI BM.GPU.H100.8. - **Healthy cards are never reported.** A port on a card at or above its group's mode, or on a card that shares a PCI card with a management NIC, is not reported however long it stays down. If a later poll finds the card at or above the mode, for example once its other ports finish training, the port is dropped. - **Unreadable devices.** While a device is unreadable the peer comparison does not run, so a port on a card with an active port waits for a complete poll: its card may be at its group's mode. A port on a card with no active port is reported, since no peer vote could find that card healthy, unless one of the card's own functions is unreadable. If the monitor cannot find the PCI card of an unreadable device, every card waits. A card with an unreadable function also keeps its ports even when no compute device was read on it, because the unreadable function can be its compute device. - **Severity.** An InfiniBand port still `DOWN` is fatal with `REPLACE_VM`, `Polling` included, the same result the homogeneity check gives a card below its mode. `INIT` and `ARMED` stay non-fatal, so a subnet manager outage at boot does not mark every node fatal. A RoCE port is reported only when `DOWN`, which is fatal. - **Ports still training at boot are not reported.** A port that comes up before the delay is dropped. - **Survives restarts.** The suppression time is kept in the state file, so a pod restart does not restart the wait. This includes a restart while the reboot baseline is still owed because a device is unreadable. A reboot or discovery-scope change resets it, and the next first poll decides again. - **Recorded while off.** The monitor saves the suppression time even when the option is off. If you turn the option on later, a port that has stayed down since its suppression is reported at the first poll after the delay, without a reboot. - **Upgrades.** Versions before this option did not save a suppression time. After an upgrade, a port suppressed by the older version is picked up only after the next reboot or discovery-scope change. ## Poll Stall Detection Reports when the monitor itself cannot observe NIC state. If a state or counter poll has been in flight for `pollStallDeadline`, the monitor publishes an unhealthy, non-fatal `NICPollStallCheck` event (recommended action `NONE`). Once no poll is stalled it publishes the healthy event. No check runs while a publish is waiting for the deployment platform connector, so a connector outage is not reported as a NIC stall. For the same condition as a metric, use `time() - nic_health_monitor_poll_cycle_last_completed_timestamp_seconds`. A stall event that fails to publish is retried until delivered, even if the poll has completed in the meantime, so the episode is still recorded. The first poll to complete after the monitor starts also publishes a healthy event, which closes a stall left open when the liveness probe restarted the container. A stalled poll usually means the host is stalling the sysfs reads, for example through lock contention in the network stack. Restarting the container does not fix that, and without this check the only trace is the restart count. ```yaml nic-health-monitor: pollStallDeadline: "10s" pollStallStoreOnly: true ``` Keep `pollStallDeadline` well below the liveness restart window, which is about 60 to 90 seconds into a stall, or the stall is never reported. `"0s"` disables detection. `pollStallStoreOnly` emits the events as `STORE_ONLY` whatever `processingStrategy` is, because the cause is the host rather than a NIC. ## Character-Device Check Tuning ### charDeviceCheck.issm Controls whether `InfiniBandCharDeviceCheck` expects the per-port `issm` character device. `umad` and `uverbs` are always required and are not configurable; only `issm` is, because whether `issm` nodes are created is architecture- and provider-dependent. ```yaml nic-health-monitor: charDeviceCheck: issm: never # never (default) | always ``` | Value | Behavior | |-------|----------| | `never` (default) | Never expect `issm`. `umad`/`uverbs` still cover the RDMA-fatal cases, and a platform that legitimately does not create `issm` nodes (e.g. GB300) is never falsely flagged. | | `always` | Expect one `issm` per InfiniBand-mode port. Enable only where `issm` nodes are guaranteed to be created (behavior prior to this option). | **Why this exists:** on newer hardware such as GB300 the platform does not create `issm` device nodes, so the previous unconditional check falsely marked every such GPU node `REPLACE_VM`. `issm` presence cannot be inferred from any port attribute — on a fabric run by an external subnet manager every compute port reads `SM_DISABLED` (`cap_mask` bit `0x400`) whether or not `issm` exists — so the check is an explicit opt-in rather than an inferred default. Changing this value forces a one-time baseline reconciliation on the next pod start (when there are outstanding `issm` conditions to clear), so stale `issm` conditions are cleared immediately rather than being held until the next reboot. ## NIC Discovery Filtering ### nicExclusionRegex Comma-separated regex patterns for device names to exclude from discovery. Applied after vendor detection and VF filtering, before NUMA-based management NIC exclusion. ```yaml nic-health-monitor: nicExclusionRegex: "^veth.*,^docker.*,^br-.*,^lo$" ``` Use this to suppress virtual interfaces, bridge devices, or loopback from appearing as potential NICs. ### nicInclusionRegexOverride When non-empty, bypasses all automatic NIC discovery and monitors only devices whose names match these comma-separated regex patterns. All automatic filters — vendor detection, VF filtering, NUMA-based management exclusion, and `nicExclusionRegex` — are skipped. Pinned devices are reported unconditionally: the first-poll peer-evidence gate and the card homogeneity check do not apply. ```yaml nic-health-monitor: nicInclusionRegexOverride: "" # Default: empty (auto-discovery enabled) ``` > **Warning**: Changing `nicInclusionRegexOverride` or `nicExclusionRegex` resets the monitor's persisted port and device state on the next pod start (healthy baselines are re-emitted). Counter snapshots and latched fatal counter breaches are preserved across scope changes. **When to use**: Emergency override when automatic discovery misclassifies a NIC on an unusual platform. In normal deployments, leave empty. ## Sysfs Paths Where the monitor reads network and InfiniBand device state. The DaemonSet mounts the host's `/sys` under these paths, so the values must match the host mounts in the pod spec. ```yaml nic-health-monitor: sysClassNetPath: "/nvsentinel/sys/class/net" sysClassInfinibandPath: "/nvsentinel/sys/class/infiniband" ``` Change these only when you alter where the chart mounts host `/sys`. A path that does not exist leaves the monitor with no devices to discover. ## Counter Detection ### Enable/Disable Enables or disables counter-based degradation monitoring globally. ```yaml nic-health-monitor: counterDetection: enabled: true ``` ### Counter Profiles Counter names are validated against a hardcoded allowlist in the monitor. Operators can enable or disable individual counters, tune thresholds, and change velocity windows. Sysfs paths, severity, recommended action, and event descriptions are owned by code and are not configurable here. ```yaml nic-health-monitor: counterDetection: enabled: true counters: - name: link_downed enabled: true thresholdType: delta threshold: 0 ``` #### Counter Profile Fields | Field | Type | Description | |-------|------|-------------| | `name` | string | Counter identifier from the allowlist (see below) | | `enabled` | bool | Whether this counter is monitored (default: `true`) | | `thresholdType` | string | `delta` (absolute change per poll) or `velocity` (rate per `velocityUnit`) | | `threshold` | number | Numeric threshold; breach when exceeded | | `velocityUnit` | string | For velocity thresholds: `second`, `minute`, or `hour` | #### Default Counter Configuration ```yaml nic-health-monitor: counterDetection: enabled: true counters: # Fatal counters — any increment triggers REPLACE_VM - name: link_downed # Port training failure enabled: true thresholdType: delta threshold: 0 - name: excessive_buffer_overrun_errors # Lossless contract violated enabled: true thresholdType: delta threshold: 0 - name: local_link_integrity_errors # Physical errors exceed hardware cap enabled: true thresholdType: delta threshold: 0 - name: rnr_nak_retry_err # Connection severed by retry exhaustion enabled: true thresholdType: delta threshold: 0 # Fatal PHY threshold - name: symbol_error_fatal # IBTA BER spec violation (> 120/hour = fatal) enabled: true thresholdType: velocity threshold: 120.0 velocityUnit: hour # PHY degradation — non-fatal - name: symbol_error # PHY bit errors (> 10/sec = degradation) enabled: true thresholdType: velocity threshold: 10.0 velocityUnit: second - name: link_error_recovery # Link retraining / micro-flapping enabled: true thresholdType: velocity threshold: 5.0 velocityUnit: minute # Transport — non-fatal - name: port_rcv_errors # Malformed packets / CRC errors enabled: true thresholdType: velocity threshold: 10.0 velocityUnit: second - name: local_ack_timeout_err # ACK timeout (fabric path issue or remote crash) enabled: true thresholdType: velocity threshold: 1.0 velocityUnit: second # Congestion — non-fatal - name: port_xmit_discards # TX discards due to flow control breakdown enabled: true thresholdType: velocity threshold: 100.0 velocityUnit: second # RoCE-specific — non-fatal - name: roce_slow_restart # Victim flow oscillation (grey failure indicator) enabled: true thresholdType: velocity threshold: 10.0 velocityUnit: second # Interface level — non-fatal - name: carrier_changes # OS-visible link instability enabled: true thresholdType: delta threshold: 2 ``` #### Allowed Counter Names Counters outside this list are rejected at startup with a validation error. **Standard IB counters** (`counters/`): `excessive_buffer_overrun_errors`, `link_downed`, `link_error_recovery`, `local_link_integrity_errors`, `port_rcv_discards`, `port_rcv_errors`, `port_rcv_remote_physical_errors`, `port_rcv_switch_relay_errors`, `port_xmit_discards`, `port_xmit_wait`, `symbol_error`, `symbol_error_fatal` **Extended counters** (`hw_counters/`): `implied_nak_seq_err`, `local_ack_timeout_err`, `out_of_sequence`, `packet_seq_err`, `req_transport_retries_exceeded`, `rnr_nak_retry_err`, `roce_slow_restart` **Ethernet statistics** (`statistics/`): `carrier_changes`, `rx_crc_errors`, `rx_errors`, `rx_missed_errors`, `tx_carrier_errors`, `tx_errors` ### Counter Customization Examples #### Stricter BER threshold ```yaml nic-health-monitor: counterDetection: counters: - name: symbol_error_fatal enabled: true thresholdType: velocity threshold: 60.0 # Stricter than the default 120/hour velocityUnit: hour ``` #### Add an allowed counter not enabled by default ```yaml nic-health-monitor: counterDetection: counters: - name: rx_missed_errors enabled: true thresholdType: velocity threshold: 10.0 velocityUnit: second ``` #### Disable a noisy counter ```yaml nic-health-monitor: counterDetection: counters: - name: port_xmit_wait enabled: false ``` > **Note**: Specifying a counter list in your values overrides the full default list. If you want to add or modify one counter, include the complete default list alongside your changes. ## NIC Driver Error Detection (Syslog Health Monitor) NIC driver and firmware errors are monitored by the **Syslog Health Monitor** DaemonSet, not the NIC Health Monitor. Enable the `SysLogsNICDriverError` check and configure pattern selection in the Syslog Health Monitor's Helm values: ```yaml syslog-health-monitor: enabledChecks: - SysLogsXIDError - SysLogsSXIDError - SysLogsGPUFallenOff - SysLogsNICDriverError # Add this to enable NIC driver log monitoring ``` ### NIC Driver Pattern Configuration Pattern regexes, severity, and recommended actions are owned by code. The following Helm/YAML configuration selects which patterns are active and optionally overrides the per-pattern processing strategy: ```yaml syslog-health-monitor: nicDriverDetection: patterns: - name: cmd_exec_timeout # Fatal: firmware hung, driver cannot issue commands enabled: true processingStrategy: EXECUTE_REMEDIATION - name: health_poll_failed # Fatal: firmware heartbeat lost enabled: true processingStrategy: EXECUTE_REMEDIATION - name: unrecoverable_err # Fatal: hardware admission of failure enabled: true processingStrategy: EXECUTE_REMEDIATION - name: mlx5_napi_soft_lockup # Fatal: CPU wedged in NIC NAPI poll loop enabled: true processingStrategy: STORE_ONLY # Start in shadow mode; graduate to EXECUTE_REMEDIATION after validation - name: netdev_watchdog # Non-fatal: TX queue stall with auto-recovery enabled: true - name: mlx5_tx_timeout_detected # Non-fatal: TX timeout (driver-reported) enabled: true - name: mlx5_rx_timeout_detected # Non-fatal: RX timeout (driver-reported) enabled: true - name: port_module_high_temp # Non-fatal: thermal warning enabled: true - name: pci_power_insufficient # Non-fatal: PCIe power negotiation enabled: true - name: module_unplugged # Non-fatal: SFP/transceiver removed enabled: true - name: access_reg_failed # Non-fatal: monitoring tool conflict noise enabled: true ``` See [Syslog Health Monitor Configuration](/nvsentinel/v1.30.0/configuration/syslog-health-monitor) for complete syslog-health-monitor options. ## Health Events Analyzer Rules Repeated non-fatal NIC events are escalated to `CONTACT_SUPPORT` by the Health Events Analyzer. Enable these rules in the health-events-analyzer Helm values: ```yaml health-events-analyzer: enableRepeatedNICDegradationRule: true # 3 non-fatal counter events on same NIC+port in 1 hour enableRepeatedNICDriverErrorRule: true # 3 non-fatal syslog events of same pattern on same node in 1 hour ``` ## Scheduling Configure pod placement for the NIC Health Monitor DaemonSet. ```yaml nic-health-monitor: nodeSelector: {} tolerations: [] affinity: {} ``` The DaemonSet runs on all nodes by default. Use `nodeSelector` or `affinity` to limit it to nodes with monitored NICs.