Skip to navigation
ReferenceCheck Catalog

InfiniBand Checks

View as Markdown

InfiniBand covers InfiniBand network health: port state, error rates, and link integrity.

rdma_linkflap_fatal

Detects mlx5 RDMA interface link outages that lasted at least min_outage_seconds, parsed from the system journal. Fails CRITICAL once the count of such outages reaches fatal_flap_threshold.

PropertyValue
Provenancenvidia
Condition scopeCLUSTER
Execution classPASSIVE
Scheduling cadenceCONTINUOUS
Maturitystable
Execution contextsOPERATOR_INFRA, TENANT_GUEST
ArchitecturesGB200, GB300
Required signalsoutage_durations

Thresholds

NameValueUnitCategoryDescription
fatal_flap_threshold1countDEFAULTNumber of outages at or above min_outage_seconds that fails the check. Override per fleet.
min_outage_seconds30secondsDEFAULTMinimum down→up outage length, in seconds, for an RDMA link transition to count as a fatal flap. Override per fleet.

Rules

  • CRITICAL when size(outage_durations.filter(d, d >= min_outage_seconds)) >= fatal_flap_threshold
    • Message: RDMA link outage(s) lasting at least {minoutageseconds}sreachedthefatal−flapthresholdof\{min_outage_seconds\}s reached the fatal-flap threshold of {fatal_flap_threshold}. See flap_events for the affected interfaces.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required