NVLink Checks

NVLink covers NVLink interconnect health: replay counters, link errors, and fabric state.

NVLink Active Links

nvlink_active

Verifies each GPU has no inactive NVLinks.

Thresholds

Rules

• CRITICAL when inactive_count > max_inactive_nvlinks
  • Message: GPU $\{gpu_id\}:${inactive_count} inactive NVLinks (allowed: ${max_inactive_nvlinks}).

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

NVLink Bandwidth, Count, and Speed

nvlink_bandwidth_count_speed

Verifies each GPU’s NVLink link count and per-link speed match expected values.

Thresholds

Rules

• CRITICAL when link_count != expected_link_count
  • Message: GPU $\{gpu_id\}: expected${expected_link_count} NVLinks, found ${link_count}.
• CRITICAL when speed_mbytes_per_sec != expected_speed_mbytes_per_sec
  • Message: GPU $\{gpu_id\}: expected${expected_speed_mbytes_per_sec} MB/s per NVLink, found ${speed_mbytes_per_sec} MB/s.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

NVLink Capabilities

nvlink_capabilities

Verifies every NVLink on each GPU reports the capabilities required for multi-GPU communication (P2P, system-memory access, P2P and system-memory atomics, SLI, and link validity).

Rules

• CRITICAL when all_links_support_p2p == false
  • Message: GPU ${gpu_id}: one or more NVLinks do not support P2P.
• CRITICAL when all_links_support_sysmem_access == false
  • Message: GPU ${gpu_id}: one or more NVLinks do not support system memory access.
• CRITICAL when all_links_support_p2p_atomics == false
  • Message: GPU ${gpu_id}: one or more NVLinks do not support P2P atomics.
• CRITICAL when all_links_support_sysmem_atomics == false
  • Message: GPU ${gpu_id}: one or more NVLinks do not support system memory atomics.
• CRITICAL when all_links_support_sli == false
  • Message: GPU ${gpu_id}: one or more NVLinks do not support SLI.
• CRITICAL when all_links_support_link == false
  • Message: GPU ${gpu_id}: one or more NVLinks are not reported as supported (Link capability).

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

NVLink Fabric Registration

nvlink_fabric

Verifies each GPU has completed NVLink fabric registration with a successful status. Incomplete or failed registration cuts the GPU off from multi-GPU NVLink communication.

Rules

• CRITICAL when fabric_registration_incomplete == true
  • Message: GPU ${gpu_id}: NVLink fabric registration has not completed.
• CRITICAL when fabric_registration_failed == true
  • Message: GPU ${gpu_id}: NVLink fabric registration completed with a failure status.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

NVLink Fabric Health

nvlink_fabric_health

Monitors per-GPU NVLink fabric health flags from NVML. Degraded bandwidth, in-progress route recovery, an unhealthy route, or active access timeout recovery indicate a fabric problem affecting multi-GPU communication.

Rules

• CRITICAL when fabric_bandwidth_degraded == true
  • Message: GPU ${gpu_id}: NVLink fabric bandwidth is not Full.
• CRITICAL when fabric_route_recovery_in_progress == true
  • Message: GPU ${gpu_id}: NVLink fabric route recovery is in progress.
• CRITICAL when fabric_route_unhealthy == true
  • Message: GPU ${gpu_id}: NVLink fabric has an unhealthy route.
• CRITICAL when fabric_access_timeout_recovery == true
  • Message: GPU ${gpu_id}: NVLink fabric access timeout recovery is active.

Recommended action: operational state drain, then reset_gpu → restart_os → hold

NVLink P2P Connectivity

nvlink_p2p

Verifies each GPU has healthy peer-to-peer NVLink connectivity to every other GPU.

Thresholds

Rules

• CRITICAL when unhealthy_p2p_peer_count > max_unhealthy_p2p_peers
  • Message: GPU $\{gpu_id\}:${unhealthy_p2p_peer_count} peer GPUs unreachable over NVLink P2P (allowed: ${max_unhealthy_p2p_peers}).

Recommended action: operational state drain, then reset_gpu → restart_os → service_required