GPU Checks

GPU covers GPU device health: driver-reported events (XID), thermal state, driver status, power, and clock behavior.

GPU Average Power Draw

gpu_average_power_draw

Verifies each GPU’s one-second average power draw is within the expected range.

Thresholds

Rules

• CRITICAL when average_power_draw_w < min_power_w
  • Message: GPU $\{gpu_id\}: average power draw${average_power_draw_w} W is below minimum ${min_power_w} W.
• CRITICAL when average_power_draw_w > max_power_w
  • Message: GPU $\{gpu_id\}: average power draw${average_power_draw_w} W is above maximum ${max_power_w} W.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

GPU Compute Apps

gpu_compute_apps

Fails when any compute process still holds a GPU.

Rules

• CRITICAL when compute_process_count > 0
  • Message: GPU $\{gpu_id\}:${compute_process_count} compute process(es) still holding the GPU: ${compute_process_cmdlines}

Recommended action: operational state drain, then restart_os → service_required

GPU Power Limit

gpu_power_limit

Verifies each GPU’s current and enforced power limits.

Thresholds

Rules

• CRITICAL when power_limit_w != expected_power_limit_w
  • Message: GPU $\{gpu_id\}: current power limit${power_limit_w} W, expected ${expected_power_limit_w} W.
• CRITICAL when enforced_power_limit_w != expected_power_limit_w
  • Message: GPU $\{gpu_id\}: enforced power limit${enforced_power_limit_w} W, expected ${expected_power_limit_w} W.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

GPU Power State

gpu_power_state

Verifies each GPU is operating at the expected performance state.

Thresholds

Rules

• CRITICAL when gpu_pstate != expected_gpu_pstate
  • Message: GPU $\{gpu_id\}: performance state P${gpu_pstate}, expected P${expected_gpu_pstate}.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required

GPU Sensor Throttle

gpu_sensor_throttle

Monitors GPU hardware throttling via NVML clocks-event-reasons. Active hardware slowdown, thermal slowdown, or power-brake slowdown indicates the GPU is being clamped by hardware protection logic.

Rules

• CRITICAL when hw_thermal_slowdown == true
  • Message: GPU ${gpu_id}: hardware thermal slowdown is active. GPU is throttling to avoid thermal damage.
• CRITICAL when hw_power_brake_slowdown == true
  • Message: GPU ${gpu_id}: hardware power-brake slowdown is active. External power protection is clamping the GPU.
• CRITICAL when hw_slowdown == true
  • Message: GPU ${gpu_id}: hardware slowdown is active. GPU is being clamped by hardware protection logic.

Recommended action: operational state drain, then reset_gpu → restart_os → service_required