> This page is for Health Validation.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/health-validation/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/health-validation/_mcp/server.

# GPU Checks

GPU covers GPU device health: driver-reported events (XID), thermal state, driver status, power, and clock behavior.

## GPU Average Power Draw

`gpu_average_power_draw`

Verifies each GPU's one-second average power draw is within the expected range.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`average_power_draw_w`</nobr> |

**Thresholds**

| Name | Value | Unit | Category | Description |
|---|---|---|---|---|
| <nobr>`max_power_w`</nobr> | 1500 | W | <nobr>`DEFAULT`</nobr> | Maximum expected one-second average power draw per GPU. Override per fleet. |
| <nobr>`min_power_w`</nobr> | 65 | W | <nobr>`DEFAULT`</nobr> | Minimum expected one-second average power draw per GPU. Override per fleet. |

**Rules**

- **CRITICAL** when `average_power_draw_w < min_power_w`
  - Message: GPU $\{gpu_id\}: average power draw $\{average_power_draw_w\} W is below minimum $\{min_power_w\} W.
- **CRITICAL** when `average_power_draw_w > max_power_w`
  - Message: GPU $\{gpu_id\}: average power draw $\{average_power_draw_w\} W is above maximum $\{max_power_w\} W.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## GPU Compute Apps

`gpu_compute_apps`

Fails when any compute process still holds a GPU.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`PROLOG`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr>, <nobr>`A100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`compute_process_count`</nobr>, <nobr>`compute_process_cmdlines`</nobr> |

**Rules**

- **CRITICAL** when `compute_process_count > 0`
  - Message: GPU $\{gpu_id\}: $\{compute_process_count\} compute process(es) still holding the GPU: $\{compute_process_cmdlines\}

**Recommended action:** operational state `drain`, then `restart_os` → `service_required`

## GPU Power Limit

`gpu_power_limit`

Verifies each GPU's current and enforced power limits.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`PROLOG`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`power_limit_w`</nobr>, <nobr>`enforced_power_limit_w`</nobr> |

**Thresholds**

| Name | Value | Unit | Category | Description |
|---|---|---|---|---|
| <nobr>`expected_power_limit_w`</nobr> | GB200: 1200, GB300: 1400, H100: 700 | W | <nobr>`DEFAULT`</nobr> | Current and enforced power limit expected for every GPU. |

**Rules**

- **CRITICAL** when `power_limit_w != expected_power_limit_w`
  - Message: GPU $\{gpu_id\}: current power limit $\{power_limit_w\} W, expected $\{expected_power_limit_w\} W.
- **CRITICAL** when `enforced_power_limit_w != expected_power_limit_w`
  - Message: GPU $\{gpu_id\}: enforced power limit $\{enforced_power_limit_w\} W, expected $\{expected_power_limit_w\} W.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## GPU Power State

`gpu_power_state`

Verifies each GPU is operating at the expected performance state.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr>, <nobr>`A100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`gpu_pstate`</nobr> |

**Thresholds**

| Name | Value | Unit | Category | Description |
|---|---|---|---|---|
| <nobr>`expected_gpu_pstate`</nobr> | 0 | pstate | <nobr>`DEFAULT`</nobr> | Performance state every GPU must be in, as the numeric level N in P&lt;N> (0 = P0, maximum performance). Override per fleet. |

**Rules**

- **CRITICAL** when `gpu_pstate != expected_gpu_pstate`
  - Message: GPU $\{gpu_id\}: performance state P$\{gpu_pstate\}, expected P$\{expected_gpu_pstate\}.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## GPU Sensor Throttle

`gpu_sensor_throttle`

Monitors GPU hardware throttling via NVML clocks-event-reasons. Active hardware slowdown, thermal slowdown, or power-brake slowdown indicates the GPU is being clamped by hardware protection logic.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`PROLOG`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr>, <nobr>`H200`</nobr>, <nobr>`A100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`hw_slowdown`</nobr>, <nobr>`hw_thermal_slowdown`</nobr>, <nobr>`hw_power_brake_slowdown`</nobr> |

**Rules**

- **CRITICAL** when `hw_thermal_slowdown == true`
  - Message: GPU $\{gpu_id\}: hardware thermal slowdown is active. GPU is throttling to avoid thermal damage.
- **CRITICAL** when `hw_power_brake_slowdown == true`
  - Message: GPU $\{gpu_id\}: hardware power-brake slowdown is active. External power protection is clamping the GPU.
- **CRITICAL** when `hw_slowdown == true`
  - Message: GPU $\{gpu_id\}: hardware slowdown is active. GPU is being clamped by hardware protection logic.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`