> This page is for Health Validation.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/health-validation/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/health-validation/_mcp/server.

# NVLink Checks

NVLink covers NVLink interconnect health: replay counters, link errors, and fabric state.

## NVLink Active Links

`nvlink_active`

Verifies each GPU has no inactive NVLinks.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr>, <nobr>`A100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`inactive_count`</nobr> |

**Thresholds**

| Name | Value | Unit | Category | Description |
|---|---|---|---|---|
| <nobr>`max_inactive_nvlinks`</nobr> | 0 | count | <nobr>`DEFAULT`</nobr> | Maximum inactive NVLinks tolerated per GPU before the GPU is considered failing. |

**Rules**

- **CRITICAL** when `inactive_count > max_inactive_nvlinks`
  - Message: GPU $\{gpu_id\}: $\{inactive_count\} inactive NVLinks (allowed: $\{max_inactive_nvlinks\}).

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## NVLink Bandwidth, Count, and Speed

`nvlink_bandwidth_count_speed`

Verifies each GPU's NVLink link count and per-link speed match expected values.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`link_count`</nobr>, <nobr>`speed_mbytes_per_sec`</nobr> |

**Thresholds**

| Name | Value | Unit | Category | Description |
|---|---|---|---|---|
| <nobr>`expected_link_count`</nobr> | 18 | count | <nobr>`DEFAULT`</nobr> | Expected NVLinks per GPU. GB200 exposes 18 NVLink 5 links. |
| <nobr>`expected_speed_mbytes_per_sec`</nobr> | GB200: 53125, GB300: 53125, H100: 26562, other: 53125 | MB/s | <nobr>`DEFAULT`</nobr> | Expected per-link NVLink speed in MegaBytes per second. GB200 NVLink 5 at 53125, GB300 at 53125, H100 NVLink 4 at 26562. |

**Rules**

- **CRITICAL** when `link_count != expected_link_count`
  - Message: GPU $\{gpu_id\}: expected $\{expected_link_count\} NVLinks, found $\{link_count\}.
- **CRITICAL** when `speed_mbytes_per_sec != expected_speed_mbytes_per_sec`
  - Message: GPU $\{gpu_id\}: expected $\{expected_speed_mbytes_per_sec\} MB/s per NVLink, found $\{speed_mbytes_per_sec\} MB/s.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## NVLink Capabilities

`nvlink_capabilities`

Verifies every NVLink on each GPU reports the capabilities required for multi-GPU communication (P2P, system-memory access, P2P and system-memory atomics, SLI, and link validity).

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`all_links_support_p2p`</nobr>, <nobr>`all_links_support_sysmem_access`</nobr>, <nobr>`all_links_support_p2p_atomics`</nobr>, <nobr>`all_links_support_sysmem_atomics`</nobr>, <nobr>`all_links_support_sli`</nobr>, <nobr>`all_links_support_link`</nobr> |

**Rules**

- **CRITICAL** when `all_links_support_p2p == false`
  - Message: GPU $\{gpu_id\}: one or more NVLinks do not support P2P.
- **CRITICAL** when `all_links_support_sysmem_access == false`
  - Message: GPU $\{gpu_id\}: one or more NVLinks do not support system memory access.
- **CRITICAL** when `all_links_support_p2p_atomics == false`
  - Message: GPU $\{gpu_id\}: one or more NVLinks do not support P2P atomics.
- **CRITICAL** when `all_links_support_sysmem_atomics == false`
  - Message: GPU $\{gpu_id\}: one or more NVLinks do not support system memory atomics.
- **CRITICAL** when `all_links_support_sli == false`
  - Message: GPU $\{gpu_id\}: one or more NVLinks do not support SLI.
- **CRITICAL** when `all_links_support_link == false`
  - Message: GPU $\{gpu_id\}: one or more NVLinks are not reported as supported (Link capability).

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## NVLink Fabric Registration

`nvlink_fabric`

Verifies each GPU has completed NVLink fabric registration with a successful status. Incomplete or failed registration cuts the GPU off from multi-GPU NVLink communication.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`DOMAIN`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`fabric_registration_incomplete`</nobr>, <nobr>`fabric_registration_failed`</nobr> |

**Rules**

- **CRITICAL** when `fabric_registration_incomplete == true`
  - Message: GPU $\{gpu_id\}: NVLink fabric registration has not completed.
- **CRITICAL** when `fabric_registration_failed == true`
  - Message: GPU $\{gpu_id\}: NVLink fabric registration completed with a failure status.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`

## NVLink Fabric Health

`nvlink_fabric_health`

Monitors per-GPU NVLink fabric health flags from NVML. Degraded bandwidth, in-progress route recovery, an unhealthy route, or active access timeout recovery indicate a fabric problem affecting multi-GPU communication.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`DOMAIN`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`fabric_bandwidth_degraded`</nobr>, <nobr>`fabric_route_recovery_in_progress`</nobr>, <nobr>`fabric_route_unhealthy`</nobr>, <nobr>`fabric_access_timeout_recovery`</nobr> |

**Rules**

- **CRITICAL** when `fabric_bandwidth_degraded == true`
  - Message: GPU $\{gpu_id\}: NVLink fabric bandwidth is not Full.
- **CRITICAL** when `fabric_route_recovery_in_progress == true`
  - Message: GPU $\{gpu_id\}: NVLink fabric route recovery is in progress.
- **CRITICAL** when `fabric_route_unhealthy == true`
  - Message: GPU $\{gpu_id\}: NVLink fabric has an unhealthy route.
- **CRITICAL** when `fabric_access_timeout_recovery == true`
  - Message: GPU $\{gpu_id\}: NVLink fabric access timeout recovery is active.

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `hold`

## NVLink P2P Connectivity

`nvlink_p2p`

Verifies each GPU has healthy peer-to-peer NVLink connectivity to every other GPU.

| Property | Value |
|---|---|
| Provenance | <nobr>`nvidia`</nobr> |
| Condition scope | <nobr>`NODE`</nobr> |
| Execution class | <nobr>`PASSIVE`</nobr> |
| Scheduling cadence | <nobr>`CONTINUOUS`</nobr> |
| Maturity | <nobr>`stable`</nobr> |
| Execution contexts | <nobr>`OPERATOR_INFRA`</nobr>, <nobr>`TENANT_GUEST`</nobr> |
| Architectures | <nobr>`GB200`</nobr>, <nobr>`GB300`</nobr>, <nobr>`H100`</nobr>, <nobr>`A100`</nobr> |
| Required signals | <nobr>`gpu_id`</nobr>, <nobr>`unhealthy_p2p_peer_count`</nobr> |

**Thresholds**

| Name | Value | Unit | Category | Description |
|---|---|---|---|---|
| <nobr>`max_unhealthy_p2p_peers`</nobr> | 0 | count | <nobr>`DEFAULT`</nobr> | Maximum peer GPUs unreachable over NVLink P2P tolerated per GPU before the GPU is considered failing. |

**Rules**

- **CRITICAL** when `unhealthy_p2p_peer_count > max_unhealthy_p2p_peers`
  - Message: GPU $\{gpu_id\}: $\{unhealthy_p2p_peer_count\} peer GPUs unreachable over NVLink P2P (allowed: $\{max_unhealthy_p2p_peers\}).

**Recommended action:** operational state `drain`, then `reset_gpu` → `restart_os` → `service_required`