Xid Checks | NVIDIA Health Validation

Xid covers NVRM Xid fault events logged by the NVIDIA driver to the kernel ring buffer. Xid measurements are collected separately from GPU telemetry and route only to Xid checks.

Single-rule checks reading `xid_counts`

These 19 checks share the properties below and each has one rule.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`

Check	Name	Condition	Severity	Recommended action
`gpu_xid_46`	GPU Xid 46 (GPU stopped processing)	`46 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_48`	GPU Xid 48 (double bit ECC error)	`48 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_62`	GPU Xid 62 (internal micro-controller halt)	`62 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_64`	GPU Xid 64 (GPU memory remapping failure)	`64 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_79`	GPU Xid 79 (GPU has fallen off the bus)	`79 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_95`	GPU Xid 95 (uncontained memory error)	`95 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_109`	GPU Xid 109 (context switch timeout error)	`109 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_110`	GPU Xid 110 (security fault error)	`110 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_119`	GPU Xid 119 (GSP RPC timeout)	`119 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_120`	GPU Xid 120 (GSP error)	`120 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_140`	GPU Xid 140 (ECC unrecovered error)	`140 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_142`	GPU Xid 142 (NVENC3 error)	`142 in xid_counts`	CRITICAL	`drain`: `service_required`
`gpu_xid_143`	GPU Xid 143 (GPU initialization error)	`143 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_151`	GPU Xid 151 (key rotation error)	`151 in xid_counts`	CRITICAL	`drain`: `restart_os`
`gpu_xid_154`	GPU Xid 154 (GPU recovery action changed)	`154 in xid_counts`	INFO	`drain`: `xid_154`
`gpu_xid_155`	GPU Xid 155 (NVLink SW-defined error)	`155 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_156`	GPU Xid 156 (resource retirement event)	`156 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_158`	GPU Xid 158 (GPU fatal timeout)	`158 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_159`	GPU Xid 159 (CHI non-data error)	`159 in xid_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`

Single-rule checks reading `sub_xid_code_counts`

These 20 checks share the properties below and each has one rule.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`sub_xid_code_counts`

Check	Name	Condition	Severity	Recommended action
`gpu_xid_149_1`	GPU Xid 149.1 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.1" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_2`	GPU Xid 149.2 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.2" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_2a`	GPU Xid 149.2A (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.2A" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_2b`	GPU Xid 149.2B (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.2B" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_4`	GPU Xid 149.4 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.4" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_10`	GPU Xid 149.10 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.10" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_11`	GPU Xid 149.11 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.11" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_12`	GPU Xid 149.12 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.12" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_15`	GPU Xid 149.15 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.15" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_16`	GPU Xid 149.16 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.16" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_20`	GPU Xid 149.20 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.20" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_21`	GPU Xid 149.21 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.21" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_22`	GPU Xid 149.22 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.22" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_23`	GPU Xid 149.23 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.23" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_24`	GPU Xid 149.24 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.24" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_25`	GPU Xid 149.25 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.25" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_26`	GPU Xid 149.26 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.26" in sub_xid_code_counts`	WARNING	none
`gpu_xid_149_28`	GPU Xid 149.28 (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.28" in sub_xid_code_counts`	WARNING	none
`gpu_xid_149_a`	GPU Xid 149.A (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.A" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`
`gpu_xid_149_f`	GPU Xid 149.F (NETIR_LINK_EVT/NETIR_LINK_DOWN)	`"149.F" in sub_xid_code_counts`	CRITICAL	`drain`: `reset_gpu` → `service_required`

GPU Xid 144 (NVLink SAW error)

gpu_xid_144

Detects NVRM Xid 144 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 144 in fatal_xid_counts
- Message: GPU Xid 144 detected: NVLink SAW error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 144 in xid_counts && xid_counts[144] > (144 in fatal_xid_counts ? fatal_xid_counts[144] : 0)
- Message: GPU Xid 144 detected: NVLink SAW error (Nonfatal).

GPU Xid 145 (NVLink RLW error)

gpu_xid_145

Detects NVRM Xid 145 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 145 in fatal_xid_counts
- Message: GPU Xid 145 detected: NVLink RLW error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 145 in xid_counts && xid_counts[145] > (145 in fatal_xid_counts ? fatal_xid_counts[145] : 0)
- Message: GPU Xid 145 detected: NVLink RLW error (Nonfatal).

GPU Xid 146 (NVLink TLW error)

gpu_xid_146

Detects NVRM Xid 146 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 146 in fatal_xid_counts
- Message: GPU Xid 146 detected: NVLink TLW error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 146 in xid_counts && xid_counts[146] > (146 in fatal_xid_counts ? fatal_xid_counts[146] : 0)
- Message: GPU Xid 146 detected: NVLink TLW error (Nonfatal).

GPU Xid 147 (NVLink TREX error)

gpu_xid_147

Detects NVRM Xid 147 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 147 in fatal_xid_counts
- Message: GPU Xid 147 detected: NVLink TREX error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 147 in xid_counts && xid_counts[147] > (147 in fatal_xid_counts ? fatal_xid_counts[147] : 0)
- Message: GPU Xid 147 detected: NVLink TREX error (Nonfatal).

GPU Xid 148 (NVLink NVLPW_CTRL error)

gpu_xid_148

Detects NVRM Xid 148 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 148 in fatal_xid_counts
- Message: GPU Xid 148 detected: NVLink NVLPW_CTRL error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 148 in xid_counts && xid_counts[148] > (148 in fatal_xid_counts ? fatal_xid_counts[148] : 0)
- Message: GPU Xid 148 detected: NVLink NVLPW_CTRL error (Nonfatal).

GPU Xid 149 (NVLink NETIR error)

gpu_xid_149

Detects NVRM Xid 149 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 149 in fatal_xid_counts
- Message: GPU Xid 149 detected: NVLink NETIR error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 149 in xid_counts && xid_counts[149] > (149 in fatal_xid_counts ? fatal_xid_counts[149] : 0)
- Message: GPU Xid 149 detected: NVLink NETIR error (Nonfatal).

GPU Xid 150 (NVLink MSE error)

gpu_xid_150

Detects NVRM Xid 150 in the system journal since boot, differentiated by the Fatal/Nonfatal severity NVRM assigns each occurrence.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`, `fatal_xid_counts`

Rules

CRITICAL when 150 in fatal_xid_counts
- Message: GPU Xid 150 detected: NVLink MSE error (Fatal).
- Action: operational state drain, then reset_gpu → service_required
WARNING when 150 in xid_counts && xid_counts[150] > (150 in fatal_xid_counts ? fatal_xid_counts[150] : 0)
- Message: GPU Xid 150 detected: NVLink MSE error (Nonfatal).

GPU Xid Warnings

gpu_xid_warning

Detects application-level and otherwise recoverable NVRM Xid errors in the system journal since boot, grouped by their NVIDIA Xid-catalog immediate-action resolution bucket.

Property	Value
Provenance	`nvidia`
Condition scope	`NODE`
Execution class	`PASSIVE`
Scheduling cadence	`CONTINUOUS`
Maturity	`stable`
Execution contexts	`OPERATOR_INFRA`, `TENANT_GUEST`
Architectures	`GB200`, `GB300`
Required signals	`xid_counts`

Rules

WARNING when xid_counts.exists(k, k in [8, 11, 13, 25, 31, 32, 39, 40, 41, 60, 68, 69, 70, 71, 72, 75, 76, 77, 82, 83, 84, 85, 86, 88, 89, 94, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 139])
- Message: Application-level GPU Xid error detected. Restart the faulting application; send the node for service if the error persists.
- Action: operational state drain, then restart_app → service_required
WARNING when xid_counts.exists(k, k in [45])
- Message: GPU Xid 45 (preemptive removal) detected. Follow the Xid 45 workflow.
- Action: operational state drain, then workflow_xid_45
WARNING when xid_counts.exists(k, k in [78])
- Message: GPU Xid 78 (vGPU start error) detected. Update GPU software/firmware.
- Action: operational state drain, then update_swfw

Recommended action: operational state drain, then restart_app → service_required

Single-rule checks reading xid_counts

Single-rule checks reading sub_xid_code_counts

GPU Xid 144 (NVLink SAW error)

GPU Xid 145 (NVLink RLW error)

GPU Xid 146 (NVLink TLW error)

GPU Xid 147 (NVLink TREX error)

GPU Xid 148 (NVLink NVLPW_CTRL error)

GPU Xid 149 (NVLink NETIR error)

GPU Xid 150 (NVLink MSE error)

GPU Xid Warnings

Single-rule checks reading `xid_counts`

Single-rule checks reading `sub_xid_code_counts`