GPU Operational Event Catalog#

Warning

GPU Operational Events are a preview feature and remain under active development. Category values, module signatures, module event codes, severities, context payloads, and the set of published events may change before the interface is declared stable. Treat the identifiers in this document as valid for the driver release documented here, and re-validate consumers against each driver release. Legacy Xid reporting remains supported for production monitoring during the preview period. See Release Notes for compatibility changes.

This page lists the shared event category registry and the GPU Operational Events published in R620. See Analyzing GPU Operational Events for monitoring interfaces and how to interpret an event, and GPU Operational Event Formats for the formats events are published in.

Event Categories#

The category registry is shared by all reporting modules. A consumer can act on the category alone when it does not recognize a specific module event code.

Category	Value	Indicates
`TIMEOUT`	`0x0001`	An operation did not complete within its expected time budget.
`MEMORY_INTEGRITY_ERROR`	`0x0002`	A data integrity error in memory or cache, such as ECC, parity, or poison.
`INTERCONNECT_ERROR`	`0x0003`	An error on a link between the GPU and another device, such as NVLink, PCIe, or C2C.
`CORRUPTION_DETECTED`	`0x0004`	Corrupted driver or device state was detected.
`SECURITY_ERROR`	`0x0005`	A security policy violation or an authentication failure.
`POWER_ERROR`	`0x0006`	A power delivery or power management fault.
`THERMAL_ERROR`	`0x0007`	A thermal limit was reached or a thermal sensor failed.
`FIRMWARE_FAULT`	`0x0008`	A fault detected by firmware running on the device.
`ENGINE_ERROR`	`0x0009`	An error reported by a GPU engine.
`RESOURCE_EXHAUSTED`	`0x000A`	A repair, spare, or tracking resource has been used up.
`TRANSLATION_FAULT`	`0x000B`	An address translation or MMU fault.
`UNCLASSIFIED_ERROR`	`0x7FFF`	An error that has not yet been assigned a specific category.
`INITIALIZATION`	`0x8000`	Component initialization progress or completion.
`SHUTDOWN`	`0x8001`	Component shutdown or teardown.
`STATE_CHANGED`	`0x8002`	An operational state transition.
`CONFIG_CHANGED`	`0x8003`	A configuration change.
`RESOURCE_RETIREMENT`	`0x8004`	A resource has been marked for retirement or repair.
`RESOURCE_THRESHOLD`	`0x8005`	A resource usage threshold was crossed.
`TELEMETRY`	`0xFFFE`	Operational telemetry.
`DEBUG`	`0xFFFF`	Engineering and debug data.

This release publishes events in ten of these categories: TIMEOUT, MEMORY_INTEGRITY_ERROR, INTERCONNECT_ERROR, FIRMWARE_FAULT, RESOURCE_EXHAUSTED, UNCLASSIFIED_ERROR, INITIALIZATION, STATE_CHANGED, RESOURCE_RETIREMENT, and RESOURCE_THRESHOLD. The remaining values are reserved for other event categories.

Note

In this release, the nvidia-smi event-log decoder renders category 0x0008 as “Engine Software Error” and 0x0009 as “Engine Hardware Error”. These values correspond to FIRMWARE_FAULT and ENGINE_ERROR, respectively. Decode categories from the numeric value.

Event Log Levels#

The log level describes the urgency of the report independently from the event severity. Values increase with urgency, so a subscription filter can use “at least this urgent” semantics. A filter value of 0 matches every log level.

Log Level	Value	Indicates
`TELEMETRY`	`10`	Operational telemetry.
`DIAGNOSTIC`	`20`	Diagnostic information. NVML names this level `NVML_GPU_OPERATIONAL_EVENT_LOG_LEVEL_DIAG`.
`NOTICE`	`30`	A notable condition that does not require immediate action.
`WARNING`	`40`	A condition that may require operator attention.
`ERROR`	`50`	A condition that requires operator attention.

Event Severities#

The severity identifies the impact of the event relative to its scope. Values increase with severity, so a subscription filter can use “at least this severe” semantics. A filter value of 0 matches every severity.

Severity	Value	Indicates
`INFORMATIONAL`	`10`	The event reports state, telemetry, or progress.
`CORRECTED`	`20`	The condition was corrected without recovery action by the operator.
`RECOVERABLE`	`30`	The condition affected operation, and recovery is possible.
`FATAL`	`40`	The condition caused loss of the scoped resource.

When a GPU Operational Event is encoded as CPER, the GOE severity is remapped to a host-scoped CPER severity. See GOE Severity to CPER Severity Mapping.

Event Scopes#

The scope identifies the resource affected by the event.

Scope	Value	Indicates
`SYSTEM`	`0`	The whole system.
`DEVICE`	`1`	One GPU device.
`GPU_INSTANCE`	`2`	One MIG GPU instance.
`COMPUTE_INSTANCE`	`3`	One MIG compute instance.
`ENGINE`	`4`	One GPU engine.

Published Events#

This release publishes the events listed below. To identify an event you have decoded, find its module signature and module event code in the lookup table, then follow the event name to its entry. Not all Xids emitted by the driver have been fully converted to GPU Operational Events yet; more support will be added in future releases. To determine the actions to take, look up the event’s legacy Xid in the Xid Errors catalog.

Where severity is shown as FATAL or RECOVERABLE, the driver selects the severity at runtime according to the impact of the reported condition. Some events emit more than one legacy Xid; these are listed together. Event-specific context may be opaque, depending on the output format.

Event Lookup#

Module	Code	Event	Legacy Xid	Category
`GPU`	`0x0001`	DRIVER_INIT	None	`INITIALIZATION`
`GPU-BUS`	`0x0002`	C2C_CONTAINMENT	179	`INTERCONNECT_ERROR`
`GPU-BUS`	`0x0003`	GPU_FALLEN_OFF_BUS	79	`INTERCONNECT_ERROR`
`GPU-FSP`	`0x0001`	BOOT_TIMEOUT	143	`TIMEOUT`
`GPU-FSP`	`0x0002`	FUSE_ERROR	143	`FIRMWARE_FAULT`
`GPU-GSP`	`0x0001`	RPC_TIMEOUT	119	`TIMEOUT`
`GPU-GSP`	`0x0002`	LIBOS_HEARTBEAT_TIMEOUT	120	`TIMEOUT`
`GPU-GSP`	`0x0003`	GSP_RM_HEARTBEAT_TIMEOUT	120	`TIMEOUT`
`GPU-GSP`	`0x0004`	FIRMWARE_FAULT	120	`FIRMWARE_FAULT`
`GPU-GSP`	`0x0005`	POISON	140	`MEMORY_INTEGRITY_ERROR`
`GPU-MEMSYS`	`0x0001`	MAINT_OP_ERROR	175	`MEMORY_INTEGRITY_ERROR`
`GPU-NVLINK`	`0x0001`	ALI_TRAINING_FAILURE	136	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0001`	IR2SAW_INTERRUPT	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0002`	MSE_DEGRADED	150	`FIRMWARE_FAULT`
`GPU-NVLINK`	`0x0002`	CROSS_CONTAIN	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0003`	MSE_WATCHDOG_TIMEOUT	150	`FIRMWARE_FAULT`
`GPU-NVLINK`	`0x0003`	ENCRYPT_ERROR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0004`	SW_LINK_DOWN	155	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0004`	REMOTE_HW_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0005`	FATAL_ERROR_NVL3	74	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0005`	LOCAL_STO_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0006`	PRIV_ERROR	137	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0006`	LOCAL_PRIV_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0007`	SAW_ERROR_GPU_FATAL	144	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0007`	REMOTE_PRIV_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0008`	SAW_ERROR_NON_FATAL_WARNING	144	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0008`	DMEM_PARITY_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0009`	SAW_ERROR_NON_FATAL_CONTAINED	144	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0009`	IMEM_PARITY_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000A`	RLW_ERROR_GPU_FATAL	145	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000A`	MPU_PARITY_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000B`	RLW_ERROR_NON_FATAL_WARNING	145	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000B`	REG_PARITY_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000C`	RLW_ERROR_APPLICATION_FATAL	145	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000C`	TRAFFIC_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000D`	TLW_ERROR_GPU_FATAL	146	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000D`	ICACHE_PARITY_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000E`	TLW_ERROR_NON_FATAL_WARNING	146	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000E`	DCACHE_PARITY_ERR	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000F`	TLW_ERROR_NON_FATAL_CONTAINED	146	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x000F`	IR2SAW_RECOVERABLE	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0010`	TREX_ERROR_NON_FATAL_WARNING	147	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0010`	CROSS_CONTAIN_RECOVERABLE	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0011`	NVLPW_ERROR_NON_FATAL_WARNING	148	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0011`	LOCAL_STO_ERR_RECOVERABLE	170	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0012`	NETIR_ERROR_NON_FATAL_WARNING	149	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0013`	NETIR_ERROR_GPU_FATAL	149	`INTERCONNECT_ERROR`
`GPU-NVLINK`	`0x0014`	NETIR_ERROR_LINK_FATAL	149	`INTERCONNECT_ERROR`
`GPU-PCIE`	`0x0001`	FATAL_TIMEOUT	167	`INTERCONNECT_ERROR`
`GPU-PCIE`	`0x0002`	CORRECTABLE_ERROR_FAULT	176	`INTERCONNECT_ERROR`
`GPU-PCIE`	`0x0003`	RESET_FAILURE	182	`INTERCONNECT_ERROR`
`GPU-PSHC`	`0x0001`	PSHC_REENGAGED	162	`STATE_CHANGED`
`GPU-PSHC`	`0x0002`	PSHC_DISENGAGED	163	`STATE_CHANGED`
`GPU-PSHC`	`0x0003`	PSHC_LOW_LIFETIME	164	`RESOURCE_THRESHOLD`
`GPU-PSHC`	`0x0004`	PSHC_ZERO_LIFETIME	165	`RESOURCE_THRESHOLD`
`GPU-RAS`	`0x0001`	ROW_REMAPPING_PENDING	63	`RESOURCE_RETIREMENT`
`GPU-RAS`	`0x0002`	ROW_REMAPPING_FAILURE_TABLE_FULL	64	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x0003`	ROW_REMAPPING_FAILURE_BANK_FULL	64	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x0004`	ROW_REMAPPING_FAILURE_INTERNAL_ERROR	64	`UNCLASSIFIED_ERROR`
`GPU-RAS`	`0x0005`	ROW_REMAPPING_FAILURE_RESERVED_ROW	64	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x0006`	ROW_REMAPPING_FAILURE_PAGE_OFFLINE_FAILURE	64	`UNCLASSIFIED_ERROR`
`GPU-RAS`	`0x0007`	DRAM_RETIREMENT_FAILURE_INTERNAL_ERROR	64	`UNCLASSIFIED_ERROR`
`GPU-RAS`	`0x0008`	DRAM_RETIREMENT_FAILURE_INFOROM_FULL	64	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x0009`	DRAM_RETIREMENT_FAILURE_HW_LIMIT	64	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x000A`	DRAM_RETIREMENT_FAILURE_NO_SPARE	64	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x000B`	LTS_REPAIR_PENDING	160	`RESOURCE_RETIREMENT`
`GPU-RAS`	`0x000C`	LTS_REPAIR_FAILURE	161	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x000D`	MEMORY_CHANNEL_REPAIR_PENDING	160	`RESOURCE_RETIREMENT`
`GPU-RAS`	`0x000E`	MEMORY_CHANNEL_REPAIR_FAILURE	161	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x000F`	SPARING_REPAIR_PENDING	156	`RESOURCE_RETIREMENT`
`GPU-RAS`	`0x0011`	SPARING_REPAIR_NOT_POSSIBLE	157	`RESOURCE_EXHAUSTED`
`GPU-RAS`	`0x0013`	ECC_RESIDUAL_UNCORRECTABLE_ERROR	140	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0014`	DRAM_ECC_INTR_STORM	92	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0015`	SM_ECC_INTR_STORM	92	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0016`	BANK_REMAPPING_PENDING	177	`RESOURCE_RETIREMENT`
`GPU-RAS`	`0x0017`	LTC_FATAL_ERROR	95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0018`	LRC_FATAL_ERROR	48, 172, 95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0019`	DRAM_ERROR	48, 171	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x001A`	DRAM_SIDEBAND_ERROR	48, 171	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x001B`	DRAM_CBC_FATAL_ERROR	48, 171, 95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x001C`	DRAM_CBC_NON_FATAL_ERROR	48, 171, 94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x001D`	LTC_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x001E`	LTC_UNSUPPORTED_CLIENT_POISON_UNCONTAINED_ERROR	95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x001F`	LTC_UNSUPPORTED_CLIENT_POISON_CONTAINED_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0020`	PCIE_INGRESS_POISON_CONTAINED_ERROR	94	`INTERCONNECT_ERROR`
`GPU-RAS`	`0x0021`	PCIE_INGRESS_POISON_UNCONTAINED_ERROR	95	`INTERCONNECT_ERROR`
`GPU-RAS`	`0x0022`	PCIE_EGRESS_POISON_ERROR	95	`INTERCONNECT_ERROR`
`GPU-RAS`	`0x0023`	FBHUB_FATAL_POISON_ERROR	95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0024`	FBHUB_NON_FATAL_POISON_ERROR	95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0025`	MMU_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0026`	GSP_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0027`	COPY_ENGINE_POISON_CONTAINED_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0028`	COPY_ENGINE_POISON_UNCONTAINED_ERROR	95	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0029`	PMU_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x002A`	SEC2_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x002B`	FBFALCON_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x002C`	VIDEO_ENGINE_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x002D`	SM_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x002E`	CONTEXT_SWITCH_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x002F`	GCC_POISON_ERROR	94	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0030`	PCIE_UNCORRECTABLE_ERROR	48, 172	`INTERCONNECT_ERROR`
`GPU-RAS`	`0x0031`	COPY_ENGINE_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0032`	MMU_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0033`	FECS_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0034`	SM_CONTAINED_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0035`	GCC_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0036`	CONTEXT_SWITCH_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0037`	GPCMMU_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0038`	GSP_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x0039`	LPSEC_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x003A`	NVLINK_UNCORRECTABLE_ERROR	48, 172	`INTERCONNECT_ERROR`
`GPU-RAS`	`0x003B`	PMU_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x003C`	SEC2_UNCORRECTABLE_ERROR	48, 172	`MEMORY_INTEGRITY_ERROR`
`GPU-RAS`	`0x003D`	SPARING_EVENT	185	`STATE_CHANGED`
`GPU-RC`	`0x0001`	CHANNEL_RESET	Varies	`UNCLASSIFIED_ERROR`

Event Details#

Entries are grouped by source module signature, then ordered by category value and module event code. NVLink secure events follow the other NVLink events.

GPU#

DRIVER_INIT#

Code:: 0x0001
Category:: INITIALIZATION
Severity:: INFORMATIONAL
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: None

GPU driver initialization completed for the device. Carries driver and device metadata.

GPU-BUS#

C2C_CONTAINMENT#

Code:: 0x0002
Category:: INTERCONNECT_ERROR
Severity:: FATAL or RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 179

C2C containment was triggered on the chip-to-chip link to the host CPU. Carries the containment code.

GPU_FALLEN_OFF_BUS#

Code:: 0x0003
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 79

The GPU stopped responding to register reads and is no longer accessible on the bus. Only emitted on Unix platforms.

GPU-FSP#

BOOT_TIMEOUT#

Code:: 0x0001
Category:: TIMEOUT
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 143

The driver timed out polling for FSP boot completion during GPU initialization.

FUSE_ERROR#

Code:: 0x0002
Category:: FIRMWARE_FAULT
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 143

The FSP fuse error check failed. Carries the fuse status.

GPU-GSP#

RPC_TIMEOUT#

Code:: 0x0001
Category:: TIMEOUT
Severity:: FATAL or RECOVERABLE
Log level:: ERROR or WARNING
Scope:: DEVICE
Legacy Xid:: 119

The driver timed out waiting for an RPC response from GSP. Carries the RPC function, name, sequence number, and wait duration.

LIBOS_HEARTBEAT_TIMEOUT#

Code:: 0x0002
Category:: TIMEOUT
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 120

The LibOS heartbeat from the GSP microcontroller stopped. Emitted only when GSP timeout classification is enabled; disabled by default.

GSP_RM_HEARTBEAT_TIMEOUT#

Code:: 0x0003
Category:: TIMEOUT
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 120

The GSP-RM heartbeat stopped. Emitted only when GSP timeout classification is enabled; disabled by default.

POISON#

Code:: 0x0005
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 140

GSP consumed poisoned data.

FIRMWARE_FAULT#

Code:: 0x0004
Category:: FIRMWARE_FAULT
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 120

GSP firmware reported a fault.

GPU-MEMSYS#

MAINT_OP_ERROR#

Code:: 0x0001
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 175

A GPU memory subsystem maintenance operation timed out. Carries the operation type.

GPU-NVLINK#

The SAW, RLW, TLW, TREX, NVLPW, and NETIR events carry the link ID, interrupt information, error status, and debug registers. Their severity and log level distinguish warnings, contained errors, and fatal errors.

ALI_TRAINING_FAILURE#

Code:: 0x0001
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 136

NVLink ALI link training failed. Carries the affected link IDs and debug registers.

SW_LINK_DOWN#

Code:: 0x0004
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 155

An NVLink went down while it was active. Carries the link ID.

FATAL_ERROR_NVL3#

Code:: 0x0005
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 74

A fatal NVLink error occurred. Carries the link ID and debug registers.

PRIV_ERROR#

Code:: 0x0006
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 137

An NVLink privileged-register access failed. Carries the link ID.

SAW_ERROR_GPU_FATAL#

Code:: 0x0007
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 144

A fatal NVLink SAW error requires a GPU reset.

SAW_ERROR_NON_FATAL_WARNING#

Code:: 0x0008
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 144

NVLink SAW reported a nonfatal warning.

SAW_ERROR_NON_FATAL_CONTAINED#

Code:: 0x0009
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 144

A contained, recoverable NVLink SAW error occurred.

RLW_ERROR_GPU_FATAL#

Code:: 0x000A
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 145

A fatal NVLink RLW error requires a GPU reset.

RLW_ERROR_NON_FATAL_WARNING#

Code:: 0x000B
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 145

NVLink RLW reported a nonfatal warning.

RLW_ERROR_APPLICATION_FATAL#

Code:: 0x000C
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 145

An NVLink RLW error affected an application while the GPU remains recoverable.

TLW_ERROR_GPU_FATAL#

Code:: 0x000D
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 146

A fatal NVLink TLW error requires a GPU reset.

TLW_ERROR_NON_FATAL_WARNING#

Code:: 0x000E
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 146

NVLink TLW reported a nonfatal warning.

TLW_ERROR_NON_FATAL_CONTAINED#

Code:: 0x000F
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 146

A contained, recoverable NVLink TLW error occurred.

TREX_ERROR_NON_FATAL_WARNING#

Code:: 0x0010
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 147

NVLink TREX reported a nonfatal warning.

NVLPW_ERROR_NON_FATAL_WARNING#

Code:: 0x0011
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 148

NVLink NVLPW reported a nonfatal warning.

NETIR_ERROR_NON_FATAL_WARNING#

Code:: 0x0012
Category:: INTERCONNECT_ERROR
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 149

NVLink NETIR reported a nonfatal warning.

NETIR_ERROR_GPU_FATAL#

Code:: 0x0013
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 149

A fatal NVLink NETIR error requires a GPU reset.

NETIR_ERROR_LINK_FATAL#

Code:: 0x0014
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 149

A fatal NVLink NETIR error affected a link.

MSE_DEGRADED#

Code:: 0x0002
Category:: FIRMWARE_FAULT
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 150

The NVLink MSE transport entered a degraded state. The driver stops MSE communication until it is reloaded.

MSE_WATCHDOG_TIMEOUT#

Code:: 0x0003
Category:: FIRMWARE_FAULT
Severity:: FATAL or RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 150

The NVLink MSE watchdog expired. Carries the error status and debug registers.

The following secure events carry the secure interrupt status, link ID, error-log enable bits, and error status.

IR2SAW_INTERRUPT#

Code:: 0x0001
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA501

A fatal NVLink IR2SAW secure interrupt occurred.

CROSS_CONTAIN#

Code:: 0x0002
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA502

A fatal NVLink cross-containment error occurred.

ENCRYPT_ERROR#

Code:: 0x0003
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA503

An NVLink encryption error occurred.

REMOTE_HW_ERR#

Code:: 0x0004
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA504

An NVLink remote hardware error occurred.

LOCAL_STO_ERR#

Code:: 0x0005
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA505

An NVLink local STO error occurred.

LOCAL_PRIV_ERR#

Code:: 0x0006
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA506

An NVLink local privileged-register error occurred.

REMOTE_PRIV_ERR#

Code:: 0x0007
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA507

An NVLink remote privileged-register error occurred.

DMEM_PARITY_ERR#

Code:: 0x0008
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA508

An NVLink data-memory parity error occurred.

IMEM_PARITY_ERR#

Code:: 0x0009
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA509

An NVLink instruction-memory parity error occurred.

MPU_PARITY_ERR#

Code:: 0x000A
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA50A

An NVLink MPU parity error occurred.

REG_PARITY_ERR#

Code:: 0x000B
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA50B

An NVLink register parity error occurred.

TRAFFIC_ERR#

Code:: 0x000C
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA50C

An NVLink secure-traffic error occurred.

ICACHE_PARITY_ERR#

Code:: 0x000D
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA50D

An NVLink instruction-cache parity error occurred.

DCACHE_PARITY_ERR#

Code:: 0x000E
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA50E

An NVLink data-cache parity error occurred.

IR2SAW_RECOVERABLE#

Code:: 0x000F
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA501

A recoverable NVLink IR2SAW secure interrupt occurred.

CROSS_CONTAIN_RECOVERABLE#

Code:: 0x0010
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA502

A recoverable NVLink cross-containment error occurred.

LOCAL_STO_ERR_RECOVERABLE#

Code:: 0x0011
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 170
Context type:: 0xA505

A recoverable NVLink local STO error occurred.

GPU-PCIE#

FATAL_TIMEOUT#

Code:: 0x0001
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 167

A fatal timeout occurred in the PCIe endpoint layer. Carries the error type and register value.

CORRECTABLE_ERROR_FAULT#

Code:: 0x0002
Category:: INTERCONNECT_ERROR
Severity:: CORRECTED
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 176

The PCIe correctable error rate exceeded a threshold. Carries the error-rate tuple identifiers. Native CPER output omits the event-specific and legacy Xid contexts.

RESET_FAILURE#

Code:: 0x0003
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 182

A PCIe reset failed. Carries the reset status and recovery action.

GPU-PSHC#

PSHC_REENGAGED#

Code:: 0x0001
Category:: STATE_CHANGED
Severity:: INFORMATIONAL
Log level:: DIAGNOSTIC
Scope:: DEVICE
Legacy Xid:: 162

The power-smoothing controller reengaged.

PSHC_DISENGAGED#

Code:: 0x0002
Category:: STATE_CHANGED
Severity:: INFORMATIONAL
Log level:: DIAGNOSTIC
Scope:: DEVICE
Legacy Xid:: 163

The power-smoothing controller disengaged because of a slowdown, timeout, or exhausted lifetime. Carries the reason and, for a slowdown, the slowdown mask.

PSHC_LOW_LIFETIME#

Code:: 0x0003
Category:: RESOURCE_THRESHOLD
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 164

The power-smoothing controller lifetime fell below a threshold. Carries the threshold percentage.

PSHC_ZERO_LIFETIME#

Code:: 0x0004
Category:: RESOURCE_THRESHOLD
Severity:: INFORMATIONAL
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 165

The power-smoothing controller lifetime reached zero. The controller will be disabled.

GPU-RAS#

ECC_RESIDUAL_UNCORRECTABLE_ERROR#

Code:: 0x0013
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL or RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 140

An uncorrectable ECC error was detected outside the normal handling path.

DRAM_ECC_INTR_STORM#

Code:: 0x0014
Category:: MEMORY_INTEGRITY_ERROR
Severity:: CORRECTED
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 92

Correctable ECC error interrupts were disabled on a framebuffer partition because of a high error rate.

SM_ECC_INTR_STORM#

Code:: 0x0015
Category:: MEMORY_INTEGRITY_ERROR
Severity:: CORRECTED
Log level:: WARNING
Scope:: DEVICE
Legacy Xid:: 92

An SM correctable ECC interrupt storm was detected.

LTC_FATAL_ERROR#

Code:: 0x0017
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 95

An uncontained L2 data, tag, or compression-tag error requires a GPU reset.

LRC_FATAL_ERROR#

Code:: 0x0018
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172, 95

An uncontained LRC SRAM error requires a GPU reset.

DRAM_ERROR#

Code:: 0x0019
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 171

An uncorrectable DRAM error occurred.

DRAM_SIDEBAND_ERROR#

Code:: 0x001A
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 171

An uncorrectable DRAM sideband error occurred.

DRAM_CBC_FATAL_ERROR#

Code:: 0x001B
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 171, 95

An uncontained DRAM compression-tag error requires a GPU reset.

DRAM_CBC_NON_FATAL_ERROR#

Code:: 0x001C
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 171, 94

A contained DRAM compression-tag error occurred.

LTC_ERROR#

Code:: 0x001D
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable L2 SRAM error occurred.

LTC_UNSUPPORTED_CLIENT_POISON_UNCONTAINED_ERROR#

Code:: 0x001E
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 95

An L2 client that does not support poison reported an uncontained error.

LTC_UNSUPPORTED_CLIENT_POISON_CONTAINED_ERROR#

Code:: 0x001F
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 94

An L2 client that does not support poison reported a contained error.

FBHUB_FATAL_POISON_ERROR#

Code:: 0x0023
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 95

An uncontained framebuffer hub poison error requires a GPU reset.

FBHUB_NON_FATAL_POISON_ERROR#

Code:: 0x0024
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 95

A framebuffer hub poison error occurred without requiring an immediate GPU reset.

MMU_POISON_ERROR#

Code:: 0x0025
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

The MMU reported a contained poison error.

GSP_POISON_ERROR#

Code:: 0x0026
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 94

GSP reported a contained poison error that requires a GPU reset.

COPY_ENGINE_POISON_CONTAINED_ERROR#

Code:: 0x0027
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

A copy-engine user channel reported a contained poison error.

COPY_ENGINE_POISON_UNCONTAINED_ERROR#

Code:: 0x0028
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 95

A copy-engine kernel channel reported an uncontained poison error requiring a GPU reset.

PMU_POISON_ERROR#

Code:: 0x0029
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

PMU reported a contained poison error that requires a GPU reset.

SEC2_POISON_ERROR#

Code:: 0x002A
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

SEC2 reported a contained poison error that requires a GPU reset.

FBFALCON_POISON_ERROR#

Code:: 0x002B
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

The framebuffer Falcon reported a contained poison error that requires a GPU reset.

VIDEO_ENGINE_POISON_ERROR#

Code:: 0x002C
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

A video engine reported a contained poison error.

SM_POISON_ERROR#

Code:: 0x002D
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

An SM reported a contained poison error.

CONTEXT_SWITCH_POISON_ERROR#

Code:: 0x002E
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

The context-switch engine reported a contained poison error.

GCC_POISON_ERROR#

Code:: 0x002F
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 94

GCC reported a contained poison error.

COPY_ENGINE_UNCORRECTABLE_ERROR#

Code:: 0x0031
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in a copy engine.

MMU_UNCORRECTABLE_ERROR#

Code:: 0x0032
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in the MMU.

FECS_UNCORRECTABLE_ERROR#

Code:: 0x0033
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in FECS.

SM_CONTAINED_UNCORRECTABLE_ERROR#

Code:: 0x0034
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 48, 172

A contained uncorrectable SRAM error occurred in an SM.

GCC_UNCORRECTABLE_ERROR#

Code:: 0x0035
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in GCC.

CONTEXT_SWITCH_UNCORRECTABLE_ERROR#

Code:: 0x0036
Category:: MEMORY_INTEGRITY_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in GPCCS.

GPCMMU_UNCORRECTABLE_ERROR#

Code:: 0x0037
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in GPCMMU.

GSP_UNCORRECTABLE_ERROR#

Code:: 0x0038
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in GSP-RISC-V.

LPSEC_UNCORRECTABLE_ERROR#

Code:: 0x0039
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in LPSEC.

PMU_UNCORRECTABLE_ERROR#

Code:: 0x003B
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in the PMU Falcon or RISC-V core.

SEC2_UNCORRECTABLE_ERROR#

Code:: 0x003C
Category:: MEMORY_INTEGRITY_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in SEC2-RISC-V.

PCIE_INGRESS_POISON_CONTAINED_ERROR#

Code:: 0x0020
Category:: INTERCONNECT_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 94

A contained PCIe ingress poison error occurred.

PCIE_INGRESS_POISON_UNCONTAINED_ERROR#

Code:: 0x0021
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 95

An uncontained PCIe ingress poison error requires a GPU reset.

PCIE_EGRESS_POISON_ERROR#

Code:: 0x0022
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 95

An uncontained PCIe egress poison error requires a GPU reset.

PCIE_UNCORRECTABLE_ERROR#

Code:: 0x0030
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in PCIe.

NVLINK_UNCORRECTABLE_ERROR#

Code:: 0x003A
Category:: INTERCONNECT_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 48, 172

An uncorrectable SRAM error occurred in NVLink.

ROW_REMAPPING_FAILURE_TABLE_FULL#

Code:: 0x0002
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

Row remapping failed because the row remapping table is full.

ROW_REMAPPING_FAILURE_BANK_FULL#

Code:: 0x0003
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

Row remapping failed because all reserved rows for the bank are already remapped.

ROW_REMAPPING_FAILURE_RESERVED_ROW#

Code:: 0x0005
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

Row remapping failed because the target is a reserved row.

DRAM_RETIREMENT_FAILURE_INFOROM_FULL#

Code:: 0x0008
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

DRAM retirement failed because the InfoROM object is full.

DRAM_RETIREMENT_FAILURE_HW_LIMIT#

Code:: 0x0009
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

DRAM retirement failed because a hardware limit was reached.

DRAM_RETIREMENT_FAILURE_NO_SPARE#

Code:: 0x000A
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

DRAM retirement failed because no spare remains for retirement.

LTS_REPAIR_FAILURE#

Code:: 0x000C
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 161

L2 slice repair failed because no spare L2 slices remain.

MEMORY_CHANNEL_REPAIR_FAILURE#

Code:: 0x000E
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 161

Memory channel repair failed because no spare channels remain.

SPARING_REPAIR_NOT_POSSIBLE#

Code:: 0x0011
Category:: RESOURCE_EXHAUSTED
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 157

A repair using spare resources is not possible.

ROW_REMAPPING_FAILURE_INTERNAL_ERROR#

Code:: 0x0004
Category:: UNCLASSIFIED_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

Row remapping failed due to an internal driver error.

ROW_REMAPPING_FAILURE_PAGE_OFFLINE_FAILURE#

Code:: 0x0006
Category:: UNCLASSIFIED_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

The row is already pending remapping. The remap occurs on the next GPU reset.

DRAM_RETIREMENT_FAILURE_INTERNAL_ERROR#

Code:: 0x0007
Category:: UNCLASSIFIED_ERROR
Severity:: FATAL
Log level:: ERROR
Scope:: DEVICE
Legacy Xid:: 64

DRAM retirement failed due to an internal driver error.

SPARING_EVENT#

Code:: 0x003D
Category:: STATE_CHANGED
Severity:: RECOVERABLE
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 185

A sparing operation was reported.

ROW_REMAPPING_PENDING#

Code:: 0x0001
Category:: RESOURCE_RETIREMENT
Severity:: RECOVERABLE
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 63

A new DRAM row was marked for remapping. The remap activates on the next GPU reset.

LTS_REPAIR_PENDING#

Code:: 0x000B
Category:: RESOURCE_RETIREMENT
Severity:: RECOVERABLE
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 160

An L2 slice and its pair were marked for repair. The Xid 160 message gives the location and the action needed to activate the repair.

MEMORY_CHANNEL_REPAIR_PENDING#

Code:: 0x000D
Category:: RESOURCE_RETIREMENT
Severity:: RECOVERABLE
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 160

A memory channel and its pair were marked for repair. The Xid 160 message gives the location and the action needed to activate the repair.

SPARING_REPAIR_PENDING#

Code:: 0x000F
Category:: RESOURCE_RETIREMENT
Severity:: RECOVERABLE
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 156

A repair using spare resources is pending.

BANK_REMAPPING_PENDING#

Code:: 0x0016
Category:: RESOURCE_RETIREMENT
Severity:: RECOVERABLE
Log level:: NOTICE
Scope:: DEVICE
Legacy Xid:: 177

A new DRAM bank was marked for remapping. The remap activates on the next GPU reset.

GPU-RC#

CHANNEL_RESET#

Code:: 0x0001
Category:: UNCLASSIFIED_ERROR
Severity:: RECOVERABLE
Log level:: ERROR
Scope:: ENGINE
Legacy Xid:: Varies

Robust Channel recovery reset a channel or engine. The Legacy Xid context identifies the originating Xid. Additional recovery details are carried in an opaque context.