GPU Operational Event Formats#

Warning

GPU Operational Events are a preview feature and remain under active development. Category values, module signatures, module event codes, severities, context payloads, and the set of published events may change before the interface is declared stable. Treat the identifiers in this document as valid for the driver release documented here, and re-validate consumers against each driver release. Legacy Xid reporting remains supported for production monitoring during the preview period. See Release Notes for compatibility changes.

GPU Operational Events are published in several formats. These describe the same underlying events, but the context data available in each format can differ.

Format	Typical Consumers
Structured Event Records	NVML subscribers
UEFI CPER	BMC collection, `nvidia-smi event-log`, `nvidia-smi cper`, and `nvmlSystemGetCPER_v1`
Linux Tracepoints	Kernel tracing with the open GPU kernel modules
Legacy Xid Kernel Log Lines	Existing `dmesg` and syslog parsers

See Analyzing GPU Operational Events for how to monitor events and how to interpret categories and published codes.

Structured Event Records#

Structured records expose the event identity and attribution as discrete fields. Through NVML, a successful nvmlEventSetWait_v3 returns records whose dataType is NVML_EVENT_DATA_TYPE_GPU_OPERATIONAL_EVENT. Those records carry:

Module signature, category, and module event code
Severity, log level, scope, and originator
Device UUID and attribution
Trace ID, group cursor, group position, and instance ID
Timestamp and per-event / group attributes

groupSize counts the events visible to the subscriber’s scope, and groupIndex is their zero-based position within that group.

Context payloads follow the event. Each context has a type, a format version, and a payload size.

Domain-independent types carry generically decodable data:

Type	Contents
`0x0000`	Opaque bytes
`0x0001`	Key/value pairs, 64-bit values
`0x0002`	Key/value pairs, 32-bit values
`0x0003`	Value array, 64-bit values
`0x0004`	Value array, 32-bit values

GPU domain types begin at 0x8001; 0x8000 is reserved. The types defined in this release are:

Type	Name	Contents
`0x8001`	Legacy Xid	Legacy Xid code and formatted message
`0x9001`	GPU Timeout Data	Timeout-specific fields such as waited duration
`0xA001`	GPU Initialization Metadata	Driver and device metadata reported at initialization
`0xA101`	GSP RPC Timeout Data	RPC function, name, sequence number, and wait duration
`0xA201`	FSP Boot Timeout Data	FSP boot polling state
`0xA202`	FSP Fuse Error Data	FSP fuse status
`0xA301`	NVLink ALI Training Failure Data	Affected link IDs and debug registers
`0xA302`	NVLink MSE Error Data	MSE error status and debug registers
`0xA303`	NVLink Software Link-Down Data	Link ID and eight unused debug words
`0xA304`	NVLink NVL3 Fatal Error Data	Link ID and seven debug words
`0xA305`	NVLink Privilege Error Data	Link ID
`0xA306`	NVLink Blackwell Error Data	Link ID (`-1` if not link-specific), error status, interrupt information, and eight debug words
`0xA307`	NVLink SAW Error Data	64 reserved bytes
`0xA308`	NVLink RLW Error Data	64 reserved bytes
`0xA309`	NVLink TLW Error Data	64 reserved bytes
`0xA30A`	NVLink TREX Error Data	64 reserved bytes
`0xA30B`	NVLink NVLPW Control Error Data	64 reserved bytes
`0xA30C`	NVLink NETIR Error Data	64 reserved bytes
`0xA501`	Secure NVLink IR2SAW Interrupt Data	Secure NVLink error fields
`0xA502`	Secure NVLink Cross-Containment Data	Secure NVLink error fields
`0xA503`	Secure NVLink Encryption Error Data	Secure NVLink error fields
`0xA504`	Secure NVLink Remote Hardware Error Data	Secure NVLink error fields
`0xA505`	Secure NVLink Local STO Error Data	Secure NVLink error fields
`0xA506`	Secure NVLink Local Privilege Error Data	Secure NVLink error fields
`0xA507`	Secure NVLink Remote Privilege Error Data	Secure NVLink error fields
`0xA508`	Secure NVLink DMEM Parity Error Data	Secure NVLink error fields
`0xA509`	Secure NVLink IMEM Parity Error Data	Secure NVLink error fields
`0xA50A`	Secure NVLink MPU Parity Error Data	Secure NVLink error fields
`0xA50B`	Secure NVLink Register Parity Error Data	Secure NVLink error fields
`0xA50C`	Secure NVLink Traffic Error Data	Secure NVLink error fields
`0xA50D`	Secure NVLink Instruction Cache Parity Error Data	Secure NVLink error fields
`0xA50E`	Secure NVLink Data Cache Parity Error Data	Secure NVLink error fields
`0xA601`	PCIe Fatal Timeout Data	Timeout source and register value
`0xA602`	PCIe Correctable Error Fault Data	Correctable-error rate tuple number, NERD number, and tuple version
`0xA603`	PCIe Reset Failure Data	Reset status and recommended recovery action

Private contexts use opaque type 0x0000 and do not have a published field layout.

nvidia-smi event-log prints structured events in human-readable form.

Linux Tracepoints#

R620 adds five tracepoints in the nvidia trace system: nvidia_gpu_error, nvidia_gpu_warning, nvidia_gpu_notice, nvidia_gpu_diagnostic, and nvidia_gpu_telemetry. The event’s log level selects the tracepoint. These are available with the open GPU kernel modules.

The raw trace record carries the PCI device, UUID, module signature and event code, category, severity, scope, originator, event and group attributes, log level, trace ID, instance ID, timestamp, and module, chiplet, and MIG attribution. It does not carry contexts or group cursor, size, or index. The formatted trace line shows a subset of those fields.

UEFI CPER#

GPU Operational Events are encoded as UEFI Common Platform Error Records. See Appendix N of the UEFI Specification for the CPER format. CPER is the format used for out-of-band delivery to a BMC and for retained history access through nvmlSystemGetCPER_v1 and nvidia-smi cper.

One CPER record represents one event group. Each event in the group becomes one section of type NVIDIA Event Section, GUID 9068e568-6ca0-11f0-aeaf-159343591eac. The Creator ID and Notification Type GUIDs listed below are version-5 GUIDs derived from the NVIDIA namespace ea219640-18b9-4c91-8e5c-58eeacd9b33f. Match these GUIDs by value.

The NVIDIA Event Section begins with a 32-byte common header, followed by a device-type-specific event info structure, followed by the context list. The event identity maps as follows:

CPER Field	Source	Notes
Source Device Type	`GPU` (1)	Selects the event info layout.
Event Type (`u16`)	Category	Value from the category registry.
Event Sub-type (`u16`)	Module Event Code	Specific event within the reporting module.
Source Module Signature (`char[16]`)	Module Signature	NUL-terminated ASCII string naming the source module.
Event Trace ID (`u64`)	Trace ID	Full 64-bit value including the originator prefix.
Error Record Header `recordId`	Group Cursor	The driver writes the group cursor into `recordId`, so the same value identifies the event group in both the structured record and the CPER record.

Event info for the GPU device type is 16 bytes, version 1.0 (0x0100). It adds the event originator, the device PDI, and source partition and subpartition fields reserved for future use. Each context has a 16-byte header, version 0.0, and is padded to 16-byte alignment. contextSize includes the header and padding; dataSize counts only the payload bytes.

CPER uses the same context type registry, but context encoding depends on the reporting path. Native event serializers encode most module-specific contexts as opaque type 0x0000, version 0. Legacy Xid and initialization metadata retain their typed encodings (0x8001 and 0xA001, version 1.0). Native PCIe correctable-error serialization emits the event section but omits its correctable-error and Legacy Xid contexts.

When the host imports a firmware event, CPER preserves the context types and versions from the imported structured record. A module context can therefore be typed on this path and opaque on another. Decode each context using its own header; use structured records for the published module-specific layouts when CPER supplies opaque data.

The Error Record Header identifies the producer through a Creator ID and describes how the condition was discovered through a Notification Type:

Role	GUID	Meaning
Creator ID	`5a572ae4-9a04-5e55-a86f-dbfdfc0165e8`	Record created by the host driver on the physical function.
Notification Type	`0eeaf365-3740-5271-ae58-10f3b7c09c4a`	Reported through a GPU hardware interrupt.
Notification Type	`e0fd0b2d-ba62-59c1-9744-9dc8bf313650`	Detected by on-device firmware fault handling.
Notification Type	`3a90eac6-481d-5cd7-b57d-0a92315bff8f`	Inferred from a timeout waiting on GPU hardware or firmware.
Notification Type	`cb2668a9-0508-5e36-bbc9-4d6c7d2dc808`	Detected by a software validation or invariant check.

GOE Severity to CPER Severity Mapping

GOE severity is evaluated at the event’s explicit scope, such as a device or engine. CPER severity uses the UEFI encoding — recoverable (0), fatal (1), corrected (2), informational (3) — and is always evaluated at the host scope. Encoding a GOE as CPER therefore re-scopes the impact to the host, so the CPER severity can differ from the GOE severity.

The driver maps GOE fatal and GOE recoverable to CPER recoverable, GOE corrected to CPER corrected, and GOE informational to CPER informational. GPU-fatal conditions leave the host recoverable, so the driver emits CPER recoverable for them and does not emit CPER fatal. The NVIDIA Event Section carries identity and attribution fields, while GOE severity and scope remain on the structured event record. For GPU-relative impact, read those fields from the structured record; they are not recoverable from the CPER severity alone.

Group attributes map to Error Record Header flags: recovered, previous error, and simulated.

Legacy Xid Kernel Log Lines#

Contexts of type 0x8001 carry a legacy Xid code and its formatted message. These produce the traditional kernel log line:

NVRM: Xid (PCI:0000:41:00): 63, Row Remapper: New row (0x00000000fbc12000) marked for remapping, reset gpu to activate.

Most events that report an Xid also deliver that code and message as a 0x8001 context inside structured records and CPER, so a programmatic consumer may read them without parsing dmesg.