GPU Operational Event Formats#
Warning
GPU Operational Events are a preview feature and remain under active development. Category values, module signatures, module event codes, severities, context payloads, and the set of published events may change before the interface is declared stable. Treat the identifiers in this document as valid for the driver release documented here, and re-validate consumers against each driver release. Legacy Xid output remains the stable contract for production monitoring during the preview period.
GPU Operational Events are published in several formats. Every format describes the same underlying events; consumers choose the format that fits their collection path.
Format |
Typical Consumers |
|---|---|
Structured Event Records |
NVML subscribers and |
UEFI CPER |
Out-of-band BMC collection, |
Legacy Xid Kernel Log Lines |
Existing |
See Analyzing GPU Operational Events for how to monitor events and how to interpret categories and published codes.
Structured Event Records#
Structured records expose the event identity and attribution as discrete fields. Through NVML,
a successful nvmlEventSetWait_v3 returns records whose dataType is
NVML_EVENT_DATA_TYPE_GPU_OPERATIONAL_EVENT. Those records carry:
Module signature, category, and module event code
Severity, log level, scope, and originator
Device UUID and attribution
Trace ID, group cursor, group position, and instance ID
Timestamp and per-event / group attributes
Context payloads follow the event. Each context has a type and a size. Bit 15 of the type partitions the space: types with bit 15 clear are domain-independent encodings, and types with bit 15 set are specific to the GPU domain.
Domain-independent types carry generically decodable data:
Type |
Contents |
|---|---|
|
Opaque bytes |
|
Key/value pairs, 64-bit values |
|
Key/value pairs, 32-bit values |
|
Value array, 64-bit values |
|
Value array, 32-bit values |
GPU domain types begin at 0x8001; 0x8000 is reserved. The types defined in this release
are:
Type |
Name |
Contents |
|---|---|---|
|
Legacy Xid |
Legacy Xid code and formatted message |
|
GPU Timeout Data |
Timeout-specific fields such as waited duration |
|
GPU Initialization Metadata |
Driver and device metadata reported at initialization |
|
GSP RPC Timeout Data |
RPC function, name, sequence number, and wait duration |
|
FSP Boot Timeout Data |
FSP boot polling state |
|
FSP Fuse Error Data |
FSP fuse status |
|
NVLink ALI Training Failure Data |
Affected link IDs and debug registers |
|
NVLink MSE Error Data |
MSE error status and debug registers |
|
NVLink Software Link-Down Data |
Link ID of the link that went down |
Types are allocated to reporting modules in blocks: 0xA000–0xA0FF for events common to
the GPU, 0xA100–0xA1FF for GSP, 0xA200–0xA2FF for FSP, 0xA300–0xA3FF for
NVLink, and 0xA400–0xA4FF for Robust Channel. A consumer that does not recognize a
context type skips it using the size field, which keeps a consumer working across releases that
add types.
Legacy Xid is the one context type that NVML decodes into named fields, through
nvmlEventSetGetGpuOperationalEventContextLegacyXid_v1. NVML reports every other context
through its raw-data interfaces.
nvidia-smi event-log renders the same structured fields in human-readable form.
UEFI CPER#
GPU Operational Events are encoded as UEFI Common Platform Error Records. See
Appendix N of the UEFI Specification
for the CPER format. CPER is the format used for out-of-band delivery to a BMC and for retained
history access through nvmlSystemGetCPER_v1 and nvidia-smi cper.
One CPER record represents one event group. Each event in the group becomes one section of type
NVIDIA Event Section, GUID 9068e568-6ca0-11f0-aeaf-159343591eac. The Creator ID and
Notification Type GUIDs listed below are version-5 GUIDs derived from the NVIDIA namespace
ea219640-18b9-4c91-8e5c-58eeacd9b33f. Match these GUIDs by value.
The NVIDIA Event Section begins with a 32-byte common header, followed by a device-type-specific event info structure, followed by the context list. The event identity maps as follows:
CPER Field |
Source |
Notes |
|---|---|---|
Source Device Type |
|
Selects the event info layout. |
Event Type ( |
Category |
Value from the category registry. |
Event Sub-type ( |
Module Event Code |
Specific event within the reporting module. |
Source Module Signature ( |
Module Signature |
NUL-terminated ASCII string naming the source module. |
Event Trace ID ( |
Trace ID |
Full 64-bit value including the originator prefix. |
Error Record Header |
Group Cursor |
The driver writes the group cursor into |
Event info for the GPU device type adds the event originator, the device PDI, and the source
partition and subpartition that locate the event within the device. Each context is emitted with
a 16-byte header and padded to 16-byte alignment;
context format types match the structured record context types, so a legacy Xid context appears
as CPER context format type 0x8001.
The Error Record Header identifies the producer through a Creator ID and describes how the condition was discovered through a Notification Type:
Role |
GUID |
Meaning |
|---|---|---|
Creator ID |
|
Record created by GSP firmware on the physical function. |
Creator ID |
|
Record created by the host driver on the physical function. |
Notification Type |
|
Reported through a GPU hardware interrupt. |
Notification Type |
|
Detected by on-device firmware fault handling. |
Notification Type |
|
Inferred from a timeout waiting on GPU hardware or firmware. |
Notification Type |
|
Detected by a software validation or invariant check. |
GOE Severity to CPER Severity Mapping
GOE severity is evaluated at the event’s explicit scope, such as a device or engine. CPER severity uses the UEFI encoding — recoverable (0), fatal (1), corrected (2), informational (3) — and is always evaluated at the host scope. Encoding a GOE as CPER therefore re-scopes the impact to the host, so the CPER severity can differ from the GOE severity.
The driver maps GOE fatal and GOE recoverable to CPER recoverable, GOE corrected to CPER corrected, and GOE informational to CPER informational. GPU-fatal conditions leave the host recoverable, so the driver emits CPER recoverable for them and does not emit CPER fatal. The NVIDIA Event Section carries identity and attribution fields, while GOE severity and scope remain on the structured event record. For GPU-relative impact, read those fields from the structured record; they are not recoverable from the CPER severity alone.
Group attributes map to Error Record Header flags: recovered, previous error, and simulated.
Legacy Xid Kernel Log Lines#
Contexts of type 0x8001 carry a legacy Xid code and its formatted message. These produce the
traditional kernel log line:
NVRM: Xid (PCI:0000:41:00 GPU-I:03): 63, Row Remapper: New row (0x00000000fbc12000) marked for remapping, reset gpu to activate.
Most events that report an Xid also deliver that code and message as a 0x8001 context inside
structured records and CPER, so a programmatic consumer may read them without parsing dmesg.
The exception are some NVLink events in this release, which emit two separate GPU Operational
Events for a single underlying occurrence: one that contains the event data formatted as an Xid
context, and one that contains the event data as a structured NVLink event context. In that case,
only one Xid line is emitted to kernel logs.