GPU Operational Event Formats#

Warning

GPU Operational Events are a preview feature and remain under active development. Category values, module signatures, module event codes, severities, context payloads, and the set of published events may change before the interface is declared stable. Treat the identifiers in this document as valid for the driver release documented here, and re-validate consumers against each driver release. Legacy Xid output remains the stable contract for production monitoring during the preview period.

GPU Operational Events are published in several formats. Every format describes the same underlying events; consumers choose the format that fits their collection path.

Format

Typical Consumers

Structured Event Records

NVML subscribers and nvidia-smi event-log

UEFI CPER

Out-of-band BMC collection, nvidia-smi cper, and nvmlSystemGetCPER_v1

Legacy Xid Kernel Log Lines

Existing dmesg and syslog parsers

See Analyzing GPU Operational Events for how to monitor events and how to interpret categories and published codes.

Structured Event Records#

Structured records expose the event identity and attribution as discrete fields. Through NVML, a successful nvmlEventSetWait_v3 returns records whose dataType is NVML_EVENT_DATA_TYPE_GPU_OPERATIONAL_EVENT. Those records carry:

  • Module signature, category, and module event code

  • Severity, log level, scope, and originator

  • Device UUID and attribution

  • Trace ID, group cursor, group position, and instance ID

  • Timestamp and per-event / group attributes

Context payloads follow the event. Each context has a type and a size. Bit 15 of the type partitions the space: types with bit 15 clear are domain-independent encodings, and types with bit 15 set are specific to the GPU domain.

Domain-independent types carry generically decodable data:

Type

Contents

0x0000

Opaque bytes

0x0001

Key/value pairs, 64-bit values

0x0002

Key/value pairs, 32-bit values

0x0003

Value array, 64-bit values

0x0004

Value array, 32-bit values

GPU domain types begin at 0x8001; 0x8000 is reserved. The types defined in this release are:

Type

Name

Contents

0x8001

Legacy Xid

Legacy Xid code and formatted message

0x9001

GPU Timeout Data

Timeout-specific fields such as waited duration

0xA001

GPU Initialization Metadata

Driver and device metadata reported at initialization

0xA101

GSP RPC Timeout Data

RPC function, name, sequence number, and wait duration

0xA201

FSP Boot Timeout Data

FSP boot polling state

0xA202

FSP Fuse Error Data

FSP fuse status

0xA301

NVLink ALI Training Failure Data

Affected link IDs and debug registers

0xA302

NVLink MSE Error Data

MSE error status and debug registers

0xA303

NVLink Software Link-Down Data

Link ID of the link that went down

Types are allocated to reporting modules in blocks: 0xA000–0xA0FF for events common to the GPU, 0xA100–0xA1FF for GSP, 0xA200–0xA2FF for FSP, 0xA300–0xA3FF for NVLink, and 0xA400–0xA4FF for Robust Channel. A consumer that does not recognize a context type skips it using the size field, which keeps a consumer working across releases that add types.

Legacy Xid is the one context type that NVML decodes into named fields, through nvmlEventSetGetGpuOperationalEventContextLegacyXid_v1. NVML reports every other context through its raw-data interfaces.

nvidia-smi event-log renders the same structured fields in human-readable form.

UEFI CPER#

GPU Operational Events are encoded as UEFI Common Platform Error Records. See Appendix N of the UEFI Specification for the CPER format. CPER is the format used for out-of-band delivery to a BMC and for retained history access through nvmlSystemGetCPER_v1 and nvidia-smi cper.

One CPER record represents one event group. Each event in the group becomes one section of type NVIDIA Event Section, GUID 9068e568-6ca0-11f0-aeaf-159343591eac. The Creator ID and Notification Type GUIDs listed below are version-5 GUIDs derived from the NVIDIA namespace ea219640-18b9-4c91-8e5c-58eeacd9b33f. Match these GUIDs by value.

The NVIDIA Event Section begins with a 32-byte common header, followed by a device-type-specific event info structure, followed by the context list. The event identity maps as follows:

CPER Field

Source

Notes

Source Device Type

GPU (1)

Selects the event info layout.

Event Type (u16)

Category

Value from the category registry.

Event Sub-type (u16)

Module Event Code

Specific event within the reporting module.

Source Module Signature (char[16])

Module Signature

NUL-terminated ASCII string naming the source module.

Event Trace ID (u64)

Trace ID

Full 64-bit value including the originator prefix.

Error Record Header recordId

Group Cursor

The driver writes the group cursor into recordId, so the same value identifies the event group in both the structured record and the CPER record.

Event info for the GPU device type adds the event originator, the device PDI, and the source partition and subpartition that locate the event within the device. Each context is emitted with a 16-byte header and padded to 16-byte alignment; context format types match the structured record context types, so a legacy Xid context appears as CPER context format type 0x8001.

The Error Record Header identifies the producer through a Creator ID and describes how the condition was discovered through a Notification Type:

Role

GUID

Meaning

Creator ID

7d3f1537-bfc6-583a-997b-a754778e60c4

Record created by GSP firmware on the physical function.

Creator ID

5a572ae4-9a04-5e55-a86f-dbfdfc0165e8

Record created by the host driver on the physical function.

Notification Type

0eeaf365-3740-5271-ae58-10f3b7c09c4a

Reported through a GPU hardware interrupt.

Notification Type

e0fd0b2d-ba62-59c1-9744-9dc8bf313650

Detected by on-device firmware fault handling.

Notification Type

3a90eac6-481d-5cd7-b57d-0a92315bff8f

Inferred from a timeout waiting on GPU hardware or firmware.

Notification Type

cb2668a9-0508-5e36-bbc9-4d6c7d2dc808

Detected by a software validation or invariant check.

GOE Severity to CPER Severity Mapping

GOE severity is evaluated at the event’s explicit scope, such as a device or engine. CPER severity uses the UEFI encoding — recoverable (0), fatal (1), corrected (2), informational (3) — and is always evaluated at the host scope. Encoding a GOE as CPER therefore re-scopes the impact to the host, so the CPER severity can differ from the GOE severity.

The driver maps GOE fatal and GOE recoverable to CPER recoverable, GOE corrected to CPER corrected, and GOE informational to CPER informational. GPU-fatal conditions leave the host recoverable, so the driver emits CPER recoverable for them and does not emit CPER fatal. The NVIDIA Event Section carries identity and attribution fields, while GOE severity and scope remain on the structured event record. For GPU-relative impact, read those fields from the structured record; they are not recoverable from the CPER severity alone.

Group attributes map to Error Record Header flags: recovered, previous error, and simulated.

Legacy Xid Kernel Log Lines#

Contexts of type 0x8001 carry a legacy Xid code and its formatted message. These produce the traditional kernel log line:

NVRM: Xid (PCI:0000:41:00 GPU-I:03): 63, Row Remapper: New row (0x00000000fbc12000) marked for remapping, reset gpu to activate.

Most events that report an Xid also deliver that code and message as a 0x8001 context inside structured records and CPER, so a programmatic consumer may read them without parsing dmesg. The exception are some NVLink events in this release, which emit two separate GPU Operational Events for a single underlying occurrence: one that contains the event data formatted as an Xid context, and one that contains the event data as a structured NVLink event context. In that case, only one Xid line is emitted to kernel logs.