Is this page helpful?

Analyzing GPU Operational Events#

Warning

GPU Operational Events are a preview feature and remain under active development. Category values, module signatures, module event codes, severities, context payloads, and the set of published events may change before the interface is declared stable. Treat the identifiers in this document as valid for the driver release documented here, and re-validate consumers against each driver release. Legacy Xid reporting remains supported for production monitoring during the preview period. See Release Notes for compatibility changes.

Overview#

A GPU Operational Event (GOE) is a structured, machine-readable record that the NVIDIA GPU driver publishes whenever it detects a condition worth reporting to a system operator: an error, a recovery action, a resource retirement, or a notable state change. Each event carries an identity, a severity, an impact scope, and a list of typed context payloads that describe the specific occurrence.

GPU Operational Events give system software a programmatic interface for the same conditions that the driver reports as Xid messages in the kernel log. Each event carries its identity in discrete numeric fields.

Events that have a legacy Xid equivalent continue to produce NVRM: Xid (PCI:...) kernel log lines. The structured event and the legacy Xid line describe the same occurrence. See Release Notes for changes to individual Xid messages.

For the actions to take in response to a condition, see the Xid Errors documentation. Each event in Published Events lists its legacy Xid, which selects the corresponding entry in the Xid catalog and its recommended immediate and investigatory actions.

Availability#

This document describes GPU Operational Events in R620. NVML support is available on Linux, on Turing and later GPUs. Structured subscriptions, retained CPER queries, and the nvidia-smi commands below require administrative privileges (CAP_SYS_ADMIN on Linux).

Event Identity#

Event identity consists of a module signature, category, and module event code.

Field

Type

Description

Module Signature

Character array

ASCII string naming the source module, for example GPU-GSP. Namespaces the event code.

Category

16-bit integer

Class of condition, from the category registry in Event Categories. Shared across all modules.

Module Event Code

16-bit integer

Specific event within the reporting module. Together with the module signature and category, locates matching entries in Published Events.

Each event additionally carries the fields below.

Field

Description

Severity

Impact relative to the event scope. See Event Severities.

Log Level

Urgency of the report, orthogonal to severity. See Event Log Levels.

Scope

What the event affects. See Event Scopes.

Originator

Which driver or firmware component originated the event.

Device UUID

UUID of the GPU that reported the event.

Instance ID

Originator-scoped sequence number.

Timestamp

Microsecond wall clock.

Trace ID

Correlation identifier shared by related events that describe one incident.

Group Cursor

64-bit value that identifies the event group. Also maps to the CPER Error Record Header recordId for the retained CPER record.

Group Position

Zero-based index within the group visible to the subscriber.

Group Size

Number of events in the group visible to the subscriber.

Attributes

Per-event flags that describe properties of the event.

Group Attributes

Group-level flags that describe properties of the event group.

Attribution

Fields for locating the event within the device. R620 does not report MIG instance IDs for GPU Operational Events.

The trace ID correlates related events in a causal chain; the group cursor identifies one exported group. NVML delivers one event per successful wait. Use groupCursor to associate those records, groupIndex to order them, and groupSize to check completeness. See GPU Operational Event Formats for how these fields appear in each published format.

Monitoring GPU Operational Events#

The driver retains recent event groups in an in-memory Operational Event log. The interfaces below expose event records for monitoring and diagnosis.

Command Line#

nvidia-smi event-log prints retained events since driver load in a human-readable form. It reports events for every GPU visible to NVML, skipping inaccessible GPUs during enumeration. event-log accepts -h / --help. Resolve the numeric values it prints through Published Events. The decoder has the following limitations in this release:

  • Module event codes print as hexadecimal values without names.

  • Categories 0x0008 and 0x0009 still print as Engine Software Error and Engine Hardware Error. In R620 these mean Firmware Fault and Engine Error, respectively. Use the numeric category and this release’s catalog.

  • Contexts other than Legacy Xid print their numeric type and no payload. Legacy Xid contexts print the Xid code.

  • Severity reports the CPER severity of the record rather than the event’s own severity, so a GPU-fatal event appears here as Recoverable. See UEFI CPER for the mapping, and read the severity in a structured record for GPU-relative impact.

nvidia-smi cper prints retained history as Base64-encoded raw CPER records, one record per Base64 block, for forwarding to an external collector. cper accepts -h / --help and --no-wrap. Output wraps at 76 columns by default; --no-wrap emits each CPER record on a single line.

NVML#

NVML exposes GPU Operational Events through two interfaces. See the NVML API Reference Guide for full function prototypes.

Structured event subscription. Register a GPU with nvmlEventSetRegisterGpuOperationalEvents_v1, then retrieve events with nvmlEventSetWait_v3. Records with dataType set to NVML_EVENT_DATA_TYPE_GPU_OPERATIONAL_EVENT carry the structured event. Read attached context payloads with nvmlEventSetGetContextCount_v1, nvmlEventSetGetContextInfo_v1, and nvmlEventSetGetContextData_v1. Decode a legacy Xid context with nvmlEventSetGetGpuOperationalEventContextLegacyXid_v1.

Zero-initialize nvmlGpuOperationalEventConfig_v1_t and set its uuid to the target GPU-... UUID. This subscribes to new device-wide events after registration. Optional minLogLevel and minSeverity thresholds filter whole groups; zero disables each filter. These thresholds do not remove lower-level companion events from a matching group.

Read or copy contexts before the next successful nvmlEventSetWait_v3 on that event set: the context accessors refer to the last returned event, not a caller-owned event copy. The typed Legacy Xid accessor returns the Xid code; use the raw context for its message.

The same event set can subscribe to legacy Xid events and GPU Operational Events. When a GPU Operational Event carries a Legacy Xid context, use that Xid value as a correlation hint to the matching Xid event, kernel log line, and catalog entry.

Retained CPER history. nvmlSystemGetCPER_v1 returns retained CPER records as raw bytes. This is the interface that both nvidia-smi commands above use. Initialize the cursor handle to NVML_CPER_CURSOR_HANDLE_INIT and preserve it between calls. A size query uses a null buffer and zero buffer size; NVML_ERROR_INSUFFICIENT_SIZE returns the required size. A successful call with zero buffer size means no more records.

Kernel Log#

Events that carry a legacy Xid context emit their traditional kernel log line:

NVRM: Xid (PCI:0000:41:00): 63, Row Remapper: New row (0x00000000fbc12000) marked for remapping, reset gpu to activate.

The Xid prefix and numeric code remain available to existing log parsers. Simulated events add [SIMULATED] to the message. Events without a legacy Xid equivalent are visible through the structured, CPER, and out-of-band interfaces.

Linux Tracepoints#

The open kernel module exposes five tracepoints in the nvidia tracing subsystem: nvidia_gpu_error, nvidia_gpu_warning, nvidia_gpu_notice, nvidia_gpu_diagnostic, and nvidia_gpu_telemetry. Each event goes to the tracepoint for its log level. They carry event identity, severity, scope, attribution, and correlation fields; context payloads remain available through NVML and CPER.

With tracefs mounted at /sys/kernel/tracing, run as root to capture error-level events:

echo 1 > /sys/kernel/tracing/events/nvidia/nvidia_gpu_error/enable
cat /sys/kernel/tracing/trace_pipe

After stopping the capture, write 0 to the same enable file. These tracepoints are unavailable with the proprietary kernel module.

Out-of-Band#

On platforms that support out-of-band reporting, the driver forwards rendered CPER records to the BMC. Match the CPER record ID to the structured event’s group cursor to correlate reports.

Interpreting an Event#

Determine the Reporting Module and Event Code#

Read the module signature and module event code from the structured record, or Source Module and Module Event Code from nvidia-smi event-log. Use the category and context type to find the matching entry in Published Events.

Determine Impact from the Category and Severity#

The category identifies the class of condition and the severity identifies impact relative to the event’s scope. Use scope to establish what the severity applies to: a fatal event at individual GPU engine scope reports that one engine is down, while a fatal event at device scope reports that the GPU is down.

Read the Context Payloads#

Contexts carry occurrence-specific data. In this release, NVML and nvidia-smi decode the Legacy Xid code. NVML also exposes raw payloads, including the Legacy Xid message and other typed contexts. Check the source context type, format version, and size before decoding; skip unrecognized payloads using their size. See GPU Operational Event Formats for the context layouts.

Determine the Actions to Take#

For an event that lists a legacy Xid in Published Events, look the Xid up in the Xid Errors catalog, which supplies the immediate and investigatory actions for that condition. For LTS_REPAIR_PENDING and MEMORY_CHANNEL_REPAIR_PENDING, the Xid 160 message states the action needed to activate the repair.