Under Linux, the Xid error messages are placed in the location /var/log/messages.

Grep for “NVRM: Xid”to find all the Xid messages.

The following is an example of a Xid string:

[…] NVRM: GPU at 0000:03:00: GPU-b850f46d-d5ea-c752-ddf3-c4453e44d3f7 

[…] NVRM: Xid (0000:03:00): 14, Channel 00000001 

• The first Xid in the log file is preceded by a line that contains the GPU GUID and device IDs.

In the above example,

The GUID is a globally unique, immutable identifier for each GPU.

• Each subsequent Xid line contains the device ID, the Xid error, and information about the Xid.

In the above example,

NVIDIA provides two additional tools that may be helpful when dealing with Xid errors.

nvidia-smi is a command-line program that installs with the NVIDIA driver. It reports basic monitoring and configuration data about each GPU in the system. nvidia-smi can list ECC error counts (Xid 48) and indicate if a power cable is unplugged (Xid 54), among other things. Please see the nvidia-smi man page for more info. Run ‘nvidia-smi –q’ for basic output.

NVIDIA Validation Suite (NVVS) is a health checking tool that is provided as part of the GPU Deployment Kit, located at https://developer.nvidia.com/gpu-deployment-kit. NVVS can check for basic GPU health, including the presence of ECC errors, PCIe problems, bandwidth issues, and general problems with running CUDA programs. NVVS documentation is included in the GPU Deployment Kit.

The following table lists the recommended actions to take for various issues encountered.

This event is logged for general user application faults. Typically this is an out-of-bounds error where the user has walked past the end of an array, but could also be an illegal instruction, illegal register, or other case.

In rare cases, it’s possible for a hardware failure or system software bugs to materialize as XID 13.

When this event is logged, NVIDIA recommends the following:

1. Run the application in cuda-gdb or cuda-memcheck , or
2. Run the application with CUDA_DEVICE_WAITS_ON_EXCEPTION=1 and then attach later with cuda-gdb, or
3. File a bug if the previous two come back inconclusive to eliminate potential NVIDIA driver or hardware bug.

This event is logged when a fault is reported by the MMU, such as when an illegal address access is made by an applicable unit on the chip Typically these are application-level bugs, but can also be driver bugs or hardware bugs.

When this event is logged, NVIDIA recommends the following:

1. Run the application in cuda-gdb or cuda-memcheck , or
2. Run the application with CUDA_DEVICE_WAITS_ON_EXCEPTION=1 and then attach later with cuda-gdb, or
3. File a bug if the previous two come back inconclusive to eliminate potential NVIDIA driver or hardware bug.

This event is logged when a fault is reported by the DMA controller which manages the communication stream between the NVIDIA driver and the GPU over the PCI-E bus. These failures primarily involve quality issues on PCI, and are generally not caused by user application actions.

This event is logged when a user application hits a software induced fault and must terminate. The GPU remains in a healthy state.

In most cases, this is not indicative of a driver bug but rather a user application error.

This event is logged when the user application aborts and the kernel driver tears down the GPU application running on the GPU. Control-C, GPU resets, sigkill are all examples where the application is aborted and this event is created.

In many cases, this is not indicative of a bug but rather a user or system action.

This event is logged when the GPU detects that an uncorrectable error occurs on the GPU. This is also reported back to the user application. A GPU reset or node reboot is needed to clear this error.

The tool nvidia-smi can provide a summary of ECC errors. See “Tools That Provide Additional Information About Xid Errors”.