Is this page helpful?

Visibility#

Three main mechanisms provide visibility into page retirement: XID errors in system logs, the NVML API and the nvidia-smi command line tool.

XIDs#

XID errors are driver errors that are logged to the system error log. Please see the XID Whitepaper for general info on XIDs. There are three main XIDs related to dynamic page retirement:

  • XID 48: A DBE has occurred.

  • XID 63: A page has successfully been retired.

  • XID 64: A page has failed retirement due to an error.

In the system log these XIDs show up in the following forms:

  • XID 48: “XID 48 An uncorrectable double bit error (DBE) has been detected on GPU (<id>)”

  • XID 63: “XID 63 Dynamic Page Retirement: New retired page, reload the driver to activate. (<address>)”

  • XID 64: “XID 64 Dynamic Page Retirement: Fatal error, unable to retire page (<address>)”

NVML#

The NVIDIA Management Library (NVML) is a public C-based library for GPU monitoring and management. It includes APIs that report the status and count of retired pages. Refer to the NVML API doc for general info on the library.

The set of currently retired pages, and their addresses, can be retrieved using:

nvmlReturn_t nvmlDeviceGetRetiredPages (nvmlDevice_t device, nvmlPageRetirementCause_t cause, unsigned int* pageCount, unsigned long long* addresses)

Driver versions 410.72 or later provide a newer API that also returns page retirement timestamp:

nvmlReturn_t nvmlDeviceGetRetiredPages_v2 (nvmlDevice_t device, nvmlPageRetirementCause_t cause, unsigned int* pageCount, unsigned long long* addresses, unsigned long long* timestamps)

For both APIs, the nvmlPageRetirementCause_t passed is one of:

  • NVML_PAGE_RETIREMENT_CAUSE_MULTIPLE_SINGLE_BIT_ECC_ERRORS

  • NVML_PAGE_RETIREMENT_CAUSE_DOUBLE_BIT_ECC_ERROR

The current state of the driver (whether any pages are pending retirement) can be retrieved using:

nvmlReturn_t nvmlDeviceGetRetiredPagesPendingStatus (nvmlDevice_t device, nvmlEnableState_t* isPending)

nvidia-smi#

nvidia-smi is a public command line interface for GPU monitoring and management. It implements most of the NVML APIs and supports reporting the status and count of retired pages. Please see the nvidia-smi man page for general info on the tool.

The nvidia-smi tool provides:

  • the ability to list the number of retired pages–sorted by cause of retirement–and indicate whether any pages are pending retirement, and

  • the ability to list all retired page addresses.

To view the number of retired pages and the page retirement state of the driver in human readable form:

$ nvidia-smi -i <target gpu> -q -d  PAGE_RETIREMENT
     ...
     Retired pages
            Single Bit ECC   : 2
            Double Bit ECC   : 0
            Pending Page Blacklist  : No
     ...

The nvidia-smi “Pending Page Blacklist” field indicates whether a page has been recently retired and will be blacklisted on the next system reboot or driver load.

Note

The retired page count increments immediately when a page is retired and not on the next driver reload when the page is blacklisted.

If pages have been retired, the affected addresses can be viewed through nvidia-smi’s scriptable outputs, in either XML format:

$ nvidia-smi -i <target gpu> -q -x
...
 <retired_pages>
     <multiple_single_bit_retirement>
     <retired_count>2</retired_count>
          <retired_page_addresses>
               <retired_page_addresse>0xABC123</retired_page_addresse>
               <retired_page_addresse>0xDEF456</retired_page_addresse>
          </retired_page_addresses>
     </multiple_single_bit_retirement>
     <double_bit_retirement>
          <retired_count>0</retired_count>
          <retired_page_addresses></retired_page_addresses>
     </double_bit_retirement>
     <pending_retirement>No</pending_retirement>
 </retired_pages>
...

or CSV format:

$ nvidia-smi -i <target gpu> --query-retired-pages=
gpu_uuid,retired_pages.address,retired_pages.cause --format=csv
...
gpu_uuid, retired_pages.address, retired_pages.cause
GPU-d73c8888-9482-7d65-c95c-4b58c7d9eb4c, 0xABC123, Double Bit ECC
GPU-d73c8888-9482-7d65-c95c-4b58c7d9eb4c, 0xDEF456, Double Bit ECC
GPU-d73c8888-9482-7d65-c95c-4b58c7d9eb4c, 0x123ABC, Single Bit ECC
On this page