The NVIDIA® driver supports "retiring" of bad framebuffer memory cells, by retiring the page the cell belongs to. This is called "dynamic page retirement" and is done automatically for cells that are degrading in quality. This feature can improve the longevity of an otherwise good board and and is thus an important resiliency feature on supported products, especially in HPC and enterprise environments.
The marking of a page for exclusion is called "retiring", while the actual act of excluding that page from subsequent memory allocations is called "blacklisting". The NVIDIA® driver will retire a page once it has experienced a single Double Bit ECC Error (DBE) or 2 Single Bit ECC Errors (SBE). These addresses are stored in the InfoROM. When the each GPU is attached and initialized the driver will retrieve these addresses from the InfoROM, then have the framebuffer manager set these pages aside, such that they cannot be used by the driver or user applications.
Ideally, the NVIDIA® driver will catch weakening cells at the 2 SBE point and retire the page, before the cell degrades to the point of a DBE and disrupts an application.
- The NVIDIA® driver detects a DBE and reports that a DBE occurred.
- Applications will receive a DBE event notification for graceful exit, and no further context will be created on the GPU until the DBE is mapped out.
- The NVIDIA® driver logs the DBE count and address in the InfoROM.
- The NVIDIA® driver logs, in a separate list, that the page containing the DBE is to be retired.
- Upon the next reattachment of the GPU, the page is mapped out of usage.
- The NVIDIA® driver detects an SBE and reports that an SBE occurred.
- The NVIDIA® driver logs the SBE count and address in the InfoROM.
- If the SBE occurs more than once in a particular address, the driver logs, in a separate list, that the page containing that address is to be retired.
- Upon the next reattachment of the GPU, the page is mapped out of usage.
Pages that have been previously retired are blacklisted for all future allocations of the framebuffer, provided that the target GPU has been properly reattached and initialized. This can be done in several ways.
- Reattach the GPU
- Reset the GPU
- Reload the kernel module (nvidia.ko)
- Reboot the machine
Reattaching the GPU is the least invasive solution. The detachment process occurs automatically a few seconds after the last client terminates on the GPU, as long as persistence mode is not enabled. The next client that targets the GPU will trigger the driver to reattach and blacklist all marked pages.
If persistence mode is enabled the preferred solution is to reset the GPU using "nvidia-smi -i <target GPU> -r". This operation reattaches the GPU as a step in the larger process of resetting all GPU SW and HW state.
Reloading the NVIDIA® kernel module triggers reattachment of all GPUs on the machine, and thus requires the termination of all clients on all GPUs.
Dynamic page retirement is supported on the following products and environments:
- Drivers: R319 and newer
- OSes: All standard driver-supported Linux and Windows TCC platforms
- K20 and newer Tesla products
- No Quadro, GRID or Geforce products are currently supported
Three main mechanisms provide visibility into page retirement: XID errors in system logs, the NVML API and the nvidia-smi command line tool.
XID errors are driver errors that are logged to the system error log. Please see the XID Whitepaper for general info on XIDs.
- XID 48: A DBE has occurred.
- XID 63: A page has successfully been retired.
- XID 64: A page has failed retirement due to an error.
- XID 48: "XID 48 An uncorrectable double bit error (DBE) has been detected on GPU (<id>)"
- XID 63: "XID 63 Dynamic Page Retirement: New retired page, reload the driver to activate. (<address>)"
- XID 64: "XID 64 Dynamic Page Retirement: Fatal error, unable to retire page (<address>)"
The NVIDIA® Management Library (NVML) is a public C-based library for GPU monitoring and management. It includes APIs that report the status and count of retired pages. Please see the NVML API docs for general info on the library.
The set of currently retired pages, and their addresses, can be retireved using:
nvmlReturn_t nvmlDeviceGetRetiredPages (nvmlDevice_t device, nvmlPageRetirementCause_t cause, unsigned int* pageCount, unsigned long long* addresses)
The current state of the driver (whether any pages are pending retirement) can be retrieved using:
nvmlReturn_t nvmlDeviceGetRetiredPagesPendingStatus (nvmlDevice_t device, nvmlEnableState_t* isPending)
Nvidia-smi is a public command line interface for GPU monitoring and management. It implements most of the NVML APIs and supports reporting the status and count of retired pages. Please see the Nvidia-smi man page for general info on the tool.
To view the number of retired pages and the page retirement state of the driver in human readable form:
$ nvidia-smi -i <target gpu> -q -d PAGE_RETIREMENT ... Retired pages Single Bit ECC : 2 Double Bit ECC : 0 Pending : No ...
If pages have been retired the affected addresses can be viewed through nvidia-smi's scriptable outputs, either XML:
$ nvidia-smi -i <target gpu> -q -x ... <retired_pages> <multiple_single_bit_retirement> <retired_count>2</retired_count> <retired_page_addresses> <retired_page_addresse>0xABC123</retired_page_addresse> <retired_page_addresse>0xDEF456</retired_page_addresse> </retired_page_addresses> </multiple_single_bit_retirement> <double_bit_retirement> <retired_count>0</retired_count> <retired_page_addresses></retired_page_addresses> </double_bit_retirement> <pending_retirement>No</pending_retirement> </retired_pages> ...
$ nvidia-smi -i <target gpu> --query-retired-pages=gpu_uuid,retired_pages.address,retired_pages.cause --format=csv ... gpu_uuid, retired_pages.address, retired_pages.cause GPU-d73c8888-9482-7d65-c95c-4b58c7d9eb4c, 0xABC123, Double Bit ECC GPU-d73c8888-9482-7d65-c95c-4b58c7d9eb4c, 0xDEF456, Double Bit ECC GPU-d73c8888-9482-7d65-c95c-4b58c7d9eb4c, 0x123ABC, Single Bit ECC ...
There exists a race condition between logging errors to the InfoROM and ending a CUDA™ job while in persistence mode. This race condition is most often hit when shutting down in response to a DBE. The effect of this condition is that a page may fail to retire in certain corner cases.
Exiting persistence mode before rebooting the system will forcibly flush any pending writes to the InfoROM. If XID 48 is seen and XID 63 is not seen, it is recommended to exit persistence mode via the command:
% nvidia-smi -i <target GPU> -pm 0
At this point, the XID 63 should be seen and the NVML query can be used to verify the page was written to the InfoROM.
There are no current plans to fix the race condition in persistence mode, as persistence mode is replaced by the persistence daemon. The persistence daemon is not susceptible to this race condition
Is available memory reduced by retired pages?
Each retired page decreases the total memory available to applications. However, the total maximum size of memory retired in this way is only on the order of 256 KiB. This is insignificant relative to other factors, such as natural fluctuations in memory allocated internally by the NVIDIA® driver during normal operation.
What is the size of each retired page?
How many pages can be retired?
A combined total of 60 pages can be mapped out, or retired. This can be any combination of DBE and SBE pages.
How many addresses can be stored in the InfoROM?
A maximum of 192 different addresses can be stored in the InfoROM.
Can page retirement be disabled?
No, page retirement is an important reliability feature and cannot be disabled for either SBEs or DBEs. Any pages already marked as retired will continue to be excluded in all future allocations. Note though that if ECC is disabled no new memory errors will be detected and thus no new pages will be blacklisted for future retirement.
Is the SBE recurrence threshold for triggering retirement be configurable?
Is application behavior affected?
No, applications behave the same. Since pages are retired only after the driver has been restarted the act of retiring a page occurs outside the lifetime of any GPU process or application. An application running on a GPU with pages scheduled for retirement (blacklisted) will continue to see those pages in its memory space, though any page retired due to a double bit error (DBE) will necessarily cause an application to terminate. This is true even without page retirement.
Is application performance affected?
No, application performance is unaffected by either the retirement of pages or their subsequent blacklisting. Retirement is the only act taken during application execution, while the actual blacklisting event happens only after the application has terminated. As noted in the first FAQ question above, the memory impact of retired pages is also negligible.
Is memory fragmentation due to page retirement expected to impact app performance?
Fragmentation is not expected to affect performance.
Must multiple SBEs be located at the same address to trigger retirement?
Yes, multiple SBE's must be located at the exact same location (address). Multiple SBEs at different locations within the same page will not trigger page retirement.
Are "stuck" bits rewritten by the driver, or corrected on each read?
On Kepler-class GPUs and later, the driver rewrites the data to avoid stuck bits.
Are all SBE and DBE addresses tracked indefinitely?
SBE and DBE addresses are tracked indefinitely, up to the maximum number of addresses that can be stored (See Memory Impact). Additional addresses beyond the maximum are dropped.
ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, "MATERIALS") ARE BEING PROVIDED "AS IS." NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.
Information furnished is believed to be accurate and reliable. However, NVIDIA Corporation assumes no responsibility for the consequences of use of such information or for any infringement of patents or other rights of third parties that may result from its use. No license is granted by implication of otherwise under any patent rights of NVIDIA Corporation. Specifications mentioned in this publication are subject to change without notice. This publication supersedes and replaces all other information previously supplied. NVIDIA Corporation products are not authorized as critical components in life support devices or systems without express written approval of NVIDIA Corporation.
NVIDIA and the NVIDIA logo are trademarks or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.
© 2013-2015 NVIDIA Corporation. All rights reserved.