FAQ#
RMA Eligibility#
How many pages can be mapped out before the GPU should be returned for repair?
The Tesla board will continue to retire pages up until the page retirement table is full, at 64 dynamically retired memory pages. However, a board that generates 60 or more retired pages is eligible for an RMA. A Tesla card with 64 pages retired will fail the NVIDIA Field Diagnostic tool.
Additionally, if a board is found to exhibit 15 or more retired pages and continues to retire memory pages at a rate of one or more newly retired pages per week, it can be evaluated for an RMA before the 60-page RMA threshold has been reached. Please track the page retirement rate and provide that information with the returned board..
Memory Impact#
Is available memory reduced by retired pages?
Each retired page decreases the total memory available to applications. However, the total maximum size of memory retired in this way is only on the order of 4 MiB. This is insignificant relative to other factors, such as natural fluctuations in memory allocated internally by the NVIDIA driver during normal operation.
What is the size of each retired page?
64KiB
How many pages can be retired?
A combined total of 64 pages can be mapped out, or retired. This can be any combination of DBE and SBE pages.
How many addresses can be stored in the InfoROM?
The InfoROM can store at least 192 different addresses. Different GPU models and InfoROM formats may extend this beyond 192 addresses up to a maximum of 600.
Configuration#
Is ECC enabled on my GPU?
nvidia-smi can be used to show if the GPU currently has ECC enabled, and what mode is pending to be activated following a reboot.
$ nvidia-smi -q
...
Ecc Mode
Current : Disabled
Pending : Enabled
Can page retirement be disabled?
No, page retirement is an important reliability feature and cannot be disabled for either SBEs or DBEs. Any pages already marked as retired will continue to be excluded in all future allocations. Note though that if ECC is disabled no new memory errors will be detected and thus no new pages will be blacklisted for future retirement.
ECC can be disabled through nvidia-smi:
$ nvidia-smi -e 0
Disabled ECC support for GPU 00000000:06:00.0.
All done.
Reboot required.
Or re-enabled, similarly:
sudo nvidia-smi -e 1
Enabled ECC support for GPU 00000000:06:00.0.
All done.
Reboot required.
Is the SBE recurrence threshold for triggering retirement be configurable?
No
Can I disable DMA protected range (DPR) for SBEs or DBEs?
No, NVIDIA does not support disabling DPR, either entirely or for just SBEs or DBEs.
App Behavior#
Is application behavior affected?
No, applications behave the same. Since pages are retired only after the driver has been restarted the act of retiring a page occurs outside the lifetime of any GPU process or application. An application running on a GPU with pages scheduled for retirement (blacklisted) will continue to see those pages in its memory space, though any page retired due to a double bit error (DBE) will necessarily cause an application to terminate. This is true even without page retirement.
App Performance#
Is application performance affected?
No, application performance is unaffected by either the retirement of pages or their subsequent blacklisting. Retirement is the only act taken during application execution, while the actual blacklisting event happens only after the application has terminated. As noted in the first FAQ question above, the memory impact of retired pages is also negligible.
Is memory fragmentation due to page retirement expected to impact app performance?
Fragmentation is not expected to affect performance.
Driver Behavior#
Must multiple SBEs be located at the same address to trigger retirement?
Yes, multiple SBEs must be located at the exact same location (address). Multiple SBEs at different locations within the same page will not trigger page retirement.
Are “stuck” bits rewritten by the driver, or corrected on each read?
On Kepler-class GPUs and later, the driver rewrites the data to avoid stuck bits.
Are all SBE and DBE addresses tracked indefinitely?
SBE and DBE addresses are tracked indefinitely, up to the maximum number of addresses that can be stored (See Memory Impact). Additional addresses beyond the maximum are dropped.