Is this page helpful?

Implementation#

The marking of a page for exclusion is called “retiring”, while the actual act of excluding that page from subsequent memory allocations is called “blacklisting”. The NVIDIA driver will retire a page once it has experienced a single Double Bit ECC Error (DBE), or 2 Single Bit ECC Errors (SBE) on the same address. These addresses are stored in the InfoROM. When each GPU is attached and initialized the driver will retrieve these addresses from the InfoROM, then have the framebuffer manager set these pages aside, such that they cannot be used by the driver or user applications.

Note

Retiring of pages may only occur when ECC is enabled. However, once a page has been retired it will always be blacklisted by the driver, even if ECC is later disabled.

Ideally, the NVIDIA driver will catch weakening cells at the 2 SBE point and retire the page, before the cell degrades to the point of a DBE and disrupts an application.

Use Case 1: DBE Detected#

  1. The NVIDIA driver detects a DBE and reports that a DBE occurred.

  2. Applications will receive a DBE event notification for graceful exit, and no further context will be created on the GPU until the DBE is mapped out.

  3. The NVIDIA driver logs the DBE count and address in the InfoROM.

    Page retirement occurs and the nvidia-smi Retired Pages ‘Double Bit ECC’ field is incremented.

    The nvidia-smi ‘Pending Page Blacklist’ status becomes ‘YES’.

  4. The NVIDIA driver logs, in a separate list, that the page containing the DBE is to be retired.

  5. Upon the next reattachment of the GPU, the page is mapped out of usage.

    Page blacklisting occurs and nvidia-smi ‘Pending Page Blacklist’ status becomes ‘NO’.

Note

The DBE addresses and counts are preserved across driver reloads.

Use Case 2: Two SBEs Detected at the Same Location#

  1. The NVIDIA driver detects an SBE and reports that an SBE occurred.

  2. The NVIDIA driver logs the SBE count and address in the InfoROM.

  3. If the SBE occurs more than once in a particular address, the driver logs, in a separate list, that the page containing that address is to be retired.

    Page retirement occurs and the nvidia-smi Retired Pages ‘Single Bit ECC’ field is incremented.

    The nvidia-smi ‘Pending Page Blacklist’ status becomes ‘YES’.

  4. Upon the next reattachment of the GPU, the page is mapped out of usage.

    Page blacklist occurs and the nvidia-smi ‘Pending Page Blacklist’ status becomes ‘NO’.

Note

Unlike the DBE case, applications continue to run uninterrupted.

Note

The SBE addresses and counts are preserved across driver reloads.