20.0-20.2 Only: CUDA or deep learning applications might fail with memory errors after vGPU migration

Description

After a Linux vGPU VM is rebooted and then successfully migrated on VMware vSphere ESXi, any CUDA or deep learning application in the VM might fail with an out-of-memory or CUDA memory allocation error. This issue can occur even when sufficient GPU memory is available.

This issue affects only the migrated VM. The hypervisor host and other VMs are unaffected. This issue is not specific to a particular GPU.

Messages similar to the following examples are written to the kernel log in the guest VM:

NVRM: _scrubWaitAndSave: Timed out when waiting for scrub jobs to finish.
NVRM: nvCheckOkFailedNoLog: Check failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from _scrubWaitAndSave(pScrubber, pList, requiredItemsToSave) @ mem_scrub.c:626

Status

Resolved in NVIDIA vGPU software 20.3

Ref. #

6133450