Troubleshooting#

This chapter describes basic troubleshooting steps.

Known Issues#

Before troubleshooting or filing a bug report, review the release notes that accompany each driver release, for information about known issues with the current release, and potential workarounds.

Verifying the NVIDIA vGPU Software License Status of a Licensed Client#

After configuring a client with an NVIDIA vGPU software license, verify the license status by displaying the licensed product name and status.

To verify the license status of a licensed client, run nvidia-smi with the -q or --query option from the licensed client, not the hypervisor host. If the product is licensed, the expiration date is shown in the license status.

nvidia-smi -q
==============NVSMI LOG==============

Timestamp                                 : Wed Nov 23 10:52:59 2022
Driver Version                            : 525.60.06
CUDA Version                              : 12.0

Attached GPUs                             : 2
GPU 00000000:02:03.0
    Product Name                          : NVIDIA A2-8Q
    Product Brand                         : NVIDIA RTX Virtual Workstation
    Product Architecture                  : Ampere
    Display Mode                          : Enabled
    Display Active                        : Disabled
    Persistence Mode                      : Enabled
    MIG Mode
        Current                           : Disabled
        Pending                           : Disabled
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : N/A
        Pending                           : N/A
    Serial Number                         : N/A
    GPU UUID                              : GPU-ba5b1e9b-1dd3-11b2-be4f-98ef552f4216
    Minor Number                          : 0
    VBIOS Version                         : 00.00.00.00.00
    MultiGPU Board                        : No
    Board ID                              : 0x203
    Board Part Number                     : N/A
    GPU Part Number                       : 25B6-890-A1
    Module ID                             : N/A
    Inforom Version
        Image Version                     : N/A
        OEM Object                        : N/A
        ECC Object                        : N/A
        Power Management Object           : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GSP Firmware Version                  : N/A
    GPU Virtualization Mode
        Virtualization Mode               : VGPU
        Host VGPU Mode                    : N/A
    vGPU Software Licensed Product
        Product Name                      : NVIDIA RTX Virtual Workstation
        License Status                    : Licensed (Expiry: 2022-11-23 10:41:16 GMT)
    …
    …

Licensing Event Logs#

Significant licensing events are logged to help troubleshoot issues. Error events are logged with an error code to help you to determine the underlying cause of the error. License acquisition events are logged with the name and version of the licensed product, for example, GRID-Virtual-WS,2.0.

Examples of significant licensing events that are logged are as follows:

Acquisition of a license
Return of a license
Expiration of a license
Failure to acquire a license
License state changes between the unlicensed restricted state (20 mins), unlicensed state (24 hours), and licensed state

Windows Licensing Event Logs#

On Windows, licensing events are logged in the plain-text file %SystemDrive%\Users\Public\Documents\NvidiaLogging\Log.NVDisplay.Container.exe.log.

The log file is rotated when its size reaches 16 MB. A new log file is created and the old log file is renamed to Log.NVDisplay.Container.exe.log1. Each time the log file is rotated, the number in the file name of each existing old log file is increased by 1. The oldest log file is deleted when the number of log files exceeds 16.

Logging is enabled by default. To disable logging, create the Windows Registry key EnableLogging (DWORD) with the value zero at the path HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\nvlddmkm\Global\GridLicensing.

License state changes are reported in Event Viewer under Windows Logs in the Application category. To open Event Viewer, press Windows+R, type eventvwr.msc, and press ENTER.

Linux Licensing Event Logs#

On Linux, logging is enabled by default. The licensing events are logged in the standard activity log in the /var/log directory. The name of the log file in this directory depends on the Linux distribution, for example:

For Red Hat Enterprise Linux and CentOS distributions, the file name is messages.
For Ubuntu distributions, the file name is syslog.

You can use the standard features of the operating system to rotate and purge these logs.

Log Messages for Acquisition and Return of a License and Transient Network Failure#

This example shows log messages for the following sequence of events:

Acquisition of a license when a VM is started
Return of a license when the VM is shut down
Failure to communicate with the license server when the VM is started when the license server is unreachable
Acquisition of a license after communications with the license server are restored

Mon Oct 22 20:53:29 2018:<1>:License acquired successfully. (Info: http://gridlicense1.example.com:7070/request; GRID-Virtual-WS,2.0)
Mon Oct 22 20:54:22 2018:<1>:End Logging
Mon Oct 22 20:54:24 2018:<0>:License returned successfully
Mon Oct 22 20:54:24 2018:<0>:End Logging
Mon Oct 22 20:55:36 2018:<1>:Failed server communication. Please ensure your system has network connectivity to the license server
Mon Oct 22 20:55:49 2018:<1>:Failed server communication. Please ensure your system has network connectivity to the license server
Mon Oct 22 20:56:01 2018:<1>:Failed server communication. Please ensure your system has network connectivity to the license server
Mon Oct 22 20:56:19 2018:<1>:Failed server communication. Please ensure your system has network connectivity to the license server
Mon Oct 22 20:56:31 2018:<1>:Failed server communication. Please ensure your system has network connectivity to the license server
Mon Oct 22 20:56:44 2018:<1>:Failed server communication. Please ensure your system has network connectivity to the license server
Mon Oct 22 20:56:47 2018:<1>:License acquired successfully. (Info: http://gridlicense1.example.com:7070/request; GRID-Virtual-WS,2.0)
Mon Oct 22 21:36:11 2018:<1>:End Logging
Mon Oct 22 21:36:12 2018:<0>:License returned successfully
Mon Oct 22 21:36:12 2018:<0>:End Logging

Log Messages for Failure to Acquire a License#

This example shows log messages for the failure to acquire a license because no licenses of the requested type are available.

Mon Oct 22 22:26:36 2018:<1>:Error : Server URL : http://gridlicense1.example.com:7070/request - 
[1,7E2,2,0[70000030,0,702C3]]
Insufficient count for the requested feature.
Mon Oct 22 22:28:15 2018:<1>:Error : Server URL : http://gridlicense1.example.com:7070/request - 
[1,7E2,2,0[70000030,0,702C3]]
Insufficient count for the requested feature.
Mon Oct 22 22:29:55 2018:<1>:Error : Server URL : http://gridlicense1.example.com:7070/request - 
[1,7E2,2,0[70000030,0,702C3]]
Insufficient count for the requested feature.
Mon Oct 22 22:31:34 2018:<1>:Error : Server URL : http://gridlicense1.example.com:7070/request - 
[1,7E2,2,0[70000030,0,702C3]]
Insufficient count for the requested feature.

Troubleshooting Steps#

If an NVIDIA vGPU software system fails to obtain a license, troubleshoot the failure as follows:

Examine the log of significant licensing events for information about the cause of the failure. For details, see Licensing Event Logs.
Check that the license server address and port number are correctly configured.
Run a network ping test from the NVIDIA vGPU software system to the license server address to verify that the system has network connectivity to the license server.
Verify that the date and time are configured correctly on the NVIDIA vGPU software system. If the time is set inaccurately or is adjusted backwards by a large amount, the system may fail to obtain a license.
Verify that the license server in use has available licenses of the type required by the NVIDIA vGPU software feature the NVIDIA vGPU software system is configured to use.

Maximum Allowed Network Adapters on a VM Exceeded#

If a VM is configured with too many network adapters, the VM might fail to acquire or return a license. The exact number of network adapters allowed on a VM depends on your system. In testing, it was verified that a VM with 40 network adapters could successfully acquire a license.

If a VM fails to acquire or return a license because the VM is configured with too many network adapters, the following error message is written to the licensing event log file:

Failed to update local trusted store - Maximum buffer size exceeded

For information about this log file, refer to Licensing Event Logs.

How to recover from such a failure depends on the guest OS that is running in the VM as explained in the following topics:

Windows: Recovering from a Failure Caused by Too Many Network Adapters
Linux: Recovering from a Failure Caused by Too Many Network Adapters

Windows: Recovering from a Failure Caused by Too Many Network Adapters#

If a Windows VM fails to acquire or return a license because the VM is configured with too many network adapters, reduce the number of network adapters on the VM.

Stop the NvDisplayContainer service.
Remove some network adapters from the VM.
Delete all files in the %SystemDrive%\Program Files\NVIDIA Corporation\vGPU Licensing\TrustedStorage folder, which contains the trusted storage database files.
Start the NvDisplayContainer service.

Linux: Recovering from a Failure Caused by Too Many Network Adapters#

If a Linux VM fails to acquire or return a license because the VM is configured with too many network adapters, reduce the number of network adapters on the VM.

Stop the nvidia-gridd service.
```
# sudo service nvidia-gridd stop
```
Remove some network adapters from the VM.
Delete all files in the /var/lib/nvidia/vGPULicensing/ directory, which contains the trusted storage database files.
```
# sudo \rm -rf /var/lib/nvidia/vGPULicensing/*
```
Start the nvidia-gridd service.
```
# sudo service nvidia-gridd start
```