Is this page helpful?

Installation and Upgrade Issues#

GPU Operator Installation Failed - Clean Uninstallation Process#

If the GPU Operator is installed using Helm, always use the helm uninstall command to ensure proper cleanup of all GPU Operator artifacts. Manually deleting components using kubectl can leave stale entries, leading to installation failures with errors about invalid ownership metadata or namespace mismatches.

Next Steps

  1. Automated Cleanup:

    1. Run: helm uninstall --wait gpu-operator nvidia/gpu-operator -n gpu-operator

    2. Replace the namespace if necessary.

  2. Manual Cleanup:

    1. Delete Custom Resource Definitions and deployments.

    2. Remove namespace, ClusterRoles, and ClusterRoleBindings.

For more detailed instructions and additional information, visit the full article here.

GPU Operator Installation Fails on Kubernetes Version Greater Than 1.25 if Pod Security Policy is Enabled#

The GPU Operator installation fails on Kubernetes versions 1.25 and later due to the removal of the PodSecurityPolicy (PSP) API. Users attempting to install GPU Operator versions below 22.09 with PSP enabled encounter the following error: Error: INSTALLATION FAILED: unable to build Kubernetes objects from release manifest: resource mapping not found for name: "gpu-operator-restricted" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

Next Steps

  1. Disable PSP during installation using the following command: helm install --wait gpu-operator nvaie/gpu-operator-1-3 -n gpu-operator --set driver.image=vgpu-guest-driver-1-3 --set psp.enabled=false

  2. Upgrade to the latest GPU Operator version that supports Kubernetes 1.25+.

For more detailed instructions and additional information, visit the full article here.

Linux Driver Installation Issue on Ubuntu 22.04.2 LTS#

The Linux GRID driver fails to install on Ubuntu 22.04.2 LTS with kernel version 6.5.0-41 due to:

  1. There is a mismatch between the GCC version used to build the kernel and the installed GCC version.

  2. An unsupported GCC option -ftrivial-auto-var-init=zero in GCC-11.

Next Steps

  1. Fix GCC version mismatch:

    1. Install GCC-12

    2. Run the NVIDIA installer

    3. Alternatively, update symbolic links for GCC to point to GCC-12

  2. Resolve GCC option issues: Ensure the compiler version matches the kernel build version to avoid unsupported options.

For more detailed instructions and additional information, visit the full article here.

In-Place Upgrade Stuck on Querying DLS Upgrade Workflow#

The DLS 3.3.0 in-place upgrade process gets stuck at Step 3 due to a certificate mismatch between the DLS UI and the In-Place Upgrade Service.

Next Steps

  1. Access the upgrade page using the DLS IP (e.g, https://<DLS_IP>:8443).

  2. Use a browser with lower security settings (e.g., Firefox with Standard settings).

For more detailed instructions and additional information, visit the full article here.

Installation or Upgrade Failure on ESXi Hosts with 500 MB Bootbank#

Installation failures may occur on ESXi hosts that use a legacy 500 MB bootbank partition. This issue arises when the partition lacks sufficient space for components being installed, including NVIDIA vGPU host drivers.

Error signatures include:

  • vLCM upgrade failure:

    Host not compatible with the image. The bootbank partition on the host has a capacity of <size>, the image requires <size>. Remove unnecessary components from the image.

  • esxcli software install failure:

    [InstallationError] The pending transaction requires 524 MB free space, however the maximum supported size is 489 MB.

For detailed instructions and additional information on how to solve this issue, visit the full article here.

NVIDIA vGPU Software Graphics Driver Is Installed Manually on Each Windows VM Instead of Being Deployed Automatically#

When deploying the NVIDIA vGPU guest driver across multiple Windows VMs, users may not be aware that the installer supports silent installation via command-line switches. Without this, the driver must be installed interactively on each VM individually. Silent installation enables automated, unattended deployment through scripts or configuration management tools.

Next Steps

  1. Extract the files from the graphics driver installation package and navigate to the extracted directory.

  2. Run the silent installation command:

    setup.exe -s -n Display.Driver

  3. To capture installation logs, use:

    setup.exe -s -n Display.Driver -log:"C:\test1" -loglevel:6

    Logs are written to the C:\test1 folder, which is created automatically. The log is cryptic and intended for NVIDIA support analysis. Expect many non-critical error entries in the log; these are normal.

  4. Check the installer exit code. Both exit codes below indicate a successful installation:

    Exit code

    Meaning

    0

    Successful installation

    1

    Successful installation, reboot required

  5. After installation, configure NLS licensing by deploying the client configuration token to each VM. More information on how to generate a client configuration token is available here.

How to Upgrade Individual GPU Operator Components#

The GPU Operator deploys each of its components as individual pods. The GPU Operator provides a way to upgrade each component individually by updating the ClusterPolicy Custom Resource Definition (CRD) instance using kubectl, without requiring a full Helm chart upgrade.

Next Steps

  1. Before upgrading, verify that the target component version is validated for your GPU Operator release. Refer to the component matrix.

  2. Find the available versions for the component you want to upgrade in NVIDIA GPU Cloud (NGC). For example, for DCGM, navigate to https://catalog.ngc.nvidia.com/orgs/nvidia/teams/cloud-native/containers/dcgm and use the Copy Image Path dropdown or the Tags tab to find the target version tag.

  3. Edit the ClusterPolicy CRD to update the component version:

    kubectl edit clusterpolicies.nvidia.com

    Use oc instead of kubectl if you are using OpenShift.

  4. In the ClusterPolicy spec, locate the section for the component you want to upgrade, and update the version: field to the new version tag.

  5. Save the changes. Wait for the cluster to upgrade the component pod to the new version.

For more detailed instructions and additional information, visit the full article here.