Advanced Options#

This section contains information on some advanced setup scenarios which are not covered in the basic instructions above.

Switching between Driver Module Flavors#

Use the following steps to switch between the NVIDIA driver proprietary and open module flavors on your system.

Note

Replace XXX with the NVIDIA driver branch number such as 580.

Amazon Linux 2023, KylinOS 10, Red Hat Enterprise Linux 8/9, Rocky Linux 8/9, Oracle Linux 8/9

To switch between proprietary and open kernel modules:

# dnf -y module switch-to nvidia-driver:<stream> --allowerasing

Red Hat Enterprise Linux 10, Rocky Linux 10, Fedora 42

To switch from proprietary to open:

# dnf install --allowerasing nvidia-open

To switch from open to proprietary:

# dnf install --allowerasing cuda-drivers

If you have done a desktop or compute-only installation, you can just switch the kernel module package. For example to go from proprietary to open:

# dnf install --allowerasing kmod-nvidia-open-dkms

Azure Linux 2/3

Only the open kernel modules are supported, no switching is possible.

Debian 12

To switch from proprietary to open:

# apt install --autoremove --purge nvidia-open

To switch from open to proprietary:

# apt install --autoremove --purge cuda-drivers

Ubuntu 22.04/24.04

To switch from proprietary to open:

# apt install --autoremove --purge nvidia-open-580 nvidia-driver-580-open

To switch from open to proprietary:

# apt install --autoremove --purge cuda-drivers-580 nvidia-driver-580

OpenSUSE Leap 15, SUSE Linux Enterprise Server 15

To switch from proprietary to open:

# zypper install --details --force-resolution nvidia-open

To switch from open to proprietary:

# zypper install --details --force-resolution cuda-drivers

Meta Packages#

Meta packages are rpm/deb packages which contain no (or few) files but have multiple dependencies. They are used to install many CUDA packages when you may not know the details of the packages you want. The following table lists the meta packages. Not all of the packages listed below are available on every distribution. Sometime they are provided by other packages, sometimes they are not needed as there are other mechanisms offered by the distribution to provide branch segregation (ex. DNF modules).

Table 7 Kernel Modules Meta Packages#

Meta Package

Purpose

nvidia-open

Installs all NVIDIA Open GPU kernel modules Driver packages. Handles upgrading to the next version of the driver packages when they are released.

nvidia-open-580

Installs all NVIDIA Open GPU kernel modules Driver packages. Will not upgrade beyond the 580.xxx branch drivers.

cuda-drivers

Installs all NVIDIA proprietary kernel modules Driver packages. Handles upgrading to the next version of the Driver packages when they are released.

cuda-drivers-580

Installs all NVIDIA proprietary kernel modules Driver packages. Will not upgrade beyond the 580.xx branch drivers.

Package Upgrades#

When a new version is available, a normal package update command specific for the distribution should suffice in upgrading the driver. The various differences in distributions would take care of obsolency and package switching when performing upgrades to a different branch.

The following section shows some examples of the commands that can be performed.

Red Hat Enterprise Linux 8/9, Rocky Linux 8/9, Oracle Linux 8/9, KylinOS 10, Amazon Linux 2023#

When upgrading the driver to the same stream:

# dnf update

When upgrading the driver to a different stream:

# dnf module reset nvidia-driver
# dnf module enable nvidia-driver:<stream>
# dnf update --allowerasing

Or when switching from proprietary modules to open modules:

# dnf module reset nvidia-driver
# dnf module enable nvidia-driver:<stream>
# dnf install nvidia-open --allowerasing

Red Hat Enterprise Linux 10, Rocky Linux 10, Fedora 42#

If DNF locking is configured on the sytem, please adjust the configuration or remove the lock entirely. Please refer to the DNF 4/5 paragraph of the Version locking section for more information.

When upgrading the driver, whether configured to a version locked branch or the latest available, the command to execute is always the same; what matters is the DNF version lock configuration:

# dnf update

When switching from proprietary modules to open modules:

# dnf install nvidia-open --allowerasing

This will remove any package which would have dependencies removed.

Azure Linux 2/3#

When upgrading the driver to the same stream:

# tdnf update

When upgrading the driver to a different stream:

# tdnf install --allowerasing nvidia-open-580

This will remove any package which would have dependencies removed.

SUSE Enterprise Linux Server 15, OpenSUSE Leap 15#

When upgrading the driver to the same stream:

# zypper update --details

When upgrading the driver to a different stream:

# zypper install --details --force-resolution nvidia-open-580

This will remove any package which would prevent you from reaching the target.

Debian 12#

If APT pinning is configured on the sytem, please adjust the configuration or remove the pinning entirely. Please refer to the APT paragraph of the Version locking section for more information.

When upgrading the driver, whether configured to a pinned branch or the latest available, the command to execute is always the same; what matters is the APT pinning configuration:

# apt dist-upgrade

Or a bit more aggressive, to take into consideration held back packages that might have changed betweend driver releases, for example to upgrade from nvidia-open-565 to nvidia-open-580:

# apt dist-upgrade --autoremove --purge nvidia-open-580

This will remove any package which would have dependencies removed.

Ubuntu 22.04/24.04#

When updating packages of the same driver branch or without a specific tracking branch:

# apt dist-upgrade

Or a bit more aggressive, to take into consideration held back packages that might have changed betweend driver releases, for example to upgrade from nvidia-open-565 to nvidia-open-580:

# apt dist-upgrade --autoremove --purge nvidia-open-580

This will remove any package which would have dependencies removed.

Red Hat Enterprise Linux 8/9 Precompiled Streams#

Precompiled streams offer an optional method of streamlining the installation process. The advantages of precompiled streams:

  • Signed by Red Hat: allows Secure Boot and kernel signature validation to be completely enabled

  • Precompiled: faster boot up after driver and/or kernel updates

  • Pre-tested: kernel and driver combination has been validated

  • Removes compiler and associated dependencies: no compiler installation required

  • Removes DKMS dependency: enabling EPEL repository not required

  • Removes kernel-devel and kernel-headers dependencies: no black screen if matching packages are missing

When using precompiled drivers, a plugin for the DNF package manager is enabled that prevents system breakages by preventing upgrades to a kernel for which no precompiled driver yet exists. This can delay the application of kernel updates, but ensures that a tested kernel and driver combination is always used. A warning is displayed by dnf during that upgrade situation:

NVIDIA driver: some kernel packages have been filtered due to missing precompiled modules.
Please run "dnf nvidia-plugin" as a command to see a report on the filter being applied.

Additional information is shown for all kernels and precompiled modules that are available for the system by running the plugin as a standlone command. For example:

# dnf nvidia-plugin
Last metadata expiration check: 0:27:16 ago on Wed 20 Nov 2024 08:09:53 PM CET.

Installed kernel(s):
  kernel-core-5.14.0-503.14.1.el9_5.x86_64
  kernel-core-5.14.0-427.42.1.el9_4.x86_64

Available kernel(s):
  kernel-core-5.14.0-503.11.1.el9_5.x86_64
  kernel-core-5.14.0-503.14.1.el9_5.x86_64

Available driver(s):
  nvidia-driver-3:565.57.01-1.el9.x86_64
  nvidia-driver-cuda-3:565.57.01-1.el9.x86_64

Available kmod(s):
  kmod-nvidia-3:565.57.01-2.el9.x86_64

Packaging templates and instructions are provided on GitHub to allow you to maintain your own precompiled kernel module packages for custom kernels and derivative Linux distros: NVIDIA/yum-packaging-precompiled-kmod

To use the new driver packages on RHEL 8 or RHEL 9:

  1. First, ensure that the necessary Red Hat repositories are enabled.

    Compared to the normal DKMS installation, this requires less repositories to be enabled on the system.

    Red Hat Enterprise Linux 8:

    # subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms
# subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms

    Red Hat Enterprise Linux 9:

    # subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
# subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms

  2. Choose one of the options below depending on the desired driver:

    • latest always updates to the highest versioned driver (precompiled):

      # dnf module enable nvidia-driver:latest

    • Locks the driver updates to the specified driver branch (precompiled). Replace <id> with the appropriate driver branch streams, for example 570, 560, 550, etc.:

      # dnf module enable nvidia-driver:<id>

  3. Install the preferred driver combination:

    For example:

    # dnf install nvidia-open
# dnf install nvidia-driver-cuda
# dnf install nvidia-driver nvidia-settings
# dnf install nvidia-driver nvidia-driver-cuda

Please refer to the precompiled folder in the driver repositories for more information.

Precompiled Streams Support Matrix#

This table shows an example of the supported precompiled, legacy, and open DKMS streams for each driver.

Table 8 Streams Support Matrix#

NVIDIA Driver

Precompiled Stream

Legacy DKMS Stream

Open DKMS Stream

Highest version

latest

latest-dkms

open-dkms

Locked at 580.x

580

580-dkms

580-open

Locked at 575.x

575

575-dkms

575-open

Locked at 570.x

570

570-dkms

570-open

Locked at 565.x

565

565-dkms

565-open

Locked at 560.x

560

560-dkms

560-open

Locked at 550.x

550

550-dkms

550-open

Prior to switching between module streams, first reset the DNF module:

# dnf module reset nvidia-driver

Or as an alternative:

# dnf module switch-to nvidia-driver:<stream>

Modularity Profiles#

Modularity profiles work with any supported modularity stream and allow for additional use cases. These modularity profiles are available on Amazon Linux 2023, KylinOS 10, and Red Hat Enterprise Linux 8/9.

Table 9 Modularity Profiles#

Stream

Profile

Use Case

Default

/default

Installs all the driver packages in a stream.

Kickstart

/ks

Performs unattended Linux OS installation using a config file.

NVSwitch Fabric

/fm

Installs all the driver packages plus components required for bootstrapping an NVSwitch system (including the Fabric Manager and NSCQ telemetry).

For example:

# dnf module install nvidia-driver:<stream>/default
# dnf module install nvidia-driver:<stream>/ks
# dnf module install nvidia-driver:<stream>/fm

You can install multiple modularity profiles using BASH curly brace expansion, for example:

# dnf module install nvidia-driver:latest/{default,fm}

Please refer to the Developer Blog for more information.

Red Hat Enterprise Linux 8/9 Kickstart Installation#

Edit the Kickstart configuration file as follows.

Red Hat Enterprise Linux 8, Rocky Linux 8, Oracle Linux 8

  1. Enable the EPEL repository:

    repo --name=epel --baseurl=http://download.fedoraproject.org/pub/epel/8/Everything/x86_64/

  2. Enable the CUDA repository:

    repo --name=cuda-rhel8 --baseurl=https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/

  3. If you are not using driver 570 or later, in the packages section of the Kickstart file, make sure you are using the /ks profile:

    @nvidia-driver:open-dkms/ks

This is not needed from 570 and above, you can skip the /ks profile.

  1. Perform the Post-installation Actions.

Red Hat Enterprise Linux 9, Rocky Linux 9, Oracle Linux 9

  1. Enable the EPEL repository:

    repo --name=epel --baseurl=http://download.fedoraproject.org/pub/epel/9/Everything/x86_64/

  2. Enable the CUDA repository:

    repo --name=cuda-rhel9 --baseurl=https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/

  3. If you are not using driver 570 or later, in the packages section of the Kickstart file, make sure you are using the /ks profile:

    @nvidia-driver:open-dkms/ks

  4. Perform the Post-installation Actions.

SUSE Vendor Change#

Starting with driver version 575, all the driver packages for openSUSE Leap and SUSE Enterprise Linux Server have a vendor of “NVIDIA”. The system will prevent the update of packages hosted in the other repositories unless there is an override for the vendor.

More information on the topic is available in the Vendor Change page of openSUSE.

This can be triggered on the command line:

# zypper update --allow-vendor-change
Loading repository data...
Reading installed packages...

The following package is going to be upgraded:
  dkms

The following package is going to change vendor:
  dkms  openSUSE -> NVIDIA

1 package to upgrade, 1 to change vendor.

Package download size:    58.0 KiB

Package install size change:
              |     148.7 KiB  required by packages that will be installed
     5.4 KiB  |  -  143.3 KiB  released by packages that will be removed

Backend:  classic_rpmtrans
Continue? [y/n/v/...? shows all options] (y):

Or by adding the vendor equivalence to a Zypper configuration file. For example, in the case of openSUSE, just add “NVIDIA” at the end of this configuration file:

$ cat /etc/zypp/vendors.d/00-openSUSE.conf
[main]
vendors=openSUSE,SUSE,SUSE LLC <https://www.suse.com/>,NVIDIA

And then the upgrade can happen allowing the vendor change:

# zypper update --details
Loading repository data...
Reading installed packages...

The following package is going to be upgraded:
  dkms  3.0.11-bp156.1.2 -> 3.1.5-1  noarch  test  NVIDIA

1 package to upgrade.

Package download size:    58.0 KiB

Package install size change:
              |     148.7 KiB  required by packages that will be installed
     5.4 KiB  |  -  143.3 KiB  released by packages that will be removed

Backend:  classic_rpmtrans
Continue? [y/n/v/...? shows all options] (y):

Restrict APT to Look for Specific Architectures#

Modify Ubuntu’s apt package manager to query specific architectures for specific repositories. This is useful when a foreign architecture has been added, causing “404 Not Found” errors to appear when the repository meta-data is updated.

Each repository you wish to restrict to specific architectures must have its sources.list entry modified. This is done by modifying the /etc/apt/sources.list file and any files containing repositories you wish to restrict under the /etc/apt/sources.list.d/ directory. Normally, it is sufficient to modify only the entries in /etc/apt/sources.list.

An architecture-restricted repository entry looks like:

deb [arch=<arch1>,<arch2>] <url>

For example, if you wanted to restrict a repository to only the amd64 and i386 architectures, it would look like:

deb [arch=amd64,i386] <url>

It is not necessary to restrict the deb-src repositories, as these repositories don’t provide architecture-specific packages.

For more details, see the sources.list manpage.

APT Repository File not Found#

In case of the error: E: Failed to fetch file:/var/cuda-repo File not found on Debian and Ubuntu systems.

This can occur when installing CUDA after uninstalling a different version. Use the following command before installation:

# rm -v /var/lib/apt/lists/*cuda* /var/lib/apt/lists/*nvidia*

Verbose Versions when Using APT#

Use the --verbose-versions flag, for example:

# apt install --verbose-versions nvidia-open