Upgrading BaseOS Software#

BaseOS 9 provides access to an extensive collection of software from Ubuntu and NVIDIA repositories. Before upgrading or installing software, refer to the BaseOS 9 Release Notes for the latest release information. This section covers common software management tasks, including system upgrades, GPU driver branch changes, CUDA Toolkit installations, and specialized components, such as DOCA-OFED networking software and GPUDirect Storage (GDS) support. You can perform these tasks using the apt command.

Upgrading the System#

Before installing any additional software, you should upgrade the system to the latest versions. This ensures that you can access new software releases added to the repositories since your last upgrade. For more information and instructions about enabling Ubuntu’s security maintenance, refer to Extended Security Maintenance for Ubuntu and Open Source.

Note

  • Before upgrading your system, refer to the BaseOS 9 Release Notes for the upgrade path and supported DGX systems.

  • The latest software branches appear in the package lists only after you upgrade BaseOS.

  • When you switch between software branches, such as the GPU driver or CUDA Toolkit, you must install the packages for the new branch. Depending on the software, the process then removes the existing branch or supports concurrent branches installed on a system.

Changing Your GPU Driver Branch#

NVIDIA drivers are part of the CUDA repository. For more information about the NVIDIA driver release, refer to the release notes in NVIDIA Driver Documentation.

To upgrade the GPU driver branch, use the instructions in Upgrading BaseOS Using the CLI.

Installing or Upgrading to a Newer CUDA Toolkit Release#

DGX servers are intended to be shared resources that use containers and do not have CUDA Toolkit installed by default. However, you have the option to install a qualified CUDA Toolkit release.

Although BaseOS supports all CUDA Toolkit releases that interoperate with the installed driver, BaseOS releases can include a default CUDA Toolkit release that is not necessarily the most recently released version. Unless you must use a new CUDA Toolkit version that contains the new features, remain on the default version that is included in the BaseOS release. Refer to the BaseOS 9 Release Notes for the default CUDA Toolkit release.

Important

Before you install or upgrade to any CUDA Toolkit release, ensure the release is compatible with the driver that is installed on the system. Refer to CUDA Compatibility for more information and a compatibility matrix.

CUDA Compatibility Matrix and Forward Compatibility#

Each CUDA Toolkit requires a minimum GPU driver version. This compatibility matrix is documented in CUDA Compatibility.

You can use newer CUDA Toolkits with older GPU drivers if the appropriate forward compatibility package is installed. Refer to Installing the CUDA Forward Compatibility Package.

Example:

CUDA Toolkit 12.8 requires GPU driver version 570.86.15; however, the installed GPU driver is the Release 550 GPU driver. To use CUDA Toolkit 12.8 with the older GPU driver, you must install the cuda-compat-12-8 forward compatibility package:

sudo apt install cuda-compat-12-8

You can set the LD_LIBRARY_PATH manually:

LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH

Alternatively, you can configure it automatically by modifying the /etc/ld.so.conf file or by adding a file under the /etc/ld.so.conf.d/ directory.

Checking the Currently Installed CUDA Toolkit Release#

Before installing a new CUDA Toolkit release, run the following command to check the currently installed release:

apt list --installed cuda-toolkit-*

The following example output shows that CUDA Toolkit 12.8 is installed:

$ apt list --installed cuda-toolkit-*
Listing... Done
cuda-toolkit-12-8/unknown,now 12.8.1-1 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it

Installing or Upgrading the CUDA Toolkit#

These steps help you determine which new CUDA Toolkit releases are available.

To list the available CUDA Toolkit releases:

  1. Update the local database with the latest information from the Ubuntu repository.

    sudo apt update

  2. Show all available CUDA Toolkit releases.

    apt list cuda-toolkit-*

    The following output shows CUDA Toolkit 12.5, 12.6, 12.8, 13.0, and 13.2 as available versions that can be installed:

    Listing... Done
cuda-toolkit-12-5/unknown 12.5.1-1 amd64
cuda-toolkit-12-6/unknown 12.6.3-1 amd64
cuda-toolkit-12-8/unknown 12.8.1-1 amd64
cuda-toolkit-13-0/unknown 13.0.1-1 amd64
cuda-toolkit-13-2/unknown 13.2.2-1 amd64

  3. To install or upgrade the CUDA Toolkit, run the following command:

    sudo apt install cuda-toolkit-<version>

DOCA-OFED Installation and Configuration#

DGX systems include high-performance network cards for InfiniBand or Ethernet connectivity. NVIDIA DOCA™ OFED is equivalent to MLNX_OFED, providing the same kernel drivers, user-space libraries, and management tools for NVIDIA networking products. For more information about DOCA-OFED, refer to the What Is DOCA-OFED section in MLNX_OFED to DOCA-OFED Transition Guide. For complete installation information, refer to the NVIDIA DOCA Installation Guide for Linux.

Installing the Latest DOCA-OFED Package#

Install the latest version of NVIDIA DOCA-OFED software by adding the NVIDIA DOCA-OFED repository to your system.

  1. Install the nvidia-repositories and nvidia-repo-keys packages to automate secure package access.

    sudo apt update
sudo apt install nvidia-repositories nvidia-repo-keys -y

  2. Refresh your package list to include the newly added DOCA repository.

    sudo apt update

  3. Perform a full upgrade to ensure all dependencies match the new repository.

    sudo apt full-upgrade -y

  4. Install the MLNX drivers.

    sudo apt install nvidia-system-mlnx-drivers -y

Installing a Specific Version of DOCA-OFED#

To install a specific version of NVIDIA DOCA-OFED software, configure the repository to point to the desired version.

  1. Install the nvidia-repositories and nvidia-repo-keys packages to automate secure package access.

    sudo apt update
sudo apt install nvidia-repositories nvidia-repo-keys -y

  2. Configure the DOCA-OFED repository for a specific version.

    Edit the /etc/apt/sources.list.d/doca-bos8-latest.sources file and replace baseos8-latest with your desired version number.

    • For x86_64-based DGX systems:

      For example, to use version 3.0.0-058218 on an x86_64 system:

      URIs: https://linux.mellanox.com/public/repo/doca/3.0.0-058218/ubuntu24.04/x86_64/

    • For ARM64-based DGX systems:

      For example, to use version 3.0.0-058218 on an ARM64 system:

      URIs: https://linux.mellanox.com/public/repo/doca/3.0.0-058218/ubuntu24.04/arm64-sbsa/

  3. Refresh your package list to include the newly added DOCA repository.

    sudo apt update

  4. Perform a full upgrade to ensure all dependencies match the new repository.

    sudo apt full-upgrade -y

  5. Install the MLNX drivers.

    sudo apt install nvidia-system-mlnx-drivers -y

Installing GPUDirect Storage Support#

NVIDIA Magnum IO GPUDirect Storage (GDS) enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage. This software avoids a bounce buffer through the CPU.

Note

This section applies only to the following situations:

  • You plan to use GPUDirect Storage on bare metal without Linux PCI P2PDMA.

  • You plan to upgrade or reinstall the DOCA-OFED package and the nvidia-fs module because BaseOS provides the updated versions.

Installing GDS Components#

On DGX servers (DGX B300, DGX B200, DGX H100/H200, DGX H800, and DGX A100/A800):

  • Install the nvidia-gds package.

    sudo apt install nvidia-gds

Enabling Relaxed Ordering for NVMe Drives#

The Samsung NVMe drives used in the NVIDIA DGX systems support relaxed ordering for I/O operations. Relaxed ordering enables the PCIe bus to complete transactions out of order. NVIDIA recommends enabling this setting when you use GPUDirect Storage to improve performance.

  • Run the nvidia-relaxed-ordering-nvme.sh utility.

    sudo /bin/nvidia-relaxed-ordering-nvme.sh enable

Note

This feature applies only to DGX A100/A800 systems.

Configuring NVMe Interrupt Coalescing#

The nvidia-nvme-options package, which is installed on all DGX systems, automatically configures NVMe interrupt coalescing on all Samsung and Kioxia drives at each boot. To disable this setting or manually configure the setting, run the following commands:

To disable the setting:

sudo systemctl stop nvidia-nvme-interrupt-coalescing.service
sudo systemctl disable nvidia-nvme-interrupt-coalescing.service

To configure the setting manually:

sudo /usr/bin/nvidia-nvme-interrupt-coalescing.sh enable
sudo /usr/bin/nvidia-nvme-interrupt-coalescing.sh disable

Next Steps#

Refer to Verifying a Successful GDS Installation in the NVIDIA GPUDirect Storage Installation and Troubleshooting Guide.