DGX OS 6 / Ubuntu 22.04

The NVIDIA DGX OS 6 User Guide is also available as a PDF.

• About DGX OS 6
  • DGX OS 6 Features
  • Installation and Upgrade
  • Related Documentation
  • NVIDIA Enterprise Support
• Release Guidance
  • Release Mechanisms
    • DGX OS ISO Image
    • Linux Software Repositories
  • Release Numbering Convention
  • Release Cadence
• Release Notes
  • Current Software Versions
  • Supported DGX Systems
  • Update History
    • DGX OS 6.3.1 Release: September 17, 2024
    • DGX OS 6.3.0 Release: August 9, 2024
    • DGX OS 6.2.1 Release: May 2, 2024
    • DGX OS 6.2.0 Release: March 22, 2024
    • DGX OS 6.1.0 Release: August 11, 2023
    • DGX OS 6.0.11 Release: May 17, 2023
    • DGX OS 6.0.10 Release: May 3, 2023
  • DGX OS ISO Releases
    • DGX OS ISO 6.3.1
    • DGX OS ISO 6.3.0
    • DGX OS ISO 6.2.1
    • DGX OS ISO 6.2.0
    • DGX OS ISO 6.1.0
    • DGX OS ISO 6.0.11
    • DGX OS ISO 6.0.10
• Initial Setup
  • Connecting to the DGX System
  • First Boot Setup Wizard
    • First Boot Process for DGX Servers
    • First Boot Process for DGX Station
  • Post-Installation Tasks
    • Adding Support for Additional Languages to the DGX Station
    • Configuring your DGX Station
    • Configuring your DGX Station V100
    • Enabling Multiple Users to Remotely Access the DGX System
• Reimaging the System
  • Obtaining the DGX OS ISO Image
  • Installing the DGX OS Image
    • Installing the DGX OS Image Remotely through the BMC
    • Installing the DGX OS Image from a USB Flash Drive or DVD-ROM
      • Creating a Bootable USB Flash Drive by Using the dd Command
      • Creating a Bootable USB Flash Drive by Using Akeo Rufus
    • Booting the DGX OS ISO image
  • DGX OS ISO Boot Options
    • Install DGX OS
    • Install DGX OS without Reformatting the Data RAID
    • Advanced Installation Options (Encrypted Root)
    • Boot Into a Live Environment
    • Check Disc for Defects
• Installing DGX Software on Ubuntu
  • Prerequisites
    • Ubuntu Software Requirements
    • Access to Software Repositories
  • Installation Considerations
    • System Drive Mirroring (RAID-1) [recommended]
    • Data Drive RAID-0 or RAID-5
    • System Drive Encryption [optional]
    • Data Drive Encryption [optional]
    • System Drive Partitioning
  • Installing Ubuntu
    • Booting from the Installation Media
    • Running the Ubuntu Installer
  • Installing the DGX Software Stack
    • Installing DGX System Configurations and Tools
    • Configuring Data Drives
    • Installing the GPU Driver
    • Installing the Mellanox OpenFabrics Enterprise Distribution (MLNX_OFED)
    • Installing Docker and the NVIDIA Container Toolkit
    • Installing the NVIDIA System Management (NVSM) Tool [Recommended]
    • Additional Software Installed By DGX OS
  • Next Steps and Additional Information
• Upgrading the OS
  • DGX OS 6 Release Upgrade Advisory
  • Getting Release Information for DGX Systems
  • Preparing to Upgrade the Software
    • Connect to the DGX System Console
    • Verifying the DGX System Connection to the Repositories
  • Performing a Release Upgrade from DGX OS 5
    • Upgrade DGX OS 5 to the Latest Version
    • Performing the Release Upgrade
    • Resolving Release Upgrade Conflicts
    • Verifying the Upgrade
    • Recovering from an Interrupted or Failed Update
  • Performing a Release Upgrade with Base Command Manager
  • Performing Package Upgrades
    • Enabling Extended Security Maintenance Upgrades
    • Performing Package Upgrades Using the CLI
    • Managing Software Upgrades on DGX Station
    • Performing Package Upgrades Using the GUI
    • Checking for Updates to DGX Station Software
• System Configurations
  • Network Configuration
    • Configuring Network Proxies
    • For the OS and Most Applications
    • For the apt Package Manager
  • Configuring ConnectX from InfiniBand to Ethernet
    • Determining the Current Port Configuration
    • Configuring the Port
    • Related Information for Configuring ConnectX from InfiniBand to Ethernet
  • Docker Configuration
    • Preparing the DGX System to be Used With Docker
    • Enabling Users To Run Docker Containers
    • Configuring Docker IP Addresses
    • Connectivity Requirements for NGC Containers
    • Configuring Static IP Addresses for the Network Ports
  • Managing CPU Mitigations
    • Determining the CPU Mitigation State of the DGX System
    • Disabling CPU Mitigations
    • Re-enable CPU Mitigations
  • Managing the DGX Crash Dump Feature
    • Using the Script
  • Connecting to Serial Over LAN
  • Filesystem Quotas
  • Running Workloads on Systems with Mixed Types of GPUs
    • Running with Docker Containers
    • Running on Bare Metal
  • Using Multi-Instance GPUs
  • Updating the containerd Override File for MIG configurations
  • Data Storage Configuration
    • Using Data Storage for NFS Caching
    • Using cachefilesd
    • Disabling cachefilesd
    • Changing the RAID Configuration for Data Drives
  • Running NGC Containers
    • Obtaining an NGC Account
    • Running NGC Containers with GPU Support
• Installing ConnectX-7 Firmware
• Managing Self-Encrypting Drives
  • Overview
  • Installing the Software
  • Configuring Trusted Computing
    • Determining Whether Drives Support SID
    • Enabling the TPM and Preventing the BIOS from Sending Block SID Requests
  • Initializing the System for Drive Encryption
  • Enabling Drive Locking
  • Initialization Examples
    • Example 1: Passing in the JSON File
      • Determining Which Drives Can be Managed as Self-Encrypting
      • Creating the Drive/Password Mapping JSON Files and Using it to Initialize the System
    • Example 2: Generating Random Passwords
    • Example 3: Specifying Passwords One at a Time When Prompted
  • Disabling Drive Locking
  • Enabling Drive Locking
  • Exporting the Vault
  • Erasing Your Data
  • Clearing the TPM
  • Changing Disk Passwords, Adding Disks, or Replacing Disks
  • Recovering From Lost Keys
• Managing and Upgrading Software
  • Upgrading the System
  • Changing Your GPU Branch
    • Checking the Currently Installed Driver Branch
    • Determining the New Available Driver Branches
    • Upgrading Your GPU Branch
  • Installing or Upgrading to a Newer CUDA Toolkit Release
    • CUDA Compatibility Matrix and Forward Compatibility
    • Checking the Currently Installed CUDA Toolkit Release
    • Installing or Upgrading the CUDA Toolkit
  • Installing the Mellanox OFED Drivers
    • Using the Mellanox OFED Packages
    • Using the Ubuntu OFED Packages
    • Inbox OFED vs Mellanox OFED Use Cases
    • Initialize on Alloc Performance Impact
    • Upgrading Firmware for Mellanox ConnectX Cards
      • Checking the Device Type
      • Download the New Firmware
      • Program the Firmware
  • Installing GPUDirect Storage Support
    • Prerequisites
    • Installing GDS Components for the Optimized NVIDIA Kernel
    • Installing GDS Components for the Generic Kernel
    • Enabling Relaxed Ordering for NVMe Drives
    • Next Steps
  • Selecting a different Linux kernel
    • Boot the system to the generic kernel one time
    • Boot the system to the generic kernel by default
• Known Issues
  • DGX System Device ID Not Found in /usr/share/misc/pci.ids
    • Issue
    • Workaround
  • Virtualization Not Supported
    • Issue
  • Excessive Growth of OpenSM Log Causing DGX Systems to Become Inoperable
    • Issue
    • Explanation
  • Incorrect DCGM Version After Upgrade from 5.X to 6.2.0
    • Issue
    • Workaround
  • Errors Occur When Loading Mirrored Repositories on Air-Gapped Systems
    • Issue
    • Explanation
    • Workaround
  • Reduced Network Communication Speeds on DGX H100 System
    • Issue
    • Explanation
    • Workaround
  • NVSM Raises Alerts for Missing Devices on DGX H100 System
    • Issue
    • Explanation
    • Workaround
  • DGX A800 Station/Server: mig-parted config
    • Issue
    • Workaround
  • Erroneous Insufficient Power Error May Occur for PCIe Slots
    • Issue
    • Explanation
  • Applications that call the cuCTXCreate API Might Experience a Performance Drop
    • Issue
    • Explanation
  • Incorrect nvidia-container-toolkit version after upgrade from 5.X to 6.0
    • Issue
    • Explanation
  • UBSAN error and mstconfig stack dump in kernel logs at boot
    • Issue
    • Explanation
  • The BMC Redfish interface is not active on first boot after installation
    • Issue
    • Explanation

Appendices

• DGX OS Connectivity Requirements
  • In-Band Management, Storage, and Compute Networks
  • Out-of-Band Management
• DGX Software Stack
  • NVIDIA DGX Software Packages
  • Base OS 6.3.1 Installed Packages
  • DGX Kernel Parameters
• PXE Boot Setup
  • Pre-requisites
  • Overview of the PXE Server
    • Configuring the HTTP File Directory and ISO Image
  • Mount the BaseOS 6.0.0 ISO
  • Configure the TFTP directory
  • Parameters unique to the Base OS installer
  • Configure DHCP
  • Optional: Configure CX-4/5/6/7 cards to PXE boot
  • Query UEFI PXE ROM state
  • MOFED Instructions
  • Optional: Configure the DGX-Server to PXE boot automatically
    • Add PXE to the top of the UEFI boot order
  • Configure network boot priorities
  • Make the DGX-Server PXE boot
    • Automated PXE Boot Process
    • Manual PXE Boot Process
  • Other IPMI boot options
  • Autoinstall Customizations
  • NVIDIA-Specific Autoinstall Variables
  • Common Customizations
  • Network Configuration
  • Creating a User
• Air-Gapped Installations
  • Creating a Local Mirror of the NVIDIA and Canonical Repositories
  • Creating the Mirror of the Repositories
  • Configuring the Target Air-Gapped System
• Cloud-init Configuration File
  • Modifying the Configuration File
  • Drive Partitioning
• Installing Docker Containers

Legal Information

• Third-Party License Notices
  • Micron msecli
  • Mellanox (OFED)
• Notices
  • Notice
  • Trademarks