DGX OS 5 / Ubuntu 20.04

• Introduction
  • DGX OS 5 Features
  • Overview
  • Additional Documentation
  • NVIDIA Enterprise Support
• Release Guidance
  • Release Mechanisms
    • The ISO Image
    • The Linux Software Repositories
  • Release Numbering Convention
  • Release Cadence
• Release Notes
  • Current Software Versions
  • Supported DGX Systems
  • DGX OS Releases
    • DGX OS 5.5
    • DGX OS 5.4
    • DGX OS 5.3
    • DGX OS 5.2
    • DGX OS 5.1
    • DGX OS 5.0
  • Update History
    • Update: October 17, 2023
    • DGX OS 5.5 Release: April, 2023
    • Update: November 22, 2022
    • Update: October 14, 2022
    • DGX OS 5.4 Release: August 8, 2022
    • Update: June 7, 2022
    • Update: May 17, 2022
    • DGX OS 5.3 Release: April 28, 2022
    • DGX OS 5.2 Release: February 17, 2022
    • Update: December 14, 2021
    • Update: October 26 , 2021
    • DGX OS 5.1 Release: August 26, 2021
    • Update: June 30 , 2021
    • Update: June 20 , 2021
    • Update: June 2, 2021
    • Update: May 27, 2021
    • Update: May 06, 2021
    • Update: April 20, 2021
    • Update: April 13, 2021
    • Update: March 30, 2021
    • Update: March 2, 2021
    • Update: February 23, 2021
    • Update: January 20, 2021
    • Update: December 11, 2020
    • Update: October 31, 2020 (DGX OS 5.0 Release)
  • DGX OS ISO Releases
    • DGX OS ISO 5.5.0
    • DGX OS ISO 5.4.1
    • DGX OS ISO 5.3.1
    • DGX OS ISO 5.2.0
    • DGX OS ISO 5.1.0
    • DGX OS ISO 5.0.2
    • DGX OS ISO 5.0.0
• Initial Setup
  • Connecting to the DGX System
  • First Boot Setup Wizard
    • First Boot Process for DGX Servers
    • First Boot Process for DGX Station
  • Post-Installation Tasks
    • Adding Support for Additional Languages to the DGX Station
    • Configuring your DGX Station
    • Configuring your DGX Station V100
    • Enabling Multiple Users to Remotely Access the DGX System
• Reimaging
  • Obtaining the DGX OS ISO Image
  • Installing the DGX OS Image
    • Installing the DGX OS Image Remotely through the BMC
    • Installing the DGX OS Image from a USB Flash Drive or DVD-ROM
      • Creating a Bootable USB Flash Drive by Using the dd Command
      • Creating a Bootable USB Flash Drive by Using Akeo Rufus
    • Booting the DGX OS ISO image
  • DGX OS ISO Boot Options
    • Install DGX OS and Reformat the Data RAID
    • Install DGX OS without Reformatting the Data RAID
    • Advanced Installation Options (Encrypted Root)
    • Boot Into a Live Environment
    • Check Disc for Defects
• Installing on Ubuntu
  • Prerequisites
    • Ubuntu Software Requirements
    • Access to Software Repositories
  • Installation Considerations
    • System Drive Mirroring (RAID-1) [recommended]
    • Data Drive RAID-0 or RAID-5
    • System Drive Encryption [optional]
    • Data Drive Encryption [optional]
    • System Drive Partitioning
  • Installing Ubuntu
    • Booting from the Installation Media
    • Running the Ubuntu Installer
  • Installing the DGX Software Stack
    • Installing DGX System Configurations and Tools
    • Configuring Data Drives
    • Installing the GPU Driver
    • Installing the Mellanox OpenFabrics Enterprise Distribution (MLNX_OFED)
    • Installing Docker and the NVIDIA Container Toolkit
    • Installing the NVIDIA System Management (NVSM) Tool [Recommended]
    • Additional Software Installed By DGX OS
  • Next Steps and Additional Information
• Upgrading
  • DGX OS 5 Release Upgrade Advisements
  • Getting Release Information for DGX Systems
  • Preparing to Upgrade the Software
    • Connect to the DGX System Console
    • Verifying the DGX System Connection to the Repositories
  • Rotating the GPG Keys
    • Rotating the GPG Key For a Default Installation or After Reimaging
    • Rotating the GPG Keys for the DGX Software Stack
  • Performing a Release Upgrade from DGX OS 4
    • Upgrade DGX OS 4 to the Latest Version
    • Performing the Release Upgrade
    • Resolving Release Upgrade Conflicts
    • Verifying the Upgrade
    • Recovering from an Interrupted or Failed Update
  • Performing Package Upgrades
    • Enabling Extended Security Maintenance Upgrades
    • Performing Package Upgrades Using the CLI
    • Managing Software Upgrades on DGX Station
    • Performing Package Upgrades Using the GUI
    • Checking for Updates to DGX Station Software
• System Configurations
  • Network Configuration
    • Configuration Network Proxies
    • For the OS and Most Applications
    • For the apt Package Manager
  • Docker Configuration
    • Preparing the DGX System to be Used With Docker
    • Enabling Users To Run Docker Containers
    • Configuring Docker IP Addresses
    • Connectivity Requirements for NGC Containers
    • Configuring Static IP Addresses for the Network Ports
  • Managing CPU Mitigations
    • Determining the CPU Mitigation State of the DGX System
    • Disabling CPU Mitigations
    • Re-enable CPU Mitigations
  • Managing the DGX Crash Dump Feature
    • Using the Script
  • Connecting to Serial Over LAN
  • Filesystem Quotas
  • Running Workloads on Systems with Mixed Types of GPUs
    • Running with Docker Containers
    • Running on Bare Metal
  • Using Multi-Instance GPUs
  • Data Storage Configuration
    • Using Data Storage for NFS Caching
    • Using cachefilesd
    • Disabling cachefilesd
    • Changing the RAID Configuration for Data Drives
  • Running NGC Containers
    • Obtaining an NGC Account
    • Running NGC Containers with GPU Support
• Additional Software
  • Upgrading the System
    • Changing Your GPU Branch
  • Checking the Currently Installed Driver Branch
  • Determining the New Available Driver Branches
  • Upgrading Your GPU Branch
    • Installing or Upgrading to a Newer CUDA Toolkit Release
  • CUDA Compatibility Matrix and Forward Compatibility
  • Checking the Currently Installed CUDA Toolkit Release
  • Installing or Upgrading the CUDA Toolkit
  • Installing or Upgrading GPUDirect Storage
    • Prerequisites
    • Procedure
  • Installing nvidia_peermem
• Known Issues
  • Known Issue Overview
  • Known Issues Details
    • Errors Occur When Loading Mirrored Repositories on Air-Gapped Systems
      • Issue
      • Explanation
      • Workaround
    • DGX A800 Station/Server: DCGM Diagnostics may return Skip - All
      • Issue
      • Explanation
      • Workaround
    • DGX A800 Station/Server: mig-parted config
      • Issue
      • Workaround
    • Regression of CUDA application startup performance
      • Issue
      • Explanation
    • NVSM Stress Test Logs Do Not Contain Summary Information
      • Issue
      • Explanation
    • nvidia-release-upgrade May Report That Not All Updates Have Been Installed and Exit
      • Issue
      • Explanation
    • Duplicate EFI Variable May Cause efibootmgr to Fail
      • Issue
      • Explanation
    • Erroneous Insufficient Power Error May Occur for PCIe Slots
      • Issue
      • Explanation
    • AMD Crypto Coprocessor is not Supported
      • Issue
      • Explanation
    • nvsm show alerts Reports NVSwitch PCIe Link Width is Degraded
      • Issue
      • Explanation
    • nvsm show health Reports Firmware as Not Authenticated
      • Issue
      • Explanation
    • Running NGC Containers Older than 20.10 May Produce “Incompatible MOFED Driver” Message
      • Issue
      • Explanation
    • System May Slow Down When Using mpirun
      • Issue
      • Explanation
    • Forced Reboot Hangs the OS
      • Issue
      • Explanation
    • Applications that call the cuCTXCreate API Might Experience a Performance Drop
      • Issue
      • Explanation
    • NVIDIA Desktop Shortcuts Not Updated After a DGX OS Release Upgrade
      • Issue
      • Explanation
    • Unable to Set a Separate/xinerama Mode through the xorg.conf File or through nvidia-settings
      • Issue
      • Explanation
  • Known Limitations Details
    • No RAID Partition Created After ISO Install
      • Issue
      • Explanation
    • System Services Startup Messages Appear Upon Completion of First-Boot Setup
      • Issue
      • Explanation
    • [DGX A100]: Hot-plugging of Storage Drives not Supported
      • Issue
      • Explanation and Workaround
    • [DGX A100]: Syslog Contains Numerous “SM LID is 0, maybe no SM is running” Error Messages
      • Issue
      • Explanation and Workaround
    • [DGX-2]: Serial Over LAN Does not Work After Cold Resetting the BMC
      • Issue
      • Explanation and Workaround
    • [DGX-2]: Some BMC Dashboard Quick Links Appear Erroneously
      • Issue
    • [DGX-2]: Applications Cannot be Run Immediately Upon Powering on the DGX-2
      • Issue
      • Explanation and Workaround
    • [DGX-1]: Script Cannot Recreate RAID Array After Re-inserting a Known Good SSD
      • Issue
      • Explanation and Workaround
    • [DGX Station A100] Suspend and Power Button Section Appears in Power Settings
      • Issue
      • Explanation
    • [DGX-2] NVSM Does not Detect Downgraded GPU PCIe Link
      • Issue
      • Explanation
  • Resolved Issues Details
    • NVSM Platform Displays as Unsupported
      • Issue
      • Explanation
    • NVSM Enumerates NVSwitches as 8-13 Instead of 0-5
      • Issue
      • Explanation
    • [DGX A100] A System with Encrypted rootfs May Fail to Boot if one of the M.2 drives is Corrupted
      • Issue
      • Explanation
    • NVSM Fails to Show CPU Information on Non-English Locales
      • Issue
      • Explanation
    • Driver Version Mismatch Reported
      • Issue
      • Explanation

Appendices

• DGX OS Connectivity Requirements
  • In-Band Management, Storage, and Compute Networks
  • Out-of-Band Management
• Downgrading ConnectX Firmware
  • Checking the Device Type
  • Downgrading the Firmware
• DGX Software Stack
  • NVIDIA DGX Software Packages
  • DGX Kernel Parameters
• PXE Boot Setup
  • Overview of the PXE Server
    • Setting Up dnsmasq for DHCP
    • Configure the FTP Side
    • Configuring the HTTP File Directory and ISO Image
    • Configure your DHCP Server
    • Configuring the syslinux Bootloader
    • Live Boot Parameters
    • Setup GRUB PXE Server
    • Optional: Configure the NVIDIA ConnextX cards to PXE boot
    • Query the UEFI PXE ROM State
    • Enable the UEFI PXE ROM
    • Curtin Customizations
    • Customizing the Curtin YAML
      • NVIDIA-Specific Curtin Variables
      • CHANGE_STORAGE_REG
      • CHANGE_STORAGE_ENC
      • CHANGE_BOOT_DISK_NAME_x
    • Boot the DGX System over PXE
    • Reference: Other IPMI Boot Options
• Air-Gapped Installations
  • Creating a Local Mirror of the NVIDIA and Canonical Repositories
  • Creating the Mirror of the Repositories
  • Configuring the Target Air-Gapped System
• Cloud-init Configuration File
  • Modifying the Configuration File
  • Drive Partitioning

Misc

• Third-Party License Notices
  • Micron msecli
  • Mellanox (OFED)
• Notices
  • Notice
  • Trademarks