After booting the ISO image, the Ubuntu installer should start and guide you through the installation process.

Note: The screenshots in the following steps are taken from a DGX A100. Other DGX systems have differences in drive partitioning and networking.

During the boot process of the ISO image, you might see some error messages due to older drivers, etc. They can be safely ignored.

1. Select your language at the welcome screen, then follow the instructions to select whether to update the installer (if offered) and to choose your keyboard..
   
   
   
   
2. At the Network connections screen, configure your network.
   
   
   
   The installer tries to automatically retrieve a DHCP address for all network interfaces, so you should be able to continue without any changes. However, you also have the option to manually configure the interface(s).
3. At the Guided storage configuration screen, configure the partitioning and file systems. All DGX systems are shipped preinstalled with DGX OS. The drives are, therefore, already partitioned and formatted. DGX OS installer configures a single ext4 partition for the root partition in addition to the EFI boot partition. You have the following options:
   • Keep the current partition layout and formatting [recommended]
   • Create a custom partition scheme [advanced]
   • Use a single disk with the default Ubuntu partition scheme
   Creating a new custom partition scheme with a RAID configuration is a more involved process and out of the scope for this document. Refer to the Ubuntu installation guide for more information. When you choose the option to use an entire disk, Ubuntu will only use one of the two redundant boot drives.
   Note: Note: The RAID level for the data drive can be changed after the installation of Ubuntu.
   The following instructions describe the steps for keeping the current partition layout. It still requires you to re-create and reformat the partitions.
   Note: Note: DGX-1 is using a hardware RAID controller and the RAID membership can only be configured in the RAID controller BIOS. The default configuration consists of two virtual devices:

   • The first virtual device (sda) is a single disk and used as the system drive
   • Second virtual device (sdb) consists of the rest of the disks for data.
   1. Select Custom storage layout, then click Done.
      
      
      
      
   2. Identify the system drive.

      The system drive on the DGX-2 and DGX A100 is a RAID 1 array and you should find it easily. The DGX-1 has a hardware RAID controller and you will see a single drive as sda.

   3. Select the system drive and then click Format.
      
      
      
      
   4. Set Format to ext4 (do not select “Leave formatted as <filesystem>”).
      
      
      
      
   5. Set Mount to “/”:
      
      
      
      
   6. Set the boot flag on the raw devices.

      Identify the system drives under AVAILABLE DEVICES (not the RAID array) and select “Use as Boot Device” for the first device. On DGX-2 and DGX A100 that have two drives, repeat this process for the second drive and select “Use as another Boot Device”.

      
      
      
      
      
   7. Complete the configuration.
      • RAID 0 Array: In most cases, the RAID 0 array for the data drives will have already been created from the factory. If it hasn’t been created you can either create them in the Storage configurations dialog or by using the config_raid_array tool after completing the Ubuntu installation.
      • (Optional) Enable drive encryption: Note that encryption can only be enabled during the Storage configuration. It cannot be changed after the installation. To change the encryption state again you need to reinstall the OS. To enable drive encryption, you have to create a virtual group and volume. This is out of the scope for this document. Please refer to the Ubuntu documentation for more details.
      • Swap Partition: The default installation does not define a swap partition. Linux uses any configured swap partition for temporarily storing data when the system memory is full, incurring a performance hit. With the large memory of DGX systems swapping is not recommended.

      The “FILE SYSTEM SUMMARY” at the top of the page should display the root partition on the RAID 1 drive for and a boot/efi partition (the two drives will only show up as a single entry). On DGX-1 with the hardware RAID controller, it will show the root partition on sda.

      
      
      
      
      

      Select Done and accept all changes.

4. Follow the instructions for the remaining tasks.

   Create a default user in the Profile setup dialog and choose any additional SNAP package you want to install in the Featured Server Snaps screen.

5. Wait for the installation to complete.

   Log messages are presented while the installation is running.

6. Select Reboot Now when the installation is complete to restart the system.

   After reboot, you can log in using the username and password for the user you have created above.

The NVIDIA DGX Software Stack includes system-specific configurations and tools to take advantage of the advanced DGX features. They are provided from NVIDIA repositories in the form of software packages that can be installed on top of a typical Ubuntu installation.

1. Enable the NVIDIA repositories by extracting the repository information.

   This step adds the URIs and configuration preferences to control the package versions that will be installed to the /etc/apt directory and the GPG keys for the NVIDIA repositories in the /usr/share/keyrings directory.

   curl https://repo.download.nvidia.com/baseos/ubuntu/focal/dgx-repo-files.tgz | sudo tar xzf - -C /

2. Update the internal APT database with the latest version information of all packages.

   sudo apt update

3. Recommended: Upgrade all software packages with the latest versions.

   sudo apt upgrade 

4. Install the DGX system tools and configurations.
   • For DGX-1, install the DGX-1 configurations and DGX-1 system tools:

      sudo apt install -y dgx1-system-configurations dgx1-system-tools dgx1-system-tools-extra 

   • For DGX-2, install the DGX-2 configurations and DGX-2 system tools:

      sudo apt install -y dgx2-system-configurations dgx2-system-tools dgx2-system-tools-extra

   • For DGX A100, install DGX A100 configurations and DGX A100 system tools:

      sudo apt install -y dgx-a100-system-configurations dgx-a100-system-tools dgx-a100-system-tools-extra 

5. Disable the ondemand governor to set the governor to performance mode.

   sudo systemctl disable ondemand

6. Recommended: Disable unattended upgrades.

   Ubuntu periodically checks for security and other bug fixes and automatically installs updates software packages typically overnight. Because this may be disruptive, you should regularly check for updates and install them manually.

   sudo apt purge -y unattended-upgrades

7. Recommended: Enable serial-over-lan console output.
   Note: If you have boot drive encryption enabled, the prompt for entering the passphrase and input will be over the serial console if you install this package.

   sudo apt install -y nvidia-ipmisol 

8. Optional: Modify the logrotate policy to collect more logging information (but size-limited):

   sudo apt install -y nvidia-logrotate

   The configuration changes will take effect only after rebooting the system. To minimize extra reboots, you can defer this step until after the drivers have been installed later in this document.

1. Configure the /raid partition.

   All DGX systems support RAID 0 and RAID 5 arrays.

   • To create a RAID 0 array:

     sudo /usr/bin/configure_raid_array.py -c -f 

   • To create a RAID 5 array:

     sudo /usr/bin/configure_raid_array.py -c -f -5

   The command creates the /raid mount point and RAID array, and adds a corresponding entry in /etc/fstab.
2. Optional: Install tools for managing the self-encrypting drives (SED) for the data drives on the DGX A100.

   This requires to store the keys in the TPM or use external key servers. Refer to the “Managing Self-Encrypting Drives” section in the DGX A100 User Guide for usage information.

   1. Install the nv-disk-encrypt package.

      sudo apt install -y nv-disk-encrypt

   2. Reboot the system.

      sudo reboot

3. Optional: If you wish to use your RAID array for read caching of NFS mounts, you can install cachefilesd and set the cachefs option for an NFS share.
   1. Install cachefilesd and nvidia-conf-cachefilesd.

      This will update the cachefilesd configuration to use the /raid partition.

      sudo apt install -y cachefilesd nvidia-conf-cachefilesd

   2. Enable caching on all NFS shares you want to cache by setting the fsc flag.

      Edit /etc/fstab and add the fsc flag to the mount options as shown in this example.

      <nfs_server>:<export_path> /mnt nfs rw,noatime,rsize=32768,wsize=32768,nolock,tcp,intr,fsc,nofail 0 0 

   3. Mount the NFS share.

      If the share is already mounted, use the remount option.

      mount <mount-point> -o,remount 

   4. To validate that caching is enabled, issue the following.

      cat /proc/fs/nfsfs/volumes

      Look for the text FSC=yes in the output of the command. The NFS will be mounted with caching enabled upon subsequent reboot cycles.

1. Ensure to have the latest version of the package database.

   sudo apt update

2. Display a list of all available drivers.

   sudo apt list nvidia-driver*server 

   Example Output:

   nvidia-driver-418-server/focal-updates,focal-security 418.226.00-0ubuntu0.20.04.2 amd64 
   nvidia-driver-440-server/focal-updates,focal-security 450.156.00-0ubuntu0.20.04.2 amd64
   . . . 

   The following steps install the NVIDIA CUDA driver and configure the system. Replace the release version used as an example (450) with the release you want to install. Ensure that the driver release you intend to install is supported by the GPU in the system.
3. Ensure to have the latest version of the package database.

   sudo apt update

4. Ensure you have the latest kernel version installed.

   The driver package has a dependency to the kernel and updating the database might have updated the version information.

   sudo apt install -y linux-generic

5. Install NVIDIA CUDA driver.
   • For non-NVswitch systems like DGX-1:

     sudo apt install -y nvidia-driver-470-server linux-modules-nvidia-470-server-generic libnvidia-nscq-470 nvidia-modprobe datacenter-gpu-manager nv-persistence-mode 

   • For NVswitch systems like DGX-2 and DGX A100, be sure to also install the fabric-manager package:

     sudo apt install -y nvidia-driver-470-server linux-modules-nvidia-470-server-generic libnvidia-nscq-470 nvidia-modprobe nvidia-fabricmanager-470 datacenter-gpu-manager nv-persistence-mode 

6. Enable the persistenced daemon and other services:
   • For non-NVswitch systems, such as DGX-1:

     sudo systemctl enable nvidia-persistenced nvidia-dcgm

   • For NVswitch systems like DGX-2 and DGX A100, be sure to also enable the NVIDIA fabric manager service:

     sudo systemctl enable nvidia-fabricmanager nvidia-persistenced nvidia-dcgm

7. Reboot the system to load the drivers and to update system configurations.
   1. Issue reboot.

      sudo reboot 

   2. After the system has rebooted, verify that the drivers have been loaded and are handling the NVIDIA devices.

      nvidia-smi

      The output should show all available GPUs and show the Persistence-Mode ‘On’:

      +-----------------------------------------------------------------------------+
      | NVIDIA-SMI 450.119.04 Driver Version: 450.119.04 CUDA Version: 11.4 | 
      |----------------------------+----------------------+-----------------------+ 
      | GPU Name      Persistence-M| Bus-Id        Disp.A | Volatile Uncorr.  ECC | 
      | Fan Temp Perf Pwr:Usage/Cap|         Memory-Usage | GPU-Util   Compute M. | 
      |                            |                      |                MIG M. | 
      |============================+======================+=======================| 
      | 0 Tesla V100-SXM2...    On | 00000000:06:00.0 Off |                     0 | 
      | N/A   35C  P0   42W / 300W |      0MiB / 16160MiB |   0%          Default | 
      |                            |                      |                   N/A | 
      +----------------------------+----------------------+-----------------------+ 
      | 1 Tesla V100-SXM2...    On | 00000000:07:00.0 Off |                     0 | 
      | N/A   35C  P0   44W / 300W |      0MiB / 16160MiB |   0%          Default | 
      |                            |                      |                   N/A | 
      +----------------------------+----------------------+-----------------------+ 
      ... 
      +----------------------------+----------------------+-----------------------+ 
      | 7 Tesla V100-SXM2...    On | 00000000:8A:00.0 Off |                     0 | 
      | N/A  35C  P0   43W / 300W  |      0MiB / 16160MiB |   0%          Default | 
      |                            |                      |                   N/A | 
      +----------------------------+----------------------+-----------------------+ 
      +---------------------------------------------------------------------------+ 
      | Processes:                                                                | 
      | GPU   GI   CI        PID     Type      Process name            GPU Memory | 
      |       ID   ID                                                  Usage      | 
      |===========================================================================| 
      | No running processes found                                                | 
      +---------------------------------------------------------------------------+ 

1. Install the MOFED Driver

   sudo apt install -y mlnx-ofed-all nvidia-mlnx-ofed-misc 

2. Enable and start the openibd service:

   sudo systemctl enable --now openibd 

3. Enable nv-peer-mem. This service loads the peer-mem module on every boot.

   sudo update-rc.d nv_peer_mem defaults

1. Execute the following command start a container and run the nvidia-smi tool inside the container:

   sudo docker run --gpus=all --rm nvcr.io/nvidia/cuda:11.0-base nvidia-smi

   Example Output

   +-----------------------------------------------------------------------------+
   | NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
   |-------------------------------+----------------------+----------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
   | N/A   35C    P0    42W / 300W |      0MiB / 16160MiB |      0%      Default |
   |                               |                      |                  N/A |
   +-------------------------------+----------------------+----------------------+
   |   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
   | N/A   35C    P0    44W / 300W |      0MiB / 16160MiB |      0%      Default |
   |                               |                      |                  N/A |
   +-------------------------------+----------------------+----------------------+
   ...
   +-------------------------------+----------------------+----------------------+
   |   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
   | N/A   35C    P0    43W / 300W |      0MiB / 16160MiB |      0%      Default |
   |                               |                      |                  N/A |
   +-------------------------------+----------------------+----------------------+
   
   +-----------------------------------------------------------------------------+
   | Processes:                                                                  |
   |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
   |        ID   ID                                                   Usage      |
   |=============================================================================|
   |  No running processes found                                                 |
   +-----------------------------------------------------------------------------+

2. Verify that the output shows all available GPUs and has Persistence-Mode set to On.

1. Install the NVIDIA System Management tool (NVSM):

   sudo apt install -y nvsm

2. Optional: Modify message-of-the-day (MOTD) to display NVSM health monitoring alerts and release information.

   sudo apt install -y nvidia-motd