Installing Software on Air-Gapped NVIDIA DGX-1 Systems

For security purposes, some installations require that systems be isolated from the internet or outside networks. Since most DGX-1 software updates are accomplished through an over-the-network process with NVIDIA servers, this section explains how updates can be made when using an over-the-network method is not an option. It includes a process for installing Docker containers as well.

Installing NVIDIA DGX-1 Software

One method for updating DGX-1 software on an air-gapped DGX-1 is to download the ISO image, copy it to removable media and then re-image the DGX-1 from the media. This method is available only for software versions that are available as ISO images for download.

Alternately, you can update the DGX-1 software by performing a network update from a local repository. This method is available only for software versions that are available for over-the-network updates.

Re-Imaging the System

Note:WARNING:This process destroys all data and software customizations that you have made on the DGX-1. Be sure to back up any data that you want to preserve, and push any Docker images that you want to keep to a trusted registry.
  1. Obtain the ISO image from the Enterprise Support site.
    1. Log on to the NVIDIA Enterprise Support site and click the Announcements tab to locate the DGX-1 Base OS Image ISO file.
    2. Download the image ISO file.
  2. Refer to the instructions in the Restoring the DGX-1 Software Image section for additional instructions.

Creating a Local Mirror of the NVIDIA and Canonical Repositories

The procedure below describes how to download all the necessary packages to create a mirror of the repositories that are needed to update Nvidia DGX systems. The steps are specific to versions 4.0.X and 4.1.X, but they can be edited to work with other versions. For more information on DGX OS versions and the release notes available, visit https://docs.nvidia.com/dgx/dgx-os-server-release-notes/index.html#dgx-os-release-number-scheme. For more information on how to upgrade from versions 4.0.x to 4.1.x, review the release notes: https://docs.nvidia.com/dgx/pdf/DGX-OS-server-4.1-relnotes-update-guide.pdf.

Creating the Mirrors

The instructions in this section are to be performed on a system with network access.

The following are the prerequisites.
  • A system installed with Ubuntu OS is needed to create the mirror because there are several Ubuntu tools that need to be used.
  • The system must contain enough storage space to replicate the repositories to a filesystem; the space requirement could be as high as 250GB.
  • An efficient way to move large amount of data; for example, shared storage in a DMZ, or portable USB drives that can be brought into the air-gapped area.

    The data will need to be moved to the systems that need to be updated. Make sure the portable drive is formatted using ext4 or FAT32.

  1. Make sure the storage device is attached to the system with network access and identify the mount point. Example mount point used in these instructions: /media/usb/repository
  2. Once the space requirement has been met, install the apt-mirror package. Make sure the target directory is owned by the user apt-mirror or the replication will not work.
    $ sudo apt update 
    $ sudo apt install apt-mirror 
    $ sudo chown apt-mirror:apt-mirror /media/usb/repository 
  3. Configure the path of the destination directory in /etc/apt/mirror.list and use the included list of repositories below to retrieve the packages for both Ubuntu base OS as well as the NVIDIA DGX OS packages.
    ############# config ################## 
    # 
    set base_path /media/usb/repository #/your/path/here 
    # 
    # set mirror_path $base_path/mirror 
    # set skel_path $base_path/skel 
    # set var_path $base_path/var 
    # set cleanscript $var_path/clean.sh 
    # set defaultarch <running host architecture> 
    # set postmirror_script $var_path/postmirror.sh 
    set run_postmirror 0 
    set nthreads 20 
    set _tilde 0 
    # 
    ############# end config ############## 
    # Standard Canonical package repositories: 
    deb http://security.ubuntu.com/ubuntu bionic-security main 
    deb http://security.ubuntu.com/ubuntu bionic-security universe 
    deb http://security.ubuntu.com/ubuntu bionic-security multiverse 
    deb http://archive.ubuntu.com/ubuntu/ bionic main multiverse universe 
    deb http://archive.ubuntu.com/ubuntu/ bionic-updates main multiverse universe 
    # 
    deb-i386 http://security.ubuntu.com/ubuntu bionic-security main 
    deb-i386 http://security.ubuntu.com/ubuntu bionic-security universe 
    deb-i386 http://security.ubuntu.com/ubuntu bionic-security multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu/ bionic main multiverse universe 
    deb-i386 http://archive.ubuntu.com/ubuntu/ bionic-updates main multiverse universe 
    # 
    # DGX specific repositories: 
    deb http://international.download.nvidia.com/dgx/repos/bionic bionic main restricted universe multiverse 
    deb http://international.download.nvidia.com/dgx/repos/bionic bionic-updates main restricted universe multiverse 
    deb http://international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1 main multiverse restricted universe 
    # 
    deb-i386 http://international.download.nvidia.com/dgx/repos/bionic bionic main restricted universe multiverse 
    deb-i386 http://international.download.nvidia.com/dgx/repos/bionic bionic-updates main restricted universe multiverse 
    # Only for DGX OS 4.1.0 
    deb-i386 http://international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1 main multiverse restricted universe 
    # Clean unused items 
    clean http://archive.ubuntu.com/ubuntu 
    clean http://security.ubuntu.com/ubuntu
  4. Run apt-mirror and wait for it to finish downloading content.

    This will take a long time depending on the network connection speed.

    $ sudo apt-mirror
  5. Eject the removable storage with all packages.
    $ sudo eject /media/usb/repository 

Configure the Target System

The instructions in this section are to be performed on the target system.

The following are the prerequisites.
  • The target DGX system is installed, has gone through the first boot process, and is ready to be updated with the latest packages.
  • A USB storage device is attached to the target DGX server.

    There are other ways to transfer the data that are not covered in this document as they will depend on the data center policies for the air-gapped environment.

  1. Mount the storage device on the air-gapped system to /media/usb/repository for consistency.
  2. Configure apt to use the filesystem as the repository in the file /etc/apt/sources.list by modifying the following lines.
    deb file:///media/usb/repository/mirror/security.ubuntu.com/ubuntu bionic-security main 
    deb file:///media/usb/repository/mirror/security.ubuntu.com/ubuntu bionic-security universe 
    deb file:///media/usb/repository/mirror/security.ubuntu.com/ubuntu bionic-security multiverse 
    deb file:///media/usb/repository/mirror/archive.ubuntu.com/ubuntu/ bionic main multiverse universe 
    deb file:///media/usb/repository/mirror/archive.ubuntu.com/ubuntu/ bionic-updates main multiverse universe
  3. Configure apt to use the NVIDIA DGX OS packages in the file /etc/apt/sources.list.d/dgx.list.
    deb file:///media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic main multiverse restricted universe
  4. If present, remove the file /etc/apt/sources.list.d/docker.list as it is no longer needed and it will eliminate error messages during the update process.
  5. (For DGX OS Release 4.1 and later only) Configure apt to use the NVIDIA DGX OS packages in the file /etc/apt/sources.list.d/dgx-bionic-r418-cuda10-1-repo.list.
    deb file:///media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic/ bionic-r418+cuda10.1 main multiverse restricted universe
  6. Edit the file /etc/apt/preferences.d/nvidia to update the Pin parameter as follows.
    Package: * 
    #Pin: origin international.download.nvidia.com 
    Pin: release o=DGX Server 
    Pin-Priority: 600 
  7. Update the apt repository and confirm there are no errors.
    $ sudo apt update 
    
    Get:1 file:/media/usb/repository/mirror/security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 
    Get:1 file:/media/usb/repository/mirror/security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 
    Get:2 file:/media/usb/repository/mirror/archive.ubuntu.com/ubuntu bionic InRelease [242 kB] 
    Get:2 file:/media/usb/repository/mirror/archive.ubuntu.com/ubuntu bionic InRelease [242 kB] 
    Get:3 file:/media/usb/repository/mirror/archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 
    Get:4 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1 InRelease [13.0 kB] 
    Get:5 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic InRelease [13.1 kB] 
    Get:3 file:/media/usb/repository/mirror/archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 
    Get:4 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1 InRelease [13.0 kB] 
    Get:5 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic InRelease [13.1 kB] 
    Hit:6 https://download.docker.com/linux/ubuntu bionic InRelease Get:7 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1/multiverse amd64 Packages [10.1 kB] 
    Get:8 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1/restricted amd64 Packages [10.3 kB] 
    Get:9 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic-r418+cuda10.1/restricted i386 Packages [516 B] 
    Get:10 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic/multiverse amd64 Packages [44.5 kB] 
    Get:11 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic/multiverse i386 Packages [8,575 B] 
    Get:12 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic/restricted i386 Packages [745 B] 
    Get:13 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic/restricted amd64 Packages [8,379 B] 
    Get:14 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic/universe amd64 Packages [2,946 B] 
    Get:15 file:/media/usb/repository/mirror/international.download.nvidia.com/dgx/repos/bionic bionic/universe i386 Packages [496 B] 
    Reading package lists... Done 
    Building dependency tree 
    Reading state information... Done 
    249 packages can be upgraded. Run 'apt list --upgradable' to see them. 
    $ 
  8. Upgrade the system using the newly configured local repositories.
    $ sudo apt full-upgrade 

Installing Docker Containers

This method applies to Docker containers hosted on the NVIDIA DGX Container Registry, and requires that you have an active DGX account.

  1. On a system with internet access, log in to the DGX Container Registry by entering the following command and credentials.
    $ docker login nvcr.io
    Username: $oauthtoken 
    Password: apikey

    Type “$oauthtoken” exactly as shown for the Username. This is a special username that enables API key authentication. In place of apikey, paste in the API Key text that you obtained from the DGX website.

  2. Enter the docker pull command, specifying the image registry, image repository, and tag:
    $ docker pull nvcr.io/nvidia/repository:tag
  3. Verify the image is on your system using docker images.
    $ docker images
  4. Save the Docker image as an archive. .
    $ docker save nvcr.io/nvidia/repository:tag > framework.tar
  5. Transfer the image to the air-gapped system using removable media such as a USB flash drive.
  6. Load the NVIDIA Docker image.
     $ docker load –i framework.tar
  7. Verify the image is on your system.
    $ docker images