PXE Boot Setup

The DGX-Server UEFI BIOS supports PXE boot. Several manual customization steps are required to get PXE to boot the Base OS image.

Caution

This document is meant to be used as a reference. Explicit instructions are not given to configure the DHCP, FTP, and TFTP servers. It is expected that the end user’s IT team will configure them to fit within their company’s security guidelines.

Prerequesites

  • A TFTP and FTP server are setup

    • IP address is <FTP IP>

    • Fully qualified host is <FTP host>

This document is intended to provide detailed step-by-step instructions on how to set up a PXE boot environment for DGX systems. The examples are based on a DGX A100. There are several major components of the solution:

  • DHCP server: dnsmasq (https://thekelleys.org.uk/dnsmasq/doc.html) is used in this doc.

  • TFTP server: dnsmasq is also used as a TFTP server.

  • HTTP server: HTTP server is used to transfer large files, such is ISO image and initrd. Alternatively, FTP can also be used for this purpose. HTTP in used this doc.

  • Syslinux: Linux bootloader software package. section

Overview of the PXE Server

The PXE Server is divided up into the following areas:

  • Bootloader (syslinux)

  • FTP contents (i.e. the ISO image and its extracted contents)

Note that when comparing the Base OS 5.x contents to previous versions, there are comparatively very few files in the ISO. The rough directory structure on the FTP server will look like (where 5.x.y is the version you’re installing)

/local/ftp
└── dgxbaseos-5.x.y
    ├── base_os_5.x.y.iso
    ├── boot/
    │   └── grub
    │       └── bios.img
    │       └── boot.catdefault
    │       └── grub.cfg
    └── EFI
    │   └── efiboot.imgg
    ├── initrd
    ├── live/
    │       └── filesystem.squashfs
    ├── md5sum.txt
    ├── UBUNTU_LIVE
    ├── vmlinuz
/local/syslinux
└── efi64
    ├── syslinux.efi
    ├── ldlinux.e64
    ├── pxelinux.cfg
    │   └── default
    ├── sdgx-5.x.y/
    │   └── vmlinuz
    │   └── initrd

The tftp-server (controlled by the xinetd service and configuration found in /etc/xinetd.d/tftp) points to the syslinux directory for when the system PXE boots. TFTP is what transfers the syslinux.efi file that is designated in the DHCP server’s dhcpd.conf file (see Configure your DHCP Server). By default after the syslinux.efi is booted it looks for a pxelinux.cfg/default file with the menu options for booting further, i.e. the kernel and initrd in that sdgx/ directory. That config file will look in the same working directory for its files as it did for the pxelinux.cfg directory. In this case it is /local/syslinux/efi64.

The following steps will assume the DHCP and PXE servers are configured to use the above directory structure. The lab admin, or whoever is in charge of deploying the PXE environment, should change the directory names and structure to fit their infrastructure.

Setting Up dnsmasq for DHCP

The DGX system BIOS does not allow you to manually specify a PXE server. You need to use the DHCP server to determine what it is and from which file to boot.

  1. Install dnsmasq.

    apt install dnsmasq

  2. Back up the config file prior to customizing it.

    mv /etc/dnsmasq.conf /etc/dnsmasq.conf.backup

  3. Edit the config file using the appropriate values for your system.

    vi /etc/dnsmasq.conf

    interface=enp226s0
bind-interfaces
domain=nvidia.com
dhcp-range=enp226s0,10.33.254.106,static,1h
dhcp-host=5c:ff:35:e1:a4:d1,DGX-A100-PM2,10.33.254.106
dhcp-option=option:router,10.33.254.1
dhcp-option=option:dns-server,8.8.8.8
enable-tftp
tftp-root=/local/syslinux/efi64/
 dhcp-boot=syslinux.efi

  4. Create and set up the local/syslinux/efi64/ folder.

    sudo mkdir -p /local/syslinux/efi64

    systemctl restart dnsmasq

    systemctl status dnsmasq

Configure the FTP Side

Mount the ISO

Assume you have a copy of the 5.x.y ISO located at “/tmp/base_os_5.x.y.iso”, and your mount point is “/mnt”:

sudo mount -o loop /tmp/base_os_5.x.y.iso /mnt

Copy the ISO and its extracted contents to the FTP Directory

Using the sample directory structure in the Overview, the FTP root directory is “/local/ftp/”, and there is a “dgxbaseos-5.x.y/” directory under it.

cp -r /mnt/* /local/ftp/dgxbaseos-5.x.y/

cp /tmp/base_os_5.x.y.iso /local/ftp/dgxbaseos-5.x.y/

Ensure all the files are owned, or are at least readable, by the “ftp” user. It’s now safe to unmount the ISO if you want to:

sudo umount /mnt

Configuring the HTTP File Directory and ISO Image

Here are the steps to configure the http file directory and ISO image for PXE boot.

The instructions produce the following file structure on the PXE server.

/local/http
└── dgxbaseos-5.x.y
    ├── base_os_5.x.y.iso
    ├── initrd
    └── vmlinuz

You should have obtained and copied the ISO image to “/tmp/dgxbaseos-5.x.y.iso”.

  1. Create the DGX OS directory.

    sudo mkdir -p /local/http/dgxbaseos-5.x.y

  2. Mount the ISO.

    Your mount point is /mnt.

    sudo mount -o loop /tmp/base_os_5.x.y.iso /mnt

  3. Copy the ISO and its extracted contents to the HTTP directory.

    sudo cp /mnt/{initrd,vmlinuz} /local/http/dgxbaseos-5.x.y/

    cp /tmp/dgxbaseos-5.0.2.iso /local/http/dgxbaseos-5.x.y/

  4. Ensure all the files are owned, or are at least readable, by the http user.

  5. You can now unmount the ISO.

    sudo umount /mnt

Configure your DHCP Server

The DGX-Server BIOS unfortunately does not allow you to specify a PXE server manually. You have to rely on the DHCP server to tell you what it is, and what file to boot from. In the DHCP server’s /etc/dhcp/dhcpd.conf file:

  • Set the “next-server” to <FTP IP_ADDR>

  • Set the “filename” to “efi64/syslinux.efi”

Again, this assumes the directory structure in the Overview.

Optional: Configure the NVIDIA ConnextX cards to PXE boot

DGX-Servers may also PXE boot using the NVIDIA ConnextX cards. If you are logged into the DGX system, you can perform this section’s steps using the /usr/sbin/mlnx_pxe_setup.bash tool, which will enable the UEFI PXE ROM of every NVIDIA ConnectX InfiniBand device found.

Otherwise, proceed with the manual steps below.

Query UEFI PXE ROM state

In order to PXE boot from the MLNX CX-4/5/6 cards, you must first enable the UEFI PXE ROM of the card you wish to PXE boot from because it is disabled by default. This needs to be performed from the DGX Server host OS itself, it can’t be done remotely. The configuration of the NVIDIA ConnextX devices is handled through the “mlxconfig” command.

To determine the device name and current configurations of the NVIDIA ConnectX cards, run sudo mlxconfig query

sudo mlxconfig query

Device #1:

----------

Device type: ConnectX4
Name: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE;
single-port QSFP28; PCIe3.0 x16; ROHS R6

Device: /dev/mst/mt4115_pciconf3
Configurations: Next Boot

...

...

EXP_ROM_UEFI_x86_ENABLE False(0)

...
...

Enable UEFI PXE ROM

The EXP_ROM_UEFI_x86_ENABLE configuration must be set to True(1) for the NVIDIA ConnectX card that you wish to PXE boot from, and reboot.

sudo mlxconfig -y -d /dev/mst/mt4115_pciconf3 set EXP_ROM_UEFI_x86_ENABLE=1

Reboot the system:

sudo reboot

Upon reboot, confirm the configuration was set.

sudo mlxconfig query

Device #1:

----------

Device type: ConnectX4
Name: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE;
single-port QSFP28; PCIe3.0 x16; ROHS R6

Device: /dev/mst/mt4115_pciconf3
Configurations: Next Boot

...
...

EXP_ROM_UEFI_x86_ENABLE True(1)

...
...

Configuring the syslinux Bootloader

These instructions install the syslinux 6.04-pre1 package. The 5.x.y OS image does not include a bootloader for PXE boot. Instead, the syslinux from kernel.org is used. The syslinux 6.04-pre1 package is installed because fixes in 6.04 are required to support PXE booting from the Mellanox ethernet cards as an option. PXE boot from the Mellanox cards also require at least 16.23.1020 level firmware with:

  • FW 16.23.1020

  • PXE 3.5.0504

  • UEFI 14.16.0017

These instructions produce the following file structure on the PXE server.

/local/syslinux
└── efi64
    ├── ldlinux.e64
    ├── libcom32.c32
    ├── libutil.c32
    ├── menu.c32
    ├── pxelinux.0
    ├── pxelinux.cfg
    │   └── default
    ├── syslinux.efi
    └── vesamenu.c32

  1. Install Syslinux 6.04-pre1

    1. Download and extract the syslinux archive.

      cd /local

      wget https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/Testing/6.04/syslinux-6.04-pre1.tar.gz

      sudo tar -xvf syslinux-6.04-pre1.tar.gz

    2. Copy the syslinux files.

      sudo ln -s /local/syslinux-6.04-pre1 /local/syslinux

      sudo mkdir /local/syslinux/efi64/pxelinux.cfg

      sudo cp syslinux-6.04-pre1/bios/core/pxelinux.0 syslinux/efi64/

      sudo cp syslinux-6.04-pre1/efi64/efi/syslinux.efi syslinux/efi64/

      sudo cp syslinux-6.04-pre1/efi64/com32/elflink/ldlinux/ldlinux.e64 syslinux/efi64/

      sudo cp syslinux-6.04-pre1/efi64/com32/menu/vesamenu.c32 syslinux/efi64/

      sudo cp syslinux-6.04-pre1/efi64/com32/menu/menu.c32 syslinux/efi64/

      sudo cp syslinux-6.04-pre1/efi64/com32/libutil/libutil.c32 syslinux/efi64

      sudo cp syslinux-6.04-pre1/efi64/com32/lib/libcom32.c32 syslinux/efi64/

  2. Edit PXE boot default configuration file as follows and make sure you update the IP address of the http server:

    sudo vi /local/syslinux/efi64/pxelinux.cfg/default
default menu
prompt 0
timeout 300
menu title DGX Install Images
label dgxbaseos-5.x.y
menu label DGXBaseOS 5.x.y (Base OS)
kernel http://<http-address>/dgxbaseos-5.x.y/vmlinuz
initrd http://<http-address>/dgxbaseos-5.x.y/initrd
append boot=live config components union=overlay noswap noeject ip::enp226s0:dhcp ethdevice-timeout=60 fetch=http://<http-address>/dgxbaseos-5.x.y/DGXOS-5.x.y-<release-labels>.iso apparmor=0 elevator=noop nvme-core.multipath=n nouveau.modeset=0 rebuild-raid offwhendone console=tty0 console=ttyS1,115200n8

The kernel boot parameters should match the contents of the corresponding ISO’s boot menu, found in /mnt/boot/grub/grub.cfg.

Live Boot Parameters

Here is some information about the live boot parameters specified in the pxelinux.cfg/default file.

Note

The man-pages for all of the live-boot parameters used above (and more) can be found here

The following parameters can be used to configure your system’s network interfaces:

ip=: Tells the initramfs to automatically configure the system’s interfaces by using DHCP. If only one interface is connected to the network, this interface should be enough. If multiple interfaces are connected to the network, then the initramfs will go with the first interface that receives a reply.

In this case, you might need to provide a more specific configuration. For instance, ip::enp1s0f0:dhcp tells the initramfs to configure the enp1s0f0 interface by using DHCP.

ethdevice-timeout: Allows you to specify how long to wait for the ethernet devices to get configured. The example above gives the system 60 seconds to configure its interfaces before giving up.

The following parameters are unique to the DGX OS Installer:

rebuild-raid: If the data RAID is specified, tells the installer to rebuild the RAID. An installation from the factory should always specify this parameter, but otherwise, it is optional.

curtin-encrypt-root: Sets up LUKS encryption on the root filesystem using a well-known passphrase.

no-mlnx-fw-update: Skips updating the NVIDIA ConnectX cards firmware during the installation.

Setup GRUB PXE Server

Instead of using syslinux to boot, you can modify the PXE server setup outlined earlier in this document to use GRUB instead. As a reminder, “the lab admin, or whoever is in charge of deploying the PXE environment, should change the directory names and structure to fit their infrastructure.” In addition, the required GRUB binaries should be downloaded and placed in the correct directories depending on the lab admin’s chosen infrastructure.

Download GRUB Packages

Create a new directory /local/grub2/ and then download the relevant packages with the correct architecture specified. In this example we will be using x86_64:

yum install --downloadonly shim-x64.x86_64 grub2-efi-x64.x86_64

The RPMs end up in: /var/cache/yum/x86_64/7/base/packages and /var/cache/yum/x86_64/7/updates/packages. Unpack the RPMs with the following commands:

rpm2cpio grub2-efi-x64-2.02-0.86.el7.centos.x86_64.rpm | cpio -idmv

rpm2cpio shim-x64-15-8.el7.x86_64.rpm | cpio -idmv

Copy the following binaries from the unpacked RPMs to /local/grub2/:

  • shim.efi

  • shimx64.efi

  • grubx64.efi

    Make a copy of shimx64.efi in /local/grub2/ and rename the copy as bootx64.efi.

    Modify DHCP Configuration

    In the DHCP server’s /etc/dhcp/dhcpd.conf file:

    • Leave the “next-server” as <FTP IP_ADDR>

    • {.p dir=“ltr”} Change the “filename” entries that point to “syslinux.efi” to the following:

if option architecture = 00:06 {
 filename "grub2/bootia32.efi";
} elsif option architecture = 00:07 {
 filename "grub2/bootx64.efi";
} elsif option architecture = 00:09 {
 filename "grub2/bootx64.efi";
} elsif option architecture = 00:0b {
 filename "grub2/grubaa64.efi";
} else {
 filename "pxelinux.0";
}

Modify TFTP Configuration

Modify /etc/xinetd.d/tftp:

  • Point the tftp-server to the grub2 directory for when the system PXE boots

  • Make the tftp-server transfer the new .efi files from the grub2 directory which are specified in the DHCP server’s dhcpd.conf file

    Once *.efi is booted it should look in the same directory for the corresponding grub.cfg file that points to vmlinuz and initrd

    Example:

    /local/http
└── http/
    ├── dgxbaseos-5.x.y/
    │   └── base_os_5.x.y.iso
└── ftp/
    ├── dgxbaseos-5.x.y/
    │   └── base_os_5.x.y.iso
    ├── boot/
    │   └── grub/
    │    │    └── bios.img
    │    │    └── boot.cat
    │    │    └── grub.cfg
    ├── EFI/
    │   └── efiboot.img
    ├── live/
    │   └── filesystem.squashfs
    ├── md5sum.rtxt
    ├── UBUNTU_LIVE
 - dgxbaseos-5.x.z/
    │   └── base_os_5.x.z.iso
    ├── boot/
    │   └── grub/
    │    │    └── bios.img
    │    │    └── boot.cat
    │    │    └── grub.cfg
    ├── EFI/
    │   └── efiboot.img
    ├── live/
    │   └── filesystem.squashfs
    ├── md5sum.rtxt
    ├── UBUNTU_LIVE
- dgxbaseos-5.a.b/
    ├── boot/
    │   └── grub/
    │    │    └── bios.img
    │    │    └── boot.cat
    │    │    └── grub.cfg
    ├── base_os_5.a.b.iso
    ├── boot/
    │   └── grub/
    │    │    └── bios.img
    │    │    └── boot.cat
    │    │    └── grub.cfg
    ├── EFI/
    │   └── efiboot.img
    ├── live/
    │   └── filesystem.squashfs
    ├── md5sum.rtxt
    ├── UBUNTU_LIVE
- grub2/
    ├── sdgx-5.x.y/
       └── vmlinuz
       └── initrd
    ├── grub.cfg
    ├── grubx64.efi
    ├── shim.efi
    ├── shimx64.efi

The contents of the grub2/grub.cfg file should look something like:

set default=0
set timeout=-1
insmod all_video
menuentry 'dgxbaseos-5.x.y' {
linux /sdgx-5.x.y/vmlinuz boot=live config components union=overlay noswap noeject ip= ethdevice-timeout=60
initrd /sdgx-5.x.y/initrd
}

Optional: Configure the NVIDIA ConnextX cards to PXE boot

DGX-Servers may also PXE boot using the MLNX CX-4/5/6 cards. If you are logged into the DGX-Server host OS, and running DGX Base OS 4.4 or later, then you can perform this section’s steps using the /usr/sbin/mlnx_pxe_setup.bash tool, which will enable the UEFI PXE ROM of every MLNX Infiniband device found.

Otherwise, proceed with the manual steps below.

Query the UEFI PXE ROM State

Here are the steps to query the current state of the UEFI PXE ROM. To PXE boot from the MLNX CX-4/5/6 cards, enable the UEFI PXE ROM of the card from which you want to PXE boot.

By default, the UEFI PXE ROM is disabled.

Note

This process cannot be completed done remotely, because it needs to be performed from the host OS.

The configuration of the MLNX CX devices is handled through the [mlxconfig] command.

To determine the device name and current configurations of the MLNX CX cards, run the [sudo mlxconfig query]

sudo mlxconfig query

Device #1:
----------

Device type:    ConnectX4
Name:           MCX455A-ECA_Ax
Description:    ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; single-port QSFP28; PCIe3.0 x16; ROHS R6
Device:         /dev/mst/mt4115_pciconf3

Configurations:                              Next Boot
         ...
         ...
         EXP_ROM_UEFI_x86_ENABLE             False(0)
         ...
         ...

Enable the UEFI PXE ROM

Here is some information about how to enable the UEFI PXE ROM.

  1. Ensure that the EXP_ROM_UEFI_x86_ENABLE configuration is set to True(1) for the MLNX CX card from which you want to PXE boot.

    sudo mlxconfig -y -d /dev/mst/mt4115_pciconf3 set EXP_ROM_UEFI_x86_ENABLE=1

  2. Reboot the system:

    sudo reboot

  3. Wait for the system to reboot and confirm that the configuration was set.

    sudo mlxconfig query
Device #1:
----------
Device type:    ConnectX4
Name:           MCX455A-ECA_Ax
Description:    ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; single-port QSFP28; PCIe3.0 x16; ROHS R6
Device:         /dev/mst/mt4115_pciconf3
Configurations:                              Next Boot
         ...
         ...
         EXP_ROM_UEFI_x86_ENABLE             True(1)
         ...
         ...

Curtin Customizations

Here is some information about customizing the curtin files.

For some deployments, you may want to use your own curtin files. This section will describe some sections contained in the built-in curtin files as well as how to perform some common customizations.

Customizing the Curtin YAML

The curtin file is located in the filesystem.squashfs. Creating a touchless install, such as by seeding the initial userid and password, and skipping the oem-config steps, is accomplished by customizing the curtin file. (nooemconfig boot parameter described above does initial userid/password seeding and oem-config skip. For any other customization to curtin yaml, follow the steps below).

  1. Get the filesystem.squashfs from the ISO

  2. Run “unsquashfs filesystem.squashfs”

  3. Copy the curtin file for the platform you’re trying to PXE install from the squashfs-root/curtin/ directory into a webserver or FTP server.

  4. Customize the curtin file

  5. Use the force-curtin parameter in your PXE bootloader entry to point to the URL of your new curtin file

More info about curtin can be found here: https://curtin.readthedocs.io/en/latest/index.html

An example curtin file for performing touchless installs can be found below:

apt:
  preserve_sources_list: false
  sources_list: |
    # network: will be available to target from early stage
    deb [trusted=yes] http://127.0.0.1/ /
  conf: |
    Dpkg::Options {
      "--force-confdef";
      "--force-confold";
    };

install:
  log_file: /var/log/curtin.log
  error_tarfile: /var/log/curtin/curtin-error-logs.tar
  post_files:
    - /var/log/curtin.log
    - /var/log/syslog
  save_install_config: /var/log/curtin-conf.yaml
  save_install_log: /var/log/curtin-install.log
  umount: disabled

kernel:
  package: linux-generic

storage:
  version: 1
  config:
  - id: CHANGE_BOOT_DISK_NAME_1
    type: disk
    ptable: gpt
    path: CHANGE_BOOT_DISK_PATH_1
    name: osdisk
    wipe: superblock-recursive
    grub_device: 1
  - id: CHANGE_BOOT_DISK_NAME_1-part1
    type: partition
    device: CHANGE_BOOT_DISK_NAME_1
    number: 1
    size: 512MB
    flag: boot
  - id: CHANGE_BOOT_DISK_NAME_1-part2
    type: partition
    device: CHANGE_BOOT_DISK_NAME_1
    number: 2
    size: CHANGE_BOOT_DISK_SIZE_1
  - id: CHANGE_BOOT_DISK_NAME_1-part1-fs1
    type: format
    fstype: fat32
    label: efi
    volume: CHANGE_BOOT_DISK_NAME_1-part1
  - id: CHANGE_BOOT_DISK_NAME_1-part2-fs1
    type: format
    fstype: ext4
    label: root
    volume: CHANGE_BOOT_DISK_NAME_1-part2
  - id: CHANGE_BOOT_DISK_NAME_1-mount2
    type: mount
    path: /
    device: CHANGE_BOOT_DISK_NAME_1-part2-fs1
    options: errors=remount-ro
    passno: 1
  - id: boot-mount1
    type: mount
    path: /boot/efi
    device: CHANGE_BOOT_DISK_NAME_1-part1-fs1
    passno: 1

swap:
  size: 0

power_state:
  mode: CHANGE_POWER_STATE
  delay: 5
  message: Installation Complete

sources:
  05_primary:
    uri: "file:///curtin/ubuntu-20.04-server-cloudimg-amd64-root.tar.xz"
    type: "tgz"

reporting:
  mylistener:
    type: journald
    identifier: "curtin-journald"
    level: DEBUG

#early_commands:
#  90_mount_curtin: mount -o bind /curtin/repo/ /var/www/html/
#  91_chown_www: chown www-data:www-data /curtin/repo /var/www/html

late_commands:
  # SSH is configured differently in the cloud images so reconfigure it here to get back
  # to the default settings
  10_conf_ssh: ["curtin", "in-target", "--", "sh", "-c", "dpkg-reconfigure openssh-server"]
  11_conf_ssh: ["curtin", "in-target", "--", "sh", "-c", "sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config"]

  23_apt_update: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt update -y"]
  #24_install_pkgs: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends CHANGE_INSTALL_PKGS"]
  #25_unattended_upgrades: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt purge -y unattended-upgrades"]

  # Rewrite sources list
  30_write_sources: ["curtin", "in-target", "--", "sh", "-c", "echo deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse > /etc/apt/sources.list"]
  31_write_sources: ["curtin", "in-target", "--", "sh", "-c", "echo deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse >> /etc/apt/sources.list"]
  32_write_sources: ["curtin", "in-target", "--", "sh", "-c", "echo deb http://security.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse >> /etc/apt/sources.list"]

  # [bug 3099036]: create auth.log now so the fail2ban service can start
  33_touch_auth: ["curtin", "in-target", "--", "sh", "-c", "sudo -u syslog -g adm touch /var/log/auth.log && chmod 644 /var/log/auth.log || true"]

  # Disable release-upgrade prompt
  40_disable_upgrade: ["curtin", "in-target", "--", "sh", "-c", "sed -i -e 's/^Prompt=.*/Prompt=never/' /etc/update-manager/release-upgrades"]

  # Create user
  50_create_user: ["curtin", "in-target", "--", "sh", "-c",
    "adduser --disabled-password --gecos \"\" --quiet lab"]
  51_add_user_group: ["curtin", "in-target", "--", "sh", "-c",
    "usermod -aG sudo lab"]
  52_set_user_password: ["curtin", "in-target", "--", "sh", "-c",
    "echo 'lab:labuser' | chpasswd"]

  # Copy logs from the install environment into the target
  90_mkdir_installer_logs: mkdir -p $TARGET_MOUNT_POINT/var/log/installer
  91_copy_logs: cp -rf /var/log/* $TARGET_MOUNT_POINT/var/log/installer/

NVIDIA-Specific Curtin Variables

Here is some information about NVIDIA-specific curtin variables.

The curtin files contained in the ISO are platform-specific, and serve as a good starting point for custom versions. Many of them contain variables, prefixed with CHANGE_, that get substituted by the installer.

CHANGE_STORAGE_REG

This gets removed and uncommented when the boot parameter curtin-encrypt-root is not present. Uncommenting this stanza results in the standard disk partitioning scheme without LUKS encryption.

CHANGE_STORAGE_ENC

This gets removed and uncommented when the boot parameter “curtin-encrypt-root” is present. Uncommenting this stanza results in an encrypted root partition.

CHANGE_BOOT_DISK_NAME_x

This is a disk-name, without the “/dev” prefix. There may be multiple ones (i.e. CHANGE_BOOT_DISK_NAME_1 and CHANGE_BOOT_DISK_NAME_2) for platforms that expect a RAIDed boot device as is the case for DGX-2 and DGX A100. The installer will find the appropriate disk name to substitute here. Alternatively, the “force-bootdisk” parameter can be used to specify the disk name(s).

CHANGE_BOOT_DISK_PATH_x: This is the same as the CHANGE_BOOT_DISK_NAME_x variable above, except that is prefixed with /dev/.

CHANGE_POWER_STATE: This tells the installer whether to halt or reboot the machine after the install is complete. It is controlled by the “offwhendone” boot parameter.

CHANGE_DESC_PLATFORM: The installer will substitute this with a platform-specific descriptive name.

CHANGE_SERIAL_NUMBER: The installer will substitute this with the serial number reported by dmidecode

CHANGE_MLNX_FWUPD: This value either gets removed or substituted with “RUN_FW_UPDATER=no” by the installer. This is done based on whether or not the “no-mlnx-fw-update” boot parameter is present. This determines whether or not the installer will update the FW on Mellanox networking cards that are present on the machine.

CHANGE_INSTALL_PKGS: The installer will substitute this value with a list of packages specific to the platform. The lists of packages are specified by the *-pkgs files in the squashfs.

CHANGE_REBUILD_RAID: This gets replaced with either “true” or “false” based on whether or not the “rebuild-raid” boot parameter is present.

CHANGE_IPMISOL: This gets replaced with either “true” or “false” based on whether or not the “curtin-encrypt-root” boot parameter is present. When we set the system up with encryption, we also undo the IPMI serial-over-LAN configuration to ensure that the LUKS passphrase prompt shows up on the console rather than the serial-over-LAN interface.

While it is possible to replace these values on your own, we strongly recommend letting the installer handle this.

Boot the DGX System over PXE

You can power cycle the DGX A100 through BMC GUI, or, alternatively, use “ipmitool” to set pxe boot and reboot the client DGX A100:

ipmitool -I lanplus -H <DGX-BMC-IP> -U <username> -P <password> chassis bootdev pxe options=efiboot

ipmitool -I lanplus -H <DGX-BMC-IP> -U <username> -P <password> chassis power reset

Troubleshooting

During the PXE boot process, you may see following errors from client BMC:

BootCurrent: 0018
Timeout: 1 seconds
BootOrder: 0000,0006,0007,0008,000C,000D,000E,000F,0016,0017,0001
Boot0000* ubuntu
Boot0001  UEFI: Built-in EFI Shell
Boot0004* UEFI: PXE IPv4 Intel(R) I210 Gigabit  Network Connection
Boot0005* UEFI: PXE IPv6 Intel(R) I210 Gigabit  Network Connection
Boot0006* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:5F:44:7C
Boot0007* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:9E
Boot0008* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:8E
Boot0009* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:6B:CA
Boot000A* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:9A
Boot000B* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:5F:51:D8
Boot000C* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:6A:FA
Boot000D* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:C2
Boot000E* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:4A:62:60
Boot000F* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:4A:62:61
Boot0016* UEFI: PXE IPv6 Mellanox Network Adapter - 0C:42:A1:4A:62:60
Boot0017* UEFI: PXE IPv6 Mellanox Network Adapter - 0C:42:A1:4A:62:61
Invalid BootOrder order entry value0018
                                     ^
efibootmgr: entry 0018 does not exist
Unexpected error while running command.
Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpsgpxpe9k/target', 'efibootmgr', '-o', '0018,0000,0006,0007,0008,000C,000D,000E,000F,0016,0017,0001']
Exit code: 8

The solution is to wait for the Ubuntu console to appear, login as root without password, then run the following commands to make the Intel I210 NIC as the first NIC to try PXE boot:

efibootmgr -o 0004,0000,0006,0007,0008,0009,000A,000B, 000C,000D,000E,000F

Reference: Other IPMI Boot Options

Here is some information about other IPMI boot options.