The DGX-Server UEFI BIOS supports PXE boot. Several manual customization steps are required to get PXE to boot the Base OS image.
This document is meant to be used as a reference. Explicit instructions are not given to configure the DHCP, FTP, and TFTP servers. It is expected that the end user’s IT team will configure them to fit within their company’s security guidelines.
Prerequesites
A TFTP and FTP server are setup
IP address is <FTP IP>
Fully qualified host is <FTP host>
This document is intended to provide detailed step-by-step instructions on how to set up a PXE boot environment for DGX systems. The examples are based on a DGX A100. There are several major components of the solution:
DHCP server: dnsmasq (https://thekelleys.org.uk/dnsmasq/doc.html) is used in this doc.
TFTP server: dnsmasq is also used as a TFTP server.
HTTP server: HTTP server is used to transfer large files, such is ISO image and initrd. Alternatively, FTP can also be used for this purpose. HTTP in used this doc.
Syslinux: Linux bootloader software package. section
The PXE Server is divided up into the following areas:
Bootloader (syslinux)
FTP contents (i.e. the ISO image and its extracted contents)
Note that when comparing the Base OS 5.x contents to previous versions, there are comparatively very few files in the ISO. The rough directory structure on the FTP server will look like (where 5.x.y is the version you’re installing)
/local/ftp
└── dgxbaseos-5.x.y
├── base_os_5.x.y.iso
├── boot/
│ └── grub
│ └── bios.img
│ └── boot.catdefault
│ └── grub.cfg
└── EFI
│ └── efiboot.imgg
├── initrd
├── live/
│ └── filesystem.squashfs
├── md5sum.txt
├── UBUNTU_LIVE
├── vmlinuz
/local/syslinux
└── efi64
├── syslinux.efi
├── ldlinux.e64
├── pxelinux.cfg
│ └── default
├── sdgx-5.x.y/
│ └── vmlinuz
│ └── initrd
The tftp-server (controlled by the xinetd service and configuration found in /etc/xinetd.d/tftp) points to the syslinux directory for when the system PXE boots. TFTP is what transfers the syslinux.efi file that is designated in the DHCP server’s dhcpd.conf file (see Configure your DHCP Server). By default after the syslinux.efi is booted it looks for a pxelinux.cfg/default file with the menu options for booting further, i.e. the kernel and initrd in that sdgx/ directory. That config file will look in the same working directory for its files as it did for the pxelinux.cfg directory. In this case it is /local/syslinux/efi64.
The following steps will assume the DHCP and PXE servers are configured to use the above directory structure. The lab admin, or whoever is in charge of deploying the PXE environment, should change the directory names and structure to fit their infrastructure.
Setting Up dnsmasq for DHCP
The DGX system BIOS does not allow you to manually specify a PXE server. You need to use the DHCP server to determine what it is and from which file to boot.
Install dnsmasq.
$ apt install dnsmasq
Back up the config file prior to customizing it.
$ mv /etc/dnsmasq.conf /etc/dnsmasq.conf.backup
Edit the config file using the appropriate values for your system.
$ vi /etc/dnsmasq.conf
interface=enp226s0 bind-interfaces domain=nvidia.com dhcp-range=enp226s0,10.33.254.106,static,1h dhcp-host=5c:ff:35:e1:a4:d1,DGX-A100-PM2,10.33.254.106 dhcp-option=option:router,10.33.254.1 dhcp-option=option:dns-server,8.8.8.8 enable-tftp tftp-root=/local/syslinux/efi64/ dhcp-boot=syslinux.efi
Create and set up the
local/syslinux/efi64/folder.$ sudo mkdir -p /local/syslinux/efi64
$ systemctl restart dnsmasq
$ systemctl status dnsmasq
XXXXXXXXXXXXX
Configure the FTP Side
Mount the ISO
Assume you have a copy of the 5.x.y ISO located at “/tmp/base_os_5.x.y.iso”, and your mount point is “/mnt”:
$ sudo mount -o loop /tmp/base_os_5.x.y.iso /mnt
Copy the ISO and its extracted contents to the FTP Directory
Using the sample directory structure in the Overview, the FTP root directory is “/local/ftp/”, and there is a “dgxbaseos-5.x.y/” directory under it.
$ cp -r /mnt/* /local/ftp/dgxbaseos-5.x.y/
$ cp /tmp/base_os_5.x.y.iso /local/ftp/dgxbaseos-5.x.y/
Ensure all the files are owned, or are at least readable, by the “ftp” user. It’s now safe to unmount the ISO if you want to:
$ sudo umount /mnt
Configuring the HTTP File Directory and ISO Image
Here are the steps to configure the http file directory and ISO image for PXE boot.
The instructions produce the following file structure on the PXE server.
/local/http
└── dgxbaseos-5.x.y
├── base_os_5.x.y.iso
├── initrd
└── vmlinuz
You should have obtained and copied the ISO image to
“/tmp/dgxbaseos-5.x.y.iso”.
Create the DGX OS directory.
$ sudo mkdir -p /local/http/dgxbaseos-5.x.y
Mount the ISO.
Your mount point is
/mnt.$ sudo mount -o loop /tmp/base_os_5.x.y.iso /mnt
Copy the ISO and its extracted contents to the HTTP directory.
$ sudo cp /mnt/{initrd,vmlinuz} /local/http/dgxbaseos-5.x.y/
$ cp /tmp/dgxbaseos-5.0.2.iso /local/http/dgxbaseos-5.x.y/
Ensure all the files are owned, or are at least readable, by the
httpuser.You can now unmount the ISO.
$ sudo umount /mnt
Configure your DHCP Server
The DGX-Server BIOS unfortunately does not allow you to specify a PXE
server manually. You have to rely on the DHCP server to tell you what it
is, and what file to boot from. In the DHCP server’s
/etc/dhcp/dhcpd.conf file:
Set the “next-server” to <FTP IP_ADDR>
Set the “filename” to “efi64/syslinux.efi”
Again, this assumes the directory structure in the Overview.
Optional: Configure the NVIDIA ConnextX cards to PXE boot
DGX-Servers may also PXE boot using the NVIDIA ConnextX cards. If you are
logged into the DGX system, you can perform this section’s steps using the
/usr/sbin/mlnx_pxe_setup.bash tool, which will enable the UEFI PXE ROM
of every NVIDIA ConnectX InfiniBand device found.
Otherwise, proceed with the manual steps below.
Query UEFI PXE ROM state
In order to PXE boot from the MLNX CX-4/5/6 cards, you must first enable the UEFI PXE ROM of the card you wish to PXE boot from because it is disabled by default. This needs to be performed from the DGX Server host OS itself, it can’t be done remotely. The configuration of the NVIDIA ConnextX devices is handled through the “mlxconfig” command.
To determine the device name and current configurations of the NVIDIA ConnectX
cards, run $ sudo mlxconfig query
$ sudo mlxconfig query
Device #1:
----------
Device type: ConnectX4
Name: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE;
single-port QSFP28; PCIe3.0 x16; ROHS R6
Device: /dev/mst/mt4115_pciconf3
Configurations: Next Boot
...
...
EXP_ROM_UEFI_x86_ENABLE False(0)
...
...
Enable UEFI PXE ROM
The EXP_ROM_UEFI_x86_ENABLE configuration must be set to True(1) for the
NVIDIA ConnectX card that you wish to PXE boot from, and reboot.
$ sudo mlxconfig -y -d /dev/mst/mt4115_pciconf3 set EXP_ROM_UEFI_x86_ENABLE=1
Reboot the system:
$ sudo reboot
Upon reboot, confirm the configuration was set.
$ sudo mlxconfig query
Device #1:
----------
Device type: ConnectX4
Name: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE;
single-port QSFP28; PCIe3.0 x16; ROHS R6
Device: /dev/mst/mt4115_pciconf3
Configurations: Next Boot
...
...
EXP_ROM_UEFI_x86_ENABLE True(1)
...
...
Configuring the syslinux Bootloader
These instructions install the syslinux 6.04-pre1 package. The 5.x.y OS image does not include a bootloader for PXE boot. Instead, the syslinux from kernel.org is used. The syslinux 6.04-pre1 package is installed because fixes in 6.04 are required to support PXE booting from the Mellanox ethernet cards as an option. PXE boot from the Mellanox cards also require at least 16.23.1020 level firmware with:
FW 16.23.1020PXE 3.5.0504UEFI 14.16.0017
These instructions produce the following file structure on the PXE server.
/local/syslinux
└── efi64
├── ldlinux.e64
├── libcom32.c32
├── libutil.c32
├── menu.c32
├── pxelinux.0
├── pxelinux.cfg
│ └── default
├── syslinux.efi
└── vesamenu.c32
Install Syslinux 6.04-pre1
Download and extract the syslinux archive.
$ cd /local
$ wget https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/Testing/6.04/syslinux-6.04-pre1.tar.gz
$ sudo tar -xvf syslinux-6.04-pre1.tar.gz
Copy the syslinux files.
$ sudo ln -s /local/syslinux-6.04-pre1 /local/syslinux
$ sudo mkdir /local/syslinux/efi64/pxelinux.cfg
$ sudo cp syslinux-6.04-pre1/bios/core/pxelinux.0 syslinux/efi64/
$ sudo cp syslinux-6.04-pre1/efi64/efi/syslinux.efi syslinux/efi64/
$ sudo cp syslinux-6.04-pre1/efi64/com32/elflink/ldlinux/ldlinux.e64 syslinux/efi64/
$ sudo cp syslinux-6.04-pre1/efi64/com32/menu/vesamenu.c32 syslinux/efi64/
$ sudo cp syslinux-6.04-pre1/efi64/com32/menu/menu.c32 syslinux/efi64/
$ sudo cp syslinux-6.04-pre1/efi64/com32/libutil/libutil.c32 syslinux/efi64
$ sudo cp syslinux-6.04-pre1/efi64/com32/lib/libcom32.c32 syslinux/efi64/
Edit PXE boot default configuration file as follows and make sure you update the IP address of the http server:
$ sudo vi /local/syslinux/efi64/pxelinux.cfg/default default menu prompt 0 timeout 300 menu title DGX Install Images label dgxbaseos-5.x.y menu label DGXBaseOS 5.x.y (Base OS) kernel http://<http-address>/dgxbaseos-5.x.y/vmlinuz initrd http://<http-address>/dgxbaseos-5.x.y/initrd append boot=live config components union=overlay noswap noeject ip::enp226s0:dhcp ethdevice-timeout=60 fetch=http://<http-address>/dgxbaseos-5.x.y/DGXOS-5.x.y-<release-labels>.iso apparmor=0 elevator=noop nvme-core.multipath=n nouveau.modeset=0 rebuild-raid offwhendone console=tty0 console=ttyS1,115200n8
The kernel boot parameters should match the contents of the
corresponding ISO’s boot menu, found in /mnt/boot/grub/grub.cfg.
Live Boot Parameters
Here is some information about the live boot parameters specified in the
pxelinux.cfg/default file.
The man-pages for all of the live-boot parameters used above (and more) can be found here
The following parameters can be used to configure your system’s network interfaces:
ip=: Tells the initramfs to automatically configure the system’s
interfaces by using DHCP. If only one interface is connected to the
network, this interface should be enough. If multiple interfaces are
connected to the network, then the initramfs will go with the first
interface that receives a reply.
In this case, you might need to provide a more specific configuration.
For instance,
ip::enp1s0f0:dhcp tells the initramfs to configure the enp1s0f0
interface by using DHCP.
ethdevice-timeout: Allows you to specify how long to wait for the ethernet devices to get configured. The example above gives the system 60 seconds to configure its interfaces before giving up.
The following parameters are unique to the DGX OS Installer:
rebuild-raid: If the data RAID is specified, tells the installer to rebuild the RAID. An installation from the factory should always specify this parameter, but otherwise, it is optional.
curtin-encrypt-root: Sets up LUKS encryption on the root filesystem using a well-known passphrase.
no-mlnx-fw-update: Skips updating the NVIDIA ConnectX cards firmware during the installation.
Setup GRUB PXE Server
Instead of using syslinux to boot, you can modify the PXE server setup outlined earlier in this document to use GRUB instead. As a reminder, “the lab admin, or whoever is in charge of deploying the PXE environment, should change the directory names and structure to fit their infrastructure.” In addition, the required GRUB binaries should be downloaded and placed in the correct directories depending on the lab admin’s chosen infrastructure.
Download GRUB Packages
Create a new directory /local/grub2/ and then download the relevant packages with the correct architecture specified. In this example we will be using x86_64:
$ yum install --downloadonly shim-x64.x86_64 grub2-efi-x64.x86_64
The RPMs end up in: /var/cache/yum/x86_64/7/base/packages and /var/cache/yum/x86_64/7/updates/packages. Unpack the RPMs with the following commands:
$ rpm2cpio grub2-efi-x64-2.02-0.86.el7.centos.x86_64.rpm | cpio -idmv
$ rpm2cpio shim-x64-15-8.el7.x86_64.rpm | cpio -idmv
Copy the following binaries from the unpacked RPMs to /local/grub2/:
shim.efi
shimx64.efi
grubx64.efi
Make a copy of shimx64.efi in /local/grub2/ and rename the copy as bootx64.efi.
Modify DHCP Configuration
In the DHCP server’s /etc/dhcp/dhcpd.conf file:
Leave the “next-server” as <FTP IP_ADDR>
{.p dir=“ltr”} Change the “filename” entries that point to “syslinux.efi” to the following:
if option architecture = 00:06 {
filename "grub2/bootia32.efi";
} elsif option architecture = 00:07 {
filename "grub2/bootx64.efi";
} elsif option architecture = 00:09 {
filename "grub2/bootx64.efi";
} elsif option architecture = 00:0b {
filename "grub2/grubaa64.efi";
} else {
filename "pxelinux.0";
}
Modify TFTP Configuration
Modify /etc/xinetd.d/tftp:
Point the tftp-server to the grub2 directory for when the system PXE boots
Make the tftp-server transfer the new .efi files from the grub2 directory which are specified in the DHCP server’s dhcpd.conf file
Once *.efi is booted it should look in the same directory for the corresponding grub.cfg file that points to vmlinuz and initrd
Example:
/local/http └── http/ ├── dgxbaseos-5.x.y/ │ └── base_os_5.x.y.iso └── ftp/ ├── dgxbaseos-5.x.y/ │ └── base_os_5.x.y.iso ├── boot/ │ └── grub/ │ │ └── bios.img │ │ └── boot.cat │ │ └── grub.cfg ├── EFI/ │ └── efiboot.img ├── live/ │ └── filesystem.squashfs ├── md5sum.rtxt ├── UBUNTU_LIVE - dgxbaseos-5.x.z/ │ └── base_os_5.x.z.iso ├── boot/ │ └── grub/ │ │ └── bios.img │ │ └── boot.cat │ │ └── grub.cfg ├── EFI/ │ └── efiboot.img ├── live/ │ └── filesystem.squashfs ├── md5sum.rtxt ├── UBUNTU_LIVE - dgxbaseos-5.a.b/ ├── boot/ │ └── grub/ │ │ └── bios.img │ │ └── boot.cat │ │ └── grub.cfg ├── base_os_5.a.b.iso ├── boot/ │ └── grub/ │ │ └── bios.img │ │ └── boot.cat │ │ └── grub.cfg ├── EFI/ │ └── efiboot.img ├── live/ │ └── filesystem.squashfs ├── md5sum.rtxt ├── UBUNTU_LIVE - grub2/ ├── sdgx-5.x.y/ └── vmlinuz └── initrd ├── grub.cfg ├── grubx64.efi ├── shim.efi ├── shimx64.efi
The contents of the grub2/grub.cfg file should look something like:
set default=0
set timeout=-1
insmod all_video
menuentry 'dgxbaseos-5.x.y' {
linux /sdgx-5.x.y/vmlinuz boot=live config components union=overlay noswap noeject ip= ethdevice-timeout=60
initrd /sdgx-5.x.y/initrd
}
Optional: Configure the NVIDIA ConnextX cards to PXE boot
DGX-Servers may also PXE boot using the MLNX CX-4/5/6 cards. If you are
logged into the DGX-Server host OS, and running DGX Base OS 4.4 or
later, then you can perform this section’s steps using the
/usr/sbin/mlnx_pxe_setup.bash tool, which will enable the UEFI PXE ROM
of every MLNX Infiniband device found.
Otherwise, proceed with the manual steps below.
Query the UEFI PXE ROM State
Here are the steps to query the current state of the UEFI PXE ROM. To PXE boot from the MLNX CX-4/5/6 cards, enable the UEFI PXE ROM of the card from which you want to PXE boot.
By default, the UEFI PXE ROM is disabled.
This process cannot be completed done remotely, because it needs to be performed from the host OS.
The configuration of the MLNX CX devices is handled through the [mlxconfig] command.
To determine the device name and current configurations of the MLNX CX cards, run the [sudo mlxconfig query]
$ sudo mlxconfig query
Device #1:
----------
Device type: ConnectX4
Name: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; single-port QSFP28; PCIe3.0 x16; ROHS R6
Device: /dev/mst/mt4115_pciconf3
Configurations: Next Boot
...
...
EXP_ROM_UEFI_x86_ENABLE False(0)
...
...
Enable the UEFI PXE ROM
Here is some information about how to enable the UEFI PXE ROM.
Ensure that the
EXP_ROM_UEFI_x86_ENABLEconfiguration is set toTrue(1)for the MLNX CX card from which you want to PXE boot.$ sudo mlxconfig -y -d /dev/mst/mt4115_pciconf3 set EXP_ROM_UEFI_x86_ENABLE=1
Reboot the system:
$ sudo reboot
Wait for the system to reboot and confirm that the configuration was set.
$ sudo mlxconfig query Device #1: ---------- Device type: ConnectX4 Name: MCX455A-ECA_Ax Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; single-port QSFP28; PCIe3.0 x16; ROHS R6 Device: /dev/mst/mt4115_pciconf3 Configurations: Next Boot ... ... EXP_ROM_UEFI_x86_ENABLE True(1) ... ...
Curtin Customizations
Here is some information about customizing the curtin files.
For some deployments, you may want to use your own curtin files. This section will describe some sections contained in the built-in curtin files as well as how to perform some common customizations.
Customizing the Curtin YAML
The curtin file is located in the filesystem.squashfs. Creating a touchless install, such as by seeding the initial userid and password, and skipping the oem-config steps, is accomplished by customizing the curtin file. (nooemconfig boot parameter described above does initial userid/password seeding and oem-config skip. For any other customization to curtin yaml, follow the steps below).
Get the filesystem.squashfs from the ISO
Run “unsquashfs filesystem.squashfs”
Copy the curtin file for the platform you’re trying to PXE install from the squashfs-root/curtin/ directory into a webserver or FTP server.
Customize the curtin file
Use the force-curtin parameter in your PXE bootloader entry to point to the URL of your new curtin file
More info about curtin can be found here: https://curtin.readthedocs.io/en/latest/index.html
An example curtin file for performing touchless installs can be found here: https://gitlab-master.nvidia.com/jamien/jamien-work/-/blob/master/curtin/fast-curtin.yaml
NVIDIA-Specific Curtin Variables
Here is some information about NVIDIA-specific curtin variables.
The curtin files contained in the ISO are platform-specific, and serve
as a good starting point for custom versions. Many of them contain
variables, prefixed with CHANGE_, that get substituted by the
installer.
CHANGE_STORAGE_REG
This gets removed and uncommented when the boot parameter curtin-encrypt-root is not present. Uncommenting this stanza results in the standard disk partitioning scheme without LUKS encryption.
CHANGE_STORAGE_ENC
This gets removed and uncommented when the boot parameter “curtin-encrypt-root” is present. Uncommenting this stanza results in an encrypted root partition.
CHANGE_BOOT_DISK_NAME_x
This is a disk-name, without the “/dev” prefix. There may be multiple
ones (i.e. CHANGE_BOOT_DISK_NAME_1 and CHANGE_BOOT_DISK_NAME_2) for
platforms that expect a RAIDed boot device as is the case for DGX-2 and
DGX A100. The installer will find the appropriate disk name to
substitute here. Alternatively, the “force-bootdisk” parameter can
be used to specify the disk name(s).
CHANGE_BOOT_DISK_PATH_x: This is the same as the CHANGE_BOOT_DISK_NAME_x
variable above, except that is prefixed with /dev/.
CHANGE_POWER_STATE: This tells the installer whether to halt or reboot the machine after the install is complete. It is controlled by the “offwhendone” boot parameter.
CHANGE_DESC_PLATFORM: The installer will substitute this with a platform-specific descriptive name.
CHANGE_SERIAL_NUMBER: The installer will substitute this with the serial number reported by dmidecode
CHANGE_MLNX_FWUPD: This value either gets removed or substituted with “RUN_FW_UPDATER=no” by the installer. This is done based on whether or not the “no-mlnx-fw-update” boot parameter is present. This determines whether or not the installer will update the FW on Mellanox networking cards that are present on the machine.
CHANGE_INSTALL_PKGS: The installer will substitute this value with a list of packages specific to the platform. The lists of packages are specified by the *-pkgs files in the squashfs.
CHANGE_REBUILD_RAID: This gets replaced with either “true” or “false” based on whether or not the “rebuild-raid” boot parameter is present.
CHANGE_IPMISOL: This gets replaced with either “true” or “false” based on whether or not the “curtin-encrypt-root” boot parameter is present. When we set the system up with encryption, we also undo the IPMI serial-over-LAN configuration to ensure that the LUKS passphrase prompt shows up on the console rather than the serial-over-LAN interface.
While it is possible to replace these values on your own, we strongly recommend letting the installer handle this.
Boot the DGX System over PXE
You can power cycle the DGX A100 through BMC GUI, or, alternatively, use “ipmitool” to set pxe boot and reboot the client DGX A100:
$ ipmitool -I lanplus -H <DGX-BMC-IP> -U <username> -P <password> chassis bootdev pxe options=efiboot
$ ipmitool -I lanplus -H <DGX-BMC-IP> -U <username> -P <password> chassis power reset
Troubleshooting
During the PXE boot process, you may see following errors from client BMC:
BootCurrent: 0018
Timeout: 1 seconds
BootOrder: 0000,0006,0007,0008,000C,000D,000E,000F,0016,0017,0001
Boot0000* ubuntu
Boot0001 UEFI: Built-in EFI Shell
Boot0004* UEFI: PXE IPv4 Intel(R) I210 Gigabit Network Connection
Boot0005* UEFI: PXE IPv6 Intel(R) I210 Gigabit Network Connection
Boot0006* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:5F:44:7C
Boot0007* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:9E
Boot0008* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:8E
Boot0009* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:6B:CA
Boot000A* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:9A
Boot000B* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:5F:51:D8
Boot000C* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:6A:FA
Boot000D* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:74:74:C2
Boot000E* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:4A:62:60
Boot000F* UEFI: PXE IPv4 Mellanox Network Adapter - 0C:42:A1:4A:62:61
Boot0016* UEFI: PXE IPv6 Mellanox Network Adapter - 0C:42:A1:4A:62:60
Boot0017* UEFI: PXE IPv6 Mellanox Network Adapter - 0C:42:A1:4A:62:61
Invalid BootOrder order entry value0018
^
efibootmgr: entry 0018 does not exist
Unexpected error while running command.
Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpsgpxpe9k/target', 'efibootmgr', '-o', '0018,0000,0006,0007,0008,000C,000D,000E,000F,0016,0017,0001']
Exit code: 8
The solution is to wait for the Ubuntu console to appear, login as root without password, then run the following commands to make the Intel I210 NIC as the first NIC to try PXE boot:
$ efibootmgr -o 0004,0000,0006,0007,0008,0009,000A,000B, 000C,000D,000E,000F
Reference: Other IPMI Boot Options
Here is some information about other IPMI boot options.
For more information about specifying boot order by using the
ipmicommand, refer to the ipmitool man page, and search for the following sections:The
chassiscommandThe
bootdevsubcommand
For more information about the IPMI specification, see: https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html