Cloud-init Configuration File#

This section provides instructions for creating a cloud-init configuration file for the Ubuntu Automated Server Installation.

Modifying the Configuration File#

The following instructions outline the example configuration file. The example does not work as is and requires additional modifications as described in the following sections. For more information, refer to Ubuntu Automated Server Installation.

Begin the configuration file with the following header:
```
#cloud-config
autoinstall:
  version: 1
```

Define a default user (the example uses Ubuntu), localization, and keyboard layout.

##
## Set initial system and user information
## use mkpasswd -m sha-512 <password> to create a password
##
  identity:
    realname: DGX Ubuntu User
    hostname: dgx-host
    password: <PASSWORD HASH>
    username: ubuntu
  locale: en_US
  keyboard:
    layout: en
    variant: us
  reporting:
    builtin:
      type: print

The network section describes the network configuration and supports fixed addresses, DHCP, and various other network options. The names of the network interfaces are system-dependent. These are the primary management ports for various DGX systems. For example:
- DGX A100: enp226s0
- DGX H100/H200: eno3
- DGX B200: eno3
- DGX B300: eno1
- DGX GB200: enP5p9s0
- DGX GB300: enP5p9s0
```
##
## Network Configuration
##
  network:
    version: 2
    ethernets:
      eno1:
        dhcp4: yes
```
Update the Subiquity installer to the edge channel.
```
refresh-installer:
  channel: edge
  update: yes
```

Provide details about the additional NVIDIA repositories.

##
## Enable this for using the remote repositories
##
  apt:
    <Repository details for the CUDA Compute and DGX Repository>

    conf: |
      Dpkg::Options {
        "--force-confdef";
        "--force-confold";
      };

Configure storage for the target DGX platform.

The next section describes the storage configuration, including swap configuration and drive partitioning. Setting the size to 0 disables the swap partition. Refer to Drive Partitioning.

The reorder_uefi flag tells the installer not to change the boot order to place the currently booted entry (BootCurrent) to the first option.
```
##
## Storage Configuration
##
  storage:
    config:
      <Platform-specific partition configuration>
    swap:
      size: 0
    grub:
      reorder_uefi: false
```

Enable the SSH server.

You can also set a default SSH key.

##
## SSH Server
##
  ssh:
    install-server: yes
    allow-pw: yes

Provide a list of packages that should be installed.

Refer to the comments in this text for instructions on changing the package names for specific DGX systems and on enabling or disabling features.

##
## Packages
##
  packages:

##
## NVIDIA DGX system configurations and system tools
##
    - nvidia-system-core
    - nvidia-system-utils
    - nvidia-system-extra
    - nvidia-system-extra-localization
    - nvidia-system-diag
    - dgx-system-utils
    - nvidia-mig-manager
    - open-nvdebug
    - nvfwupd
    - dgx-server-grub
    - linux-tools-nvidia

## Add these packages for DGX B200 and DGX B300 systems
#    - collectx-bringup
#    - nvvct

##
## NVIDIA GPU driver and tools
## DGX OS 8.0 uses the release-qualified R595 driver stack
##
    - linux-nvidia
    - nvidia-driver-pinning-595
    - nvidia-open
    - libnvidia-nscq
    - nvidia-modprobe
    - nvidia-fabricmanager-direct
    - datacenter-gpu-manager-4-cuda13
    - nvidia-persistenced

## Add these NVLink packages for DGX B200 and DGX B300 systems
#    - nvlsm
#    - libnvsdm

##
## NVIDIA repositories and DOCA-OFED
##
    - nvidia-repositories
    - dgx-repositories
    - nvidia-system-mlnx-drivers
    - doca-ofed

##
## NVIDIA container support
## To run GPU containers, include nvidia-container-toolkit. After the
## system boots, configure Docker with:
##   sudo nvidia-ctk runtime configure --runtime=docker
##   sudo systemctl restart docker
##
    - docker.io
    - nvidia-container-toolkit

##
## NVIDIA system management tools
##
    - nvsm

Add any additional software packages you want to install during autoinstall.

Finally, add a list of additional commands to be executed at the end of the installation.

Disable unattended upgrades
Configure the MLNX services
Enable the NVIDIA Fabric Manager, NVIDIA Persistence Daemon, DCGM, and OpenIBD services

##
## Commands executed after completion of the installation
##
  late-commands:
    - curtin in-target --target=/target -- apt purge -y unattended-upgrades
    - curtin in-target --target=/target -- mlnx_pxe_setup.bash
    - curtin in-target --target=/target -- systemctl enable openibd.service
    - curtin in-target --target=/target -- systemctl disable srp_daemon.service srptools.service
    - curtin in-target --target=/target -- systemctl enable nvidia-fabricmanager.service nvidia-persistenced.service nvidia-dcgm.service

Drive Partitioning#

Before using an autoinstall storage configuration, identify the system boot drives and record their serial numbers or WWNs. For example:

lsblk -d -o NAME,SERIAL,WWN,SIZE,MODEL

Use a stable identifier in each Curtin disk action. Do not use a device path such as /dev/sda because device names can change between boots.

DGX A100, DGX H100/H200, DGX B200, and DGX B300 systems normally use two system drives in a RAID 1 configuration. The following example shows the general unencrypted layout:

Create a GPT partition table on each system drive.
Create a 512 MB EFI System Partition on each drive and mark both as boot devices.
Use the remaining space on each drive for a RAID partition.
Create the root file system on a RAID 1 device constructed from those partitions.
Mount one EFI System Partition at /boot/efi. Both EFI partitions remain bootable.

Warning

The wipe: superblock-recursive setting destroys existing data on the matched drives. Replace both serial-number placeholders and verify them on the target system before installation.

storage:
  swap:
    size: 0
  grub:
    reorder_uefi: false
  config:
    - id: osdisk1
      type: disk
      ptable: gpt
      serial: "<SYSTEM_DISK_1_SERIAL>"
      wipe: superblock-recursive
    - id: osdisk2
      type: disk
      ptable: gpt
      serial: "<SYSTEM_DISK_2_SERIAL>"
      wipe: superblock-recursive
    - id: osdisk1-efi
      type: partition
      device: osdisk1
      number: 1
      size: 512MB
      flag: boot
      grub_device: true
    - id: osdisk2-efi
      type: partition
      device: osdisk2
      number: 1
      size: 512MB
      flag: boot
      grub_device: true
    - id: osdisk1-root
      type: partition
      device: osdisk1
      number: 2
      size: -1
      flag: raid
    - id: osdisk2-root
      type: partition
      device: osdisk2
      number: 2
      size: -1
      flag: raid
    - id: md-root
      type: raid
      name: md0
      raidlevel: 1
      devices:
        - osdisk1-root
        - osdisk2-root
    - id: osdisk1-efi-format
      type: format
      fstype: fat32
      label: boot
      volume: osdisk1-efi
    - id: osdisk2-efi-format
      type: format
      fstype: fat32
      label: boot
      volume: osdisk2-efi
    - id: root-format
      type: format
      fstype: ext4
      volume: md-root
    - id: root-mount
      type: mount
      path: /
      device: root-format
      options: errors=remount-ro
      passno: 1
    - id: efi-mount
      type: mount
      path: /boot/efi
      device: osdisk1-efi-format
      passno: 1

If the target system intentionally uses one system drive, remove the second disk and its partitions, do not create the RAID action, and format and mount the remaining root partition directly.

Do not include the DGX data drives in this storage configuration unless you intend to erase and reconfigure them. You can configure the data-drive RAID array after installation as described in Configuring Data Drives.

Storage layouts can vary with hardware revisions and site requirements. Test the completed autoinstall file on a non-production system of each target model before deploying it to a cluster.