Is this page helpful?

Part 3: Configuring BCM to Boot DGX Nodes with the OS Image#

The steps in this part are the same for all distributions. Category names, image names, and the disk setup XML reflect the RHEL example used throughout this guide. Substitute your own naming conventions as appropriate.

  1. Assign the node to the RHEL category.

    The cm-image step above automatically created a dgx-rhel9-x86_64 category with the dgx-rhel9u6-x86_64 software image assigned to it.

    Verify this in cmsh:

    # At the OS/shell level on the BCM head node, run

cmsh
[clio-bcm11-head01]% category
[clio-bcm11-head01->category]% ls

Name (key)                 Software image           Nodes
-------------------------- ------------------------ --------
default-ubuntu2404-x86_64  default-image            1
dgx                        dgx-image                0
dgx-b200-slurm             dgx-b200-slurm-image     4
dgx-rhel9-x86_64           dgx-rhel9u6-x86_64       0
slogin                     slogin-image             2

    Assign the target node to the RHEL category:

    [clio-bcm11-head01->device]% use dgx-01
[clio-bcm11-head01->device[dgx-01]]% set category dgx-rhel9-x86_64
[clio-bcm11-head01->device*[dgx-01*]]% commit

  2. Set the bootloader to GRUB.

    To prevent the DGX B200 node from powering off during PXE boot after the Network Boot Program (NBP) file downloads, set the bootloader to GRUB on the category:

    # At the OS/shell level on the BCM head node, run

cmsh
[clio-bcm11-head01]% category
[clio-bcm11-head01->category]% use dgx-rhel9-x86_64
[clio-bcm11-head01->category[dgx-rhel9-x86_64]]% set bootloader grub
[clio-bcm11-head01->category*[dgx-rhel9-x86_64*]]% commit

  3. Add required kernel modules to the software image.

    # At the OS/shell level on the BCM head node, run

cmsh
[clio-bcm11-head01]% softwareimage
[clio-bcm11-head01->softwareimage]% use dgx-rhel9u6-x86_64
[clio-bcm11-head01->softwareimage[dgx-rhel9u6-x86_64]]% append kernelparameters " iommu=pt"
[clio-bcm11-head01->softwareimage[dgx-rhel9u6-x86_64]]% kernelmodules
[clio-bcm11-head01->softwareimage[dgx-rhel9u6-x86_64]->kernelmodules]% add bonding
[clio-bcm11-head01->softwareimage*[dgx-rhel9u6-x86_64*]->kernelmodules*[bonding*]]% add raid0
[clio-bcm11-head01->softwareimage*[dgx-rhel9u6-x86_64*]->kernelmodules*[raid0*]]% add raid1
[clio-bcm11-head01->softwareimage*[dgx-rhel9u6-x86_64*]->kernelmodules*[raid1*]]% softwareimage commit

# Successfully committed 1 SoftwareImages

  4. Configure BMC settings.

    Mirror the BMC settings from your existing dgx-b200 nodes to ensure power control works from BCM:

    # At the OS/shell level on the BCM head node, run

cmsh
[clio-bcm11-head01]% category
[clio-bcm11-head01->category]% use dgx-rhel9-x86_64
[clio-bcm11-head01->category[dgx-rhel9-x86_64]]% bmcsettings
[clio-bcm11-head01->category[dgx-rhel9-x86_64]->bmcsettings]% set password xxxxx
[clio-bcm11-head01->category*[dgx-rhel9-x86_64*]->bmcsettings*]% commit

  5. Configure the disk setup.

    Create the disk setup definition file at /cm/local/apps/cmd/etc/htdocs/disk-setup/dgx-disksetup.xml. This file tells BCM how to partition and format the NVMe drives on the DGX node during provisioning. The layout uses two drives (nvme0n1 and nvme1n1) for the root filesystem (/) in RAID 1.

    Both drives hold an identical copy of the OS, so if one fails the node keeps running, the remaining eight drives (nvme2n1– nvme9n1) are combined in RAID 0 and mounted at /raid.

    All eight drives act as one large volume, maximizing storage capacity for workload data.

    # At the OS/shell level on the BCM head node, run

nano /cm/local/apps/cmd/etc/htdocs/disk-setup/dgx-disksetup.xml

    <!--Copy the contents below into the new xml file -->

<?xml version="1.0" encoding="UTF-8"?>
<diskSetup>
   <device>
      <blockdev>/dev/nvme0n1</blockdev>
      <partition id="boot1" partitiontype="esp">
            <size>512M</size>
            <type>linux</type>
            <filesystem>fat</filesystem>
            <mountPoint>/boot/efi</mountPoint>
            <mountOptions>defaults,noatime,nodiratime</mountOptions>
      </partition>
      <partition id="slash1">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme1n1</blockdev>
      <partition id="boot2" partitiontype="esp">
            <size>512M</size>
            <type>linux</type>
            <filesystem>fat</filesystem>
            <mountOptions>defaults,noatime,nodiratime</mountOptions>
      </partition>
      <partition id="slash2">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme2n1</blockdev>
      <partition id="raid1" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme3n1</blockdev>
      <partition id="raid2" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme4n1</blockdev>
      <partition id="raid3" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme5n1</blockdev>
      <partition id="raid4" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme6n1</blockdev>
      <partition id="raid5" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme7n1</blockdev>
      <partition id="raid6" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme8n1</blockdev>
      <partition id="raid7" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <device>
      <blockdev>/dev/nvme9n1</blockdev>
      <partition id="raid8" partitiontype="esp">
            <size>max</size>
            <type>linux raid</type>
      </partition>
   </device>
   <raid id="slash">
      <member>slash1</member>
      <member>slash2</member>
      <level>1</level>
      <filesystem>ext4</filesystem>
      <mountPoint>/</mountPoint>
      <mountOptions>defaults,noatime,nodiratime</mountOptions>
   </raid>
   <raid id="raid">
      <member>raid1</member>
      <member>raid2</member>
      <member>raid3</member>
      <member>raid4</member>
      <member>raid5</member>
      <member>raid6</member>
      <member>raid7</member>
      <member>raid8</member>
      <level>0</level>
      <filesystem>ext4</filesystem>
      <mountPoint>/raid</mountPoint>
      <mountOptions>defaults,noatime,nodiratime</mountOptions>
   </raid>
</diskSetup>

    Apply the disk setup to the category:

    # At the OS/shell level on the BCM head node, run

cmsh
[clio-bcm11-head01]% category
[clio-bcm11-head01->category]% use dgx-rhel9-x86_64
[clio-bcm11-head01->category[dgx-rhel9-x86_64]]% set disksetup /cm/local/apps/cmd/etc/htdocs/disk-setup/dgx-disksetup.xml
[clio-bcm11-head01->category*[dgx-rhel9-x86_64*]]% commit

  6. Verify the Base Network Configuration Survived the Import.

    This file was created in the image root before the tarball was built (Part 1, Step 11). Confirm it is present in the BCM-managed image after import:

    # At the OS/shell level on the BCM head node, run

ls /cm/images/dgx-rhel9u6-x86_64/etc/sysconfig/network

    If it is missing and the image was built without it, create it now:

    # At the OS/shell level on the BCM head node, run

mkdir -p /cm/images/dgx-rhel9u6-x86_64/etc/sysconfig/
echo "NETWORKING=yes" > /cm/images/dgx-rhel9u6-x86_64/etc/sysconfig/network

  7. Trigger a full install and reboot the node.

    Note

    If the DGX host used to build the image in Part 1 is not in a stateless state when brought up as a node within BCM, wipe the Linux disk and reformat the RAID array before rebooting to prevent PXE boot errors.

    # At the OS/shell level on the BCM head node, run

cmsh
[clio-bcm11-head01]% device
[clio-bcm11-head01->device]% use dgx-01
[clio-bcm11-head01->device[dgx-01]]% set nextinstallmode FULL
[clio-bcm11-head01->device*[dgx-01*]]% commit
[clio-bcm11-head01->device[dgx-01]]% reboot

# Reboot in progress for: dgx-01

Important

To provision additional DGX nodes with the RHEL image, repeat Steps 1 and 7 for each node. Steps 2–6 are category and image-level settings that only need to be configured once.