Mixed Architecture Setup#

In general, the architecture that is installed on the BCM head node is the default OS image available for other connected servers (nodes). When nodes use a different architecture, an OS image compiled for that architecture needs to be created or imported. For example, If the BCM head node itself is an x86 server, an ARM/aarch64 OS image would be created to support any ARM/aarch64 nodes. The cm-image tool is specifically used for creating different OS/ARCH images (software image, node-installer image and cm-shared image).

Note

  • The BCM head node architecture can be either x86 or aarch64/ARM. The process is the same in either case.

  • The cm-image tool uses QEMU to emulate the other architecture if necessary. This can take a long time (approximately 4 hours).

The cm-image tool manages three components — software image, node-installer, and cm-shared — each of which has two parts:

  • A filesystem directory (for example, /cm/images/default-image-<distro>-<arch>)

  • A BCM entity in cmsh. All three components have entries under cmsh → fspart; the software image also appears under cmsh → softwareimage.

Method 1 — Import Pre-compiled Images (fastest)#

The following example imports x86_64 archives on an aarch64 BCM head node, as specified by the DGX SuperPOD reference architecture. If your BCM head node is x86_64 and your target nodes are aarch64, download the aarch64 archives and replace x86_64 with aarch64 in the commands.

  1. From the DGX OS release page in the NVIDIA Enterprise Support Portal, download these three precompiled archives for the target architecture. The page lists them under Additional files that may be required when importing images:

    • The archive with the bcni-ubuntu filename prefix contains the node installer.

    • The archive with the bcmn-ubuntu filename prefix contains the default software image.

    • The archive with the bcm-cm-shared-ubuntu filename prefix contains the shared binaries.

    Choose all three archives for the same Ubuntu release and target CPU architecture. Use the bcmn-ubuntu default-image archive here, not the separate bcmn-dgx-os- DGX OS archive.

    Set these variables to the full paths of the files you downloaded. Replace the example paths with the actual filenames and use the same shell for the remaining commands:

    BCNI_ARCHIVE=/path/to/downloaded/bcni-archive.tar.gz
    BCMN_ARCHIVE=/path/to/downloaded/bcmn-archive.tar.gz
    CM_SHARED_ARCHIVE=/path/to/downloaded/bcm-cm-shared-archive.tar.gz
    

    The remaining commands use ubuntu2404 as an example. If the downloaded archives are for another Ubuntu release, use its distribution identifier in the directory names and --distro option instead.

  2. Extract the node-installer archive into /cm/node-installer-ubuntu2404-x86_64:

    mkdir -p /cm/node-installer-ubuntu2404-x86_64
    tar -xzf "$BCNI_ARCHIVE" -C /cm/node-installer-ubuntu2404-x86_64
    
  3. Extract the default software image archive into /cm/images/default-image-ubuntu2404-x86_64:

    mkdir -p /cm/images/default-image-ubuntu2404-x86_64
    tar -xzf "$BCMN_ARCHIVE" -C /cm/images/default-image-ubuntu2404-x86_64
    
  4. Extract the shared binaries archive into /cm/shared-ubuntu2404-x86_64:

    mkdir -p /cm/shared-ubuntu2404-x86_64
    tar -xzf "$CM_SHARED_ARCHIVE" -C /cm/shared-ubuntu2404-x86_64
    
  5. Register the extracted components and architecture with BCM. The --add-only option does not create the images again:

    cm-image --verbose create all --arch x86_64 --distro ubuntu2404 --add-only
    

Import the DGX OS archive separately with cm-create-image as described in Method 1 — Import the DGX OS 7 image from a tar.gz (recommended).

Troubleshooting#

The troubleshooting examples show x86_64 components. If you imported aarch64 archives, identify the corresponding aarch64 entries.

If Step 5 fails at the archOS stage and the components are intact, you can resume by running only the archOS step:

cm-image --verbose create all --arch x86_64 --distro ubuntu2404 --add-archos

If --add-archos fails with an “arch/OS does not match” error, the error message identifies the failing component. That component must be recreated:

  1. Remove the component’s BCM entities and filesystem directory.

    In cmsh → fspart, remove the failing component (there may be more than one):

    cmsh; fspart; remove <component>; commit
    
    Example: identifying fspart components to remove
    [a06-u02-bcm-01->fspart]% ls | grep x86
    /cm/images/default-image-ubuntu2404-x86_64       image           default-image-ubuntu2404-x86_64
    /cm/images/default-image-ubuntu2404-x86_64/boot  boot            default-image-ubuntu2404-x86_64:boot
    /cm/node-installer-ubuntu2404-x86_64             node-installer
    /cm/shared-ubuntu2404-x86_64                    cm-shared
    [a06-u02-bcm-01->fspart]%
    

    If the failed component is the software image, also remove it from cmsh → softwareimage:

    cmsh; softwareimage; remove -d <softwareimage>; commit
    
    Example: identifying the software image to remove
    [a06-u02-bcm-01->softwareimage]% ls | grep x86
    default-image-ubuntu2404-x86_64     /cm/images/default-image-ubuntu2404-x86_64     6.8.0-51-generic        0
    [a06-u02-bcm-01->softwareimage]%
    

    Then remove the filesystem directory and verify it is gone:

    rm -rf <filesystem directory>
    
    Example: verifying removal
    root@a06-u02-bcm-01:~# ls /cm/images/default-image-ubuntu2404-x86_64
    ls: cannot access '/cm/images/default-image-ubuntu2404-x86_64': No such file or directory
    
    root@a06-u02-bcm-01:~# ls /cm/node-installer-ubuntu2404-x86_64
    ls: cannot access '/cm/node-installer-ubuntu2404-x86_64': No such file or directory
    
    root@a06-u02-bcm-01:~# ls /cm/shared-ubuntu2404-x86_64
    ls: cannot access '/cm/shared-ubuntu2404-x86_64': No such file or directory
    
  2. Extract the correct archive for the failed component again. Repeat the extraction command from Method 1: step 2 for the node installer, step 3 for the software image, or step 4 for the shared binaries.

    All three archives must be for the same Ubuntu release and CPU architecture. For example, an ubuntu2204 node-installer archive used with ubuntu2404 images causes the “arch/OS does not match” error.

  3. Recreate only the failed component by re-running cm-image create. If using cm-image create all, answer n when prompted for any components that are already correct.

    # software image
    cm-image create swimage --arch x86_64 --distro ubuntu2404 --add-only
    
    # node-installer
    cm-image create node-installer --arch x86_64 --distro ubuntu2404 --add-only
    
    # /cm/shared
    cm-image create cmshared --arch x86_64 --distro ubuntu2404 --add-only
    
  4. After you recreate the component, run --add-archos to complete the setup:

    cm-image --verbose create all --arch x86_64 --distro ubuntu2404 --add-archos
    

Method 2 — Create images from ISO or a vanilla tar.gz#

  1. Download a BCM installation ISO as described in Acquire BCM ISO to create images for the other architecture, or obtain a vanilla/base-distribution tarball. Download a vanilla tarball from Base Distributions. Navigate to the microarchitecture of the desired OS and download the appropriate .tar.gz file.

  2. Use cm-image to use an .iso or a tar.gz as the source of the image.

    • If using an .iso, the command is:

      cm-image create all --arch x86_64 (or aarch64) --source /root/bcm-11.0-ubuntu2404.iso --distro ubuntu2404 --air-gapped
      
    • If using a vanilla/base-distribution .tar.gz, the command is:

      cm-image create all --arch x86_64 (or aarch64) --source /root/UBUNTU2404.tar.gz --distro ubuntu2404 --air-gapped
      

Note

  • To create an aarch64/ARM image, use –arch aarch64. It is necessary to use a basetar/iso created for the aarch64/ARM architecture.

  • The –air-gapped option skips the connectivity check to the BCM repos, assuming all the packages are present locally.

Caution

  • If a package dependency/conflict failure occurs, use the -j option to exclude the problem package. For example, if the libglapi-mesa package is causing a conflict, use the following command:

    cm-image create all --arch x86_64 --source /root/bcm-11.0-ubuntu2404.iso --distro ubuntu2404 --air-gapped -j libglapi-mesa
    

Method 3 — Generate an image with cm-image#

This method will create an image from scratch; however, it still requires an x86 or aarch64 basetar/iso to create the image depending on the image architecture that is to be generated. QEMU emulation is used in all cases where the image architecture is different from the BCM head node architecture. This process can take a long time.

cm-image create all --bootstrap -d ubuntu2404 -a x86_64 (or aarch64)

Created Directories After the Import is Completed#

This is what a working mixed architecture setup should look like:

Example: Mixed architecture (ARM BCM head node) - default Images

cmsh -c "softwareimage;list"
+-------------------------------+-----------------------------------------------+------------------------+-------+
| Name (key)                    | Path (key)                                    | Kernel version         | Nodes |
+-------------------------------+-----------------------------------------------+------------------------+-------+
| default-image                 | /cm/images/default-image                      | 6.8.0-51-generic-64k   | 0     |
| default-image-ubuntu2404-x86_64| /cm/images/default-image-ubuntu2404-x86_64   | 6.8.0-51-generic       | 0     |
+-------------------------------+-----------------------------------------------+------------------------+-------+

Example: Mixed architecture (ARM BCM head node) - default Categories

cmsh -c "category;list"
+-------------------------------+------------------------------------+-------+
| Name (key)                    | Software Image                     | Nodes |
+-------------------------------+------------------------------------+-------+
| default-ubuntu2404-aarch64    | default-image                      |  0    |
| default-ubuntu2404-x86_64     | default-image-ubuntu2404-x86_64    |  0    |
| dgx                           | dgx-image                          |  0    |
+-------------------------------+------------------------------------+-------+

Example: Mixed architecture (Arm BCM head node) - /cm/shared

cmsh -c "device use master;fsmounts;list"
+------------------------------------------+-------------------------------------+-----------+
| Device                                   | Mountpoint (key)                    | Filesystem|
+------------------------------------------+-------------------------------------+-----------+
| 7.241.16.39:/home                        | /home                               | nfs       |
| 7.241.16.39:/cm_shared/ubuntu2404-x86_64 | /cm/shared-ubuntu2404-x86_64        | nfs       |
| 7.241.16.39:/cm_shared/ubuntu2404-aarch64| /cm/shared-ubuntu2404-aarch64       | nfs       |
+------------------------------------------+-------------------------------------+-----------+

Note

This mixed architecture configuration may not appear in fsmounts until after HA setup is complete.

Example: Mixed architecture (ARM BCM head node) - fsexports

Look at fsexports to show that both x86 and aarch64 node-installers are present as well as the /cm/shared directories.

cmsh -c "device use master;fsexports;list"
+-----------------------------------------------+-----------------------------------------------+-----------+-------+-------+----------+
| Name (key)                                   | Path                                          | Network   | Hosts | Write | Disabled |
+-----------------------------------------------+-----------------------------------------------+-----------+-------+-------+----------+
| /var/spool/burn@internalnet                  | /var/spool/burn                               | internalnet| yes  | no    |          |
| /home@internalnet                            | /home                                         | internalnet| yes  | no    |          |
| /cm/shared-ubuntu2404-x86_64@internalnet     | /cm/shared-ubuntu2404-x86_64                  | internalnet| yes  | no    |          |
| /cm/shared-ubuntu2404-aarch64@internalnet    | /cm/shared-ubuntu2404-aarch64                 | internalnet| yes  | no    |          |
| /cm/node-installer-ubuntu2404-x86_64@internalnet | /cm/node-installer-ubuntu2404-x86_64        | internalnet| no   | no    |          |
| /cm/node-installer-ubuntu2404-x86_64/certificat+ | /cm/node-installer-ubuntu2404-x86_64/certificat+ | internalnet| yes | no    |          |
| /cm/node-installer-ubuntu2404-aarch64@internaln+ | /cm/node-installer-ubuntu2404-aarch64        | internalnet| no   | no    |          |
| /cm/node-installer-ubuntu2404-aarch64/certifica+ | /cm/node-installer-ubuntu2404-aarch64/certifica+ | internalnet| yes | no    |          |
+-----------------------------------------------+-----------------------------------------------+-----------+-------+-------+----------+

Note

There will be an fsexports entry for each network that is used for node provisioning.