Mixed Architecture Setup#
In general, the architecture that is installed on the BCM head node is the default OS image available for other connected servers (nodes). When nodes use a different architecture, an OS image compiled for that architecture needs to be created or imported. For example, If the BCM head node itself is an x86 server, an ARM/aarch64 OS image would be created to support any ARM/aarch64 nodes. The cm-image tool is specifically used for creating different OS/ARCH images (software image, node-installer image and cm-shared image).
Note
The BCM head node architecture can be either x86 or aarch64/ARM. The process is the same in either case.
The cm-image tool uses QEMU to emulate the other architecture if necessary. This can take a long time (approximately 4 hours).
The cm-image tool manages three components — software image, node-installer, and
cm-shared — each of which has two parts:
A filesystem directory (for example,
/cm/images/default-image-<distro>-<arch>)A BCM entity in
cmsh. All three components have entries undercmsh→fspart; the software image also appears undercmsh→softwareimage.
Method 1 — Import Pre-compiled Images (fastest)#
The following example imports x86_64 archives on an aarch64 BCM head node,
as specified by the DGX SuperPOD reference architecture. If your BCM head node
is x86_64 and your target nodes are aarch64, download the aarch64
archives and replace x86_64 with aarch64 in the commands.
From the DGX OS release page in the NVIDIA Enterprise Support Portal, download these three precompiled archives for the target architecture. The page lists them under Additional files that may be required when importing images:
The archive with the
bcni-ubuntufilename prefix contains the node installer.The archive with the
bcmn-ubuntufilename prefix contains the default software image.The archive with the
bcm-cm-shared-ubuntufilename prefix contains the shared binaries.
Choose all three archives for the same Ubuntu release and target CPU architecture. Use the
bcmn-ubuntudefault-image archive here, not the separatebcmn-dgx-os-DGX OS archive.Set these variables to the full paths of the files you downloaded. Replace the example paths with the actual filenames and use the same shell for the remaining commands:
BCNI_ARCHIVE=/path/to/downloaded/bcni-archive.tar.gz BCMN_ARCHIVE=/path/to/downloaded/bcmn-archive.tar.gz CM_SHARED_ARCHIVE=/path/to/downloaded/bcm-cm-shared-archive.tar.gz
The remaining commands use
ubuntu2404as an example. If the downloaded archives are for another Ubuntu release, use its distribution identifier in the directory names and--distrooption instead.Extract the node-installer archive into
/cm/node-installer-ubuntu2404-x86_64:mkdir -p /cm/node-installer-ubuntu2404-x86_64 tar -xzf "$BCNI_ARCHIVE" -C /cm/node-installer-ubuntu2404-x86_64
Extract the default software image archive into
/cm/images/default-image-ubuntu2404-x86_64:mkdir -p /cm/images/default-image-ubuntu2404-x86_64 tar -xzf "$BCMN_ARCHIVE" -C /cm/images/default-image-ubuntu2404-x86_64
Extract the shared binaries archive into
/cm/shared-ubuntu2404-x86_64:mkdir -p /cm/shared-ubuntu2404-x86_64 tar -xzf "$CM_SHARED_ARCHIVE" -C /cm/shared-ubuntu2404-x86_64
Register the extracted components and architecture with BCM. The
--add-onlyoption does not create the images again:cm-image --verbose create all --arch x86_64 --distro ubuntu2404 --add-only
Import the DGX OS archive separately with cm-create-image as described in Method 1 — Import the DGX OS 7 image from a tar.gz (recommended).
Troubleshooting#
The troubleshooting examples show x86_64 components. If you imported
aarch64 archives, identify the corresponding aarch64 entries.
If Step 5 fails at the archOS stage and the components are intact, you can resume by running only the archOS step:
cm-image --verbose create all --arch x86_64 --distro ubuntu2404 --add-archos
If --add-archos fails with an “arch/OS does not match” error, the error message
identifies the failing component. That component must be recreated:
Remove the component’s BCM entities and filesystem directory.
In
cmsh→fspart, remove the failing component (there may be more than one):cmsh; fspart; remove <component>; commitExample: identifying fspart components to remove
[a06-u02-bcm-01->fspart]% ls | grep x86 /cm/images/default-image-ubuntu2404-x86_64 image default-image-ubuntu2404-x86_64 /cm/images/default-image-ubuntu2404-x86_64/boot boot default-image-ubuntu2404-x86_64:boot /cm/node-installer-ubuntu2404-x86_64 node-installer /cm/shared-ubuntu2404-x86_64 cm-shared [a06-u02-bcm-01->fspart]%
If the failed component is the software image, also remove it from
cmsh→softwareimage:cmsh; softwareimage; remove -d <softwareimage>; commitExample: identifying the software image to remove
[a06-u02-bcm-01->softwareimage]% ls | grep x86 default-image-ubuntu2404-x86_64 /cm/images/default-image-ubuntu2404-x86_64 6.8.0-51-generic 0 [a06-u02-bcm-01->softwareimage]%
Then remove the filesystem directory and verify it is gone:
rm -rf <filesystem directory>Example: verifying removal
root@a06-u02-bcm-01:~# ls /cm/images/default-image-ubuntu2404-x86_64 ls: cannot access '/cm/images/default-image-ubuntu2404-x86_64': No such file or directory root@a06-u02-bcm-01:~# ls /cm/node-installer-ubuntu2404-x86_64 ls: cannot access '/cm/node-installer-ubuntu2404-x86_64': No such file or directory root@a06-u02-bcm-01:~# ls /cm/shared-ubuntu2404-x86_64 ls: cannot access '/cm/shared-ubuntu2404-x86_64': No such file or directory
Extract the correct archive for the failed component again. Repeat the extraction command from Method 1: step 2 for the node installer, step 3 for the software image, or step 4 for the shared binaries.
All three archives must be for the same Ubuntu release and CPU architecture. For example, an
ubuntu2204node-installer archive used withubuntu2404images causes the “arch/OS does not match” error.Recreate only the failed component by re-running
cm-image create. If usingcm-image create all, answernwhen prompted for any components that are already correct.# software image cm-image create swimage --arch x86_64 --distro ubuntu2404 --add-only # node-installer cm-image create node-installer --arch x86_64 --distro ubuntu2404 --add-only # /cm/shared cm-image create cmshared --arch x86_64 --distro ubuntu2404 --add-only
After you recreate the component, run
--add-archosto complete the setup:cm-image --verbose create all --arch x86_64 --distro ubuntu2404 --add-archos
Method 2 — Create images from ISO or a vanilla tar.gz#
Download a BCM installation ISO as described in Acquire BCM ISO to create images for the other architecture, or obtain a vanilla/base-distribution tarball. Download a vanilla tarball from Base Distributions. Navigate to the microarchitecture of the desired OS and download the appropriate .tar.gz file.
Use cm-image to use an .iso or a tar.gz as the source of the image.
If using an .iso, the command is:
cm-image create all --arch x86_64 (or aarch64) --source /root/bcm-11.0-ubuntu2404.iso --distro ubuntu2404 --air-gappedIf using a vanilla/base-distribution .tar.gz, the command is:
cm-image create all --arch x86_64 (or aarch64) --source /root/UBUNTU2404.tar.gz --distro ubuntu2404 --air-gapped
Note
To create an aarch64/ARM image, use –arch aarch64. It is necessary to use a basetar/iso created for the aarch64/ARM architecture.
The –air-gapped option skips the connectivity check to the BCM repos, assuming all the packages are present locally.
Caution
If a package dependency/conflict failure occurs, use the -j option to exclude the problem package. For example, if the libglapi-mesa package is causing a conflict, use the following command:
cm-image create all --arch x86_64 --source /root/bcm-11.0-ubuntu2404.iso --distro ubuntu2404 --air-gapped -j libglapi-mesa
Method 3 — Generate an image with cm-image#
This method will create an image from scratch; however, it still requires an x86 or aarch64 basetar/iso to create the image depending on the image architecture that is to be generated. QEMU emulation is used in all cases where the image architecture is different from the BCM head node architecture. This process can take a long time.
cm-image create all --bootstrap -d ubuntu2404 -a x86_64 (or aarch64)
Created Directories After the Import is Completed#
This is what a working mixed architecture setup should look like:
Example: Mixed architecture (ARM BCM head node) - default Images
cmsh -c "softwareimage;list"
+-------------------------------+-----------------------------------------------+------------------------+-------+
| Name (key) | Path (key) | Kernel version | Nodes |
+-------------------------------+-----------------------------------------------+------------------------+-------+
| default-image | /cm/images/default-image | 6.8.0-51-generic-64k | 0 |
| default-image-ubuntu2404-x86_64| /cm/images/default-image-ubuntu2404-x86_64 | 6.8.0-51-generic | 0 |
+-------------------------------+-----------------------------------------------+------------------------+-------+
Example: Mixed architecture (ARM BCM head node) - default Categories
cmsh -c "category;list"
+-------------------------------+------------------------------------+-------+
| Name (key) | Software Image | Nodes |
+-------------------------------+------------------------------------+-------+
| default-ubuntu2404-aarch64 | default-image | 0 |
| default-ubuntu2404-x86_64 | default-image-ubuntu2404-x86_64 | 0 |
| dgx | dgx-image | 0 |
+-------------------------------+------------------------------------+-------+
Example: Mixed architecture (Arm BCM head node) - /cm/shared
cmsh -c "device use master;fsmounts;list"
+------------------------------------------+-------------------------------------+-----------+
| Device | Mountpoint (key) | Filesystem|
+------------------------------------------+-------------------------------------+-----------+
| 7.241.16.39:/home | /home | nfs |
| 7.241.16.39:/cm_shared/ubuntu2404-x86_64 | /cm/shared-ubuntu2404-x86_64 | nfs |
| 7.241.16.39:/cm_shared/ubuntu2404-aarch64| /cm/shared-ubuntu2404-aarch64 | nfs |
+------------------------------------------+-------------------------------------+-----------+
Note
This mixed architecture configuration may not appear in fsmounts until after HA setup is complete.
Example: Mixed architecture (ARM BCM head node) - fsexports
Look at fsexports to show that both x86 and aarch64 node-installers are present as well as the /cm/shared directories.
cmsh -c "device use master;fsexports;list"
+-----------------------------------------------+-----------------------------------------------+-----------+-------+-------+----------+
| Name (key) | Path | Network | Hosts | Write | Disabled |
+-----------------------------------------------+-----------------------------------------------+-----------+-------+-------+----------+
| /var/spool/burn@internalnet | /var/spool/burn | internalnet| yes | no | |
| /home@internalnet | /home | internalnet| yes | no | |
| /cm/shared-ubuntu2404-x86_64@internalnet | /cm/shared-ubuntu2404-x86_64 | internalnet| yes | no | |
| /cm/shared-ubuntu2404-aarch64@internalnet | /cm/shared-ubuntu2404-aarch64 | internalnet| yes | no | |
| /cm/node-installer-ubuntu2404-x86_64@internalnet | /cm/node-installer-ubuntu2404-x86_64 | internalnet| no | no | |
| /cm/node-installer-ubuntu2404-x86_64/certificat+ | /cm/node-installer-ubuntu2404-x86_64/certificat+ | internalnet| yes | no | |
| /cm/node-installer-ubuntu2404-aarch64@internaln+ | /cm/node-installer-ubuntu2404-aarch64 | internalnet| no | no | |
| /cm/node-installer-ubuntu2404-aarch64/certifica+ | /cm/node-installer-ubuntu2404-aarch64/certifica+ | internalnet| yes | no | |
+-----------------------------------------------+-----------------------------------------------+-----------+-------+-------+----------+
Note
There will be an fsexports entry for each network that is used for node provisioning.