Is this page helpful?

Part 1: Building an OS Image for BCM#

Important

Example OS: RHEL 9.6. The steps in this part use RHEL-specific tooling (dnf, subscription-manager, .repo files). For other distributions, substitute the equivalent bootstrap tool, package manager, and repository mechanism. The image root structure and tarball format expected by BCM are the same across distributions.

Prerequisites#

The first three items below are RHEL-specific. For other distributions, substitute the equivalent installation media and any required subscription or license. Items 4 and 5 apply to all distributions.

Before starting, ensure the following are in place:

  1. A DGX node available for bare-metal RHEL installation. This node will be repurposed as the build host.

  2. A RHEL 9.6 DVD ISO downloaded from the Red Hat Customer Portal. An active Red Hat subscription or Red Hat Developer Subscription is required. RHEL subscription and software access must be coordinated with your organization’s RHEL tenant administrators.

    Note

    The boot ISO is a network-based installer — unlike the full DVD ISO, it does not bundle packages locally and pulls all packages from Red Hat’s Content Delivery Network (CDN) at install time. This guide uses the DVD ISO. If you use boot ISO instead, ensure the DGX node has network access to Red Hat’s CDN during installation.

  3. A bootable USB drive flashed with the RHEL 9.6 DVD ISO.

  4. The cluster head node connected to the internet. This is required to pull upstream RHEL package repositories during BCM image import.

  5. BCM installed and operational on the cluster head node.

Phase 1: Bare-Metal RHEL Installation#

Note

All commands in Phase 1 should be performed on the DGX host (at the OS/shell level) set aside for the bare metal RHEL installation.

  1. Flash the USB and boot the DGX node.

    Flash the RHEL 9.6 DVD ISO to a USB drive, then boot the DGX node from it.

  2. Install RHEL 9.6.

    Follow the steps in the Installing Red Hat Enterprise Linux on DGX guide to complete the bare-metal installation.

  3. Register the system and export entitlement certificates.

    After booting into the installed RHEL 9.6 system, register with Red Hat Subscription Manager. The exported certificates will be needed when importing the final image into BCM.

    # 1. Register the system:
subscription-manager register --username <your-rh-username> --password <your-rh-password>

# 2. Attach a subscription:
subscription-manager attach --auto

# 3. Export the 3 cert files — note the paths for use later
ls /etc/pki/entitlement/    # <ID>.pem and <ID>-key.pem
ls /etc/rhsm/ca/            # redhat-uep.pem

  4. Pin the host release and verify repositories.

    # 1. Pin the release
subscription-manager release --set=9.6

# 2. Verify repositories:
dnf repolist

    If baseos, appstream, and codeready-builder appear in the output, they are already enabled — skip Step 5.

  5. Enable required repositories (if not already enabled).

    subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

Phase 2: Bootstrap the Image Root#

In this phase, the target OS image is constructed entirely within an isolated directory on the DGX build host system set up in Phase 1 without modifying the host OS itself. The host’s package manager and repositories are leveraged to populate the image root with a minimal, self-contained environment: the directory structure is initialized, repositories are made available inside it, and the required packages are installed into it. The result is a fully formed OS root, ready to be packaged and imported into BCM.

Note

All Phase 2 commands must be executed on the same DGX host (at the OS/shell level) used for the bare-metal RHEL installation in Phase 1.

  1. Create the image root directory.

    Ensure at least 10 GiB of free space, then create the directory:

    mkdir -p /srv/rhel9-dgx-image
df -h /srv

  2. Bootstrap the RHEL @core image.

    dnf --installroot=/srv/rhel9-dgx-image \
   --releasever=9.6 \
   --nogpgcheck \
   install @core

  3. Copy host repositories into the image.

    The image root has no repos of its own yet. Expose the host’s repos to it:

    mkdir -p /srv/rhel9-dgx-image/etc/yum.repos.d
cp /etc/yum.repos.d/*.repo /srv/rhel9-dgx-image/etc/yum.repos.d/
# Answer 'y' if prompted to overwrite redhat.repo

  4. Install required base packages.

    dnf --installroot=/srv/rhel9-dgx-image \
   --releasever=9.6 \
   --nogpgcheck \
   install \
      openssh-server \
      NetworkManager \
      less \
      sudo \
      which \
      tar \
      gzip \
      setup \
      glibc-common \
      grub2-tools \
      grub2-tools-minimal \
      grubby \
      authselect \
      sssd-common

  5. Install Extra Packages for Enterprise Linux (EPEL) and BCM required packages.

    dnf --installroot=/srv/rhel9-dgx-image \
   --releasever=9.6 \
   --nogpgcheck \
   install -y \
   https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

dnf --installroot=/srv/rhel9-dgx-image \
   --releasever=9 \
   --nogpgcheck \
   install -y \
      libatomic_ops \
      libXxf86vm-devel \
      mariadb-devel \
      mesa-libOSMesa \
      opensm-devel \
      perl-Data-UUID \
      perl-HTTP-Daemon \
      perl-Pod-Parser \
      perl-Unicode-EastAsianWidth \
      texinfo

  6. Create the base network configuration.

    mkdir -p /srv/rhel9-dgx-image/etc/sysconfig/
echo "NETWORKING=yes" > /srv/rhel9-dgx-image/etc/sysconfig/network

Phase 3: Add NVIDIA Repositories and GPG Keys#

BCM requires NVIDIA driver packages during image creation. Without the NVIDIA repo configured, cm-image create will fail at repo validation.

Note

All commands in Phase 3 must be executed on the same DGX host (at the OS/shell level) used for bootstrapping the image root in Phase 2.

  1. Download the NVIDIA DGX local repo packages.

    Download the packages that apply to your release from the NVIDIA DGX EL9 local repo page (requires an NVIDIA enterprise account). This guide uses release EL9-26.02. BCM installs the latest RHEL kernel, which resolves to 9.7, and these repos are compatible with that release. The required RPMs are:

    • dgx-local-repo-26.02-3.el9.x86_64.rpm

    • nvidia-driver-local-repo-rhel9-580.95.05-1.0-1.x86_64.rpm

    • cuda-repo-rhel9-13-0-local-13.0.2_580.95.05-1.x86_64.rpm

  2. Copy the RPMs into the image and install them.

    # 1. Copy the RPMs into the image
mkdir -p /srv/rhel9-dgx-image/tmp/
cp ./dgx-local-repo-26.02-3.el9.x86_64.rpm \
   ./nvidia-driver-local-repo-rhel9-580.95.05-1.0-1.x86_64.rpm \
   ./cuda-repo-rhel9-13-0-local-13.0.2_580.95.05-1.x86_64.rpm \
   /srv/rhel9-dgx-image/tmp/

# 2. Install the RPMs
dnf --installroot=/srv/rhel9-dgx-image --nogpgcheck install -y \
   /srv/rhel9-dgx-image/tmp/dgx-local-repo-26.02-3.el9.x86_64.rpm \
   /srv/rhel9-dgx-image/tmp/nvidia-driver-local-repo-rhel9-580.95.05-1.0-1.x86_64.rpm \
   /srv/rhel9-dgx-image/tmp/cuda-repo-rhel9-13-0-local-13.0.2_580.95.05-1.x86_64.rpm

  3. Verify NVIDIA repos are present.

    dnf --installroot=/srv/rhel9-dgx-image repolist | grep -i nvidia

# Expected output:
# nvidia-dgx-9-local                     NVIDIA DGX EL9
# nvidia-driver-local-rhel9-580.95.05    nvidia-driver-local-rhel9-580.95.05

  4. Import GNU Privacy Guard (GPG) Keys into the Image.

    To keep GPG checking intact and avoid building an image with security verification disabled, mount the required filesystems and import all keys.

    # 1. Mount required filesystems
mount -t proc proc /srv/rhel9-dgx-image/proc
mount -t sysfs sys /srv/rhel9-dgx-image/sys
mount -o bind /dev /srv/rhel9-dgx-image/dev

# 1. Find all GPG keys inside the image — note the paths returned
find /srv/rhel9-dgx-image/var -name "*.pub" -o -name "RPM-GPG-KEY*"

    Import each key using the paths from the output above:

    # 1. Red Hat key
rpm --root /srv/rhel9-dgx-image \
   --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

# 2. NVIDIA driver key
rpm --root /srv/rhel9-dgx-image \
   --import /srv/rhel9-dgx-image/var/nvidia-driver-local-repo-rhel9-580.95.05/<key>.pub

# 3. CUDA key
rpm --root /srv/rhel9-dgx-image \
   --import /srv/rhel9-dgx-image/var/cuda-repo-rhel9-13-0-local/<key>.pub

  5. Verify all keys were imported.

    There should be four keys: one for Red Hat, two for the NVIDIA driver, and one for CUDA.

    rpm --root /srv/rhel9-dgx-image -qa gpg-pubkey*

  6. Add resolv.conf if not present.

    Before running package commands inside the image, ensure DNS is configured. If /etc/resolv.conf is missing, DNS lookups from within the chroot will fail:

    test -f /srv/rhel9-dgx-image/etc/resolv.conf || \
   cp /etc/resolv.conf /srv/rhel9-dgx-image/etc/resolv.conf

  7. Confirm GPG now works by retrying a package install without –nogpgcheck:

    chroot /srv/rhel9-dgx-image /bin/bash -c "dnf install -y openssh-server"

  8. Unmount filesystems before proceeding because these must not be packed into the tarball.

    umount /srv/rhel9-dgx-image/dev
umount /srv/rhel9-dgx-image/sys
umount /srv/rhel9-dgx-image/proc

Phase 4: Prepare the Image for Export#

Note

All commands in this Phase must be executed on the same DGX host (at the OS/shell level) used for Phase 1-3.

  1. Sanity check the image root.

    chroot /srv/rhel9-dgx-image /usr/bin/bash -c "uname -a; cat /etc/os-release"

#Expected: RHEL 9.6 release info printed cleanly.

  2. Fix /dev/null inside the Image.

    Character devices like /dev/null do not survive a tar/extract cycle correctly. They get recreated as plain empty files, which breaks systemd-nspawn and cm-image create on the head node.

    ls -l /srv/rhel9-dgx-image/dev/null
# If the output shows -rw-r--r--, the device node was created as a plain file. Recreate it as a character device:
rm -f /srv/rhel9-dgx-image/dev/null
mknod -m 0666 /srv/rhel9-dgx-image/dev/null c 1 3

# Verify — should show: crw-rw-rw- 1 root root 1, 3
ls -l /srv/rhel9-dgx-image/dev/null

  3. Copy Red Hat entitlement certificates into the image.

    These are the cert files exported in Phase 1, Step 3.

    # Create the entitlement paths inside the image
mkdir -p /srv/rhel9-dgx-image/etc/pki/entitlement
mkdir -p /srv/rhel9-dgx-image/etc/rhsm/ca

# Copy the cert files
cp /etc/pki/entitlement/<ID>.pem /srv/rhel9-dgx-image/etc/pki/entitlement/
cp /etc/pki/entitlement/<ID>-key.pem /srv/rhel9-dgx-image/etc/pki/entitlement/
cp /etc/rhsm/ca/redhat-uep.pem /srv/rhel9-dgx-image/etc/rhsm/ca/

  4. Pin the RHEL release inside the image.

    This is essential because BCM runs dnf inside the image chroot during image creation. The subscription manager reads /etc/rhsm/release to determine which release version to request from the Red Hat CDN.

    echo "9.6" | sudo tee /srv/rhel9-dgx-image/etc/rhsm/release

  5. Create placeholder mount directories.

    cm-chroot-sw-img* needs to mount into these directories later. Add them as empty placeholders before creating the tarball.

    mkdir -p /srv/rhel9-dgx-image/proc
mkdir -p /srv/rhel9-dgx-image/sys
mkdir -p /srv/rhel9-dgx-image/run
mkdir -p /srv/rhel9-dgx-image/tmp
chmod 1777 /srv/rhel9-dgx-image/tmp

    Verify these directories are empty before proceeding:

    ls /srv/rhel9-dgx-image/dev
ls /srv/rhel9-dgx-image/run
ls /srv/rhel9-dgx-image/proc
ls /srv/rhel9-dgx-image/sys
ls /srv/rhel9-dgx-image/tmp

  6. Create the tarball.

    cd /srv/rhel9-dgx-image
tar --numeric-owner \
   -czf /srv/rhel9-dgx-image.tar.gz .

# After it completes, verify the tarball was created:
ls -lh /srv/rhel9-dgx-image.tar.gz