Skip to navigation

In-place Upgrades

View as Markdown

3.1 Introduction

In contrast with a parallel upgrade (Chapter 2), with an in-place upgrade the production cluster is taken out of operation while the upgrade takes place.

This chapter covers the procedure for an in-place upgrade.

3.2 Important: Upgrading Particular Applications

This section has important notes on upgrading particular applications.

3.2.1 Workload Managers (WLM)

Torque, SGE and PBSPro CE workload managers are not supported in BCM 10, and hence the relevant packages will be removed as part of the upgrade.

Unversioned WLM packages (such as pbspro/pbspro-client/pbspro-ce/pbspro-ce-client or slurm/slurm-client) will be removed and the administrator can install a versioned WLM package of their choice (such as pbspro2022/pbspro2022-client or slurm22.05/slurm22.05-client).

On RHEL and SLES distributions, unsupported versioned WLM packages are not removed automatically. They must be removed manually and where relevant new versioned WLM packages must be installed.

3.2.1.1 Upgrading WLMs In General From Bright 9.2

Workload manager configurations must be disabled using cm-wlm-setup before proceeding with the upgrade.

After the upgrade has been completed, workload management must be re-deployed using cm-wlm-setup.

3.2.1.2 Slurm Upgrades

Slurm versions older than Slurm 23.02 are not supported in BCM 10. When upgrading from Bright 9.2, Slurm must be upgraded to at least Slurm 23.02 from the Bright 9.2 repositories before moving to BCM 10.

Upgrading Slurm on a BCM 10 cluster is described in section 7.5.1.2 of the Administrator Manual.

3.2.1.3 Slurm DBD Performance Improvement

Starting with the BCM 10.24.05 release, some MySQL configuration settings have been changed to improve the performance of the slurmdbd service. The changes are not automatically applied during the cluster upgrade. The MySQL configuration must therefore be manually adjusted on the head nodes, after cm-upgrade has completed. The /etc/my.cnf on the head node, and also on the secondary head node in an HA-enabled cluster, should have its settings updated as follows:

innodb_buffer_pool_size=4096M
innodb_log_file_size=64M
innodb_lock_wait_timeout=900

3.2.1.4 PBS Pro Upgrades

PBS Pro versions older than PBS Pro 2022 are not supported in BCM 10. Below is an overview of the steps that can be taken to upgrade PBS Pro:

  • Stop the pbsserver service on the head node

    cmsh% device
    cmsh% foreach -l pbsproserver ( services; stop pbsserver )

  • Remove the old packages from the headnodes

    $ yum/zypper/apt remove pbspro<old version>*

  • Remove the old packages from the software images

    $ cm-chroot-sw-img /cm/images/<software image>
    ...
    [root@<software image> /]# yum/zypper/apt remove pbspro<old version>*

  • Install the new packages in the headnode

    $ yum/zypper/apt install pbspro<new version> pbspro<new version>-client

  • Install the new packages in the software images

    $ cm-chroot-sw-img /cm/images/<software image>
    ...
    ...
    [root@<software image> /]# yum/zypper/apt install pbspro<new version>-client

  • Change the version of the WLM in the wlm object

    cmsh% wlm use pbspro
    cmsh% set version <new version>
    cmsh% commit

  • Start pbsserver

    cmsh% device
    cmsh% foreach -l pbsproserver ( services; start pbsserver )

3.2.2 Older OpenMPI versions 1.x Unsupported

The 1.x series of OpenMPI is not supported in BCM 10. These packages are removed during the upgrade. Administrators must install OpenMPI 4 packages manually, and rebuild the applications that use them after the upgrade has been completed.

3.2.3 Kubernetes Package Upgrades

Supported Kubernetes deployments must be removed before upgrading and re-deployed after the upgrade.

  • Remove the old packages from the headnodes

    $ yum/zypper/apt remove cm-kubernetes* cm-etcd cm-kube-diagnose cm-containerd

  • Remove the old packages from the software images

    $ cm-chroot-sw-img /cm/images/<software image>
    ...
    [root@<software image> /]# yum/zypper/apt remove cm-kubernetes* cm-etcd \
     cm-kube-diagnose cm-containerd

For more details please refer to chapter 4 of the BCM 10 Containerization Manual.

3.2.4 JupyterHub Database Upgrade

The JupyterHub database schema must be manually upgraded in each JupyterHub node after the main package upgrade has completed. The following commands can be run to upgrade the JupyterHub database:

module load jupyter
jupyterhub upgrade-db--config=/cm/local/apps/jupyter/conf/jupyterhub_config.py \
  --db sqlite:////cm/local/apps/jupyter/run/jupyterhub.sqlite

3.2.5 Docker Registry/Harbor Package Upgrades

When upgrading a cluster using the BCM ISO, if the Docker registry or Harbor packages were previously installed, then they do not get upgraded automatically on RPM-based distributions. On Ubuntu based clusters the packages get removed automatically.

The packages must be manually installed and upgraded using the package managers (apt/zypper/yum).

If Docker registry was installed, then the cm-docker-registry package must be upgraded/installed.

If Harbor was installed, then the packages to upgrade/install are: cm-harbor cm-docker cm-docker-compose

Also note that if they were deployed on a different node than the head node, then the packages must be upgraded/installed in the relevant software image(s).

Harbor versions older than 2.8 are not supported in BCM 10. When upgrading from Bright clusters running an older Harbor version, remove it with cm-container-registry-setup and re-deploy it after the upgrade.

3.2.6 OpenStack Integration

  • Bright OpenStack deployments must be removed (using cm-openstack-setup) before upgrading. OpenStack integration is no longer supported in BCM 10, and cannot be re-deployed after the upgrade. All older Bright OpenStack packages and dependencies must be removed prior to starting the upgrade. Please contact support (https://enterprise-support.nvidia.com/s/create-case) for further assistance.

3.3 Important: Package Upgrade Dependency Issues

The upgrade process will not only upgrade CMDaemon and its dependencies, but it will also upgrade other packages. This means that old packages will not be available from the new BCM repositories.

In some cases, this will require recompiling the user applications to use the upgraded versions of the compilers and the libraries. Also, the configurations of the old packages will not be copied automatically to the new packages, which means that the administrator will have to adjust the configuration from the old packages to suit the new packages manually.

Extra base distribution packages may be installed by yum/zypper/apt-get in order to resolve dependencies that might arise as a result of the upgrade. Hence the base distribution repositories must be reachable. This means that the clusters that run the Enterprise Linux distributions (RHEL and SLES) must be subscribed to the appropriate software channels.

Packages in /cm/shared are upgraded, but the administrator should be aware of the following:

  • If /cm/shared is installed in the local partition, then the packages are upgraded. This may not be desirable for users that wish to retain the old behavior.

  • If /cm/shared is mounted from a separate partition, then unmounting it will prevent upgrades to the mounted partition, but will allow new packages to be installed in /cm/shared within the local partition. This may be desirable for the administrator, who can later copy over updates from the local /cm/shared to the remote /cm/shared manually according to site-specific requirements. Since unmounting of mounted /cm/shared is carried out by default, a local /cm/shared will have files from any packages installed there upgraded. According to the yum database, the system is then upgraded even though the files are misplaced in the local partition. However, the newer packages can only be expected to work properly if their associated files are copied over from the local partition to the remote partition.

3.4 Important: Upgrading Ubuntu Clusters

After upgrading the cluster, the Ubuntu compute nodes must initially be re-provisioned with a FULL install. To achieve that, the following cmsh example command will update each Ubuntu compute node. In this example, 750 nodes are updated:

% device foreach -n node001..node750 (set nextinstallmode FULL); device commit

Important note: a FULL install will remove any local data on the compute nodes. Thus the administrator must perform the necessary steps to back up the data on the compute nodes before the upgrade and restore it afterward.

3.5 Known Issues

  • On clusters with Jupyter deployed, after the upgrade, users might find themselves being asked to login twice. The first time when the user will be asked to login to JupyterHub, and a second time, where the user is asked about a “CMDaemon login”. In such cases, the user should once again present the credentials. This extra login request is harmless, and after the credentials are given a second time, it is unlikely that users will be queried again. The cause of this issue is that the Jupyterhub component that interacts with CMDaemon can lose connection to CMDaemon, causing it to prompt for an extra login.

  • On some base distributions, depending on the version of the installed packages, when starting cm-upgrade, a YAMLLoadWarning message may be printed when its configuration files are loaded. The warning message can be safely ignored.

  • In some cases, in particular HA setup with a shared storage, it is possible that the slurmdbd.conf.template configuration file is overwritten after the upgrade where the package manager would not preserve a backup copy of the locally modified file. As a result, when Slurm WLM is later setup with cm-wlm-setup CMDaemon will not start slurmctld service because it will detect slurmdbd service is not able to connect to the mysql database. To resolve the issue, the administrator needs to run /cm/shared/apps/slurm/var/cm/cm-restore-db-password to set up a new MySQL password for the Slurm user, and should then restart the slurmdbd service.

  • In some cases because /cm/shared may be temporarily not accessible from the cloned primary head node when it boots for the first time, if workload managers are setup on the cluster, then CMDaemon may create and leave module files called no-version. For example:
    slurm/slurm/no-version
    To resolve the issue, the leftover no-version module files under /cm/shared/modulefiles/ can be deleted.

  • In some cases at the end of the head node or the software image(s) upgrade, the cm-upgrade script may report a list of packages that were not upgraded to their BCM 10 versions. This can occur in cases where the package manager was not able to resolve the list of packages to find a suitable candidate at the time of the upgrade of all packages.

    An example here the OpenBLAS packages upgrades for BCM. In BCM prior to BCM 10, the package was called openblas, packaged by BCM developers. In BCM 10 the BCM package was replaced by cm-openblas (also packaged by BCM developers). The issue occurs since the base distro also can provide a (non-BCM) OpenBLAS package. If cm-upgrade reports that the BCM OpenBLAS package is not upgraded, then the administrator is advised to swap the OpenBLAS package manually with the cm-openblas package.

    While cm-upgrade makes a best effort to swap or upgrade the packages to their BCM 10 versions, in some cases - notably on Ubuntu, it is also possible some packages such as cmburn, hdf518, or hpl are left in their 9.2 versions. The administrator is advised to manually swap the cm9.2 packages (for example cmburn-9.2-392_cm9.2) with the cm10.0 packages (for example cmburn_10.0-391_cm10.0).

  • On clusters with cluster extension to AWS deployed, the cloud director and cloud nodes might fail to start after the upgrade. This can happen when there is an entry

            <blockdev mode="cloud">/dev/nvme1n1</blockdev>

    in the disksetup. The issue can be resolved by removing that entry from the disksetup for the cloud director and cloud nodes.

  • On clusters that have PBSPro configured, and upgraded from 9.2 to BCM 10, new jobs submitted by users will get job ids that were already used. This is particularly relevant for users that use BCM monitoring data for billing purposes.

  • Disabling Kubernetes as part of the parallel-upgrade procedure results in an apparent failure during the stage when etcd data is cleaned up, because the compute nodes are not reachable from the parallel setup. This is expected since the parallel setup is isolated from the production setup. To resolve the issue, the administrator should answer a prompt so that the failed stage is skipped. It is also recommended to re-provision the compute nodes of the upgraded parallel setup before attempting to set up Kubernetes or etcd again, so that any leftover data from before the upgrade is cleaned during the provisioning.

3.6 Upgrading Using A BCM DVD/ISO

When using a BCM DVD/ISO to perform the upgrade, it is important to use a DVD/ISO that is not older than 10.24.01. The DVD/ISO version can be found (assuming that the DVD/ISO is mounted under /mnt/cdrom) with a find command such as:

# find /mnt/cdrom -type d -name '10.24.01'
/mnt/cdrom/data/packages/10.24.01

3.7 Enable Upgrade

3.7.1 Enable The Upgrade Repo, And Install The Upgrade Package

Enabling the upgrade repository and installing the upgrade package can be carried out for the various distributions as follows:

RHEL8 derivatives:

# yum-config-manager \
  --add-repo http://support.brightcomputing.com/upgrade/10.0/rhel/8/updates
# yum --nogpgcheck install cm-upgrade

RHEL9 derivatives:

# yum-config-manager \
--add-repo http://support.brightcomputing.com/upgrade/10.0/rhel/9/updates
# yum --nogpgcheck install cm-upgrade

SLES 15:

# zypper addrepo \
http://support.brightcomputing.com/upgrade/10.0/sles/15/updates cm-upgrade-10
# zypper install cm-upgrade

Ubuntu 20.04:

# cat <<EOF > /etc/apt/sources.list.d/cm-upgrade-10.list
deb [trusted=yes] https://support.brightcomputing.com/upgrade/10.0/ubuntu/2004/ ./
EOF
# apt update
# apt install cm-upgrade

Ubuntu 22.04:

# cat <<EOF > /etc/apt/sources.list.d/cm-upgrade-10.list
deb [trusted=yes] https://support.brightcomputing.com/upgrade/10.0/ubuntu/2204/ ./
EOF
# apt update
# apt install cm-upgrade

3.7.2 Load The Environment Module

# module load cm-upgrade/10.0

3.8 Perform Upgrade

The steps for carrying out the upgrade are as follows:

3.8.1 Power Off Nodes

  • Power off regular nodes

  • Terminate cloud nodes and cloud directors, and remove the cloud extension(s)

3.8.2 Apply Updates To Head Node

  • RHEL derivatives:

    # yum update

  • SLES derivatives:

    # zypper up

  • Ubuntu:

    # apt-get update
    # apt-get upgrade --with-new-pkgs

3.8.3 Apply Updates To Software Images

Before updating and upgrading a software image, it may be worth creating an image backup by cloning the image (here). If the cluster after the upgrade has an image that has an issue, then comparing it against the backup can help troubleshoot the issue.

For each software image, updates can be applied as follows:

  • RHEL derivatives:

    # yum --installroot /cm/images/<software image> update

  • SLES derivatives:

    # zypper --root /cm/images/<software image> up

  • Ubuntu:

    # cm-chroot-sw-img /cm/images/<software image>
    # apt-get update
    # apt-get upgrade --with-new-pkgs
    # exit

3.8.4 Apply Updates To The node-installer Image

  • RHEL derivatives:

    # yum --installroot /cm/node-installer update

  • SLES derivatives:

    # zypper --root /cm/node-installer up

  • Ubuntu:

    # cm-chroot-sw-img /cm/node-installer
    # apt-get update
    # apt-get upgrade --with-new-pkgs
    # exit

3.8.5 Upgrade Head Nodes To BCM 10

Important: this must be run on both head nodes in a high availability setup. Recommended: Upgrade the active head node first and then the passive head node.

Depending on availability, the upgrade can be run either over the network, or using a DVD/ISO:

  • Upgrade using repositories accessible over the network

    # cm-upgrade

  • Upgrade using a BCM DVD/ISO

    # cm-upgrade -b /root/bcm-10.0-rocky8u8.iso

In an HA setup, after upgrading both the head nodes, resync the databases. Run the following from the active head node (it is very important to complete this step before moving to the next one):

# cmha dbreclone <secondary>

3.8.6 Reboot Head Node(s)

The head node(s) must be rebooted before proceeding to run the post-upgrade actions.

3.8.7 Post-upgrade Head Node Actions

Important: these must be run on both head nodes in an HA setup

# module load cm-upgrade/10
# cm-post-upgrade -m

3.8.8 Upgrade The Software Image(s) To BCM 10

Important: the upgrade to the software images must be done only on the active head node. Depending on availability, the upgrade can be run either over the network, or using a DVD/ISO:

  • Upgrade using repositories accessible over the network

    # cm-upgrade -i all

  • Upgrade using a BCM DVD/ISO

    # cm-upgrade -i all -b /root/bcm-10.0-rocky8u8.iso

If the software images are not under the standard location—/cm/images/ on the head node—then the option -a should be used:

# cm-upgrade -a /apps/images -i <name of software image>
# cm-upgrade -a /apps/images -i <name of software image> -b /root/bcm-10.0-rocky8u8.iso

If the software images are located under multiple locations, the target locations can be specified as a comma-separated list:

# cm-upgrade -a '/apps/images,/opt/images' -i all

3.8.8.1 Btrfs Image Upgrades

If the software image is a Btrfs image, then the revision (section 16.4 of the Administrator Manual) images cannot be used, because they use read-only filesystems. This issue extends to all references in category or node images, such as @latest and @number

Only the original, unrevisioned, parent image can be upgraded with the cm-upgrade and cm-post-upgrade steps with a Btrfs filesystem.

A workaround would be to create a new non-CoW image from a trusted revision, and upgrading that trusted non-CoW version.

3.8.9 Upgrade /cm/node-installer To BCM 10

This needs to be performed only on the primary active head node, either over the network, or using a BCM DVD/ISO.

  • Upgrade using repositories accessible over the network:

    # cm-upgrade -x

  • Upgrade using a BCM DVD/ISO:

    # cm-upgrade -x -b /root/bcm-10.0-rocky8u8.iso

3.8.10 Post-upgrade Software Images Actions

Important: these must be run only on the active head node.

# cm-post-upgrade -i all

If the software images are not under the standard location, which is /cm/images/ on the head node, then the option -a should be used:

# cm-post-upgrade -a /apps/images -i all
# cm-post-upgrade -a /apps/images -i <name of software image>

If the software images are located under multiple locations, then the target locations can be specified as a comma-separated list:

# cm-post-upgrade -a '/apps/images,/opt/images' -i all

3.8.11 (HA) Allow For The Provisioning Requests To The Secondary Head Node To Complete

Allow between 15 and 30 seconds for CMDaemon to schedule and start provisioning updates of /tftpboot and /cm/node-installer from the primary to the secondary head node. Then check and wait until the provisioning requests with the secondary head node as destination have completed using:

# cmsh -c 'softwareimage provisioningstatus -r'

If the provisioning requests with a destination node the secondary head node continue to be listed for a prolonged period of time and do not disappear from the list, then the node-installer and/or tftpboot on the secondary head node are out of sync from the primary head node, which can result in issues with provisioning of the compute nodes. Please contact support (https://enterprise-support.nvidia.com/s/create-case) for further assistance.

3.8.12 The Cluster Is Now Upgraded To BCM 10

It is recommended to provision the compute nodes at least once with BCM 10 before setting up any new integrations. In the case of Ubuntu base distro, it is recommended to initially perform a FULL install/provisioning of the compute nodes as noted in the preceding.

3.9 In-place Upgrade Of An Edge Director Or A Compute Node

The recommended way of upgrading the compute nodes is to upgrade the head node and software image(s) as described in section 3.8, and to allow the node-installer (when the compute nodes are booted) to re-provision the compute nodes with the upgraded software. For clusters with an edge setup, this usually means a new edge ISO needs to be generated on the head node, transferred to the edge site, and then the edge director node needs to be reconfigured to boot from the ISO, which will start the node-installer.

In some special cases, however, the administrator may elect to manually carry out an in-place upgrade of a compute node or edge director, instead of using the node installer to automatically upgrade the node. With the manual in-place upgrade, the administrator sets up the node to receive the updates directly from the software image (through imageupdate) without the involvement of the node installer.

Other than the cases where the node installer does not start on (re)boot by default, such as for the edge director, the in-place upgrade of a compute node/edge director does not offer any advantage over the standard approach to simply upgrade the head node and the software image(s).

3.9.1 Important Note About Services Running On The In-place Upgraded Compute Node/Edge Director

An in-place upgrade of a compute node will generally restart only the CMDaemon service. Any other service that may be affected by the upgrade of the packages, especially in the case of packages in /cm/shared (such as WLM), which are upgraded as part of the head node(s) upgrade, will need to be restarted manually by the administrator, or a reboot will be required.

It is therefore recommended to use the standard approach to upgrade the software image(s) on the head node, boot the nodes with the node installer, and allow the node installer to upgrade and configure the compute nodes as usual.

Nevertheless the administrator may find it more convenient in special cases such as edge sites, which have edge directors, to perform an in-place upgrade. The edge directors nodes are typically configured to boot from the local disk and by default do not start the node installer on (re)boot. An in-place upgrade of the edge directors will then allow the administrator to upgrade the software on the directors without transferring ISOs and reconfiguring the directors to boot from ISOs. To address the issue with the running services, the edge directors can be rebooted at the end of the upgrade and while they will still not start the node installer, they will boot into the upgraded software.

3.9.2 In-place Edge Director Upgrade Procedure

Unless otherwise stated, all actions are performed locally on the edge director

3.9.2.1 Apply Updates To The Edge Director

  • RHEL derivatives:

    # yum update

  • SLES derivatives:

    # zypper up

  • Ubuntu:

    # apt-get update
    # apt-get upgrade --with-new-pkgs

3.9.2.2 Back Up The File System

It is recommended to perform a file system backup of the edge director

3.9.2.3 Stop CMDaemon and cm-nfs-checker services

# systemctl stop cmd
# systemctl stop cm-nfs-checker

3.9.2.4 Upgrade The Head Node(s) As Described In The Preceding

Follow the steps to upgrade the head nodes, software image, and node installer as described in the preceding sections “Enable Upgrade” (section 3.7) and “Perform Upgrade” (section 3.8), and which are performed on the head node(s) of the cluster. Because CMDaemon on the edge director is stopped, the edge director does not need to be powered off while the upgrade of the head nodes takes place.

3.9.2.5 Upgrade CMDaemon To BCM 10

After the upgrade of the head nodes is completed, the package manager repositories files on the edge director must be manually updated to point to the BCM 10 repositories. This means at all occurrences of 9.2 in the repositories are replaced with with 10.0. Note: Leave the usernames such as in, for example, “username=cm92user”, unchanged.

Then, upgrade CMDaemon as follows:

  • RHEL derivatives:

    Update the /etc/yum.repos.d/cm.repo file to point to the 10.0 repositories.

    On RHEL, typically yum/dnf can manage to resolve most of the dependencies, and the administrator can execute:

    # yum clean all
    # rpm -e --nodeps cm-config-cm
    # yum install cmdaemon

  • SLES derivatives:

    Update the /etc/zypp/repos.d/Cluster_Manager_Updates.repo file to point to the 10.0 repositories.

    Execute

    # zypper clean --all
    # zypper install cmdaemon

    A problem may be reported by zypper with upgrading cmdaemon to version 10.0, due to a requirement to install cm-config-cm version 10.0. Typically the first solution, Solution 1, offered by zypper can be selected, which is to deinstall a certain set of Bright packages (prefixed by cm9.2) in order to upgrade CMDaemon to version 10.0.

  • Ubuntu:

    Update the /etc/apt/sources.list.d/cm.list file to point to the 10.0 repositories.

    Also update the /etc/apt/auth.conf.d/cm.conf file, to point to the 10.0 repositories.

    Create a new file /etc/apt/preferences.d/99-tmp-cmd-upgrade with content (without any space at the beginning of the lines):

    Package: *
    Pin: origin "updates.brightcomputing.com"
    Pin-Priority: 1001

    The administrator may also find it convenient to choose a geographically-close mirror in the cm.list file. But note that the local change will be overwritten to the settings in the software image, once imageupdate is run in a step later on.

    Execute

    # apt-get update
    # apt-get install cmdaemon

3.9.2.6 Start CMDaemon

After CMDaemon has been upgraded to BCM 10, execute:

# systemctl daemon-reload
# systemctl start cm-nfs-checker
# systemctl start cmd

3.9.2.7 Update The Edge Director Image With imageupdate

On the head node, with cmsh, verify that the edge director is UP. If the edge director is not UP, then the in-place upgrade has not been successful, and the director will need to be upgraded by booting into to the node-installer.

With cmsh on the head node, execute imageupdate for the edge director. The administrator is advised to execute a dry-run first, then to use use the synclog command to review the result and verify no unexpected local data directories or files will be affected, and then to perform the real imageupdate:

cmsh% device use edge-director
cmsh% imageupdate
cmsh% synclog
cmsh% imageupdate -w

3.9.2.8 Update The Provisioners And Recreate The Ramdisks

Execute on the head node

cmsh% softwareimage updateprovisioners;
cmsh% fspart; trigger /tftpboot; trigger /cm/node-installer; trigger /cm/shared

Allow for the provisioning requests with a destination node the in-place upgraded director to complete by monitoring the provisioning status:

cmsh% softwareimage provisioningstatus -r

For the images served by the edge director, recreate the ramdisk

cmsh% softwareimage; createramdisk -d <image-name>

If needed, the in-place upgraded edge director can now also be rebooted. It is then upgraded and can provision the edge compute nodes.