> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Parallel Upgrades ## 2.1 Introduction In contrast with the in-place upgrade (Chapter [3](/upgrade/inplaceupgrade)), for a parallel upgrade the production cluster remains operational while the upgrade takes place. This is possible with the new `cm-clone-install` and `cm-upgrade` functionality that allows the production (primary) head node to be cloned to a new node. The new node is upgraded (in parallel) outside of the cluster, and when the upgrade is completed, the old production primary head node is replaced with the (newly) upgraded head node. This last step to replace the head node will require a downtime. Alternatively, the compute nodes can be gradually moved from the old production to the new parallel-upgrade setup, while the two setups co-exist. ## 2.2 Preparations For The Parallel Upgrade Setup The following steps need to be carried out for parallel upgrades. The steps marked with HA (High Availability) need to be carried out only on clusters with (head node) HA: * Have the Bright product key and the MAC address of the spare node (which is to be used for cloning the primary active head node) ready, since they will need to be provided while cloning the primary head node to the spare node. In an air-gapped cluster the product key is not provided while cloning. It is instead used later, when a license is requested separately. * (HA) Have also the MAC address of the secondary head node ready. * Have the new IPs for the cloned primary head node on the external network(s) ready. * (HA) The primary head node must be made the active head, if it is not already. * (HA) Login to the secondary passive head node and stop CMDaemon (systemctl stop cmd). Note: make sure to stop CMDaemon on the passive secondary head node, and not on the primary active head node. * (HA) On the same secondary passive head node, edit /etc/fstab and comment out the entry for `/cm/shared`, so that on the next boot the secondary passive head node does not mount `/cm/shared` from the file server. * (HA) Shut down the secondary passive head node. The node should remain shut down until later when plugged in to the parallel setup. * Plug the spare node to the internal network of the production cluster and PXE boot it to the Bright rescue environment (select RESCUE in the PXE boot menu, in the same way as performing a clone for an HA setup) * Log in to the rescue environment and run `cm-clone-install`, providing the product key and the MAC address of this spare node. In the case of HA, the MAC address of the secondary head node must also be provided. Thus, for the non-HA case: ```bash # /cm/cm-clone-install --clone --upgrade --productkey PRODUCT-KEY \ --primarymac PRIMARY-MAC ``` and for the HA case: ```bash # /cm/cm-clone-install --clone --upgrade --productkey PRODUCT-KEY \ --primarymac PRIMARY-MAC --secondarymac SECONDARY-MAC ``` and follow the instructions. At the end of the cloning process, confirm shutting down the spare node. In the preceding the values used for the primary and secondary MAC addresses are specified as follows:\ `PRIMARY_MAC`: the MAC address of the management interface of the primary head node (i.e the one that is on the management network)\ `SECONDARY_MAC`: the MAC address of the management interface of the secondary head node (i.e the one that is on the management network) In an air-gapped cluster the preceding commands cannot complete. A license is requested over the network once the disk sync and finalization stages have finished, and if the licensing service cannot be reached, then the request fails and the run stops before the final cleanup and shutdown stage. The clone is finalized on disk, but it is not prepared for shutdown, and it is therefore not ready to be moved to the parallel setup. For an air-gapped cluster the `--skip-request-license` option is therefore used in place of `--productkey`. No product key is given on the command line. Thus, for the non-HA case: ```bash # /cm/cm-clone-install --clone --upgrade --skip-request-license \ --primarymac PRIMARY-MAC ``` For the HA case, `--secondarymac` is added in the same way as previously. The remaining stages, including the cleanup and the confirmation to shut down the spare node, then run as described previously. The cloned head node runs without a license until one is installed on it. A license can be requested and installed separately, using the procedure for a cluster without WWW access. That procedure is described under *Requesting A License With The `request-license` Script* in the *Installation Manual*. * Unplug the spare node from the production setup/networks and plug it in the parallel setup. * In the case of `/cm/shared` on a file server (i.e. usually on an HA setup), login to the file server and make a copy of the `/cm/shared` to a new directory. Alternatively, the copy can be made to a new file server. Have the directory name and the (possibly new) file server host name ready. * Boot the spare (a.k.a. cloned primary head) node as it is plugged in the parallel (isolated) setup. The node will boot with all of its network interfaces remaining DOWN. * Login on console (ignoring possible messages about not-found modules in `/cm/shared`), and run /root/cm/update-cloned-db.py, providing the new location of `/cm/shared`: ```bash # /root/cm/update-cloned-db.py --new-cm-shared NEW-CM-SHARED-DIRECTORY-NAME \ --nas-server NAS-SERVER ``` which will setup in CMDaemon to mount the copy of `/cm/shared` made in the preceding session from NAS-SERVER:NEW-CM-SHARED-DIRECTORY-NAME or, if `/cm/shared` is not on a NAS server (i.e. no copy of `/cm/shared` was made in the preceding session), then just run: ```bash # /root/cm/update-cloned-db.py ``` The script finds values for MAC addresses, interfaces, and so on that came as part of the original primary head node, and converts those values to ones suitable for the clone. * After /root/cm/update-cloned-db.py is completed, on the cloned primary head node still logged in on the console, by using "cmsh -r localhost", change the IPs of the primary head node (master) on the external network(s). * In the case of HA: change also the shared IP (alias) of the secondary head node on the external network(s) to match the new shared IP(s) of the primary head node. * If necessary, update also other settings such as power management settings or if desired fsmounts, to reflect that in this parallel setup the primary head node is a different physical machine, which is a clone of the original primary head node. * Reboot, to allow for the settings to take effect. After the reboot, the (cloned) primary head node in the parallel setup will have the network interfaces UP. * Ensure that this primary head node in the parallel setup has correctly mounted the copied `/cm/shared` from the file server, if a copy of `/cm/shared` was made in the preceding. If `/cm/shared` is not correctly mounted, then verify the fsmount settings for the primary head node, the secondary head node, as well as for the compute nodes (typically the compute nodes fsmount settings are set in the compute nodes categories) are correct and the head node is able to connect to the file server. * (HA) Plug in the secondary head node to the parallel setup and boot it. After the secondary head node has booted, ensure it has mounted correctly the `/cm/shared` copy from the file server. * (HA) On both the (cloned) primary head node and the secondary head node, ensure the active head node is the (cloned) primary head node, the failover is in a good state and the mysql replication is working by running (on both head nodes): ```bash # cmha status ``` If any of the returned responses for `failoverping`, `mysql`, `ping`, or `status` do not report "OK", then ensure the network configuration and cabling allow the two head nodes in the parallel setup to communicate, and that they are isolated from the production setup. If this does not resolve the issue, then please contact support ([https://enterprise-support.nvidia.com/s/create-case](https://enterprise-support.nvidia.com/s/create-case)) for further assistance. * Disable/remove the Bright Cluster Manager integrations, such as with Kubernetes or WLM for some versions of Bright, as described in the important notes and other prerequisites in the preceding. The notes in sections [2.3](#23-important-considerations-for-the-parallel-upgrade-setup)–[2.8](#28-upgrading-using-a-bcm-dvdiso) should now be considered, before continuing on with the upgrade as described in section [2.9](#29-enable-upgrade). ## 2.3 Important: Considerations For The Parallel Upgrade Setup ### 2.3.1 Configuration Changes And Services On The Production Cluster Desynchronized The parallel upgrade procedure involves creating a clone of the primary head node and its isolation from the production cluster. The administrator must be aware that this means that any further configuration changes on the production cluster, further collected monitoring data, other changes in settings or data, such as the running or new WLM jobs’ state data, that is usually saved on the head node or in `/cm/shared` will not be mirrored or synced to the parallel setup after creating the clone of the primary head node, resp. the copy of the `/cm/shared` directory tree. In just the same way, services running on the original primary head node will also be started by the operating system on the cloned head node, but they will not be able to communicate with the production cluster nodes. As an example, if the cluster is running with Ceph integration with only one monitor running on the head node, then the Ceph monitor on the production head node will continue to communicate with the Ceph OSDs nodes and maintain the Ceph cluster state. In the parallel setup, the Ceph monitor service on the cloned head node cannot communicate with any node from the production cluster and will not be aware of the Ceph cluster state changes. Therefore some special services running on the parallel-setup head node may not be able to take over from the respective services running on the production head node when the old production head node is to be shut down and the upgraded parallel-setup head node is to become the production head node. In these special cases, the administrator needs to take the necessary steps to migrate the services, or the administrator needs to perform an in-place upgrade. ### 2.3.2 Local Data On The Head Node Desynchronized When the users’ home directories (`/home`) or other data are not on a file server, then the administrator will most commonly need to take the necessary steps to sync the latest changes in the users’ home, resp. other directories from the production setup to the parallel setup when the upgrade of the parallel setup is complete. ### 2.3.3 Dangling, Temporary, And Relocated Mounted Network File Systems If the users’ home directories are on a file server, then a manual sync of the data as pointed in the preceding is unnecessary, however the administrator must be aware that except for `/cm/shared` as noted below, the head node(s) in the parallel setup will retain their fsmount settings also during the upgrade. This means that also during the upgrade the head nodes from the parallel setup will attempt to mount the network file systems. If this behavior is not desired, the administrator can configure the parallel-upgrade network and cabling in such a way so that the file server(s) are also isolated from the parallel setup. Alternatively, the administrator can modify the fsmount settings after the primary head node is cloned when it is booted for the first time as described below, so that the head nodes in the parallel setup do not mount these network file systems. On the other hand, if mounting the network file systems on the head nodes in the parallel setup is required, for example so that users can log in and test the upgraded setup while the production setup is still operational in parallel, then the network configuration/cabling should allow the head node to access the file servers. When the file servers are on an external network, usually no additional steps will be required since the head nodes in the parallel setup will have full access to the external networks. If the file servers are on the internal network(s) however, because by default the cloned head node retains the same internal network IPs, and because the parallel setup’s internal networks should be isolated from the production setup, by default the parallel-setup’s head nodes cannot access the file servers or any other device on the production-cluster’s internal network(s). Additional measures such as setting up intermediate servers or re-configuring the network settings will have to be taken to ensure the parallel-setup’s nodes can access the data on the file servers when needed, or to make sure they have access to a copy of the data on different (possibly temporary) file servers. Such configuration changes are beyond the scope of this document, for further information please contact support ([https://enterprise-support.nvidia.com/s/create-case](https://enterprise-support.nvidia.com/s/create-case)) for further assistance. A special case is `/cm/shared` mounted from a file server. The clone and upgrade scripts will detect that, and the administrator will be required to copy `/cm/shared` to a new location and provide the (new) path and file server host name. The new location can be a new directory on the same file server if the file server is on an external network. If the file server is on the internal network, the simplest solution is to set up a new or temporary file server to host the required for the parallel upgrade copy of `/cm/shared`. ### 2.3.4 Cluster Extension Upgrade Not Supported Parallel upgrade of clusters extended to the cloud (Cluster Extension cloudbursting) is not supported. For these clusters only an in-place upgrade is possible, which involves the termination of all cloud nodes when the cluster extension is removed as part of the upgrade procedure. ### 2.3.5 Edge Setup Upgrade Not Supported Parallel upgrade of clusters with an edge setup is not supported. ### 2.3.6 Prerequisites * The cluster must have a previously installed and active subscription license. * Sites with hardware life licenses should contact support ([https://enterprise-support.nvidia.com/s/create-case](https://enterprise-support.nvidia.com/s/create-case)) for further assistance. * Before starting with the upgrade, the subscription license should first be unlocked via the self-service customer portal at [https://customer.brightcomputing.com/Customer-Portal?p=unlock](https://customer.brightcomputing.com/Customer-Portal?p=unlock) * A spare node is required for cloning the original (a.k.a production) primary head node. The spare node must have comparable to the current head node hardware, such as number of network cards, number of disks and size, memory, etc. After the completion of the upgrade, the spare node will become the new primary head node for the upgraded cluster. * In the case of head node HA, the secondary head node will need to be taken out of the production cluster so that it can also be upgraded. This means that for the time needed to perform the parallel upgrade, until all compute nodes are migrated to the upgraded setup, the old production cluster will run with only one head node. * If `/cm/shared` will be unmounted during the upgrade then please make sure that the contents of the local `/cm/shared` are in sync with the remote copy. * In the case of `/cm/shared` on a file server (such as in the case of head node HA), the file server must have enough space for creating a copy of the `/cm/shared` directory tree. Alternatively, `/cm/shared` can be copied to a new file server. * While performing the upgrade, the cloned primary and—in the case of HA—the secondary head node(s) need to have isolated internal network(s) which do not clash with the internal network(s) of the production cluster. The IP(s) of the upgraded head node(s) on the internal network(s) will remain the same. After the upgrade is completed, the old (not upgraded) primary head must be shut down and the upgraded head node(s) can be re-connected back in the cluster. Alternatively, compute nodes can be disconnected from the old (not upgraded) setup and then connected in the upgraded setup, which can also be performed gradually. * There is no requirement to isolate the external network(s) for the head node(s) while performing the upgrade, which means the cloned head node can still have access to the Internet. However, the cloned primary head node will require a new set of IPs on the external network(s), which are different from the current IP(s) of the primary head node on the external network(s). It is also recommended to use static IPs, and not DHCP with dynamic DNS updates on the external network, because both the original production head node and its clone in the parallel setup will have the same hostname as defined in CMDaemon. The secondary head node does not require a new set of IPs because it is simply moved (and not cloned) from the production to the parallel setup. * When upgrading the parallel setup, the head nodes require access to the new BCM package repositories. Alternatively, the BCM ISOs can be used as a source for the packages. Access to the base distro package repositories, using for example, the via internet or local repositories, is also required. Updates using the old Bright package repositories should be carried out only to bring the cluster up to date before running `cm-upgrade`. * In the case of HA, the cloned primary and the secondary head nodes will require to be able to connect to the file server for their copied `/cm/shared` directory tree. This means that if the file server is on an internal network, then—because the internal networks must be isolated—a new file server must be setup and made available on the isolated internal network. ## 2.4 Important: Upgrading Particular Applications This section has important notes on upgrading particular applications. ### 2.4.1 Workload Managers (WLM) Torque, SGE and PBSPro CE workload managers are not supported in BCM 10, and hence the relevant packages will be removed as part of the upgrade. Unversioned WLM packages (such as `pbspro`/`pbspro-client`/`pbspro-ce`/`pbspro-ce-client` or `slurm`/`slurm-client`) will be removed and the administrator can install a versioned WLM package of their choice (such as `pbspro2022`/`pbspro2022-client` or `slurm22.05`/`slurm22.05-client`). On RHEL and SLES distributions, unsupported versioned WLM packages are not removed automatically. They must be removed manually and where relevant new versioned WLM packages must be installed. #### 2.4.1.1 Upgrading WLMs In General From Bright 9.2 Workload manager configurations must be disabled using `cm-wlm-setup` before proceeding with the upgrade. After the upgrade has been completed, workload management must be re-deployed using `cm-wlm-setup`. #### 2.4.1.2 Slurm Upgrades Slurm versions older than Slurm 23.02 are not supported in BCM 10. When upgrading from Bright 9.2, Slurm must be upgraded to at least Slurm 23.02 from the Bright 9.2 repositories before moving to BCM 10. Upgrading Slurm on a BCM 10 cluster is described in [section 7.5.1.2 of the *Administrator Manual*](/admin/workloadmanagement#7512-updating-from-earlier-slurm-versions-to-slurm2311). #### 2.4.1.3 Slurm DBD Performance Improvement Starting with the BCM 10.24.05 release, some MySQL configuration settings have been changed to improve the performance of the slurmdbd service. The changes are not automatically applied during the cluster upgrade. The MySQL configuration must therefore be manually adjusted on the head nodes, after `cm-upgrade` has completed. The `/etc/my.cnf` on the head node, and also on the secondary head node in an HA-enabled cluster, should have its settings updated as follows: ```bash innodb_buffer_pool_size=4096M innodb_log_file_size=64M innodb_lock_wait_timeout=900 ``` #### 2.4.1.4 PBS Pro Upgrades PBS Pro versions older than PBS Pro 2022 are not supported in BCM 10. Below is an overview of the steps that can be taken to upgrade PBS Pro: * Stop the `pbsserver` service on the head node ```bash cmsh% device cmsh% foreach -l pbsproserver ( services; stop pbsserver ) ``` * Remove the old packages from the headnodes ```bash $ yum/zypper/apt remove pbspro* ``` * Remove the old packages from the software images ```bash $ cm-chroot-sw-img /cm/images/ ... [root@ /]# yum/zypper/apt remove pbspro* ``` * Install the new packages in the headnode ```bash $ yum/zypper/apt install pbspro pbspro-client ``` * Install the new packages in the software images ```bash $ cm-chroot-sw-img /cm/images/ ... ... [root@ /]# yum/zypper/apt install pbspro-client ``` * Change the version of the WLM in the `wlm` object ```bash cmsh% wlm use pbspro cmsh% set version cmsh% commit ``` * Start `pbsserver` ```bash cmsh% device cmsh% foreach -l pbsproserver ( services; start pbsserver ) ``` ### 2.4.2 Older OpenMPI versions 1.x Unsupported The 1.x series of OpenMPI is not supported in BCM 10. These packages are removed during the upgrade. Administrators must install OpenMPI 4 packages manually, and rebuild the applications that use them after the upgrade has been completed. ### 2.4.3 Kubernetes Package Upgrades Supported Kubernetes deployments must be removed before upgrading and re-deployed after the upgrade. * Remove the old packages from the headnodes ```bash $ yum/zypper/apt remove cm-kubernetes* cm-etcd cm-kube-diagnose cm-containerd ``` * Remove the old packages from the software images ```bash $ cm-chroot-sw-img /cm/images/ ... [root@ /]# yum/zypper/apt remove cm-kubernetes* cm-etcd \ cm-kube-diagnose cm-containerd ``` For more details please refer to chapter 4 of the BCM 10 Containerization Manual. ### 2.4.4 JupyterHub Database Upgrade The JupyterHub database schema must be manually upgraded in each JupyterHub node after the main package upgrade has completed. The following commands can be run to upgrade the JupyterHub database: ```bash module load jupyter jupyterhub upgrade-db--config=/cm/local/apps/jupyter/conf/jupyterhub_config.py \ --db sqlite:////cm/local/apps/jupyter/run/jupyterhub.sqlite ``` ### 2.4.5 Docker Registry/Harbor Package Upgrades When upgrading a cluster using the BCM ISO, if the Docker registry or Harbor packages were previously installed, then they do not get upgraded automatically on RPM-based distributions. On Ubuntu based clusters the packages get removed automatically. The packages must be manually installed and upgraded using the package managers (`apt`/`zypper`/`yum`). If Docker registry was installed, then the `cm-docker-registry` package must be upgraded/installed. If Harbor was installed, then the packages to upgrade/install are: `cm-harbor` `cm-docker` `cm-docker-compose` Also note that if they were deployed on a different node than the head node, then the packages must be upgraded/installed in the relevant software image(s). Harbor versions older than 2.8 are not supported in BCM 10. When upgrading from Bright clusters running an older Harbor version, remove it with `cm-container-registry-setup` and re-deploy it after the upgrade. ### 2.4.6 OpenStack Integration * Bright OpenStack deployments must be removed (using cm-openstack-setup) before upgrading. OpenStack integration is no longer supported in BCM 10, and cannot be re-deployed after the upgrade. All older Bright OpenStack packages and dependencies must be removed prior to starting the upgrade. Please contact support ([https://enterprise-support.nvidia.com/s/create-case](https://enterprise-support.nvidia.com/s/create-case)) for further assistance. ## 2.5 Important: Package Upgrade Dependency Issues The upgrade process will not only upgrade CMDaemon and its dependencies, but it will also upgrade other packages. This means that old packages will not be available from the new BCM repositories. In some cases, this will require recompiling the user applications to use the upgraded versions of the compilers and the libraries. Also, the configurations of the old packages will not be copied automatically to the new packages, which means that the administrator will have to adjust the configuration from the old packages to suit the new packages manually. Extra base distribution packages may be installed by `yum`/`zypper`/`apt-get` in order to resolve dependencies that might arise as a result of the upgrade. Hence the base distribution repositories must be reachable. This means that the clusters that run the Enterprise Linux distributions (RHEL and SLES) must be subscribed to the appropriate software channels. Packages in `/cm/shared` are upgraded, but the administrator should be aware of the following: * If `/cm/shared` is installed in the local partition, then the packages are upgraded. This may not be desirable for users that wish to retain the old behavior. * If `/cm/shared` is mounted from a separate partition, then unmounting it will prevent upgrades to the mounted partition, but will allow new packages to be installed in `/cm/shared` within the local partition. This may be desirable for the administrator, who can later copy over updates from the local `/cm/shared` to the remote `/cm/shared` manually according to site-specific requirements. Since unmounting of mounted `/cm/shared` is carried out by default, a local `/cm/shared` will have files from any packages installed there upgraded. According to the yum database, the system is then upgraded even though the files are misplaced in the local partition. However, the newer packages can only be expected to work properly if their associated files are copied over from the local partition to the remote partition. ## 2.6 Important: Upgrading Ubuntu Clusters After upgrading the cluster, the Ubuntu compute nodes must initially be re-provisioned with a `FULL` install. To achieve that, the following `cmsh` example command will update each Ubuntu compute node. In this example, 750 nodes are updated: ```bash % device foreach -n node001..node750 (set nextinstallmode FULL); device commit ``` Important note: a `FULL` install will remove any local data on the compute nodes. Thus the administrator must perform the necessary steps to back up the data on the compute nodes before the upgrade and restore it afterward. ## 2.7 Known Issues * On clusters with Jupyter deployed, after the upgrade, users might find themselves being asked to login twice. The first time when the user will be asked to login to JupyterHub, and a second time, where the user is asked about a "CMDaemon login". In such cases, the user should once again present the credentials. This extra login request is harmless, and after the credentials are given a second time, it is unlikely that users will be queried again. The cause of this issue is that the Jupyterhub component that interacts with CMDaemon can lose connection to CMDaemon, causing it to prompt for an extra login. * On some base distributions, depending on the version of the installed packages, when starting `cm-upgrade`, a YAMLLoadWarning message may be printed when its configuration files are loaded. The warning message can be safely ignored. * In some cases, in particular HA setup with a shared storage, it is possible that the `slurmdbd.conf.template` configuration file is overwritten after the upgrade where the package manager would not preserve a backup copy of the locally modified file. As a result, when Slurm WLM is later setup with `cm-wlm-setup` CMDaemon will not start slurmctld service because it will detect slurmdbd service is not able to connect to the mysql database. To resolve the issue, the administrator needs to run `/cm/shared/apps/slurm/var/cm/cm-restore-db-password` to set up a new MySQL password for the Slurm user, and should then restart the `slurmdbd` service. * In some cases because `/cm/shared` may be temporarily not accessible from the cloned primary head node when it boots for the first time, if workload managers are setup on the cluster, then CMDaemon may create and leave module files called `no-version`. For example:\ `slurm/slurm/no-version`\ To resolve the issue, the leftover `no-version` module files under `/cm/shared/modulefiles/` can be deleted. * In some cases at the end of the head node or the software image(s) upgrade, the `cm-upgrade` script may report a list of packages that were not upgraded to their BCM 10 versions. This can occur in cases where the package manager was not able to resolve the list of packages to find a suitable candidate at the time of the upgrade of all packages. An example here the OpenBLAS packages upgrades for BCM. In BCM prior to BCM 10, the package was called `openblas`, packaged by BCM developers. In BCM 10 the BCM package was replaced by `cm-openblas` (also packaged by BCM developers). The issue occurs since the base distro also can provide a (non-BCM) OpenBLAS package. If `cm-upgrade` reports that the BCM OpenBLAS package is not upgraded, then the administrator is advised to swap the OpenBLAS package manually with the `cm-openblas` package. While `cm-upgrade` makes a best effort to swap or upgrade the packages to their BCM 10 versions, in some cases - notably on Ubuntu, it is also possible some packages such as `cmburn`, `hdf518`, or `hpl` are left in their 9.2 versions. The administrator is advised to manually swap the `cm9.2` packages (for example `cmburn-9.2-392_cm9.2`) with the `cm10.0` packages (for example `cmburn_10.0-391_cm10.0`). * On clusters with cluster extension to AWS deployed, the cloud director and cloud nodes might fail to start after the upgrade. This can happen when there is an entry ```bash /dev/nvme1n1 ``` in the disksetup. The issue can be resolved by removing that entry from the disksetup for the cloud director and cloud nodes. * On clusters that have PBSPro configured, and upgraded from 9.2 to BCM 10, new jobs submitted by users will get job ids that were already used. This is particularly relevant for users that use BCM monitoring data for billing purposes. * Disabling Kubernetes as part of the parallel-upgrade procedure results in an apparent failure during the stage when etcd data is cleaned up, because the compute nodes are not reachable from the parallel setup. This is expected since the parallel setup is isolated from the production setup. To resolve the issue, the administrator should answer a prompt so that the failed stage is skipped. It is also recommended to re-provision the compute nodes of the upgraded parallel setup before attempting to set up Kubernetes or etcd again, so that any leftover data from before the upgrade is cleaned during the provisioning. ## 2.8 Upgrading Using A BCM DVD/ISO When using a BCM DVD/ISO to perform the upgrade, it is important to use a DVD/ISO that is not older than **10.24.01**. The DVD/ISO version can be found (assuming that the DVD/ISO is mounted under `/mnt/cdrom`) with a `find` command such as: ```bash # find /mnt/cdrom -type d -name '10.24.01' /mnt/cdrom/data/packages/10.24.01 ``` ## 2.9 Enable Upgrade ### 2.9.1 Enable The Upgrade Repo, And Install The Upgrade Package Enabling the upgrade repository and installing the upgrade package can be carried out for the various distributions as follows: **RHEL8 derivatives:** ```bash # yum-config-manager \ --add-repo http://support.brightcomputing.com/upgrade/10.0/rhel/8/updates # yum --nogpgcheck install cm-upgrade ``` **RHEL9 derivatives:** ```bash # yum-config-manager \ --add-repo http://support.brightcomputing.com/upgrade/10.0/rhel/9/updates # yum --nogpgcheck install cm-upgrade ``` **SLES 15:** ```bash # zypper addrepo \ http://support.brightcomputing.com/upgrade/10.0/sles/15/updates cm-upgrade-10 # zypper install cm-upgrade ``` **Ubuntu 20.04:** ```bash # cat < /etc/apt/sources.list.d/cm-upgrade-10.list deb [trusted=yes] https://support.brightcomputing.com/upgrade/10.0/ubuntu/2004/ ./ EOF # apt update # apt install cm-upgrade ``` **Ubuntu 22.04:** ```bash # cat < /etc/apt/sources.list.d/cm-upgrade-10.list deb [trusted=yes] https://support.brightcomputing.com/upgrade/10.0/ubuntu/2204/ ./ EOF # apt update # apt install cm-upgrade ``` ### 2.9.2 Load The Environment Module ```bash # module load cm-upgrade/10.0 ``` ## 2.10 Perform Upgrade The steps for carrying out the upgrade are as follows: ### 2.10.1 Apply Updates To Head Node * RHEL derivatives: ```bash # yum update ``` * SLES derivatives: ```bash # zypper up ``` * Ubuntu: ```bash # apt-get update # apt-get upgrade --with-new-pkgs ``` ### 2.10.2 Apply Updates To Software Images Before updating and upgrading a software image, it may be worth creating an image backup by cloning the image ([here](/admin/configuring#32022-making-some-nodes-function-differently-by-image)). If the cluster after the upgrade has an image that has an issue, then comparing it against the backup can help troubleshoot the issue. For each software image, updates can be applied as follows: * RHEL derivatives: ```bash # yum --installroot /cm/images/ update ``` * SLES derivatives: ```bash # zypper --root /cm/images/ up ``` * Ubuntu: ```bash # cm-chroot-sw-img /cm/images/ # apt-get update # apt-get upgrade --with-new-pkgs # exit ``` ### 2.10.3 Apply Updates To The `node-installer` Image * RHEL derivatives: ```bash # yum --installroot /cm/node-installer update ``` * SLES derivatives: ```bash # zypper --root /cm/node-installer up ``` * Ubuntu: ```bash # cm-chroot-sw-img /cm/node-installer # apt-get update # apt-get upgrade --with-new-pkgs # exit ``` ### 2.10.4 Upgrade Head Nodes To BCM 10 Important: this must be run on both head nodes in a high availability setup. Recommended: Upgrade the active head node first and then the passive head node. Depending on availability, the upgrade can be run either over the network, or using a DVD/ISO: * Upgrade using repositories accessible over the network ```bash # cm-upgrade ``` * Upgrade using a BCM DVD/ISO ```bash # cm-upgrade -b /root/bcm-10.0-rocky8u8.iso ``` In an HA setup, after upgrading both the head nodes, resync the databases. Run the following from the active head node (it is very important to complete this step before moving to the next one): ```bash # cmha dbreclone ``` ### 2.10.5 Reboot Head Node(s) The head node(s) must be rebooted before proceeding to run the post-upgrade actions. ### 2.10.6 Post-upgrade Head Node Actions Important: these must be run on both head nodes in an HA setup ```bash # module load cm-upgrade/10 # cm-post-upgrade -m ``` ### 2.10.7 Upgrade The Software Image(s) To BCM 10 Important: the upgrade to the software images must be done only on the active head node. Depending on availability, the upgrade can be run either over the network, or using a DVD/ISO: * Upgrade using repositories accessible over the network ```bash # cm-upgrade -i all ``` * Upgrade using a BCM DVD/ISO ```bash # cm-upgrade -i all -b /root/bcm-10.0-rocky8u8.iso ``` If the software images are not under the standard location—`/cm/images/` on the head node—then the option `-a` should be used: ```bash # cm-upgrade -a /apps/images -i # cm-upgrade -a /apps/images -i -b /root/bcm-10.0-rocky8u8.iso ``` If the software images are located under multiple locations, the target locations can be specified as a comma-separated list: ```bash # cm-upgrade -a '/apps/images,/opt/images' -i all ``` #### 2.10.7.1 Btrfs Image Upgrades If the software image is a Btrfs image, then the revision ([section 16.4 of the *Administrator Manual*](/admin/administration#164-revision-control-for-images)) images cannot be used, because they use read-only filesystems. This issue extends to all references in category or node images, such as @latest and @*number* Only the original, unrevisioned, parent image can be upgraded with the `cm-upgrade` and `cm-post-upgrade` steps with a Btrfs filesystem. A workaround would be to create a new non-CoW image from a trusted revision, and upgrading that trusted non-CoW version. ### 2.10.8 Upgrade /cm/node-installer To BCM 10 This needs to be performed only on the primary active head node, either over the network, or using a BCM DVD/ISO. * Upgrade using repositories accessible over the network: ```bash # cm-upgrade -x ``` * Upgrade using a BCM DVD/ISO: ```bash # cm-upgrade -x -b /root/bcm-10.0-rocky8u8.iso ``` ### 2.10.9 Post-upgrade Software Images Actions Important: these must be run only on the active head node. ```bash # cm-post-upgrade -i all ``` If the software images are not under the standard location, which is `/cm/images/` on the head node, then the option `-a` should be used: ```bash # cm-post-upgrade -a /apps/images -i all # cm-post-upgrade -a /apps/images -i ``` If the software images are located under multiple locations, then the target locations can be specified as a comma-separated list: ```bash # cm-post-upgrade -a '/apps/images,/opt/images' -i all ``` ### 2.10.10 (HA) Allow For The Provisioning Requests To The Secondary Head Node To Complete Allow between 15 and 30 seconds for CMDaemon to schedule and start provisioning updates of `/tftpboot` and `/cm/node-installer` from the primary to the secondary head node. Then check and wait until the provisioning requests with the secondary head node as destination have completed using: ```bash # cmsh -c 'softwareimage provisioningstatus -r' ``` If the provisioning requests with a destination node the secondary head node continue to be listed for a prolonged period of time and do not disappear from the list, then the `node-installer` and/or `tftpboot` on the secondary head node are out of sync from the primary head node, which can result in issues with provisioning of the compute nodes. Please contact support ([https://enterprise-support.nvidia.com/s/create-case](https://enterprise-support.nvidia.com/s/create-case)) for further assistance. ### 2.10.11 The Cluster Is Now Upgraded To BCM 10 It is recommended to provision the compute nodes at least once with BCM 10 before setting up any new integrations. In the case of Ubuntu base distro, it is recommended to initially perform a `FULL` install/provisioning of the compute nodes as noted in the preceding.