> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # In-place Upgrades ## 3.1 Introduction In contrast with a parallel upgrade (Chapter [2](/upgrade/parallelupgrade)), with an in-place upgrade the production cluster is taken out of operation while the upgrade takes place. This chapter covers the procedure for an in-place upgrade. ## 3.2 Existing Cluster Health Check It is a good idea to check that the existing cluster is in a healthy state before carrying out the upgrade. The head nodes health checks in particular should be looked at. Unexpected health check failures should be examined, and if required, resolved. In some cases it is expected that health checks will be in a failed state. For example, if all the compute nodes are DOWN, then the schedulers health check fails. The reason behind checking the health of the cluster before upgrading is that the health checks can detect failing services, overfull filesystems, unmounted filesystems, networking issues, time synchronization issues, and other issues. If these are unexpected then they can prevent the upgrade from completing successfully. ## 3.3 Important: Upgrading Particular Applications ### 3.3.1 Overview Of The Applications That `cm-upgrade` Does And Does Not Manage #### 3.3.1.1 Automatically Upgraded By `cm-upgrade` The following software, if managed by BCM 10, is managed and upgraded automatically by `cm-upgrade` to BCM 11: * Docker * The `cm-docker-registry` package (Harbor upgrades (section [3.3.6](#336-upgrading-the-docker-registry-and-harbor-packages)) are not supported) * Jupyter (the JupyterHub database requires some additional care (section [3.3.5](#335-upgrading-the-jupyterhub-database))). * Base View: The regular version of Base View in BCM 10 is upgraded to a new version in BCM 11. Any monitoring dashboards from the regular BCM 10 Base View version are no longer used in the BCM 11 Base View version (section [3.3.2](#332-upgrading-base-view)). * NVIDIA drivers: The defaults will be installed in non-DGX images #### 3.3.1.2 Automatically Kept As Is By `cm-upgrade` The following applications are allowed to remain during the upgrade, and are not automaticaly upgraded: * WLMs (section [3.3.3](#333-upgrading-workload-managers-wlm)) * Slurm * LSF The installation of workload managers is described in [section 7.3 of the *Administrator Manual*](/admin/workloadmanagement#73-installation-of-workload-managers), while the upgrade of the Slurm workload manager is described in [section 7.5 of the *Administrator Manual*](/admin/workloadmanagement#75-configuring-and-running-individual-workload-managers). * Kubernetes: Kubernetes installation is described in [section 4.22 of the *Containerization Manual*](/containerization/kubernetes#422-kubernetes-upgrades). * Run:Ai: Run:Ai installation options for BCM are described in [section 6.11 of the *Containerization Manual*](/containerization/kubernetesoperators#611-the-runai-operator). * NVIDIA Container Toolkit: An NVIDIA Container Toolkit installation can be carried out as part of Docker setup or a package installation ([section 2.7 of the *Containerization Manual*](/containerization/dockerengine#27-docker-setup-for-nvidia)). * other Kubernetes operators: The installation of other Kubernetes operators is described in [Chapter 6 of the *Containerization Manual*](/containerization/kubernetesoperators). #### 3.3.1.3 Manually Removed And Reinstalled During Upgrade (Not Managed By `cm-upgrade`) * Harbor: As explained earlier, Harbor upgrades (section [3.3.6](#336-upgrading-the-docker-registry-and-harbor-packages)) are not supported. The Harbor package needs to be removed from BCM version 10, and redeployed ([section 3.2 of the *Containerization Manual*](/containerization/dockerregistries#32-container-registries-setup-and-configuration)) in BCM version 11. * Cluster Extension: Needs to be removed and redeployed ([Chapter 3 of the *Cloudbursting Manual*](/cloudbursting/cexcloud)) #### 3.3.1.4 Removed From Support During Upgrade (Not Managed By `cm-upgrade`) The following software is no longer managed by BCM in BCM version 11. * [(Altair) Grid Engine (AGE): BCM support for integration ended.](https://support.brightcomputing.com/manuals/10/admin-manual.pdf#chapter.7) Possible alternatives to the AGE WLM, that BCM supports, are Slurm and LSF. * [PBS Professional and OpenPBS: BCM support for integration ended.](https://support.brightcomputing.com/manuals/10/admin-manual.pdf#chapter.7) Possible alternatives, that BCM supports, are Slurm and LSF. * [Ceph: BCM support for integration ended.](https://support.brightcomputing.com/manuals/10/admin-manual.pdf#chapter.9) * [BeeGFS: BCM support for integration ended.](https://support.brightcomputing.com/manuals/10/admin-manual.pdf#chapter.10) * Apptainer/Singularity: the `cm-apptainer` package is no longer built or provided. The upstream Apptainer packages can be used instead. * `cmjob`: BCM support ended. * Machine learning packages (RPMs and DEBs): no longer provided. * GigaIO: BCM support for integration ended. * Direct attached storage for HA setups: no longer supported. * OpenShift: BCM support for integration ended. ### 3.3.2 Upgrading Base View BCM has Base View ([section 2.4 of the *Administrator Manual*](/admin/clustermanagement#24-base-view-gui)) as the browser GUI to carry out cluster management tasks. It is available as the package `base-view`. Base View in BCM 10 had an additional, experimental and undocumented, Base View NG, available as the package `base-view-ng`. Base View NG uses a newer GUI toolkit from that of Base View in BCM 10. Base View in BCM 11 is actually just a development of Base View NG from BCM 10, and so it uses that newer toolkit too. The new Base View in BCM 11 means that an upgrade from BCM 10 to BCM 11 loses all the dashboards that users and administrators may have built for Base View in BCM 10. It is therefore a good idea to make notes on the useful dashboards of BCM 10 so that analogous ones can be constructed for BCM 11. A useful transition may be to construct the analogues in Base View NG while still on BCM 10, before upgrading to BCM 11. ### 3.3.3 Upgrading Workload Managers (WLM) Unsupported unversioned or unsupported versioned WLM packages are not removed automatically. They must be removed manually. If they are to be updated then it is recommended to install new, supported, versioned WLM packages. The supported PBSPro versions are compatible across BCM 10 and BCM 11. #### 3.3.3.1 Upgrading WLMs In General From BCM 10 WLMs are not required to be disabled using `cm-wlm-setup` before upgrading the BCM version from 10 to 11. #### 3.3.3.2 Upgrading Slurm Slurm versions older than Slurm 24.11 are not supported in BCM 11. When upgrading from BCM 10, Slurm must be upgraded to at least Slurm 24.11 in the BCM 10 repositories before moving to BCM 11. Upgrading Slurm on a BCM 10 cluster is described in [section 7.5 of the BCM 10 Administration Manual.](https://docs.nvidia.com/base-command-manager/manuals/10/admin-manual.pdf#subsection.7.5.1) **Packages:** Software images now include the same packages as the head nodes. This is because starting with BCM 11, Slurm packages keep many things in `/cm/local`, instead of `/cm/shared`. Slurm client packages, for example `slurm25.05-client`, are no longer available. So they are removed as part of the upgrade procedure. There are now separate `slurmctld` and `slurmd` packages that provide the binary and systemd unit files for each service. **Path adjustments:** There are some path adjustments for Slurm for the upgrade from BCM 10 to BCM 11, as summarized in the following table.
**Slurm item** **Version** **Path**
Prefix directory BCM10 `/cm/shared/apps/slurm`
BCM11 `/cm/local/apps/slurm`
Most configuration files BCM10 `/cm/shared/apps/slurm/var/etc/`
BCM11 `/cm/shared/apps/slurm/etc`
Munge configuration files BCM10 `/cm/shared/apps/slurm/var/etc/munge`
BCM11 `/cm/shared/apps/slurm/var/etc/munge` (unchanged)
Binaries BCM10 `/cm/shared/apps/slurm/current/bin`
BCM11 `/cm/local/apps/slurm/current/bin`
Template files such as: `slurmdbd.conf.template` , `cgroup.conf.template` BCM10 `/cm/shared/apps/slurm/var/etc`
BCM11 `/cm/local/apps/slurm/current/templates`
`statesave` files BCM10 `/cm/shared/apps/slurm/var/cm/statesave`
BCM11 `/cm/shared/apps/slurm/statesave`
Power scripts BCM10 `/cm/shared/apps/slurm/var/cm/power`
BCM11 `/cm/local/apps/slurm/current/scripts/power`
Enroot prolog BCM10 `/cm/shared/apps/slurm/var/cm`
BCM11 `/cm/shared/apps/slurm/prologs/prolog-enroot.sh`
Enroot epilog BCM10 `/cm/shared/apps/slurm/var/cm`
BCM11 `/cm/shared/apps/slurm/epilogs/epilog-enroot.sh`
#### 3.3.3.3 Slurm DBD Performance Improvement Installations from BCM 10.24.05 onward have MySQL configuration settings changes that improve the performance of the slurmdbd service. The changes are not automatically applied during updates of MySQL when updating from versions installed earlier than BCM 10.24.05. To apply the improvement to a cluster that does not have this update, the MySQL configuration must be manually adjusted on the head nodes after `cm-upgrade` has completed. The `/etc/my.cnf` file on the head node, and also on the secondary head node in an HA-enabled cluster, should have the following settings updated: ```bash innodb_buffer_pool_size=4096M innodb_log_file_size=64M innodb_lock_wait_timeout=900 ``` The default `my.cnf` file is typically symlinked by the system utility `update-alternatives` in Ubuntu and SLES, (`alternatives` in RHEL derivatives), so how it is configured should also be considered. #### 3.3.3.4 Slurm Upgrades For DGX OS 6 To DGX OS 7 For DGX, BCM 10 supports only DGX OS 6 (an Ubuntu 22.04 variant), while BCM 11 supports DGX OS 7 (an Ubuntu 24.04 variant). Since BCM point release 11.34.0, BCM also supports DGX OS 8 (an Ubuntu 26.04 variant) for `x86_64`. As part of the upgrade from BCM 10 to BCM 11, the DGX OS must be replaced with DGX OS 7. When replacing the DGX OS nodes from 6 to 7, a new software image (and `/cm/shared`) is created for the Ubuntu 2404 nodes. If Slurm and Pyxis have been deployed in a BCM 10 DGX OS 6 image, then some manual configuration changes must be done to make these work right in the BCM 11 DGX OS 7 image. These can be carried out as follows: **Changes to the MUNGE key in `/cm/shared`:** The key needs to have the correct permissions and ownership. If it does not have these, then a symptom of it is the message: ```bash munged: Error: Keyfile is insecure: "/cm/shared/apps/slurm/var/munge/keys/ munge.key" should be owned by UID 1 instead of UID 0 ``` as seen in the systemd status output: ```bash root@dgx-04:~# systemctl status munge.service × munge.service - MUNGE authentication service Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Fri 2025-10-31 03:54:29 PDT; 26s ago Docs: man:munged(8) Process: 255424 ExecStart=/usr/sbin/munged --key-file=/cm/shared/ apps/slurm/var/munge/keys/munge.key (code=exited, status=1/ FAILURE)Oct 31 03:54:29 dgx-04 systemd[1]: Starting munge.service - MUNGE authentication service... Oct 31 03:54:29 dgx-04 munged[255424]: munged: Error: Keyfile is insecure: "/cm/shared/apps/slurm/var/munge/keys/munge.key" should be owned by UID 1 instead of UID 0 Oct 31 03:54:29 dgx-04 systemd[1]: munge.service: Control process exited, code=exited, status=1/FAILURE Oct 31 03:54:29 dgx-04 systemd[1]: munge.service: Failed with result 'exit-code'. Oct 31 03:54:29 dgx-04 systemd[1]: Failed to start munge.service - MUNGE authentication service. ``` The key can be made consistent with that of the previous deployment. This can be done by changing the attributes of the files in the new DGX software image, for example `dgx-os-7.2-h100-image`, after mounting it in the appropriate shared directory on the new cluster: **Example** ```bash cm-chroot-sw-img -s /cm/shared-ubuntu2404-x86_64 \ /cm/images/dgx-os-7.2-h100-image ... chown daemon:root /cm/shared/apps/slurm/var/munge/keys/munge.key chmod 400 /cm/shared/apps/slurm/var/munge/keys/munge.key ``` **Changes to the Pyxis and Enroot files:** The files within the DGX OS 7 image can be configured as follows: Copy the Pyxis plugin file to the new software image, for example: `dgx-os-7.2-h100-image`: **Example** ```bash cp /cm/local/apps/slurm/current/lib64/slurm/spank_pyxis.so \ /cm/images/dgx-os-7.2-h100-image/cm/local/apps/slurm/current/lib64/slurm/ ``` Copy the plugstack files in `/cm/shared` over from the Ubuntu 2204 (DGX OS 6) version to the Ubuntu 2404 (DGX OS 7) version: **Example** ```bash cp /cm/shared-ubuntu2204-x86_64/apps/slurm/etc/slurm/plugstack.conf \ /cm/shared-ubuntu2404-x86_64/apps/slurm/etc/slurm/ cp /cm/shared-ubuntu2204-x86_64/apps/slurm/etc/slurm/plugstack.conf.d/pyxis.conf \ /cm/shared-ubuntu2404-x86_64/apps/slurm/etc/slurm/plugstack.conf.d/ ``` Copy the Enroot configurations in `/cm/shared` from the Ubuntu 2204 version to the Ubuntu 2404 version: **Example** ```bash cp /cm/shared-ubuntu2204-x86_64/apps/slurm/etc/enroot.conf \ /cm/shared-ubuntu2404-x86_64/apps/slurm/etc/ cp /cm/shared-ubuntu2204-x86_64/apps/slurm/etc/enroot-sysctl.conf \ /cm/shared-ubuntu2404-x86_64/apps/slurm/etc/ ``` Symlinks for the epilog, prolog, and `enroot.conf` files are created for the new software image: **Example** ```bash cm-chroot-sw-img -s /cm/shared-ubuntu2404-x86_64 /cm/images/dgx-os-7.2-h100-image/ ln -s /cm/shared/apps/slurm/prologs/prolog-enroot.sh \ /cm/local/apps/slurm/var/prologs/50-prolog-enroot.sh ln -s /cm/shared/apps/slurm/epilogs/epilog-enroot.sh \ /cm/local/apps/slurm/var/epilogs/50-epilog-enroot.sh ln -s /cm/shared/apps/slurm/etc/enroot.conf /etc/enroot/enroot.conf ``` The compute nodes can pick up the new configuration when they get rebooted. ### 3.3.4 Upgrading The Kubernetes Package Kubernetes deployments do not need removal before upgrading to BCM 11. The same version of Kubernetes runs on the BCM cluster after the upgrade. An upgrade to Kubernetes ([section 4.22 of the *Containerization Manual*](/containerization/kubernetes#422-kubernetes-upgrades)) can be carried out after the BCM upgrade. #### 3.3.4.1 Kubernetes Package Upgrades For DGX Software Images When upgrading DGX OS nodes from 6 to 7.2, a new software image (and `/cm/shared`) needs to be created (section [3.8.13](#3813-enable-mixed-os-setup-and-create-a-dgx-os-7-image)), for the nodes which will be running Ubuntu 24.04. Kubernetes on the new DGX image needs to be manually copied over from the old image. This can be done as follows: * The `kubernetes.list` file should be copied over from the old image, \<*old DGX image*>, to the new image, \<*new DGX image*>, along with its associated keyring file: **Example** ```bash [root@basecm11 ~]# cp /cm/images//etc/apt/sources.list.d/\ kubernetes.list /cm/images//etc/apt/sources.list.d/ [root@basecm11 ~]# cp /cm/images//etc/apt/keyrings/\ kubernetes-apt-keyring.gpg /cm/images/ /etc/apt/keyrings/ ``` * The packages for the existing version need to be placed in the new image (here it is called `new-dgx-image`): ```bash [root@basecm11 ~]# cm-chroot-sw-img /cm/images/new-dgx-image ... [root@new-dgx-image /]# apt-get install cm-kube-diagnose kubeadm kubectl \ kubelet kubernetes-cni cm-containerd nvidia-container-toolkit nginx \ libnginx-mod-stream ... ``` * The packages for the existing version need to be held. The following commands take care of that: ```bash [root@new-dgx-image /]# apt-mark hold kubeadm [root@new-dgx-image /]# apt-mark hold kubectl [root@new-dgx-image /]# apt-mark hold kubelet [root@new-dgx-image /]# ``` The is to exit out of the `cm-chroot-sw-img` directory. The `cm-chroot-sw-img` command is described [here](/admin/softwareimagemanagement#9412-change-root-with-chroot-then-running-the-package-manager-commands). ### 3.3.5 Upgrading The JupyterHub Database The JupyterHub database schema must be manually upgraded in each JupyterHub node after the main package upgrade has completed. The following commands can be run to upgrade the JupyterHub database: ```bash module load jupyter jupyterhub upgrade-db--config=/cm/local/apps/jupyter/conf/jupyterhub_config.py \ --db sqlite:////cm/local/apps/jupyter/run/jupyterhub.sqlite ``` ### 3.3.6 Upgrading The Docker Registry And Harbor Packages Harbor cannot be upgraded when BCM 10 is upgraded to BCM 11, its package can only be (re-)installed and (re-)deployed separately from the BCM upgrade. If the Harbor packages were installed in BCM 10 to a node other than the head node, then a re-deployment to that node is required after the upgrade to BCM 11 for Harbor. When upgrading a cluster from BCM 10 to BCM 11 the upgrade of Docker registry is supported. ## 3.4 Important: Package Upgrade Dependency Issues The upgrade process will not only upgrade CMDaemon and its dependencies, but it will also upgrade other packages. This means that old packages will not be available from the new BCM repositories. In some cases, this will require recompiling the user applications to use the upgraded versions of the compilers and the libraries. Also, the configurations of the old packages will not be copied automatically to the new packages, which means that the administrator will have to adjust the configuration from the old packages to suit the new packages manually. Extra base distribution packages may be installed by `yum`/`zypper`/`apt-get` in order to resolve dependencies that might arise as a result of the upgrade. Hence the base distribution repositories must be reachable. This means that the clusters that run the Enterprise Linux distributions (RHEL and SLES) must be subscribed to the appropriate software channels. Packages in `/cm/shared` are upgraded, but the administrator should be aware of the following: * If `/cm/shared` is installed in the local partition, then the packages are upgraded. This may not be desirable for users that wish to retain the old behavior. * If `/cm/shared` is mounted from a separate partition, then unmounting it will prevent upgrades to the mounted partition, but will allow new packages to be installed in `/cm/shared` within the local partition. This may be desirable for the administrator, who can later copy over updates from the local `/cm/shared` to the remote `/cm/shared` manually according to site-specific requirements. Since unmounting of mounted `/cm/shared` is carried out by default, a local `/cm/shared` will have files from any packages installed there upgraded. According to the yum database, the system is then upgraded even though the files are misplaced in the local partition. However, the newer packages can only be expected to work properly if their associated files are copied over from the local partition to the remote partition. ## 3.5 Known Issues ### 3.5.1 Kubernetes PVC Access Issue With The Shared Directory For Multidistro (MultiOS) With a multidistro Kubernetes cluster, there can be a known issue with accessing `/cm/shared` across the various nodes. #### 3.5.1.1 Under What Conditions Is There An Issue? The issue can occur when all of the following conditions apply: 1. The Kubernetes cluster (and worker nodes in particular) are spread across multiple DGX OS versions, due to one of the following situations: * (A) the nodes are running as multidistro nodes within the same Kubernetes cluster (running permanently as multidistro as is the intention, and not just transiently as part of an upgrade). * (B) the nodes are being upgraded, for example from Ubuntu 22 to Ubuntu 24 2. [The *Local Path Provisioner*](https://github.com/rancher/local-path-provisioner/tree/master?tab=readme-ov-file#overview) is provisioned and deployed in the Kubernetes cluster. This is a component running in Kubernetes that enables pods in every Kubernetes node to consume some kind of local storage. The `local-path-provisioner` parameter provides the local path `StorageClass`, which sets up a sub-directory under `/cm/shared`. 3. In this Kubernetes cluster, there is at least one pod using the local-path StorageClass using a PVC (PersistentVolumeClaim) #### 3.5.1.2 Why Is There An Issue? The problem is that the `/cm/shared` directory is not unique across all nodes in a BCM cluster. The directory differs based on the DGX OS version. For example, on the head node, for the `/cm/shared` mount, you have: * for Ubuntu 22 nodes: `/cm/shared-ubuntu2204-x86_64` * for Ubuntu 24 nodes: `/cm/shared-ubuntu2404-x86_64` During normal running, each DGX node only sees its specific `/cm/shared`. Now, if a pod consuming a PVC from its local-path StorageClass is rescheduled to a node from the other DGX OS, then it cannot find the volume anymore, and it creates a new one. #### 3.5.1.3 Resolving The Issue For Persistent Mixed Distributions A solution in the case of persistent mixed distributions on the nodes is to move all the pods to a new common shared directory for Kubernetes, independent of the directories offered for the specific DGX OS versions. Steps: 1. If the cluster is not HA, then * a new `/cm/shared-kube` directory can be created * a new fsexport is created on the headnode: **Example** ```bash [head]% device use master; fsexports [head->device[head]->fsexports*]% add /cm/shared-kube@internalnet [head->devi...*[/cm/shared-kube@internalnet*]]% set path /cm/shared-kube [head->devi...*[/cm/shared-kube@internalnet*]]% set network internalnet [head->devi...*[/cm/shared-kube@internalnet*]]% set write yes [head->devi...*[/cm/shared-kube@internalnet*]]% commit [head->device[head]->fsexports[/cm/shared-kube@internalnet]]% ``` 2. If the cluster is HA, then * a new directory or export can be created in the NAS to store the Kubernetes directory * if the headnode is a Kubernetes workers, or if the head node is a Kubernetes control-plane node that uses volumes using pods, then volumes will be used. Create an fsmount on the head nodes with: **Example** ```bash [head->device[head]->fsmounts]% add /cm/shared/apps/kubernetes [head->...red/apps/kubernetes*]]% set device 10.180.81.165:/brdkube [head->..*]->fsmounts*[/cm/shared/apps/kubernetes*]]% set filesystem nfs [head->device*[head*]->fsmounts*[/cm/shared/apps/kubernetes*]]% commit ``` 3. Create an fsmount in the Kubernetes worker nodes categories (and/or nodes): **Example** ```bash [head->category[default]->fsmounts]% add /cm/shared/apps/kubernetes [head->...red/apps/kubernetes]]% set device $localnfsserver:/cm/shared-kube [head->...*]->fsmounts*[/cm/shared/apps/kubernetes*]]% set filesystem nfs [head->...*]->fsmounts*[/cm/shared/apps/kubernetes*]]% commit [head->category[default]->fsmounts[/cm/shared/apps/kubernetes]]% ``` 4. Turn off all the Kubernetes worker nodes using `cmsh` This ensures that all pods are down, and not writing to the PVCs 5. On the head node, populate the whole directory with all the data from Kubernetes: **Example** ```bash rsync -ar /cm/shared-ubuntu2204-x86_64/apps/kubernetes/ \ /cm/shared/apps/kubernetes/ ``` 6. If present, copy specific PVCs from Ubuntu 24.04 as well: **Example** ```bash rsync -ar /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes/\ /cm/shared/apps/kubernetes/default/var/volumes/ ``` 7. Turn on all the Kubernetes worker nodes 8. Ensure the Kubernetes worker nodes mount the new `/cm/shared/apps/kubernetes` directory **Example** ```bash mount | grep /cm/shared ``` 9. Ensure all the pods are up and running again: **Example** ```bash kubectl get pods -A ``` 10. Afterwards, clean up the old PVCs: **Example** ```bash rm -rf /cm/shared-ubuntu2204-x86_64/apps/kubernetes/default/var/volumes/ rm -rf /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes/ ``` #### 3.5.1.4 Resolving The Issue For The Upgrade Case Only A solution in the case of an image upgrade, for example from DGX OS 6 to DGX OS 7, can be carried out as follows: Before starting the upgrade of DGX OS version, you can move all PVCs from the old to the new `/cm/shared`. We assume old nodes are running Ubuntu 22 and that they need to be upgraded to Ubuntu 24. 1. Turn off all the (Ubuntu 22.04) Kubernetes worker nodes via `cmsh` This ensures that all pods are down and not writing to the PVCs. 2. Upgrade DGX OS to Ubuntu 24.04 3. On the head node, copy all the PVCs to the other new `/cm/shared` **Example** ```bash rsync -ar /cm/shared-ubuntu2204-x86_64/apps/kubernetes/default/var/volumes/ \ /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes/ ``` 4. Turn on all the Kubernetes worker nodes and check that all the pods are up and running again (`kubectl get pods -A`) 5. Afterwards, clean up the old PVCs: **Example** ```bash rm -rf /cm/shared-ubuntu2204-x86_64/apps/kubernetes/default/var/volumes/ ``` #### 3.5.1.5 Resolving The Issue If You Already Have Duplicated PVCs A solution in the case of PVCs that have already been duplicated requires some finesse. 1. Detect all PVCs in the old and new `/cm/shared` and mapping between them. First look for current PVCs and ensure each one has the related directory in the new shared directory: * `kubectl get pvc -A` * `ls /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes/` 2. Turn off the nodes 3. Backup or move the newly created PVCs, that most likely should be deleted: **Example** ```bash mv /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes \ /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes-backup ``` 4. Now, for each PVC, you have to find the original/correct one in the former shared directory. This is a manual process, and a bit of an art. You can consider: * The directory name which matches the Kubernetes namespace and PVC name * If there are 2 or more matching directories then: * look for the latest modification date (`ls -lah`) * check the overall size (`du -hs`). Usually the largest directory has more data and should be the most recent one 5. Run an `rsync` command per PVC: **Example** ```bash rsync -ar \ /cm/shared-ubuntu2204-x86_64/apps/kubernetes/default/var/volumes/\ pvc-98975be7-4030-4f11-8680-46a38829e2e7_runai-backend_data-runai-backend-\ redis-queue-master-0/ \ /cm/shared-ubuntu2404-x86_64/apps/kubernetes/default/var/volumes/\ pvc-98975be7-4030-4f11-8680-46a38829e2e7_runai-backend_data-runai-backend-\ redis-queue-master-0/ ``` Note: the source and target directories are different (`...-ubuntu2204-...` to `...-ubuntu2404-...`). This is because the source should match what you have found out in the previous step, while the target should match what you listed initially. 6. Turn on nodes and verify pods are up and running ### 3.5.2 Miscellaneous Known Issues * On clusters with Jupyter deployed, after the upgrade, users might find themselves being asked to login twice. The first time when the user will be asked to login to JupyterHub, and a second time, where the user is asked about a "CMDaemon login". In such cases, the user should once again present the credentials. This extra login request is harmless, and after the credentials are given a second time, it is unlikely that users will be queried again. The cause of this issue is that the Jupyterhub component that interacts with CMDaemon can lose connection to CMDaemon, causing it to prompt for an extra login. * On some base distributions, depending on the version of the installed packages, when starting `cm-upgrade`, a YAMLLoadWarning message may be printed when its configuration files are loaded. The warning message can be safely ignored. * In some cases because `/cm/shared` may be temporarily not accessible from the cloned primary head node when it boots for the first time, if workload managers are setup on the cluster, then CMDaemon may create and leave module files called `no-version`. For example:\ `slurm/slurm/no-version`\ To resolve the issue, the leftover `no-version` module files under `/cm/shared/modulefiles/` can be deleted. * In some cases at the end of the head node or the software image(s) upgrade, the `cm-upgrade` script may report a list of packages that were not upgraded to their BCM 11 versions. This can occur in cases where the package manager was not able to resolve the list of packages to find a suitable candidate at the time of the upgrade of all packages. While `cm-upgrade` makes a best effort to swap or upgrade the packages to their BCM 11 versions, in some cases— notably on Ubuntu, it is also possible some packages are left in their 10.0 versions. The administrator is advised to manually swap the `cm10.0` packages with the `cm11.0` package. * On clusters with cluster extension to AWS deployed, the cloud director and cloud nodes might fail to start after the upgrade. This can happen when there is an entry ```bash /dev/nvme1n1 ``` in the disksetup. The issue can be resolved by removing that entry from the disksetup for the cloud director and cloud nodes. * On clusters that have PBSPro configured, and upgraded from BCM 10 to BCM 11, new jobs submitted by users will get job ids that were already used. This is particularly relevant for users that use BCM monitoring data for billing purposes. * Disabling Kubernetes as part of the parallel-upgrade procedure results in an apparent failure during the stage when etcd data is cleaned up, because the compute nodes are not reachable from the parallel setup. This is expected since the parallel setup is isolated from the production setup. To resolve the issue, the administrator should answer a prompt so that the failed stage is skipped. It is also recommended to re-provision the compute nodes of the upgraded parallel setup before attempting to set up Kubernetes or etcd again, so that any leftover data from before the upgrade is cleaned during the provisioning. * When upgrading the DGX OS 6 sofware image to the latest updates of DGX OS 6, there is a known issue with DKMS when a new kernel `linux-image-5.15.0-1090-nvidia` is installed. The DKMS post-install script fails for the new kernel. This is caused by duplicate entries for the `gdrdrv` module. This can be checked for with the following command: ```bash root@dgx-os-6:/# dkms status gdrdrv/2.4.4, 5.15.0-1063-nvidia, amd64: installed gdrdrv/2.4.4, 5.15.0-1090-nvidia, amd64: installed gdrdrv/2.4.4, 5.15.0-1090-nvidia, x86_64: built ``` The last two statuses are for the same driver, and the last line is `built` rather than `installed`, which means it was not installed, and leads to the post-install script failure. This issue can be resolved by removing the duplicate entry and reconfiguring the package as follows: ```bash dkms remove gdrdrv/2.4.4 -a amd64 -k 5.15.0-1090-nvidia dpkg --configure -a ``` * When checking if the compute nodes are UP, in some cases `cm-upgrade` may wrongly report that some nodes are not DOWN. The report depends on how a node has been shut down, the state of the node not being updated correctly in the CMDaemon database. In such scenarios, after confirming that the nodes are indeed DOWN, re-run `cm-upgrade` with the `--skip-node-states-check` flag to skip the check for node states: **Example** ```bash cm-upgrade --skip-node-states-check ``` * Upgrading from BCM 10 to BCM 11 for DGX SuperPOD deployments requires filesystem exports to be explicitly set for the network `dgxnet` ([here](/upgrade/parallelupgrade#21012-enable-mixed-os-setup-and-create-a-dgx-os-7-image)). * The BCM machine learning repositories may still be referenced in the repository lists. They should be removed. For example, in the Ubuntu head node and images with: ```bash rm /etc/apt/sources.list.d/cm-ml.list ``` ## 3.6 Upgrading Using A BCM DVD/ISO When using a BCM DVD/ISO to perform the upgrade, it is important to use a DVD/ISO that is not older than **11.30.0**. The DVD/ISO version can be found (assuming that the DVD/ISO is mounted under `/mnt/cdrom`) with a `find` command such as: ```bash # find /mnt/cdrom -type d -name '11.30.0' /mnt/cdrom/data/packages/11.30.0 ``` ## 3.7 Enable Upgrade ### 3.7.1 Enable The Upgrade Repo, And Install The Upgrade Package Enabling the upgrade repository and installing the upgrade package can be carried out for the various distributions as follows: **RHEL8 derivatives:** ```bash # yum-config-manager \ --add-repo http://support.brightcomputing.com/upgrade/11.0/rhel/8/updates # yum --nogpgcheck install cm-upgrade ``` **RHEL9 derivatives:** ```bash # yum-config-manager \ --add-repo http://support.brightcomputing.com/upgrade/11.0/rhel/9/updates # yum --nogpgcheck install cm-upgrade ``` **SLES 15:** ```bash # zypper addrepo \ http://support.brightcomputing.com/upgrade/11.0/sles/15/updates cm-upgrade-11 # zypper install cm-upgrade ``` **Ubuntu 22.04:** ```bash # cat < /etc/apt/sources.list.d/cm-upgrade-11.list deb [trusted=yes] https://support.brightcomputing.com/upgrade/11.0/ubuntu/2204/ ./ EOF # apt update # apt install cm-upgrade ``` **Ubuntu 24.04:** ```bash # cat < /etc/apt/sources.list.d/cm-upgrade-11.list deb [trusted=yes] https://support.brightcomputing.com/upgrade/11.0/ubuntu/2404/ ./ EOF # apt update # apt install cm-upgrade ``` ### 3.7.2 Load The Environment Module ```bash # module load cm-upgrade ``` ## 3.8 Perform Upgrade The steps for carrying out the upgrade are as follows: ### 3.8.1 Power Off Nodes * Power off regular nodes * Terminate cloud nodes and cloud directors, and remove the cloud extension(s) ### 3.8.2 Apply Updates To Head Node * RHEL derivatives: ```bash # yum update ``` * SLES derivatives: ```bash # zypper up ``` * Ubuntu: ```bash # apt-get update # apt-get upgrade --with-new-pkgs ``` ### 3.8.3 Apply Updates To Software Images Before updating and upgrading a software image, it may be worth creating an image backup by cloning the image ([here](/admin/configuring#32022-making-some-nodes-function-differently-by-image)). If the cluster after the upgrade has an image that has an issue, then comparing it against the backup can help troubleshoot the issue. For each software image, updates can be applied as follows: * RHEL derivatives: ```bash # yum --installroot /cm/images/ update ``` * SLES derivatives: ```bash # zypper --root /cm/images/ up ``` * Ubuntu: ```bash # cm-chroot-sw-img /cm/images/ # apt-get update # apt-get upgrade --with-new-pkgs # exit ``` ### 3.8.4 Apply Updates To The `node-installer` Image * RHEL derivatives: ```bash # yum --installroot /cm/node-installer update ``` * SLES derivatives: ```bash # zypper --root /cm/node-installer up ``` * Ubuntu: ```bash # cm-chroot-sw-img /cm/node-installer # apt-get update # apt-get upgrade --with-new-pkgs # exit ``` ### 3.8.5 Upgrade Head Nodes To BCM 11 Important: this must be run on both head nodes in a high availability setup. Recommended: Upgrade the active head node first and then the passive head node. Depending on availability, the upgrade can be run either over the network, or using a DVD/ISO: * Upgrade using repositories accessible over the network ```bash # cm-upgrade ``` * Upgrade using a BCM DVD/ISO ```bash # cm-upgrade -b /root/bcm-11.0-ubuntu2404.iso ``` In an HA setup, after upgrading both the head nodes, resync the databases. Run the following from the active head node (it is very important to complete this step before moving to the next one): ```bash # cmha dbreclone ``` ### 3.8.6 Reboot Head Node(s) The head node(s) must be rebooted before proceeding to run the post-upgrade actions. ### 3.8.7 Post-upgrade Head Node Actions Important: these must be run on both head nodes in an HA setup ```bash # module load cm-upgrade/11 # cm-post-upgrade -m ``` ### 3.8.8 Upgrade The Software Image(s) To BCM 11 Important: the upgrade to the software images must be done only on the active head node. Depending on availability, the upgrade can be run either over the network, or using a DVD/ISO: * Upgrade using repositories accessible over the network ```bash # cm-upgrade -i all ``` * Upgrade using a BCM DVD/ISO ```bash # cm-upgrade -i all -b /root/bcm-11.0-ubuntu2404.iso ``` If the software images are not under the standard location—`/cm/images/` on the head node—then the option `-a` should be used: ```bash # cm-upgrade -a /apps/images -i # cm-upgrade -a /apps/images -i \ -b /root/bcm-11.0-ubuntu2404.iso ``` If the software images are located under multiple locations, the target locations can be specified as a comma-separated list: ```bash # cm-upgrade -a '/apps/images,/opt/images' -i all ``` #### 3.8.8.1 Btrfs Image Upgrades If the software image is a Btrfs image, then the revision ([section 14.4 of the *Administrator Manual*](/admin/administration#144-revision-control-for-images)) images cannot be used, because they use read-only filesystems. This issue extends to all references in category or node images, such as @latest and @*number* Only the original, unrevisioned, parent image can be upgraded with the `cm-upgrade` and `cm-post-upgrade` steps with a Btrfs filesystem. A workaround would be to create a new non-CoW image from a trusted revision, and upgrading that trusted non-CoW version. ### 3.8.9 Upgrade /cm/node-installer To BCM 11 This needs to be performed only on the primary active head node, either over the network, or using a BCM DVD/ISO. * Upgrade using repositories accessible over the network: ```bash # cm-upgrade -x ``` * Upgrade using a BCM DVD/ISO: ```bash # cm-upgrade -x -b /root/bcm-11.0-ubuntu2404.iso ``` ### 3.8.10 Post-upgrade Software Images Actions Important: these must be run only on the active head node. ```bash # cm-post-upgrade -i all ``` If the software images are not under the standard location, which is `/cm/images/` on the head node, then the option `-a` should be used: ```bash # cm-post-upgrade -a /apps/images -i all # cm-post-upgrade -a /apps/images -i ``` If the software images are located under multiple locations, then the target locations can be specified as a comma-separated list: ```bash # cm-post-upgrade -a '/apps/images,/opt/images' -i all ``` ### 3.8.11 NVIDIA Driver Installation During the upgrade to BCM 11.0, `cm-upgrade` automatically installs NVIDIA GPU drivers on non-DGX software images. This is only carried out on software images. It is not carried out on the head nodes or the node-installer images. #### 3.8.11.1 Default NVIDIA Driver Version By default, `cm-upgrade` installs `NVIDIA driver version 570 (Open Kernel Modules edition)`. The following components are installed: * On Ubuntu 22.04 and 24.04: the package installed is the `nvidia-driver-570-open` meta-package plus additional components including: * Fabric Manager (for NVLink/NVSwitch management) * NSCQ library (NVIDIA NVSwitch Configuration and Query Library) * IMEX library (GPU memory sharing across different compute nodes in a multi-node system connected by NVLink) * NVSDM library (for monitoring NVSwitch devices on NVIDIA Blackwell systems) * On RHEL 9/Rocky 9: a comprehensive set of packages based on the major 570 version. The package set installed includes: * DKMS and open kernel modules for dynamic kernel support * Core driver libraries and CUDA components * Fabric Manager for multi-GPU interconnect management * GUI tools (nvidia-settings, nvidia-xconfig) for X11 display configuration Most package names are generic (for example: nvidia-driver, nvidia-fabric-manager), with the exact version version determined by the repository configuration * On SLES 15 SP6: the package installed is a set based on the major 570 version, and includes: * Open kernel modules (`nvidia-open-570`) * Fabric Manager and NSCQ library Packages are sourced from NVIDIA’s CUDA repository (version-numbered). Native SLES packages use G06 generation naming, but we use NVIDIA’s CUDA repository for consistency #### 3.8.11.2 Non-DGX Systems Only NVIDIA driver installation is only carried out on non-DGX systems. DGX systems have pre-installed NVIDIA drivers, which are maintained by NVIDIA’s own update mechanisms, and should not managed by `cm-upgrade`. The upgrade script detects DGX systems by checking for the presence of `/etc/dgx-release` in the software image. If this file exists, then NVIDIA driver installation in the DGX system is skipped. #### 3.8.11.3 Changing the NVIDIA Driver Version If you need to install a different version of the NVIDIA drivers, then you can modify the driver version either before or after the upgrade as follows: * Pre-upgrade: Edit `/cm/local/apps/cm-upgrade/11.0/conf/packages.yaml` and modify the package versions listed under the `nvidia_driver_packages` section for your distribution. * Post-upgrade: Use the standard package management tools (`yum`/`apt`/`zypper`) within the software image to remove the installed driver packages and install your preferred version. ### 3.8.12 (HA) Allow For The Provisioning Requests To The Secondary Head Node To Complete Allow between 15 and 30 seconds for CMDaemon to schedule and start provisioning updates of `/tftpboot` and `/cm/node-installer` from the primary to the secondary head node. Then check and wait until the provisioning requests with the secondary head node as destination have completed using: ```bash # cmsh -c 'softwareimage provisioningstatus -r' ``` If the provisioning requests with a destination node the secondary head node continue to be listed for a prolonged period of time and do not disappear from the list, then the `node-installer` and/or `tftpboot` on the secondary head node are out of sync from the primary head node, which can result in issues with provisioning of the compute nodes. Please contact support ([https://enterprise-support.nvidia.com/s/create-case](https://enterprise-support.nvidia.com/s/create-case)) for further assistance. ### 3.8.13 Enable Mixed OS Setup And Create A DGX OS 7 Image On BCM 10 Ubuntu 22.04 clusters can use DGX OS 6 images. DGX OS 6 images are not supported in BCM 11. A new DGX OS 7 software image must therefore be used. The image can be created using `cm-image` and `cm-create-image`. These are BCM utilities, and are described in [section 9.6 of the *Administrator Manual*](/admin/softwareimagemanagement#96-creating-a-custom-software-image). DGX OS 7 is based on Ubuntu 24.04. This then also means the mixed OS environment—multidistro or multiOS as described in [section 9.7 of the *Administrator Manual*](/admin/softwareimagemanagement#97-creating-images-for-other-distributions-and-architectures-multidistro-and-multiarch)—must be enabled, to allow provisioning of DGX OS 7 images from the Ubuntu 22.04 head node of BCM 10. Image creation can be carried out as follows: 1. A script may need to be created first: **Example** ```bash [root@basecm11 ~]# cat disable-upstream-repos.sh #!/bin/bash if [ -f "$1/etc/apt/sources.list.d/ubuntu.sources" ]; then mv $1/etc/apt/sources.list.d/ubuntu.sources \ $1/etc/apt/sources.list.d/ubuntu.sources.dvd_bak fi ``` The script is only needed for a cluster running a BCM version 11 earlier than 11.31.0. It works around a known issue that may arise from using an upstream repository in an inappropriate manner. For BCM 11.31.0 and later, the workaround is carried out automatically, and the script is not needed. 2. Use `cm-image` to create the appropriate the Ubuntu 24.04 base images and enable a mixed OS setup. The `-z` option that calls the script need only be used for BCM 11 installations earlier than BCM 11.31.0. **Example** ```bash [root@basecm11 ~]# module load cm-image [root@basecm11 ~]# cm-image --v create all -a x86_64 -d ubuntu2404 --source /path/to/bcm11.0-ubuntu24.04-dgx-os-7.2.iso \ -z disable-upstream-repos.sh ``` 3. Use `cm-create-image` to create the new DGX OS image. The `-z` option that calls the script need only be used for BCM 11 installations earlier than BCM 11.31.0. ```bash [root@basecm11 ~]# cm-create-image --cmdvd \ /root/bcm-11.0-ubuntu2404-dgx-os-7.2.iso \ --no-cm-cuda-repo --extra-pkg-group \ doca_ofed_2.9.1-3.0.2 --dgx --dgx-type dgx_b200 \ --imagename dgx-os-image-b200 \ -z disable-upstream-repos.sh ``` The `--dgx-type` option ([here](/admin/softwareimagemanagement#9622-examples-in-usage-of-cm-create-image)) must be set for every type that is to be used as an image. The possible values are: * `dgx_h100` for the H100 GPU * `dgx_b200` for the B200 GPU * `dgx_a100` for the A100 GPU * `dgx_gb200` for the GB200 GPU * `dgx_h200` for the H200 GPU * `dgx_gb300` for the GB300 GPU * `dgx_b300` for the B300 GPU The cluster administrator should create a new category for each new image. Cloning the `default-image` image from the `default` category is a convenient way to do this since it already has the correct fsmounts. The image name should then be set for the category. For example, for a B200 image `dbx-os-image-b200` that has been placed in /cm-images/: **Example** ```bash root@basecm11:~# cmsh [basecm11]% category [basecm11->category]% list Name (key) Software image Nodes ------------------------ ------------------------ -------- default default-image 1 [basecm11->category]% clone default gb200 [basecm11->category*[gb200*]]% use gb200 [basecm11->category*[gb200*]]% set softwareimage dgx-os-image-b200 ; commit ... ``` The nodes that are to use the new image should then be assigned their image. For example, a `node001` can be assigned to the `gb200` category created in the preceding example with: **Example** ```bash [basecm11->device]% set node001 category gb200 ; commit ``` Configuration overlays that were used in the old category should also be migrated as needed to the new category. Upgrading from BCM 10 to BCM 11 for DGX SuperPOD deployments requires NFS exports to be explicitly set for the network `dgxnet`, for `/cm/node-installer-ubuntu2404-x86_64` and `/cm/node-installer-ubuntu2404-x86_64/certificates`. This can be configured with the following `cmsh` session on the head node: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% device use master [basecm11->device[basecm11]]% fsexports [basecm11->...->fsexports]% add /cm/node-installer-ubuntu2404-x86_64@dgxnet [basecm11->...installer-ubuntu2404-x86_64@dgxnet*]]% set network dgxnet [basecm11->...installer-ubuntu2404-x86_64@dgxnet*]]% commit [basecm11->...installer-ubuntu2404-x86_64@dgxnet]]% .. [basecm11->...rts]% add /cm/node-installer-ubuntu2404-x86_64/certificates@dgxnet [basecm11->...-ubuntu2404-x86_64/certificates@dgxnet*]]% set network dgxnet [basecm11->...-ubuntu2404-x86_64/certificates@dgxnet*]]% set write yes [basecm11->...-ubuntu2404-x86_64/certificates@dgxnet*]]% commit [basecm11->...-ubuntu2404-x86_64/certificates@dgxnet]]% ``` The same procedure must also be carried out for that secondary head node. The new DGX OS image has no Kubernetes packages installed on it by default. Migrating an old Kubernetes installation over from the old image is described [here](/upgrade/parallelupgrade#2541-kubernetes-package-upgrades-for-dgx-software-images). ### 3.8.14 Upgrading CMDaemon Lite The BCM 10 to BCM 11 upgrade supports CMDaemon Lite ([section 2.6.7 of the *Administrator Manual*](/admin/clustermanagement#267-cmdaemon-lite)) running on cluster switches, such as Cumulus switches. After the upgrade has been carried out on the head node and the images are ready for deployment, then the switches can be rebooted to pick up their new image. * It is important not to reboot the switch that is being used to connect to the head node. After the switches have rebooted, the compute nodes can be powered on. In a cluster, mixing deployments of cm-lite-daemon for BCM 10 with those of BCM 11 is not supported. This means that the CMDaemon Lite servers that are not managed by CMDaemon, for example PCs running CMDaemon Lite in a Python environment, also need to be upgraded when upgrading the cluster itself from BCM 10 to BCM 11. That must be done manually, by removing the BCM 10 package, and installing the BCM 11 package according to the environment and OS it is running in. ### 3.8.15 The Cluster Is Now Upgraded To BCM 11 It is recommended to provision the compute nodes at least once with BCM 11 before setting up any new integrations. ## 3.9 Simple Validation Checks Some basic checks can be carried out to validate the cluster. If a check fails, then further investigation should be carried out. ### 3.9.1 Validating CMDaemon The CMDaemon service status ([section 2.6.1 of the *Administrator Manual*](/admin/clustermanagement#261-managing-and-inspecting-the-cluster-management-daemon)), can be checked with: **Example** ```bash root@basecm11:~# systemctl status cmd ● cmd.service - BCM daemon Loaded: loaded (/usr/lib/systemd/system/cmd.service; enabled; preset: enabled) Active: active (running) since Mon 2025-12-01 01:44:17 PST; 6h ago Main PID: 2547 (safe_cmd) Tasks: 77 (limit: 65535) Memory: 576.1M (high: 3.4G max: 3.6G available: 1.3G peak: 600.1M) CPU: 6min 56.963s CGroup: /system.slice/cmd.service ... ``` ### 3.9.2 Validating High Availability If the cluster has HA configured, then the cluster manager HA status ([here](/admin/ha#15421-cmha-status-querying-ha-status)), can be checked. For example, for head nodes `basecm11-a` and `basecm11-b` the response to the check would be: **Example** ```bash root@basecm11-a:~# cmha status Node Status: running in active mode basecm11-a* -> basecm11-b failoverping [ OK ] mysql [ OK ] ping [ OK ] status [ OK ] basecm11-b -> basecm11-a* failoverping [ OK ] mysql [ OK ] ping [ OK ] status [ OK ] ``` ### 3.9.3 Validating `cmsh` And Devices A check on the device statuses can be run with: **Example** ```bash root@basecm11:~# cmsh -c "ds" basecm11.............[ UP ] node001 .............[ UP ] ... ``` ### 3.9.4 Validating The Cluster Health Health check values can be retrieved from devices in the BCM monitoring system with the help of the `latesthealthdata` command ([section 10.6.3.1 of the *Administrator Manual*](/admin/monitoring#10631-using-the-latestdata-commands)): **Example** ```bash [basecm11->device]% foreach -v * (latesthealthdata) ================================= basecm11 ===================================== Measurable Parameter Type Value Age State Info ------------------ ------------ ------------ ---------- ---------- ---------- ---------- ManagedServicesOk Internal PASS 1m 39s chrootprocess OS PASS 1.23s cm-chroot-sw-img OS PASS 1m 18s cmsh Internal PASS 25m 7s cuda-dcgm OS PASS 54.8s defaultgateway Network PASS 1m 8s diskspace Disk PASS 14s dmesg OS PASS 1m 51s exports Disk PASS 9.21s interfaces Network PASS 1m 7s ldap OS PASS 1m 37s lustre Disk PASS 58.8s mounts Disk PASS 1m 40s mysql OS PASS 53m 47s ntp Internal PASS 1m 49s oomkiller OS PASS 18s ssh2node network PASS 8m 58s ================================= node001 ===================================== Measurable Parameter Type Value Age State Info ------------------ ------------ ------------ ---------- ---------- ---------- ---------- ManagedServicesOk Internal PASS 2m 54s cuda-dcgm OS PASS 2m 9s defaultgateway Network PASS 2m 13s diskspace Disk PASS 3m 28s dmesg OS PASS 3m 5s interfaces Network PASS 2m 22s ldap OS PASS 2m 51s lustre Disk PASS 2m 13s mounts Disk PASS 2m 55s ntp Internal PASS 3m 3s oomkiller OS PASS 1m 32s ssh2node network PASS 8m 43s ... ``` ### 3.9.5 Validating Base View For a dotted quad IP address \<*IP address*>, the cluster administrator can connect to: `https://`\<*IP address*>`:8081/base-view` and try logging in. If there was non-administrator access before the upgrade, then that should also be checked. ### 3.9.6 Validating The State Of The Packages On The Cluster The state of packages according to the package manager for the cluster can be checked in the head nodes and in the software images with: #### 3.9.6.1 Ubuntu Packages ```bash apt-get check dpkg --audit ``` #### 3.9.6.2 RHEL-like Distribution Packages ```bash yum check rpm -Va --nofiles ``` #### 3.9.6.3 SLES Packages ```bash zypper verify rpm -Va --nofiles ``` These commands should all normally give no output, other than processing messages in some cases. ### 3.9.7 Validating Slurm And Pyxis Slurm and Pyxis validation on the cluster ([here](/admin/workloadmanagement#slurmpyxis-installation-validation)), can be carried out as follows: #### 3.9.7.1 Validating Slurm ```bash root@basecm11:~# module load slurm root@basecm11:~# scontrol show config Configuration data as of 2025-12-02T04:06:05 AccountingStorageBackupHost = (null) AccountingStorageEnforce = none AccountingStorageHost = master AccountingStorageExternalHost = (null) AccountingStorageParameters = (null) AccountingStoragePort = 6819 ... Slurmctld(primary) at basecm11 is UP ``` #### 3.9.7.2 Validation Of Pyxis ```bash root@basecm11:~# srun --container-image=ubuntu grep PRETTY /etc/os-release pyxis: importing docker image: ubuntu pyxis: imported docker image: ubuntu PRETTY_NAME="Ubuntu 24.04.3 LTS" root@basecm11:~# ``` ### 3.9.8 Validating Kubernetes The `kube-system` pods should be inspected to see if they are running along without any real issues: **Example** ```bash root@basecm11:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-66bc5c9577-bv577 1/1 Running 0 4h24m coredns-66bc5c9577-rsfpv 1/1 Running 0 4h24m kube-apiserver-basecm11 1/1 Running 0 4h19m kube-apiserver-node001 1/1 Running 2 4h24m kube-controller-manager-basecm11 1/1 Running 0 4h23m kube-controller-manager-node001 1/1 Running 2 4h25m kube-proxy-2ckqr 1/1 Running 0 4h24m ... kube-scheduler-node001 1/1 Running 1 4h25m kube-state-metrics-6b98df4478-nhkd2 1/1 Running 0 4h19m metrics-server-f49fdc486-4ndfk 1/1 Running 0 4h19m metrics-server-f49fdc486-5rvbf 1/1 Running 0 4h19m ``` Are the operators running along fine too? ```bash root@basecm11:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE calico-system calico-apiserver-577cdf67b5-645qn 1/1 Running 0 4h49m ... calico-system calico-typha-6495c7469f-gpztc 1/1 Running 0 4h49m calico-system calico-typha-6495c7469f-rp9rz 1/1 Running 0 4h49m calico-system goldmane-7b6b78f6f-tn65t 1/1 Running 0 4h49m calico-system whisker-6c95b488ff-7s9pf 2/2 Running 0 4h49m cm local-path-provisioner-675dc9fd86-b6fg6 1/1 Running 0 4h48m cmkpm-system cmkpm-controller-manager-66b99b6699-6k6gm 1/1 Running 0 4h48m grafana grafana-grafana-operator-c65d95b7f-c5p59 1/1 Running 0 4h47m ingress-nginx ingress-nginx-controller-68994db546-7c2h9 1/1 Running 0 4h44m ingress-nginx ingress-nginx-controller-68994db546-p6pjv 1/1 Running 0 4h44m kube-system coredns-66bc5c9577-bv577 1/1 Running 0 4h51m ... ``` ## 3.10 In-place Upgrade Of An Edge Director Or A Compute Node The recommended way of upgrading the compute nodes is to upgrade the head node and software image(s) as described in section [3.8](#38-perform-upgrade), and to allow the node-installer (when the compute nodes are booted) to re-provision the compute nodes with the upgraded software. For clusters with an edge setup, this usually means a new edge ISO needs to be generated on the head node, transferred to the edge site, and then the edge director node needs to be reconfigured to boot from the ISO, which will start the node-installer. In some special cases, however, the administrator may elect to manually carry out an in-place upgrade of a compute node or edge director, instead of using the node installer to automatically upgrade the node. With the manual in-place upgrade, the administrator sets up the node to receive the updates directly from the software image (through `imageupdate`) without the involvement of the node installer. Other than the cases where the node installer does not start on (re)boot by default, such as for the edge director, the in-place upgrade of a compute node/edge director does not offer any advantage over the standard approach to simply upgrade the head node and the software image(s). ### 3.10.1 Important Note About Services Running On The In-place Upgraded Compute Node/Edge Director An in-place upgrade of a compute node will generally restart only the CMDaemon service. Any other service that may be affected by the upgrade of the packages, especially in the case of packages in `/cm/shared` (such as WLM), which are upgraded as part of the head node(s) upgrade, will need to be restarted manually by the administrator, or a reboot will be required. It is therefore recommended to use the standard approach to upgrade the software image(s) on the head node, boot the nodes with the node installer, and allow the node installer to upgrade and configure the compute nodes as usual. Nevertheless the administrator may find it more convenient in special cases such as edge sites, which have edge directors, to perform an in-place upgrade. The edge directors nodes are typically configured to boot from the local disk and by default do not start the node installer on (re)boot. An in-place upgrade of the edge directors will then allow the administrator to upgrade the software on the directors without transferring ISOs and reconfiguring the directors to boot from ISOs. To address the issue with the running services, the edge directors can be rebooted at the end of the upgrade and while they will still not start the node installer, they will boot into the upgraded software. ### 3.10.2 In-place Edge Director Upgrade Procedure Unless otherwise stated, all actions are performed locally on the edge director #### 3.10.2.1 Apply Updates To The Edge Director * RHEL derivatives: ```bash # yum update ``` * SLES derivatives: ```bash # zypper up ``` * Ubuntu: ```bash # apt-get update # apt-get upgrade --with-new-pkgs ``` #### 3.10.2.2 Back Up The File System It is recommended to perform a file system backup of the edge director #### 3.10.2.3 Stop CMDaemon and `cm-nfs-checker` services ```bash # systemctl stop cmd # systemctl stop cm-nfs-checker ``` #### 3.10.2.4 Upgrade The Head Node(s) As Described In The Preceding Follow the steps to upgrade the head nodes, software image, and node installer as described in the preceding sections “Enable Upgrade” (section [3.7](#37-enable-upgrade)) and “Perform Upgrade” (section [3.8](#38-perform-upgrade)), and which are performed on the head node(s) of the cluster. Because CMDaemon on the edge director is stopped, the edge director does not need to be powered off while the upgrade of the head nodes takes place. #### 3.10.2.5 Upgrade CMDaemon To BCM 11 After the upgrade of the head nodes is completed, the package manager repositories files on the edge director must be manually updated to point to the BCM 11 repositories. This means that all occurrences of 10.0 in the repositories are replaced with with 11.0. Note: Leave the usernames such as in, for example, “`username=cm10user`”, unchanged. Then, upgrade CMDaemon as follows for the various distributions: * RHEL derivatives: Update the `/etc/yum.repos.d/cm.repo` file to point to the 11.0 repositories. On RHEL, typically `yum`/`dnf` can manage to resolve most of the dependencies, and the administrator can execute: ```bash # yum clean all # rpm -e --nodeps cm-config-cm # yum install cmdaemon ``` * SLES derivatives: Update the `/etc/zypp/repos.d/Cluster_Manager_Updates.repo` file to point to the 11.0 repositories. Execute ```bash # zypper clean --all # zypper install cmdaemon ``` A problem may be reported by `zypper` with upgrading `cmdaemon` to version 11.0, due to a requirement to install `cm-config-cm` version 11.0. Typically the first solution, `Solution 1`, offered by zypper can be selected, which is to deinstall a certain set of BCM packages (prefixed by `cm9.2`) in order to upgrade CMDaemon to version 11.0. * Ubuntu: Update the `/etc/apt/sources.list.d/cm.list` file to point to the 11.0 repositories. Also update the `/etc/apt/auth.conf.d/cm.conf` file, to point to the 11.0 repositories. Create a new file `/etc/apt/preferences.d/99-tmp-cmd-upgrade` with content (without any space at the beginning of the lines): ```bash Package: * Pin: origin "updates.brightcomputing.com" Pin-Priority: 1001 ``` The administrator may also find it convenient to choose a geographically-close mirror in the `cm.list` file. But note that the local change will be overwritten to the settings in the software image, once `imageupdate` is run in a step later on. Execute ```bash # apt-get update # apt-get install cmdaemon ``` #### 3.10.2.6 Start CMDaemon After CMDaemon has been upgraded to BCM 11, execute: ```bash # systemctl daemon-reload # systemctl start cm-nfs-checker # systemctl start cmd ``` #### 3.10.2.7 Update The Edge Director Image With `imageupdate` On the head node, with `cmsh`, verify that the edge director is UP. If the edge director is not UP, then the in-place upgrade has not been successful, and the director will need to be upgraded by booting into to the node-installer. With `cmsh` on the head node, execute `imageupdate` for the edge director. The administrator is advised to execute a dry-run first, then to use use the synclog command to review the result and verify no unexpected local data directories or files will be affected, and then to perform the real `imageupdate`: ```bash cmsh% device use edge-director cmsh% imageupdate cmsh% synclog cmsh% imageupdate -w ``` #### 3.10.2.8 Update The Provisioners And Recreate The Ramdisks Execute on the head node ```bash cmsh% softwareimage updateprovisioners; cmsh% fspart; trigger /tftpboot; trigger /cm/node-installer; trigger /cm/shared ``` Allow for the provisioning requests with a destination node the in-place upgraded director to complete by monitoring the provisioning status: ```bash cmsh% softwareimage provisioningstatus -r ``` For the images served by the edge director, recreate the ramdisk ```bash cmsh% softwareimage; createramdisk -d ``` If needed, the in-place upgraded edge director can now also be rebooted. It is then upgraded and can provision the edge compute nodes.