NVIDIA UFM Enterprise Quick Start Guide v6.11.1
NVIDIA UFM Enterprise Quick Start Guide v6.11.1

Upgrading UFM Software

After UFM installation, UFM detects existing UFM versions previously installed on the machine and prompts you to run a clean install of the new version or to upgrade. We recommend backing up the UFM configuration before upgrading the UFM as specified in the section UFM Database and Configuration File Backup.

Note

Upgrading the UFM Enterprise software version is supported up to two previous GA software versions (GA -1 or GA -2).
For example, if you wish to upgrade to UFM Enterprise v6.11.0, it is possible to do so only from UFM Enterprise v6.9.0 or v6.10.0.

You can upgrade the UFM standalone server software for InfiniBand from the previous UFM version.

To upgrade the UFM server software:

  1. Create a temporary directory (for example /tmp/ufm).

  2. Open the UFM software zip file that you downloaded. The zip file contains the following installation files for:

    • RedHat 7/CentOS 7/OEL 7: ufm-X.X -XXX.el7.x86_64.tgz

    • RedHat 8/CentOS 8/OEL 8: ufm-X.X -XXX.el8.x86_64.tgz

    • Ubuntu 18.04: ufm-X.X -XXX.ubuntu18.x86_64.tgz

    • Ubuntu 20.04: ufm-X.X -XXX.ubuntu20.x86_64.tgz

  3. Extract the installation file for your system's OS to the temporary directory that you created.

  4. Stop the UFM server. Run:

    Copy
    Copied!
                

    /etc/init.d/ufmd stop

  5. From within the temporary directory, run the following command as root:

    Copy
    Copied!
                

    ./upgrade.sh

    Warning

    A configuration backup file will be created in the running directory (e.g. /tmp/ufm). The backup file name is ufm_X.X.X_bkp.zip (X.X.X is the previous version).

    1. Upgrade from the previous version: the existing UFM data and configuration are preserved.

    2. In case upgrade.sh script stops before completion (e.g. missing prerequisite), the upgrade procedure can be resumed by fixing the issue (e.g. installing missing prerequisite) and rerunning ./upgrade.sh again.

  6. Restart the UFM server. Run:

    Copy
    Copied!
                

    systemctl start ufm-enterprise.service

    Warning

    /etc/init.d/ufmd start - Available for backward compatibility.

  7. After the upgrade, remove the temporary directory

Warning

HA mechanisms have changed starting from UFM Enterprise v6.10.0. To upgrade from older UFM Enterprise versions (v6.9.0 and older), use the following upgrade procedure.

Follow the relevant instructions depending on your current UFM Enterprise version:

Upgrade from UFM Enterprise versions up to 6.9.X including

Instructions on Upgrading UFM HA from UFM Enterprise v6.9.0 (with old HA) to v6.10.0 or v6.11.0 (with NEW HA)

  1. Prior to upgrading, perform cleanup of the yum cache on both master and standby nodes.

    Copy
    Copied!
                

    yum clean all

    On CentOS7/RedHat7, check if there are any leftovers of the UFM inside the yum cache on both master and standby nodes.

    Copy
    Copied!
                

    find /var/cache/yum/ -name "UFM" /var/cache/yum/x86_64/7Server/UFM

    If there are UFM leftovers, remove them by running:

    Copy
    Copied!
                

    rm -rf /var/cache/yum/x86_64/7Server/UFM

  2. On the master node, open the UFM software zip file that you downloaded. The zip file contains the following installation files:

    • RedHat 7/CentOS 7/OEL 7: ufm-X.X -XXX.el7.x86_64.mofed5.tgz

    • RedHat 8/CentOS 8/OEL 8: ufm-X.X -XXX.el8.x86_64.mofed5.tgz

  3. On the master node, change the dir location to a temporary directory. Run:

    Copy
    Copied!
                

    cd /tmp

  4. On the master node, extract the installation file for your system's OS to a temporary directory.

    Copy
    Copied!
                

    tar zxvf ufm-X.X -XXX.el7.x86_64.mofed5.tgz

  5. On the master node, change directory to the installation folder:

    Copy
    Copied!
                

    cd ufm-X.X -XXX.el7.x86_64.mofed5/

  6. On the master node, run upgrade.sh:

    Copy
    Copied!
                

    ./upgrade.sh

  7. On the standby node, if the UFM is not removed in the previous step, remove UFM from the standby node:

    Copy
    Copied!
                

    /opt/ufm/uninstall.sh

  8. On the standby node, deploy the new UFM package by following steps 3 to 5.

  9. On the standby node, install UFM as SA:

    Copy
    Copied!
                

    ./install.sh

    At this point, you have UFM installed on both servers in SA (Stand Alone) Mode. Note that the previous UFM configuration files are located only on the master node.

    Warning

    The backup of previous UFM configuration is located on the master node at /tmp/ufm-X.X -XXX.el7.x86_64.mofed5/ufm_<previous_version_number>_bkp.zip.
    For example: /tmp/ufm-X.X -XXX.el7.x86_64.mofed5/ufm_6.9.1_bkp.zip
    It is recommended to copy the backup to a safe location.

  10. To perform UFM HA installation, configuration and startup, refer to Installing UFM Server Software for High Availability - follow step 2 and onwards.

Upgrade from UFM Enterprise versions from to 6.10.0 and up

You can upgrade the UFM server HA software for InfiniBand from the previous release. The upgrade is performed on both servers.

To upgrade the UFM server software:

  1. Stop UFM HA cluster

    Copy
    Copied!
                

    ufm_ha_cluster stop

  2. Open the UFM software zip file that you downloaded. The zip file contains the following installation files:

    • RedHat 7/CentOS 7/OEL 7: ufm-X.X -XXX.el7.x86_64.tgz

    • RedHat 8/CentOS 8/OEL 8: ufm-X.X -XXX.el8.x86_64.tgz

  3. Extract the installation file for your system's OS to a temporary directory.

  4. From the installation folder, run the following command:

    Copy
    Copied!
                

    ./upgrade.sh

  5. Install the latest UFM HA package. Follow steps 3-5 in Installing UFM Server Software for High Availability.

    Important

    There is no need to configure the UFM HA package after the installation procedure.

  6. After the upgrade, start the UFM HA cluster. Run:

    Copy
    Copied!
                

    ufm_ha_cluster start

You can upgrade the UFM server HA software for InfiniBand from the previous release. The upgrade is performed on both servers.

To upgrade the UFM server software:

  1. Stop UFM HA cluster:

    Copy
    Copied!
                

    master# ufm_ha_cluster stop

  2. Extract the new UFM package:

    Copy
    Copied!
                

    master# tar -xzf ufm-6.10.0-3.ubuntu18.mofed5.tgz

  3. Enter to the installation folder:

    Copy
    Copied!
                

    master# cd /tmp/ufm-6.10.0-3.ubuntu18.mofed5

  4. Run the below command to backup the previous UFM version:

    Copy
    Copied!
                

    master# ./ufm_backup.sh 6.9.1

  5. Move the generated backup file into a temporary folder:

    Copy
    Copied!
                

    master# mv ufm_6.9.1_bkp.zip /tmp/

  6. Run the following command to cleanup the UFM HA cluster on the master node:

    Copy
    Copied!
                

    master# ufm_ha_cluster cleanup

  7. Run the following command to cleanup the UFM HA cluster on the standby node:

    Copy
    Copied!
                

    slave# ufm_ha_cluster cleanup

  8. On the master node, remove the UFM HA indicator file:

    Copy
    Copied!
                

    master# rm -rf /opt/ufm/files/config_ha /opt/ufm/indicators/ufm_ha|| true

  9. On the standby node, remove the UFM HA indicator file:

    Copy
    Copied!
                

    slave# rm -rf /opt/ufm/files/config_ha /opt/ufm/indicators/ufm_ha || true

  10. On the master node, run the following upgrade command

    Copy
    Copied!
                

    master# cd /tmp/ufm-6.10.0-3.ubuntu18.mofed5 && ./upgrade.sh

  11. Wait until the upgrade procedure on the master node is done.

  12. On the standby node, run the following upgrade command:

    Copy
    Copied!
                

    slave# cd /tmp/ufm-6.10.0-3.ubuntu18.mofed5 && ./upgrade.sh

  13. On the master node, run the following command to restore the backup file of the previous version.

    Copy
    Copied!
                

    master# /opt/ufm/scripts/ufm_restore.sh -f /tmp/ufm_6.9.1_bkp.zip -u

  14. On the master node, configure UFM HA. Run:

    Copy
    Copied!
                

    master# configure_ha_nodes.sh --cluster-password 123456 --main-hostname swx-ci-29 --main-ip 10.209.36.143 --main-sync-interface enp5s0f0 --standby-hostname swx-ci-30 --standby-ip 10.209.36.45 --standby-sync-interface enp5s0f0 --virtual-ip 10.209.36.253

  15. Run the following to check the UFM HA cluster status:

    Copy
    Copied!
                

    # ufm_ha_cluster status

  16. The UFM should be up and running in HA configuration.

© Copyright 2023, NVIDIA. Last updated on Sep 8, 2023.