NVIDIA UFM Enterprise Appliance Software User Manual v1.3.1
NVIDIA UFM Enterprise Appliance Software User Manual v1.3.1

UFM Enterprise Appliance OS Upgrade

This section provides a step-by-step guide for UFM Enterprise Appliance Operating System upgrade procedure.

Each UFM Enterprise Appliance software has an additional tar file with a -omu.tar suffix (OMU stands for OS Manufacture and Upgrade). This tar file can be used to re-manufacture the server and to upgrade the operating system/software on the server.

  1. Copy the OMU tar file to a temporary directory on the server.
    UFM-APPLIANCE - ufm-appliance<version>-<revision>-omu.tar

  2. Extract the contents of the tar file to /tmp.

    Copy
    Copied!
                

    tar xf ./ufm-appliance-<version>-<revision>-omu.tar -C /tmp/

  3. Change to the extracted directory.

    Copy
    Copied!
                

    cd /tmp/ufm-appliance-<version>-<revision>-omu

  4. An upgrade script and an ISO file are included in the extracted directory.

    Copy
    Copied!
                

    ls -1 ./# ls -1 ./ ufm-os-upgrade.sh ufm-appliance-<version>-<revision>.iso

    The following flags are available in the upgrade script help.

    Copy
    Copied!
                

    # ufm-os-upgrade.sh --help ufm-os-upgrade.sh will upgrade and install OS packages.   IMPORTANT!!! a reboot is mandatory after the finalization of this script, kernel and kernel models will not work properly until the server is rebooted.   Additional SW installations will be automatically invoked after reboot, a message will pop on all open terminals with the installation status: "UFM-OS-FIRSTBOOT-FAILURE" - if installation is failed. "UFM-OS-FIRSTBOOT-SUCCESS" - if installation succeeded.   additional info will be vailable in "/var/log/ufm_os_upgrade_<UFM-OS-VERSION>.log" log file.   syntax: ufm-os-upgrade.sh [options]   options --appliance-sw-upgrade upgrade ufm_appliance SW as well, default is to upgrade OS only.   -d,--debug debug info will be visible on the screen.   -r,--reboot Automatically reboot the server when upgrade is finished. P.S. if secureboot is enabled and a new certifcate is enroled the server will not automatically reboot even if this flag is set.   -y,--yes wont prompt for user acknowledgements.   -h,--help print this help message.

    Important

    IMPORTANT!!! System reboot is mandatory once the upgrade procedure is completed. The -r flag can be used to automatically reboot the server at the end of the upgrade. Note that some kernel modules may not work properly until server reboot is performed.

  1. Stop UFM service by running the following command:

    Copy
    Copied!
                

    systemctl stop ufm-enterprise.service

  2. Run the upgrade script.

    Warning

    System reboot is mandatory once the upgrade procedure is completed. The -r flag can be used to automatically reboot the server.

    The -y flag can be supplied to skip user questions (the flag does not automatically reboot the server on its own. For auto reboot, combine with the -r flag)
    Once a secure boot certificate is updated/installed, the script will not auto reboot even if -y and -r flags are provided. That is because the addition of certificates require manual user intervention at boot (after the upgrade).
    There is a 10 seconds window to press any button when prompted during the boot procedure and insert the server root password in order to import the certificate. Further details are available in Appendix - Secure Boot Activation and Deactivation.
    The --appliance-sw-upgrade flag can be supplied to upgrade the UFM Enterprise Appliance SW as well. It is not upgraded by default.

    In the following example the server will auto reboot when upgrade is finished.

    Copy
    Copied!
                

    ./ufm-os-upgrade.sh -y -r

    An example with the --appliance-sw-upgrade flag. Note that the UFM Enterprise appliance SW will also be upgraded.

    Copy
    Copied!
                

    ./ufm-os-upgrade.sh -y -r --appliance-sw-upgrade

  3. In case a secure-boot certificate is installed/upgraded, the following warning is presented:

    1.JPG

    In that case the server does not reboot automatically, a manual configuration is required at boot (a 10 second prompt appears during the boot. For more information, refer to Appendix - Secure Boot Activation and Deactivation) To continue with the upgrade procedure, manually reboot the server from Appendix - Secure Boot Activation and Deactivation.

  4. After the reboot procedure is complete, a systemd service (ufm-os-firstboot.service) runs the remainder of the upgrade procedure. Once completed, a message is prompted to all open terminals including the status:
    "UFM-OS-FIRSTBOOT-FAILURE" - if installation is failed.
    "UFM-OS-FIRSTBOOT-SUCCESS" - if installation succeeded.
    Example:

    2.JPG

    To manually check the status, run systemctl status ufm-os-firstboot.service. If it is already finished, an error message is prompted stating that there is no such service. In that case, the log /var/log/ufm-os-firstboot.log can be checked instead.

    Copy
    Copied!
                

    systemctl status ufm-os-firstboot.service

    Example:

    3.JPG

Upgrade on HA is similar to the upgrade procedure on SA and needs to be done on both the Master and Standby nodes.

In case the Standby node is unavailable, the upgrade can be run on the Master node only, however, some additional steps will be required after the appliance is upgraded.

  1. [On the Master Node]: Copy and extract the OMU tar file to a temporary directory (Refer to UFM Enterprise Appliance Software Upgrade).

  2. [On the Standby Node]: Copy and extract the OMU tar file to a temporary directory (Refer to UFM Enterprise Appliance Software Upgrade).

  3. [On the Master Node]: Stop UFM cluster. Run:

    Copy
    Copied!
                

    ufm_ha_cluster stop

  4. [On the Master Node]: Run the upgrade script. (Run the same command on the Standby Node, no need to wait for the script to be completed, see the following step).
    In HA scenario, the upgrade script should run on both the Master and Standby nodes.

    Warning

    System reboot is mandatory once the upgrade procedure is completed. The -r flag can be used to automatically reboot the server.

    The -y flag can be supplied to skip user questions (the flag does not automatically reboot the server on its own. For auto reboot, combine with the -r flag)
    Once a secure boot certificate is updated/installed, the script will not auto reboot even if -y and -r flags are provided. That is because the addition of certificates require manual user intervention at boot (after the upgrade).
    There is a 10 seconds window to press any button when prompted during the boot procedure and insert the server root password in order to import the certificate. Further details are available in Appendix - Secure Boot Activation and Deactivation.
    The --appliance-sw-upgrade flag can be supplied to upgrade the UFM Enterprise Appliance SW as well. It is not upgraded by default.

    In the following example the server auro reboots once the upgrade procedure is completed:

    Copy
    Copied!
                

    ./ufm-os-upgrade.sh -y -r

    Example with the --appliance-sw-upgrade flag. The UFM Enterprise Appliance SW is also upgraded.

    Copy
    Copied!
                

    ./ufm-os-upgrade.sh -y -r --appliance-sw-upgrade

  5. [On the Standby Node]: Run the upgrade script.

    It is highly recommended to upgrade both servers as closely as possible to one another. There is no need to wait for the script to complete running on the Master Node .

  6. In case a Secure Boot certificate is installed/upgraded, the following warning is prompted:

    4.JPG

    The server does not auto reboot; a manual configuration is required at boot (a 10 second prompt appears at boot. Refer to Appendix - Secure Boot Activation and Deactivation.

  7. To continue with the upgrade procedure, manually reboot the server, as described in Appendix - Secure Boot Activation and Deactivation.

  8. After the reboot procedure is complete, a systemd service (ufm-os-firstboot.service) runs the remainder of the upgrade procedure. Once completed, a message is prompted to all open terminals including the status:
    "UFM-OS-FIRSTBOOT-FAILURE" - if installation is failed.
    "UFM-OS-FIRSTBOOT-SUCCESS" - if installation succeeded.
    Example:

    5.JPG

    To manually check the status, run systemctl status ufm-os-firstboot.service. If it is already finished, an error message is prompted stating that there is no such service. In that case, the log /var/log/ufm-os-firstboot.log can be checked instead.

    Copy
    Copied!
                

    systemctl status ufm-os-firstboot.service

    Example:

    6.JPG

  9. The upgrade script temporarily halts the DRBD service from migrating between the Master and Standby nodes, this is achieved by setting the cluster into maintenance-mode. The maintenance-mode is lifted when the upgrade is finished on both nodes.

    In case the Master and Standby node do not communicate for more than 5 minutes, the script gives up and will not automatically lift maintenance-mode. To manually remove maintenance-mode, run the following command (either on the Master or Standby nodes, no need to run on both).

    Check if maintenance-mode is set:

    Copy
    Copied!
                

    ufm_ha_cluster status

    7.JPG

    Unset maintenance-mode null

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.