NVIDIA UFM Cyber-AI Documentation v2.5.0

High Availability

UFM HA supports High-Availability on the host level for UFM products (UFM Enterprise/UFM Appliance/UFM CyberAI) The solution is based on pacemaker to monitor services and DRBD to sync file-system states. The HA package can be used with both bare-metal and Dockerized UFM products.

UFM HA should be installed on two machines, master and standby.

Supported Platforms

  1. Ubuntu

  2. Centos Master


Pacemaker packages

  1. pacemaker

  2. pcs

  3. corosync

DRBD Package

  • DRBD utils 8.4 or up.

ufm_ha_cluster usage


ufm_ha_cluster --help Usage: ufm_ha_cluster [-h|--help] <command> [<options>] This script manages ufm HA cluster.   OPTIONS: -h|--help Show this message   COMMANDS: config Configure HA cluster set-password Change hacluster password status Check HA cluster status failover Master node failover takeover Standby node takeover start Start HA services stop Stop HA services attach attach new standby node from cluster detach detach the old standby to cluster   For more help about each command, type: ufm_ha_cluster <command> --help

Setting HA Cluster Password

HA cluster user is a user used for pacemaker synchronization. the password for the user should be the same on both machines. To set the password, run the following command on both machines (order does not matter).


ufm_ha_cluster set-password -p <new-password>

Configuring Pacemaker and DRBD


ufm_ha_cluster config --help Usage: ufm_ha_cluster config [<options>]   The config command configures ha add-on for ufm server.   OPTIONS: -r | --role <node role> Node role (master or standby) mandatory. -n | --peer-node <node-hostname> Peer node name. mandatory. -s | --peer-sync-ip <ip address> Peer node sync ip adreess mandatory. -c | --sync-interface Local interface to be used for drbd sync mandatory. -i | --virtual-ip <virtual-ip> Cluster virtual IP. mandatory. -f | --ha-config-file <file path> HA configuration file. default: ufm-ha.conf -p | --hacluster-pwd <pwd> hacluster user password default: default password -h | --help Show this message

  1. You must run configuration script on the standby machine, then on the master machine.

  2. Running config command will not start UFM services, you have to run it directly from the master machine.

  3. Initial file system sync between master and standby may take few minutes, depending on your sync interface speed.

  4. You must wait for the sync process before starting the services. You may use the status command for monitoring the sync.

  5. If you are using high-availability for both UFM Cyber-AI and UFM Enterprise you have to change the following line in ufm-ha.conf file:


    systemd_services=ufm-cyberai systemd_services=ufm-cyberai ufm-ha-watcher ufm-enterprise

Stopping UFM Services

You may stop UFM services using the following stop command.


ufm_ha_cluster stop

Takeover Services

Takeover command can be executed on the standby machine so it will be the master.


ufm_ha_cluster takeover

Master Failover

Failover command can be executed on the master machine so it will be the standby.


ufm_ha_cluster failover

Replace HA Node

To replace old standby, detach the old standby, then configure the new standby, and attach it to the cluster.

On the master, run the detach command:


ufm_ha_cluster detach

On the new standby, run the config command, for more information, refer to ufm-cai-jobs.

On the master node, run the attach command:


Ufm_ha_cluster –n <peer_node> -s <peer_sync_ip> -p <hacluster-pwd> -c  <sync-interface>

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.