Monitoring and Troubleshooting

Check UFM Status

Run the below command on the master node:

Copy
Copied!
            

/etc/init.d/ufmd status 

Check HA Status

Run the below command:

Copy
Copied!
            

ufm_ha_cluster status  pcs status

Check DRBD Status

Run the below command:

Copy
Copied!
            

ufm_ha_cluster status 

Show DRBD Resource

Run the below command:

Copy
Copied!
            

drbdadm sh-resources 

Show DRBD Disk State

Run the below command:

Copy
Copied!
            

drbdadm dstate ha_data 

Show DRBD Role

Run the below command:

Copy
Copied!
            

drbdadm role ha_data 

Show DRBD Connectivity

Run the below command:

Copy
Copied!
            

drbdadm cstate ha_data 

Split-Brain Recovery

For automated HA solution, is it recommended to configure STONITH agents to kill (power-off) a peer node.

Step 1:

Manually choose a node which data modifications will be discarded.

It is called the split-brain victim. Choose wisely; all modifications will be lost! When in doubt, run a backup of the victim’s data before you continue.

When running a Pacemaker cluster, you can enable maintenance mode.

Copy
Copied!
            

ufm_ha_cluster enable-maintain

If the split-brain victim is in the Primary role, bring down all applications using this resource.

Now, switch the victim to the Secondary role:

Copy
Copied!
            

victim# ufm_ha_cluster reset standby

Resync starts automatically if the survivor is in a WFConnection network state. If the split-brain survivor is still in a Standalone connection state, reconnect it:

Copy
Copied!
            

survivor#  ufm_ha_cluster reset master

Now the resynchronization from the survivor (SyncSource) to the victim (SyncTarget) starts immediately. There is no full sync initiated, but all modifications on the victim will be overwritten by the survivor’s data, and modifications on the survivor will be applied to the victim.

© Copyright 2023, NVIDIA. Last updated on Feb 8, 2024.