Troubleshooting

Check UFM Status

Run the below command on the master node:

Copy
Copied!
            

/etc/init.d/ufmd status 

Check HA Status

Run the below command:

Copy
Copied!
            

ufm_ha_cluster status  pcs status 

Check DRBD Status

Run the below command:

Copy
Copied!
            

ufm_ha_cluster status 

Show DRBD Resource

Run the below command:

Copy
Copied!
            

drbdadm sh-resources 

Show DRBD Disk State

Run the below command:

Copy
Copied!
            

drbdadm dstate ha_data 

Show DRBD Role

Run the below command:

Copy
Copied!
            

drbdadm role ha_data 

Show DRBD Connectivity

Run the below command:

Copy
Copied!
            

drbdadm cstate ha_data 

Split-Brain Recovery

Step 1:

Manually choose a node which data modifications will be discarded.

It is called the split-brain victim. Choose wisely; all modifications will be lost! When in doubt, run a backup of the victim’s data before you continue.

When running a Pacemaker cluster, you can enable maintenance mode. If the split-brain victim is in the Primary role, bring down all applications using this resource. Now switch the victim to the Secondary role:

Copy
Copied!
            

victim# drbdadm secondary ha_data 

Step 2

Disconnect the resource if it’s in connection state WFConnection:

Copy
Copied!
            

victim# drbdadm disconnect ha_data 

Step 3

Force discard of all modifications on the split-brain victim:

Copy
Copied!
            

victim# drbdadm -- --discard-my-data connect resource 

For DRBD 8.4.x:

Copy
Copied!
            

victim# drbdadm connect --discard-my-data resource 

Step 4

Resync starts automatically if the survivor is in a WFConnection network state. If the split-brain survivor is still in a Standalone connection state, reconnect it:

Copy
Copied!
            

survivor# drbdadm connect resource 

Now the resynchronization from the survivor (SyncSource) to the victim (SyncTarget) starts immediately. There is no full sync initiated, but all modifications on the victim will be overwritten by the survivor’s data, and modifications on the survivor will be applied to the victim.

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.