Environment

• Product: ZStack Cloud on a dual-management-node (MN) HA deployment
• Version: Applies to all ZStack Cloud versions that use zsha2 for dual-MN HA.
• Component: ZStack Cloud management node, zsha2 dual-MN HA toolkit, MySQL database (zstack-ctl dump_mysql)
• Trigger/Condition: A power outage causes the ZStack MN service to enter an abnormal state, one MN becomes unreachable, the UI cannot be accessed, and the two MNs' databases drift out of sync.

Symptoms

• Customer environment lost power. After power was restored, one management node was offline and the UI could not be accessed.
• The secondary management node's ZStack service is in an abnormal state and the database has fallen out of sync between the two MNs.
• zsha2 status and zsha2 db-status show abnormal HA or database state.

Diagnosis

1. Run zsha2 status on both MNs to confirm the HA status and identify which node currently holds the VIP.
2. Run zsha2 db-status on both MNs to confirm that the databases are out of sync.
3. Check the management node logs for abnormal, demote, or sync error entries on the affected node to confirm that the failure is limited to one node.

Cause

A power outage can leave one MN's zsha2 and database services partially started. Once the database has drifted between the two MNs, neither side is authoritative. Demote the affected management node so that the healthy node becomes the authoritative source, and then initiate a one-way database synchronization from the healthy node.

Resolution

1. Prerequisites

• Before running any database operation, take a fresh MySQL backup on both management nodes. If the database is already showing "out of sync" symptoms, back up first before any zsha2 db-sync to avoid an unrecoverable state.
• Confirm that the management-node network (port 22 and the management-node cluster link) is healthy between the two MNs.

2. Procedure

1. Check dual-MN service and database state.

• On either MN:

     zsha2 status
     zsha2 db-status

2. Back up the database and stop the management service on both MNs.

• On each MN, dump MySQL:

     zstack-ctl dump_mysql

• Then stop the local node:

     zsha2 stop-node

3. Identify the current VIP holder.

• On either MN:

     zsha2 status

• Note which node reports VIP = yes. This node will keep the VIP.

4. Switch the VIP to the healthy node.

• On the MN that currently holds the VIP and is reporting errors, demote it:

     zsha2 demote

• This causes the VIP to move to the other, healthier MN.

5. Re-sync the database on the VIP holder.

• On the MN that now holds the VIP, run:

     zsha2 db-sync -p zstack.mysql.password

• Wait for the sync to complete before continuing.

6. Verify database status and start the dual-MN service.

• On the VIP node:

     zsha2 db-status
     zsha2 start-node

7. Final state check.

• On either MN:

     zsha2 status
     zsha2 db-status

• Both HA and database state should report healthy.

Verification

• zsha2 status shows both MNs in a healthy state with a single VIP holder.
• zsha2 db-status shows the databases in sync.
• The ZStack UI is reachable through the VIP, and you can log in successfully.
• VM workloads on the affected cluster are reachable and unaffected.

Risks and rollback

• Risk: zsha2 db-sync -p zstack.mysql.password overwrites one side's database. If the source node is the wrong one, this can destroy the most recent writes.
• Mitigation: always take zstack-ctl dump_mysql on both nodes before running db-sync. If db-sync is run on the wrong side, recover from the local dump.
• Rollback: restore MySQL from the pre-sync zstack-ctl dump_mysql backup that was taken on the losing side.