Scenario

• Product: ZCF
• Component: ZStack Cloud
• Version: 4.4.x
• Deployment: Dual management sharing hyperconverged nodes
• Trigger: Abnormal power loss or a power failure causes the two management-node databases to become desynchronized.

Symptoms

• The dual-management databases are not synchronized.
• Management services must stop briefly.
• Users cannot manage ZStack Cloud through the UI during synchronization, but online host and VM workloads are not affected.

Assessment

1. Back up the databases and disable high availability.
2. Use zsha2 db-status to identify the node with the healthy database.
3. Switch the VIP to the node with the correct database before synchronization.
4. Check database status and resource connections again after synchronization.

Cause

Abnormal power loss or a power failure desynchronizes the dual-management databases. Identify the correct database node first and run zsha2 db-sync only after switching the VIP to that node; otherwise, the abnormal state can spread.

Resolution

1. Preparation

• Risk level: Low.
• Back up databases, copy the backup offsite, and disable global high availability.

2. Back up databases and disable high availability
3. Back up the database on both management nodes:

      zstack-ctl dump_mysql --file-name beforSyncDB

• Expected result: A database backup is created before synchronization.

2. Copy the backup offsite:

      scp /var/lib/zstack/mysql-backup/beforSyncDB.gz x.x.x.x:/opt

• Expected result: An offsite rollback copy is retained.

3. Disable global high availability in the UI.

• Expected result: No additional platform action is triggered during database synchronization.

3. Stop dual management and switch the VIP
4. Stop management services:

      zsha2 stop-node

• Expected result: The environment enters the synchronization window.

2. Switch the VIP to the management node with the correct database:

      zsha2 demote

• Expected result: Synchronization runs on the correct node.

4. Synchronize databases
5. Check synchronization status:

      zsha2 db-status

• Expected result: The healthy database node is identified.

2. Synchronize databases:

      zsha2 db-sync -p <database_password>

Replace <database_password> with the actual database password.

• Expected result: Database synchronization completes.

3. Check synchronization status again:

      zsha2 db-status

• Expected result: Database states are consistent.

5. Restore dual-management services and high availability
6. Start services:

      zsha2 start-node

• Expected result: Management services are available.

2. Enable global high availability in the UI.

• Expected result: Normal platform protection is restored.

Verification

• Verify that zsha2 db-status reports normal synchronization.
• Verify normal dual-management services.
• Verify resource connections through VIP:5000.
• Verify normal online workloads.

Risks and notes

• Risk: Management services stop during synchronization and users cannot manage the platform through the UI.
• Risk: Synchronizing before identifying the correct database node can expand the fault.
• Risk level: Low.

Rollback

If synchronization fails, or resources cannot connect or are missing after synchronization:

1. Stop management services:

   zstack-ctl stop

2. Restore the previous backup on both management nodes:

   zstack-ctl restore_mysql -f /root/beforSyncDB --mysql-root-password <database_root_password>

3. Start management nodes:

   zstack-ctl start

4. Log in through VIP:5000 and check resource connections and workloads.

• Expected result: The database returns to its pre-synchronization state.