Resynchronizing a dual-MN database and upgrading the zsha2 HA suite

Environment

• Product Line: ZCF
• Version: 4.8.x
• Components: ZStack Cloud, zsha2, and MariaDB.
• Trigger Condition: MariaDB databases on the two management nodes become out of sync, requiring the zsha2 HA suite to be restored to a consistent state prior to upgrade.

Symptoms

This document describes the recovery procedure for MariaDB database out-of-sync issues between two management nodes, as well as the solution steps to upgrade and restore the zsha2 multi-node HA suite to a consistent state.

Diagnosis

1. Run zsha2 status, zsha2 db-status, and ceph -s on both management nodes to compare status between them.
2. Check zsha2 db-status on both nodes to confirm that the standby node is no longer continuously syncing the database.

Resolution

1. Preparation

• Disable 'Global HA' in the ZStack Cloud Web UI prior to performing recovery operations.

Figure 1: Disable Global HA in the ZStack Cloud Console (highlighted in red)

• Confirm that the new HA suite compressed archive is available and its MD5 checksum has been verified.

2. Database Synchronization

1. Check HA service status, database status, and storage status of both management nodes:

   zsha2 status
   zsha2 db-status
   ceph -s

• Expected Result:zsha2 status output' Zstack HA Monitor: inactive', then proceed to next step.
• Expected Result:zsha2 db-status output 'Slave_SQL_Running: No', means that the database is not synced then proceed to next step.

2. Back up the database of the two management nodes:

   zstack-ctl dump_mysql --file-name zstack-db-backup-master          (Primary Management Node)
   zstack-ctl dump_mysql --file-name zstack-db-backup-slave           (Secondary Management Node)

3. Stop the zsha2 service on both management nodes:

   zsha2 stop-node

4. Perform database synchronization on the node that is currently holding the VIP:

   zsha2 db-sync -p '<zstack-db-password>'

5. Check the database status and confirm the slave node database has returned to normal state, otherwise do not proceed:

   zsha2 db-status

• Expected Result:zsha2 db-status output 'Slave_SQL_Running: Yes'

6. Switch the VIP to another node:

   zsha2 demote

7. Perform the database synchronization again on the original slave node (which has now been promoted to the source):

   zsha2 db-sync -p '<zstack-db-password>'

8. Check the database synchronization status again:

   zsha2 db-status

• Expected Result:zsha2 db-status output 'Slave_SQL_Running: Yes'

9. Reboot the zsha2 of the two management nodes:

   zsha2 start-node

10. Verify the HA service, database synchronization and Ceph status. If the statuses are healthy, re-enable the global HA in the UI:

    zsha2 db-status
    zsha2 status
    ceph -s

• Expected Result:zsha2 status output' Zstack HA Monitor: Active'.
• Expected Result:zsha2 db-status output 'Slave_SQL_Running: Yes'

3. HA Upgrade Procedure

1. Perform a database backup again before the upgrade:

    zstack-ctl dump_mysql --file-name zstack-db-backup-master-preupgrade    (Primary Management Node)
    zstack-ctl dump_mysql --file-name zstack-db-backup-slave-preupgrade     (Secondary Management Node)

2. Check the current HA package version, and then extract the new archive on the VIP node:

    zsha2 version
    tar -zxvf Multinode-HA-Suite-zsha2_<version>.tar.gz

• Expected Result:zsha2 version showed: 'version 4.6.0.0, commit 5c8dbf4eaceab321e6dd1ef2cbd6853ab80e7931'
• Expected Result:It will extract the HA file to two separate files as zsha2 and zstack-hamon

3. Grant execution permissions:

    chmod +X zsha2 zstack-hamon

4. Execute the HA upgrade:

    ./zsha2 upgrade-ha -gencfg=true

5. Verify the upgraded version:

    zsha2 version

Verification

• Execute 'zsha2 status' to confirm HA status is healthy, with one node marked as VIP= 'yes' and the other as standby.
• Execute 'zsha2 db-status' to verify that standby database synchronization is normal.
• Execute ceph -s to confirm it returns HEALTH_OK.
• After re-enabling Global HA, verify that VM workloads and platform data remain continuously accessible.

Figure 2: Re-enable Global HA and confirm normal Management Node health status (highlighted in red)

• Execute 'zsha2 version' to confirm it reflects the new HA suite version.

Cause

Data divergence has occurred between the MariaDB databases on the two management nodes that leads to an abnormal HA status. The database must be manually synchronized at the underlying level to restore consistency before upgrading the HA suite itself.

Risks and rollback

This procedure must be executed in a dual-management-node HA deployment environment. The management UI will be temporarily unavailable during operation, and the cloud platform will trigger a failover. Perform this operation during a customer-approved maintenance window, and verify that failover and service recovery are clean before declaring success.