Scenario

• Product: ZCF
• Components: ZStack Cloud, ZStone
• ZStack Cloud version: 4.8.x
• ZStone version: 5.2.x
• Operating system: h84r
• Deployment: Three-node hyperconverged cluster with dual management nodes
• Trigger: A management node has a RAID controller failure and must be reinstalled and restored as a management and storage node.

Symptoms

• One management node in a hyperconverged cluster has a RAID controller failure.
• The node provides management services and is associated with mon, OSD, and cache resources.
• The legacy node must be removed before it is reinstalled and managed again.

Assessment

1. Confirm that the failed node requires system reinstallation.
2. After reinstallation, verify that ZStack Cloud, ZStone, high availability, and storage roles are restored.
3. Verify that platform and storage status are normal after the node rejoins the environment.

Cause

A RAID controller failure on a management node requires full reinstallation, role reconstruction, and dual-management reconstruction.

Resolution

1. Preparation

• Risk level: Medium.
• Disable platform high availability and storage data recovery.
• Back up ZStack Cloud and ZStone databases.
• Remove resources associated with the failed node in the UI and storage platform.

2. Remove legacy resources from the failed node.
3. Disable global high availability and data recovery.
4. Back up the databases:

      zstack-ctl dump_mysql
      mysqldump -uroot -p<database_password> zstone > /var/lib/zstack/mysql-backup/zstone-backup.sql

• Expected result: ZStack Cloud and storage databases are backed up.

3. In the ZStack Cloud UI, remove the mon node and compute node that require changes.

• Expected result: The platform no longer references the failed node.

4. In the storage platform, remove the corresponding OSDs and cache partitions, then delete the node. If the UI cannot delete the resources, use the following APIs:

• DELETE /api/v1/block/osd
• DELETE /api/v1/block/host
• Expected result: Storage records for the failed node are cleared.

5. Purge down OSDs on the underlying system:

      ceph osd --force purge osd.X

• Expected result: Residual OSD records are removed.

6. Delete the abnormal license record from the database on a healthy node:

      mysql -uroot -p<database_password> zstone
      delete from LicenseAppIdVO where hostIp='<failed_node_management_ip>';

• Expected result: The legacy node license binding is removed.

7. Reload the license in the UI, then back up the database again.

• Expected result: The environment is ready for reinstallation.

3. Reinstall the failed node.
4. Reinstall the operating system and configure networking.

• ISO: ZStack-x86_64-DVD-4.8.18-h84r.iso
• br_bond0: <management_network_cidr>
• br_bond2: <storage_network_cidr>
• Expected result: Baseline networking is restored.

2. Configure server time and write it to the hardware clock.

• Expected result: Time synchronization is normal.

3. Install Cloud management services and apply for a temporary license.

• Expected result: ZStack Cloud management functions are restored.

4. Change the database password:

      zstack-ctl change_mysql_password --root-password <current_database_root_password> --user-name root --new-password <new_database_root_password>

• Expected result: The database password matches the existing environment.

5. Update zstack.properties.

• Expected result: ZStack Cloud configuration points to the correct environment.

6. Reinstall ZStone 5.2.x on the failed node, with the configuration file containing only the failed-node address.

Important: Install storage before installing high availability on the healthy node; otherwise, the ZStone database might be overwritten.

• Expected result: ZStone services on the failed node are restored.

4. Reconstruct dual management and high availability.
5. On the surviving management node, install high availability:

      tar zxvf ZStack-Multinode-HA-Suite.tar.gz
      chmod +x zsha2 zstack-hamoo
      ./zsha2 sample-config > zs-install.config

Configure gateway, virtualIp, myIp, peerIp, dbRootPass, interface, and timeServer.

• Expected result: Dual-management HA configuration is ready.

2. Rewrite ZStone configuration on the failed node to use the VIP.

• Expected result: Dual-management configuration is consistent.

3. Restart zstone-master, zstnmon, zstone-ui, temporal-history / frontend / matching / worker / web, postgresql-9.6.x, zstone-prometheus, zstone-alertmanager, zstone-pushgateway, thanos-query, and thanos-sidecar.

• Expected result: Management and monitoring services are restored.

5. Restore management.
6. Add the original node, cache disks, and data disks in the storage UI.
7. Add OSDs and enable data recovery.
8. Add the corresponding node, mon node, and compute node in ZStack Cloud.
9. Re-enable high availability.

• Expected result: The failed node is managed again and dual management is restored.

Verification

• Verify ZStack Cloud and ZStone database recovery.
• Verify networking, time, licensing, and management services on the reinstalled node.
• Verify cache disks, data disks, OSDs, and mon nodes in the storage UI.
• Verify dual-management high availability.

Risks and rollback

• Risk: Incorrect sequencing during management-node reconstruction and storage-role reconstruction can affect ZStack Cloud and ZStone data consistency.
• Risk: Installing high availability before storage can overwrite the ZStone database.
• Risk level: Medium.
• Rebuilding the services has no rollback path. Resolve issues item by item based on the reported error; do not attempt a standard rollback.