Environment

• Product: ZCF.
• Component: ZStone
• Version: The source case recommends upgrading to version 4.2.2 or later to fully resolve the issue. Therefore, this article applies to recovery scenarios in versions earlier than 4.2.2 where Temporal split-brain has already occurred.
• Deployment mode: dual management nodes.
• Related services: zstone-master, temporal, bucardo, and postgresql-9.6.
• Related directories: /opt/zstone-installer/, /var/lib/pgsql/, and /var/run/bucardo.
• This can be used when Temporal split-brain has occurred in a dual-management-node environment and PostgreSQL data needs to be cleaned up before rebuilding Temporal-related databases and synchronization services.

Symptoms

• Temporal enters a split-brain state in a dual-management-node environment.
• Continuous PostgreSQL writes cause root partition usage to approach or reach 100%.
• After the root partition becomes full, the management node cannot start.

Diagnosis

1. Check root partition usage on both management nodes and confirm whether it approaches or reaches 100%.
2. Check the temporal and zstone-master service states and confirm whether both nodes are abnormal.
3. Run df -h and inspect the largest directories under the root directory to rule out log growth, image cache, or another cause of root partition exhaustion.

Root cause

After Temporal split-brain occurs, PostgreSQL keeps writing abnormal data. The writes might eventually fill the root partition and prevent the management node from starting. In the source case, the recovery method is to stop zstone-master, temporal, and bucardo on both management nodes, clean up PostgreSQL data, and then reinstall and initialize PostgreSQL, Bucardo, and the Temporal-related databases.

Resolution

1. Stop zstone-master, Temporal, and Bucardo on both management nodes

Run the following commands on both management nodes to stop zstone-master, temporal, and bucardo, and disable Bucardo from starting automatically.

systemctl stop zstone-master
systemctl stop temporal
mkdir /var/run/bucardo
pkill -9 bucardo
sed -i "/^bucardo/d" /etc/rc.local
bucardo stop

If Bucardo cannot be stopped normally, you can clean up the Bucardo runtime directory based on the site condition:

rm -rf /var/run/bucardo

2. Uninstall PostgreSQL and delete the pgsql directory on both management nodes

Run the following commands on both management nodes:

systemctl stop postgresql-9.6
yum remove -y "postgresql*"
rm -fr /var/lib/pgsql/
systemctl -a | grep postgresql

Confirm that PostgreSQL has stopped and that related services are no longer running.

3. Reinstall PostgreSQL and Bucardo, and create the database required by Temporal

Run the following commands on both management nodes.

Replace [IP] with the management network IP address of the current node.

cd /opt/zstone-installer/
yum localinstall -y ./pkgs/pg-rpms/*.rpm
./install.sh cmd setup_postgresql

cd /opt/zstone-installer/
./install.sh cmd setup_bucardo bucardo bucardo bucardobucardo [IP] 5432 -D
bucardo status

./install.sh cmd setup_temporal_db /opt/zstone-installer/pkgs/temporal/temporal_home/ [IP] 5432 bucardo bucardobucardo

cd /var/lib/pgsql
sudo -u postgres -H -- psql -c "\l"

Confirm that the databases required by Temporal have been created.

4. Enable Bucardo synchronization and Temporal on both management nodes

Run the following commands on both management nodes:

cd /opt/zstone-installer/
./install.sh cmd setup_bucardo_sync
./install.sh cmd setup_temporal

If setup_bucardo_sync reports an error on one of the nodes, you can run the following command on that node:

bucardo stop

Then, you should run the synchronization setup command again:

./install.sh cmd setup_bucardo_sync

Confirm that the synchronization status is normal on both management nodes.

5. Register the Temporal namespace

Run the following commands on one management node only:

cd /opt/zstone-installer/
./install.sh cmd setup_temporal_namespace
tctl namespace list

Confirm that the Temporal namespace has been registered successfully.

6. Start zstone-master on both management nodes

Run the following commands on both management nodes:

systemctl start zstone-master
systemctl status temporal
systemctl status zstone-master

Confirm that both temporal and zstone-master are running normally.

Verification

After the recovery, verify the following:

• The temporal service is running normally on both management nodes.
• The zstone-master service is running normally on both management nodes.
• bucardo status shows a normal synchronization status.
• tctl namespace list returns namespace information normally.
• ZStone-related business functions are restored.

Risk

• This method only cleans up PostgreSQL data after Temporal split-brain has already occurred. It does not prevent Temporal split-brain from occurring again.
• To fully resolve the issue, the source case recommends upgrading to version 4.2.2 or later.
• The source case mentions a reference method for preventing Temporal split-brain in version 4.2.2, but it is not recommended.
• Before deleting /var/lib/pgsql/, confirm whether PostgreSQL data needs to be backed up at the site.
• After deleting the PostgreSQL data directory, rollback is possible only from a usable database dump created before the operation. Do not delete the directory if the backup is not usable.