Rebuild PostgreSQL and Temporal on dual management nodes after a Temporal split-brain

Environment

• Product: ZCF.
• Component: ZStone
• Version: The source case recommends upgrading to version 4.2.2 or later to fully resolve the issue. Therefore, this article applies to recovery scenarios in versions earlier than 4.2.2 where Temporal split-brain has already occurred.
• Deployment mode: dual management nodes.
• Related services: zstone-master, temporal, bucardo, and postgresql-9.6.
• Related directories: /opt/zstone-installer/, /var/lib/pgsql/, and /var/run/bucardo.
• This can be used when Temporal split-brain has occurred in a dual-management-node environment and PostgreSQL data needs to be cleaned up before rebuilding Temporal-related databases and synchronization services.

Symptoms

• Temporal enters a split-brain state in a dual-management-node environment.
• Continuous PostgreSQL writes cause root partition usage to approach or reach 100%.
• After the root partition becomes full, the management node cannot start.

Diagnosis

1. Check root partition usage on both management nodes and confirm whether it approaches or reaches 100%.