Management node and browser times differ because of clock drift

Problem symptom

• The cloud platform time displayed in the Web UI does not match the user's browser time due to time drift on the management node.

Environment

• Product Line: ZCF
• Versions: Unspecified
• Components: ZStack Cloud
• Trigger Condition: Dual management node (Active-Standby) deployment with Global HA enabled.

How to confirm

1. Compare the time displayed in the top right corner of the ZStack Web UI with the current local browser time.
2. Run date on the management node and compare it against an external time authority.
3. Run chronyc tracking to check if chronyd is failing to sync with valid NTP peers.

Root cause

• The built-in NTP/time source on the management node is unreachable or unstable.
• Improper DNS configurations preventing chronyd from resolving external NTP hostnames.
• Hardware clock drift accumulating on long-running nodes without sync to system clock via hwclock -w.

Solution

1. Disable global HA in the Web UI

• Navigation: Go to Settings > Global settings > Global HA, and disable it.

2. Back up the database on both nodes

• Pre-check: Confirm both nodes are accessible.
• Execution commands (Run on each node individually):

  zstack-ctl dump_mysql --file-name zstack-db-backup-master   # Primary Node
  zstack-ctl dump_mysql --file-name zstack-db-backup-slave    # Standby Node

• Expected result: Database backup files are generated successfully without errors.

3. Stop the management node services on both nodes

• Execution commands (Run on both nodes):

  zsha2 stop-node

• Expected result: Management node services are stopped on both nodes by checking zsha2 status.

4. Update DNS and verify NTP hostname resolution

• Pre-check: Inspect the configured nameservers in /etc/resolv.conf.

• Execution commands:

  vim /etc/resolv.conf

• Recommended configuration (add the following entries):

  nameserver 114.114.114.114
  nameserver 8.8.8.8

• Verify Reachability:

  ping ntp1.aliyun.com

• Expected result: The hostname resolves successfully and pings to the Aliyun NTP server succeed.

5. Manually configure and sync one management node to Aliyun NTP

• Configuration file: /etc/chrony.conf on the designated node.
• Add or modify:

  server ntp1.aliyun.com iburst

• Restart the chronyd service:

  systemctl restart chronyd

6. Synchronize clocks and write to the hardware clock

• Pre-check: Ensure system time is consistent across all nodes using date, then sync system time to the hardware clock on both nodes:
• Execution (Run on both nodes):

  hwclock -w

• Expected result: Output from hwclock --show matches output from date.

7. Start management node services on both nodes

• Execution commands (Run on both nodes):

  zsha2 start-node

• Expected result: Both management nodes return to a normal state.

8. Verify host time

• Pre-check: Log in to a host that is not a management node and confirm that its date output matches the management nodes.

9. Re-enable global HA

• UI navigation: Go to Settings > Global settings > Global HA, and enable it again.

Verification

1. Run chronyc sources -v on the affected management node. The output should show that the external NTP server is selected and the offset is small (typically < 100 ms). After a node restarts, the system clock should remain correct because the hardware clock has been updated by hwclock --systohc.
2. Verify that the time in the top right corner of the Web UI matches browser time within a 1-second margin.
3. Execute hwclock --show to ensure hardware time matches system time.

Additional information

• Related commands/files: /etc/resolv.conf, /etc/chrony.conf, chronyc tracking, hwclock -w, zsha2 stop-node, zsha2 start-node, zstack-ctl dump_mysql.