Environment

• Product: ZStack Cloud on a dual-management-node (MN) HA deployment
• Version: 4.2.x~4.8.x.
• Component: chronyd, /etc/chrony.conf, and zstack.properties.
• Trigger/Condition: The platform time source (NTP) needs to be changed to a new external time server.

Symptoms

• The customer needs to change the time source for the ZStack Cloud platform.
• After a planned time source change, hosts and management nodes must keep synchronized time so that VM scheduling, log correlation, and HA decisions remain consistent.

Diagnosis

1. Confirm that the platform and all storage pools are healthy before maintenance.
2. Confirm that no backups or other long-running tasks are in progress before restarting MN services.
3. Identify the current primary MN IP — this will be the value used in chrony.serverIp.0.

Cause

The management nodes provide time synchronization for the platform. The platform uses chronyd on each management node, and chrony.serverIp.0 in zstack.properties specifies the MN with which the agents on hosts synchronize. To switch the time source, update /etc/chrony.conf on the primary MN and chrony.serverIp.0 on both MNs, and then reconnect the hosts so that they load the new configuration.

Resolution

1. Prerequisites

• Disable global VM HA before starting the change.

• Back up the ZStack MySQL database on both MNs:

  # master node
  zstack-ctl dump_mysql --file-name zstack-db-backup-master
  # slave node
  zstack-ctl dump_mysql --file-name zstack-db-backup-slave

• Confirm the new NTP server address and that it is reachable from the management network.

2. Procedure

1. List hosts.

   zstack-cli QueryHost hypervisorType=KVM fields=managementIp | grep managementIp | awk -F '"' '{print $4}'

2. Inspect the current chrony source on each host.

   for i in `zstack-cli QueryHost | grep man | awk -F "\"" '{print $4}'`; do
     ssh -i /usr/local/zstack/apache-tomcat-8.5.87/webapps/zstack/WEB-INF/classes/ansible/rsaKeys/id_rsa $i \
       "echo $i; chronyc sources -v | tail -n2"
   done

3. Configure the new NTP source on the primary management node.

   cp /etc/chrony.conf /root/chrony.conf.bak
   vi /etc/chrony.conf
   # add or update:
   server <new NTP time source> iburst

4. Update chrony.serverIp.0 on both management nodes.

   cp /usr/local/zstack/apache-tomcat/webapps/zstack/WEB-INF/classes/zstack.properties \
      /root/zstack.properties.bak
   vi /usr/local/zstack/apache-tomcat/webapps/zstack/WEB-INF/classes/zstack.properties
   # set:
   chrony.serverIp.0=<IP of the primary management node in the cluster>

5. Restart the dual-MN services and verify the configuration.

• Confirm that no backup or other task is running on the platform.
• Restart dual-MN services:

     zsha2 stop-node; zsha2 start-node

• Verify the new configuration is in effect:

     zstack-ctl show_configuration | grep -i chrony

6. Reconnect hosts.

• In the UI, reconnect each host individually so that it loads the new time source from the updated zstack.properties.

7. Restart the chronyd service on each host.

Add the management IP address of every compute host in the current environment to host_ip.txt, one address per line.

   for i in `cat host_ip.txt`; do
     ssh -i /usr/local/zstack/apache-tomcat-8.5.87/webapps/zstack/WEB-INF/classes/ansible/rsaKeys/id_rsa $i \
       "echo $i; systemctl restart chronyd"
   done

8. Verify node time sync.

   chronyc sources -v

The output must show * for the selected time source on each node.

9. Verify the Ceph cluster is healthy, then write the time to hardware clock on every node.

   for i in `zstack-cli QueryHost | grep man | awk -F "\"" '{print $4}'`; do
     ssh -i /usr/local/zstack/apache-tomcat-8.5.87/webapps/zstack/WEB-INF/classes/ansible/rsaKeys/id_rsa $i \
       "echo $i; hwclock -w"
   done

10. Re-enable global VM HA.

• In the UI, turn global VM HA back on.

Verification

• chronyc sources -v on each MN and host shows * next to the new NTP server.
• zstack-ctl show_configuration | grep -i chrony shows the new chrony.serverIp.0 value on both MNs.
• The Ceph cluster health check (UI) reports healthy.
• VM workloads and HA policy behave normally after global HA is re-enabled.

Risks and rollback

• Risk: During the change, management-node services are restarted, so the UI is briefly unavailable.
• Workload time risk: Changing a host's time might also change the time in its VM instances and affect workloads that depend on system time.
• Compatibility: Both MNs must be updated with the same chrony.serverIp.0 value (the primary MN's IP).
• Rollback: Restore /etc/chrony.conf from /root/chrony.conf.bak and zstack.properties from /root/zstack.properties.bak; restart chronyd and the MN services.