Applicable Scenario

• Product: ZCF
• Components: ZCE-X and ZStack Cloud
• ZCE-X version: 5.2.x
• ZStack Cloud version: 4.8.x
• Operating system/repository environment: 76C
• Trigger condition: the environment time zone must be changed from <source_timezone> to <target_timezone>.

Symptoms

• The time zone needs to be changed, for example from +8:00 to +7:00.
• The time change is substantial and requires a workload outage.
• The change affects system time as well as ZStack Cloud and distributed-storage service startup and shutdown.

Diagnostic Procedure

1. Confirm that workloads can be stopped before the change.
2. After changing the time zone, verify it with timedatectl and date.
3. After storage recovery, check ceph -s; after platform recovery, check VM and workload status.

Cause

A substantial time jump cannot be applied online. Put VMs, platform high availability, and the storage recovery path into a safe state before changing the time zone on all hosts, then restore storage and management services in stages.

Resolution

1. Preparation

• Risk level: medium.
• Stop workloads before the change.
• Back up the database.
• Disable ZStack Cloud high availability and shut down all VMs.

2. Stop workloads and switch the platform protection state
3. Back up the database.

• Expected result: a rollback basis is available.

2. Log in to ZStack Cloud:

• Disable ZStack Cloud high availability.
• Shut down all VMs.
• Put hosts into maintenance mode.
• Expected result: the management and workload planes are ready for the change.

3. Log in to distributed storage:

• Disable data recovery.
• Expected result: automatic recovery is not triggered during the time-zone change.

3. Stop critical storage services
4. Stop xmsd on all nodes:

   systemctl stop xmsd

• Expected result: storage management services are stopped.

2. Stop ceph-mon.target on all MON nodes:

   systemctl stop ceph-mon.target

• Expected result: monitor services are stopped.

3. Stop ceph-osd.target on all storage nodes:

   systemctl stop ceph-osd.target

• Expected result: storage data-plane services are stopped.

4. Change the time zone to <target_timezone>
5. Run on every host:

   timedatectl set-timezone <target_timezone>

• Expected result: the time zone is changed to Jakarta.

2. Check:

   timedatectl
   date

• Expected result: the time zone and system time are correct.

5. Restore storage and upgrade the front-end package
6. Start xmsd:

   systemctl start xmsd

• Expected result: storage management services are running again.

2. Check storage status:

   ceph -s

• Expected result: storage is healthy.

3. Upgrade the front-end package on every node:

   rpm -Uvh ZCE-XSCALEROS_5.2.115.6-1.g46f9e56.el7.centos.noarch.rpm --force --nodeps

• Expected result: the front-end package is upgraded.

4. Log in to distributed storage and enable data recovery.

• Expected result: the storage recovery path is enabled again.

6. Restore management nodes and workloads
7. Restart management-node services and the database:

   zsha2 stop-node
   systemctl restart mariadb
   zsha2 start-node

• Expected result: the management plane is restored.

2. Log in to ZStack Cloud:

• Enable hosts.
• Start VMs.
• Check workloads.
• Expected result: workloads are restored.

Verification

• Verify that all hosts use <target_timezone>.
• Verify that ceph -s is healthy.
• Verify that management-node and database services are restored.
• Verify that VMs start and workloads are reachable.

Risks and Notes

• Risk: this change requires a workload outage, and the time jump directly affects service continuity.
• Risk: failing to stop storage services in order and disable data recovery can cause additional issues.
• Risk level: medium.

Rollback

1. Stop workloads again.
2. Stop xmsd, ceph-mon.target, and ceph-osd.target again.
3. Change the time zone on every host except management nodes back to the original value:

   timedatectl set-timezone <source_timezone>
   timedatectl
   date

4. Start xmsd, then check:

   ceph -s

5. Restart management-node services and the database:

   zsha2 stop-node
   systemctl restart mariadb
   zsha2 start-node

6. Enable hosts, start VMs, and check workload recovery.

• Expected result: the platform returns to the original time zone.