Scenario

• Product: ZCF
• Component: ZStone
• Version: 4.2.x
• Trigger: Cache-disk data usage is high and affects cloud-volume write throughput, so cached data must be flushed to disks.

Symptoms

• Cache-disk data is highly accumulated in a ZStone 4.2.x environment.
• Cloud-volume write throughput is affected.
• A flush must be performed on storage hosts to write cached data to disks.

Assessment

1. Check the storage version and ZStone service status.
2. Monitor ceph -s continuously during the flush.
3. Confirm that the dirty value decreases gradually.

Cause

The issue is high cache-disk data usage rather than a control-plane failure. cas_policy_manager.sh is used to trigger cache flushing and write accumulated data to disks.

Resolution

1. Preparation

• Risk level: Medium.
• Stop all VM workloads.
• Put primary storage into maintenance mode.
• Back up the database and installer directory.

2. Health check and backup
3. Check the storage version:

      /opt/zstone/bin/zstone-master version

• Expected result: The version is 4.2.x.

2. Back up the ZStone database:

      mysqldump -uroot -p<database_password> zstone > /var/lib/zstack/mysql-backup/zstone-preupgrade-backup.sql

• Expected result: A database rollback baseline is available.

3. Back up the installer directory:

      cp -r /opt/zstone-installer/ /root/zstone-installer-bk/

• Expected result: An installer snapshot is retained for upgrade or rollback support.

4. Check workload and control resources.

• Confirm normal workload operation.
• Access port 4000 and confirm that storage resource information and status are expected.
• Expected result: The storage control plane is healthy.

5. Check management services:

      systemctl status zstone-master.service
      systemctl status zstone-ui.service
      systemctl status zstone-alertmanager.service
      systemctl status zstone-prometheus.service
      systemctl status zstone-pushgateway.service
      systemctl status keepalived.service

• Expected result: Related services are active/running.

6. Confirm that no asynchronous task is running.

• Expected result: The flush does not conflict with other control tasks.

3. Flush cache data
4. Upload the following script to every storage node:

      /root/cas_policy_manager.sh

• Expected result: The flush script is executable.

2. Start flushing:

      /root/cas_policy_manager.sh start_acp

• Expected result: Cache flushing starts.

3. Monitor continuously:

      ceph -s

Observe whether the dirty value decreases gradually.

• Expected result: Cached data is progressively flushed to disks.

4. Stop the task after flushing finishes:

      /root/cas_policy_manager.sh stop_acp

• Expected result: The flush task is closed out.

Verification

• Verify normal ceph -s output.
• Verify that dirty decreases gradually to the expected level.
• Verify that cloud-volume write throughput returns to normal.

Risks and rollback

• Risk: If workloads are not stopped or primary storage is not in maintenance mode, flushing can amplify workload fluctuations.
• Risk: If the state differs from expectations during flushing, stop the current operation immediately.
• Risk level: Medium.
• Preventive measure: Stop the operation immediately and analyze the issue if the outcome differs from the planned change.