Management node CPU usage is high after zops tasks become stuck

Problem symptom

The issue is caused by stuck zops (ZStack Ops) tasks that leave behind a large number of accumulated ansible-playbook worker processes.

Environment

• Product Line: ZCF
• Versions: Unspecified
• Components: Zstack Cloud
• Trigger Condition: Any ZStack Cloud deployment using zops to execute maintenance or operational tasks.

How to confirm

1. Use ps -ef | grep ansible-playbook | wc -l to view the count.
2. Initiate VM creation or other operational tasks in the ZStack Cloud UI to determine whether the operations are stuck.
3. Monitor top to confirm management node CPU usage and load average.

Root cause

• zopstask scheduling stuck: Tasks are no longer being consumed from the queue, each retry attempt spawns new ansible-playbook worker processes.
• A failing task persistently occupies the worker pool: The same failing task appears repeatedly in zops logs that leads to continuous resource consumption.
• Underlying resource contention: Contention on the management node (memory, I/O, or upstream service latency) prevents tasks from completing.

Solution

1. Confirm management node CPU usage and load average

• Run the following command on the affected management node:

  top

• Expected result: CPU usage exceeds 90%, and the load average reaches hundreds (e.g., above 200), confirming sustained CPU saturation.

2. Identify the dominant process family

• Count the number of ansible-playbook processes:

  ps -ef | grep ansible-playbook | grep -v grep | wc -l

or inspect process details:

  ps -ef | grep ansible-playbook

• Expected result: Returns hundreds or even thousands (e.g., 1,800+) of ansible-playbook processes.
• Interpretation: The continued accumulation of ansible-playbook workers indicates that the zops service is not consuming its task queue correctly.

3. Check zops task logs to locate repeatedly failing tasks

• Inspect the zops log directory (typically /var/log/zops/):

  tail -n 200 /var/log/zops/zops.log

• Expected result: A large volume of failed task logs that points to the same task or class of tasks. This task is the root cause consuming the worker pool.
• Record the failure timestamp and task name for post-incident analysis.

4. Restart the zops service

• Restart the service on the management node:

  systemctl restart zops

• Expected result: After zops restarts, the accumulated ansible-playbook processes exit automatically, and CPU utilization and load average rapidly return to the normal range.

5. Monitor the service status

• Periodically check the system load and process count after 1 hour of the restart:

  top
  ps -ef | grep ansible-playbook | wc -l

• Expected result: CPU usage stays normal (< 50%) and ansible-playbook processes remain low.

Verification

1. Confirm that ps -ef | grep ansible-playbook | wc -l returns a low, normal count.
2. Initiate VM creation or other operational tasks via the ZStack Cloud Web UI to verify that operations complete without hanging.
3. Monitor top to confirm management node CPU usage and load average remain normal.

Related information

• Related commands and tools: top, ps -ef, and systemctl restart zops.
• Related components: the zops task queue, the ansible-playbook worker pool, and management node resource monitoring.