Issue

Use this article to troubleshoot management-node directory-usage alarms and inconsistent dual-management-node alarm rules.

Environment

• Product: ZCF
• Versions: 4.x, 5.x
• Component: Cloud
• Affected objects: management nodes, alarm rules, monitoring metrics, and dual management nodes
• Scenario: A management-node directory-usage alarm is inconsistent with system-level df/du results.

Symptoms

• Management-node data-directory disk utilization triggers an alarm every half hour.
• The alarm reports directory utilization above 90%.
• Real-time inspection of /var/lib/zstack/* does not find obvious abnormal growth.
• Alarm rules can differ between the two nodes in a dual-management-node environment.

Possible causes

• Alarm rules in the in-memory state of the two management nodes are inconsistent.
• The monitoring metric and real-time system directory usage differ because of collection delay or definition differences.
• An old alarm rule remains or is not synchronized, causing periodic false positives.
• Directory utilization actually exceeds the threshold momentarily, but the observation window does not cover the peak.

Resolution

1. Review alarm information and confirm the alarm object, directory path, threshold, trigger cycle, and triggering management node.
2. Use GetMetricData to obtain the metric used by the alarm and confirm whether it exceeds 90%.
3. Cross-check actual system usage with df/du/watch.
4. Use DebugSignal or an equivalent method to obtain alarm rules in management-node memory.
5. Compare alarm rules on both management nodes and confirm whether they are consistent.

Key command

df -h
du -sh /var/lib/zstack/* 2>/dev/null
watch -n 1 -d 'du -sh /var/lib/zstack/* 2>/dev/null'
zstack-cli GetMetricData namespace=<namespace> metricName=<metricName>

Remediation

• If the in-memory rules on the two management nodes are confirmed inconsistent, restart the related management services or reload alarm rules as recommended by ZStack technical support.
• If actual directory usage is confirmed, move to the management-node disk-space cleanup procedure.
• If rule residue is present, clear or recreate the corresponding alarm and then observe the result.

Verification

• In-memory alarm rules on both management nodes are consistent.
• GetMetricData metric values and system-level df/du readings can be explained using the same definition.
• No false alarm occurs after at least one alarm cycle.

Risks and rollback

• Do not delete files under /var/lib/zstack merely because an alarm is suspected to be false.
• Before restarting management services, confirm management-node HA, task queues, and service impact.

Notes

• Before adjusting management services, databases, certificates, permissions, auditing, or security settings, confirm the HA state, service window, rollback plan, and compliance-retention requirements.
• If database modifications, underlying file operations, lock release, service restarts, driver or firmware upgrades, or bulk resource changes are required, contact ZStack technical support before proceeding.