Issue

Use this article to troubleshoot arbitration-alarm false positives and rule-period or metric-definition issues.

Environment

• Product: ZCF
• Versions: 4.x, 5.x
• Component: Cloud
• Affected objects: arbitration alarms, management-node monitoring, and ZStack/MN metrics
• Scenario: A customer custom alarm rule differs from the default arbitration alarm rule.

Symptoms

• A custom arbitration alarm triggers approximately every 35 minutes.
• The default platform arbitration alarm does not trigger.
• Ping checks find no packet loss during the alarm.
• The custom rule uses a 30-minute alarm interval and a 5-minute duration.

Possible causes

• The custom alarm rule differs from the default rule.
• The alarm-interval and duration combination causes fixed-period triggering.
• No ping packet loss does not directly mean that the platform arbitration metric is normal.
• The metricName, threshold, or object scope does not match the default rule.

Resolution

1. Export or capture the custom arbitration-alarm rule, including metricName, threshold, duration, alarm interval, and object scope.
2. Compare it with the default platform arbitration-alarm rule.
3. During the alarm window, use GetMetricData to query arbitration-related metrics in the ZStack/MN namespace.
4. Analyze the relationship between the 30-minute alarm interval, 5-minute duration, and 35-minute triggering period.
5. Adjust the custom alarm according to the default rule and observe the result.

Key command

zstack-cli GetMetricData namespace=ZStack/MN metricName=<metricName>

Remediation

• Align the custom alarm rule with the default platform arbitration-alarm rule.
• Adjust the custom alarm threshold, duration, alarm interval, and object scope as recommended by the product.
• If the customer only wants to monitor network connectivity, explain that ping and the platform arbitration metric use different definitions.

Verification

• Observe at least two alarm cycles after adjustment.
• The trigger logic of the default and custom alarms can be explained.
• When no exception exists, the custom alarm no longer triggers on a fixed interval.

Risks and rollback

• Do not disable arbitration-related alarms based only on ping results.
• Changing alarm rules affects subsequent fault-detection capability and requires customer confirmation of the alarm strategy.

Notes

• Before adjusting management services, databases, certificates, permissions, auditing, or security settings, confirm the HA state, service window, rollback plan, and compliance-retention requirements.
• For management networks, storage networks, service networks, DHCP/VNC, VLANs, bonds, port groups, or switch-side settings, retain an available management channel and preferentially perform changes during a maintenance window.
• If database modifications, underlying file operations, lock release, service restarts, driver or firmware upgrades, or bulk resource changes are required, contact ZStack technical support before proceeding.