Issue/Introduction

This article covers a persistent ceph -s clock skew warning after the cluster time is updated and synchronized in ZStone 4.4.27 (Cube).

Symptoms

• After the cluster time is updated, the system time of all nodes is synchronized.
• ceph -s continues to report a clock skew warning.

Environment

• Product line: ZCF
• Component: ZStone
• Version: 4.4.27

Possible causes

Ceph issues a warning when the clock skew between MON nodes exceeds 1 second. After the time is updated, the lease_expire time received by the standby MON from the active MON is earlier than the standby MON's current system time. This continuously triggers probably laggy (or possibly clocks are too skewed) messages and warnings. The issue does not affect storage operations.

Resolution

1. Check the clock skew warning threshold

• Command:

  ceph daemon mon.<hostname> config mon_clock_drift_allowed

• Check: Confirm the current MON clock skew warning threshold.

2. Check the node time and MON log

• Command:

  date

• Log path: /var/log/ceph/ceph-mon*.log
• Check: Confirm that all nodes' system times are synchronized and that the affected MON log contains many probably laggy (or possibly clocks are too skewed) messages.

3. Restart the affected MON service

• Command:

  systemctl restart ceph-mon@<hostname>

• Verification: Run ceph -s after the restart and confirm that the clock skew warning is cleared and the cluster status is HEALTH_OK.
• Impact: After the MON restart, the ZStack Cloud primary storage management page may briefly show a storage host reconnecting. The connection recovers automatically.