Issue/Introduction

This article covers unknown data disk states, an inability to add new disks to the storage pool, and stale PGs in a degraded ZStone 4.2.8 storage pool after a power-cycle test.

Symptoms

• After the power-cycle test, all ZStone data disks are reported as unknown.
• The storage pool cannot accept new disks.
• The storage pool is degraded and has stale PGs.

Environment

• Product line: ZCF
• Component: ZStone
• Version: 4.2.8
• Scenario: Power-cycle test environment

Possible causes

The two issues have the following causes:

1. The Temporal service hangs after the power-cycle test, which prevents disk scans from running and data disk states from being refreshed.
2. In a two-replica environment, an operator removes one OSD data disk from each of two nodes, changes the replica count to 3, and then removes the two OSDs from the storage pool. Some data is lost and some PGs enter the stale state with incomplete data.

Resolution

1. Check the Ceph cluster and PG status

• Commands:

  ceph -s
  ceph health detail
  ceph pg <pg_id> query

• Check: Confirm the overall cluster status, detailed PG exceptions, and the cause of the exception for the affected PG.

2. Check the OSD log

• Log path:

  /var/log/ceph/ceph-osd.<osd_id>.log

• Check: Inspect the logs for the OSDs associated with the affected PG and confirm the fault state.

3. Restore data disk state updates

• Check: Confirm whether the Temporal service hangs after the power-cycle test.
• If abnormal: Restart the Temporal service to restore disk scans, and then confirm that the data disk states can be refreshed.

4. Handle the stale PGs

• Scope: Use the following options only in a test environment or when the data is confirmed to be unrecoverable.
• Options: Force a PG rebuild to restore cluster availability, or rebuild the Ceph pool and the upper-layer services.