Issue/Introduction

This article covers a LARGE_OMAP_OBJECTS warning in ZStone 4.3.6 that occurs when an object storage bucket has too few shards and the omap key count on each shard exceeds the default threshold.

Symptoms

• The cluster reports an N large omap objects warning.
• ceph -s reports the LARGE_OMAP_OBJECTS warning.

Environment

• Product line: ZCF
• Component: ZStone
• Version: 4.3.6

Possible causes

The object storage bucket has too few shards. As the number of objects in the bucket grows, the omap key count on each shard exceeds the default threshold, osd_deep_scrub_large_omap_object_key_threshold=200000, and the warning is triggered during deep-scrub.

Resolution

1. Identify the affected bucket

• Commands:

  ceph health detail
  radosgw-admin bucket stats

• Check: Extract the bucket ID from the object name in the warning, and then confirm the corresponding bucket name.

2. Calculate the target shard count

• Method: Estimate one shard for every 100,000 objects and choose a prime number as the target shard count.

3. Back up the bucket index

• Command:

  radosgw-admin bi list --bucket=<bucket_name> > <bucket_name>.list.backup

• Check: Confirm that the original index data is written to the backup file.

4. Run reshard

• Command:

  radosgw-admin bucket reshard --bucket=<bucket_name> --num-shards=<num>

• Check: Run reshard with the target shard count calculated from the number of objects.

5. Check the reshard progress

• Command:

  radosgw-admin reshard status --bucket <bucket_name>

• Check: Confirm the reshard task status.

6. Wait for the warning to clear

• Expected result: After reshard completes, the warning is cleared after the next or several deep-scrub operations.
• Optional action: To mute the warning immediately, run:

  ceph health mute LARGE_OMAP_OBJECTS --sticky