Environment

• Product: ZCF
• Component: ZCE-S
• Affected objects: RGW, object gateways, and rgw_bucket_quota_ttl
• Version: 6.3.205
• Trigger condition: In a multi-RGW environment, the user quota cache may cause total_size_b to return a stale value or 0, requiring an RGW quota cache adjustment.

Symptoms

• The current version updates the user quota cache through asynchronous tasks.
• In a multi-RGW environment, the cache may intermittently cause total_size_b to return a stale value or 0.
• Set rgw_bucket_quota_ttl to 0 so that every request queries the quota in real time.

Diagnosis

1. Confirm that the environment is within scope and record versions, system architecture, operating system, deployment mode, and storage type.
2. Run ceph -s and confirm HEALTH_OK or HEALTH_WARN with warnings that do not affect the operation.
3. Check whether rgw_bucket_quota_ttl is already configured in ceph.conf on the RGW nodes.
4. After configuration, verify the value with ceph daemon ... config show | grep rgw_bucket_quota_ttl.

Cause

User quota caching is delayed in a multi-RGW environment and may return a stale value or 0. Setting rgw_bucket_quota_ttl to 0 queries the quota for every request and eliminates cache inconsistency, but adds 1 I/O operation to each write request.

Resolution

1. Prerequisites

• Risk level: Medium
• Avoid affecting existing services throughout the change.
• Confirm cluster health and back up the configuration on every RGW node.

2. Procedure

1. Confirm cluster status.

   ceph -s

• Expected result: Status is HEALTH_OK or HEALTH_WARN, and warnings do not affect the operation.

2. Back up the configuration on every RGW node.

   cp /var/lib/ceph/etc/ceph/ceph.conf /root

• Expected result: ceph.conf is backed up.

3. Log in to every RGW node and edit the configuration.

   vi /var/lib/ceph/etc/ceph/ceph.conf

• View RGW nodes from object gateway management on the port 6680 UI.
• Expected result: The configuration is open for editing.

4. Add the parameter under [global] or the corresponding RGW instance section.

   rgw_bucket_quota_ttl = 0

• Expected result: The quota cache time is 0.

5. Restart RGW on every RGW node.

   systemctl restart radosgw@<rgw-name>.service
   systemctl status radosgw@<rgw-name>.service

• Replace <rgw-name> with the actual instance name; tab completion is available.
• Expected result: Status is active (running).

6. Verify that the parameter is effective.

   ceph daemon /var/lib/ceph/var/run/ceph/ceph-client.radosgw.rgw3.asok config show | grep rgw_bucket_quota_ttl

• Expected result: The output includes rgw_bucket_quota_ttl = 0.

7. Verify business functions with a small number of object uploads and downloads.

• Expected result: Business functions operate normally.

Verification

• Run ceph -s and confirm normal cluster status.
• Confirm active (running) from systemctl status radosgw@<rgw-name>.service.
• Confirm that rgw_bucket_quota_ttl = 0 is effective.
• Upload and download a small number of objects and confirm normal access.

Risks and rollback

• Risk: Each write request incurs 1 additional I/O operation, slightly affecting object write performance.
• Risk level: Medium.
• Rollback: Restore the backed-up ceph.conf and restart RGW services.

Exception handling recommendations

If an abnormality occurs, stop subsequent operations immediately and restore the backed-up configuration as specified by the rollback plan.