Applicable scenario

• Product: ZCF
• Component: ZCE-X
• Versions: 4.2.x, 5.2.x
• Current storage version: 4.2.x
• Target storage version: 5.2.x
• Intermediate version: 4.2.x.*
• Operating system/architecture: C76 / X86
• Trigger: An online storage upgrade is required, or 4.2.x rocksdb exceptions cause a MON node to go down or dual-management database desynchronization that later self-recovers.

Symptoms

• A 4.2.x environment cannot be upgraded directly to 5.2.x.
• The 4.2.x.* intermediate version is required.
• The upgrade restarts OSDs and related services one by one and can cause storage-performance fluctuations.
• Confirmed 4.2.x rocksdb exceptions can cause a MON node to go down or dual-management database desynchronization that later self-recovers.

Determination method

1. Before upgrade, check node resources, storage-pool health, and ceph/etcd/ES state.
2. After each upgrade, restart xdc one server at a time and observe cluster recovery.
3. Confirm automatic rebalancing is switched correctly before and after upgrade.

Cause

The valid upgrade path is 4.2.x -> 4.2.x.* -> 5.2.x. V4-to-V5 upgrading involves a temporary license, thin-provisioning adjustments, and a formal license request, so cross-version direct upgrade is not supported. For confirmed rocksdb exceptions, upgrade to 5.2.x avoids MON-down and database-desynchronization issues.

Resolution

1. Preparation

• Risk level: high.
• Obtain a temporary v4-compatible v5 license, then request a formal license after upgrading to 5.2.x.
• Disable global VM HA.
• For overnight upgrades, disable automatic rebalancing first.
• Shut down service VMs during the maintenance window and put primary storage into maintenance mode.

2. Pre-upgrade health checks
3. Check all nodes:

• CPU and memory below 80%
• System-volume usage below 70%
• At least 50GB+ free on /opt and /tmp
• At least 30GB+ free on root
• Passwordless ssh from management nodes to compute nodes
• Expected result: All basic resources meet upgrade conditions.

2. Check that storage pools are healthy.

• Expected result: Restore health before upgrading if unhealthy.

3. Check the ceph/etcd/ES clusters:

• storage health is HEALTH_OK
• etcd cluster is healthy
• ES is green
• Expected result: All core clusters are healthy.

4. Check service state:

    xms-cli --user admin --password <account_password> service list

• Expected result: All services show active.

5. Check directory permissions, i attributes, umask, and MON consistency.

• Remove i attributes if present.
• Change umask to 0022 if it differs.
• Confirm hostname, ceph.conf, and ceph mon dump are consistent.
• Expected result: Upgrade-script interruption is avoided.

6. For overnight upgrade, disable automatic rebalancing:

   ceph tell mon.* injectargs "--mon_pg_auto_rebalance_enable=false"
   xms-manage conf update ceph

• Expected result: Automatic rebalancing does not amplify load fluctuations during upgrade.

3. Upgrade to 4.2.x
4. Upload and extract the installation package on the primary database node.
5. Run:

   ./upgrade.sh --skip-xdc-restart

• Expected result: The main version upgrade completes without automatically restarting xdc.

3. For confirmed rocksdb exceptions, run:

   ./upgrade.sh --skip-xdc-restart -nc

• Expected result: The main upgrade procedure for the rocksdb-exception scenario completes.

4. Restart xdc one server at a time.

• For each server, first confirm:
• systemctl status xdc is normal
• The cluster has recovered
• Then process the next server.
• Expected result: The intermediate-version switch completes safely.

5. If the upgrade is interrupted, fix the issue and continue:

   ./upgrade.sh --continue

4. Upgrade to 5.2.x
5. Repeat the health checks.
6. Upload and extract the 5.2.x installation package.
7. Run:

   ./upgrade.sh --skip-xdc-restart

• Expected result: The V5 main upgrade completes.

4. Restart xdc again one server at a time. If csi-iscsi or cinder-iscsi is used, restart nodes one by one and complete health checks on each before continuing.
5. If interrupted, continue with:

   ./upgrade.sh --continue

5. Post-upgrade completion
6. Repeat health checks after upgrade.
7. Enable primary storage after storage pools are healthy and wait for reconnection.
8. Re-enable automatic rebalancing:

   ceph tell mon.* injectargs "--mon_pg_auto_rebalance_enable=true"
   xms-manage conf update ceph

• Expected result: The parameter returns to true.

4. Set storage-pool thin-provisioning for V4-to-V5 upgrade:

   xms-cli --user admin -p <account_password> pool list
   xms-cli --user admin -p <account_password> pool set --thin-provisioned id

• Expected result: Storage-pool mode meets V5 requirements.

5. Download the request code and contact sales for a formal license.

• Expected result: The temporary license is replaced by a formal license.

Verification

• The cluster recovers health at each version stage.
• xdc is normal after restart on each server.
• Automatic rebalancing switches off and on correctly.
• Storage pools and primary storage reconnect, and licensing and thin-provisioning are complete.

Risks and rollback

• Risk: Upgrade wait timeout is 120 minutes; timeout interrupts and exits the process.
• Risk: OSDs and related services restart one by one, causing I/O fluctuations.
• Risk: An unhealthy ceph cluster before upgrade significantly increases subsequent risk.
• Risk level: high.
• Rollback: If an upgrade problem occurs, fix it and continue the upgrade rather than immediately rolling back the entire package.