Applicable scenario

• Product: ZCF
• Component: ZCE-X
• Version: 6.4.x
• Current kernel: kernel-4.19.90-89.23.v2401.ky10.x86_64
• Target kernel: kernel-4.19.90-89.25.v2401.ky10.x86_64
• Trigger: Newly added storage nodes might be used for csi-iSCSI and require XDC kernel mode during deployment.

Symptoms

• The objective is preparation for future csi-iSCSI integration, not repair of current services.
• Scope:
• Only newly expanded primary-storage nodes without workloads
• Existing VM services are unaffected
• Complete all of the following:

1. Upgrade kernels on newly expanded storage nodes
2. Upgrade storage version to 6.4.x
3. Switch xdc to kernel mode

Determination method

1. Confirm cluster health, service health, and resource headroom before upgrade.
2. Check ceph -s and uname -a after every node kernel upgrade.
3. After storage upgrade, check VERSION, service state, and MON/OSD version matching.
4. Check xdc state after service restart.

Cause

Newly expanded storage nodes must support XDC kernel mode before they are placed online for future csi-iSCSI integration. Upgrade the kernel and SDS to compatible versions before switching the XDC feature, rather than changing only xdc configuration.

Resolution

1. Upgrade the kernel on each newly expanded storage node
2. Check storage health:

   ceph -s

• Expected result: Health is normal.

2. Upload and verify the kernel archive:

   4a55105d041b08f281304076018abc4c  kernel-4.19.90-89.25.tar.gz

   tar xzvf kernel-4.19.90-89.25.tar.gz

3. Install the target kernel:

   cd kernel-4.19.90-89.25/
   yum install --disablerepo=* ./kernel-*

4. Restart the server:

   reboot

5. After restart, check:

   ceph -s
   uname -a

• Expected result: The node recovers normally with the updated kernel.

6. Process remaining newly expanded storage nodes one by one using the same steps.

• Note: Upgrade only newly expanded storage nodes, not compute nodes.

2. Pre-upgrade condition checks
3. Check cluster health:

   ceph -s

• Expected result: HEALTH_OK.

2. Check service health:

   xms-cli --user admin --password $password service list

• Expected result: All status=active.

3. Check ES:

   xms-cli --user admin --password $password -l cluster show elasticsearch

4. Continuously observe Ceph:

   watch -n1 ceph -s

5. Check that MON node host names match.
6. Check passwordless cluster access and firewall hardening.
7. Check ETCD/ES:

   xms-manage -s

8. Check Postgres replication:

   docker exec -u postgres sds-postgres psql -c "select pid, usename, client_addr, state, pg_xlog_location_diff( pg_current_xlog_location(), write_location) write_delay, pg_xlog_location_diff( pg_current_xlog_location(), flush_location) flush_delay, pg_xlog_location_diff(pg_current_xlog_location(), replay_location) replay_delay from pg_stat_replication;"

9. Check resources on all nodes:

• CPU/memory below 80%
• System-volume usage below 70%
• At least 30GB free on /opt, /tmp, and root
• Expected result: Resources meet upgrade requirements.

10. Check for prometheus port changes.
11. Check for hidden network or failed-disk risks:

   for i in `xms-cli --user admin -password $password host list | grep storage | awk -F"[ |]+" '{print $8}'`; do echo $i; ssh $i "grep -rnE 'NIC Link is Down|critical medium error|blk_update_request|I/O error' /var/log/messages"; echo -e "\n"; sleep 1; done

12. Obtain server list and check system-volume space:

   xms-cli --user admin --password $password host list | awk '{print $4}' | grep "^zstack" > hostlist
   for i in `cat hostlist`; do ssh $i "df -hT | grep -Ei 'xfs|filesystem' | grep -Ewi 'opt|tmp|/|filesystem'"; done

13. Check that /opt/sds permissions are 755:

   for i in `cat hostlist`; do ssh $i 'ls -l /opt/ | grep sds; ls -l /opt/sds'; done

14. Check important files/directories for the i attribute:

   for y in `cat hostlist`; do ssh $y "for i in /etc/hosts /etc/chrony.conf /etc/rsyslog.conf /etc/group /etc/security; do lsattr \$i; done"; done

• If present, run first:

     chattr -i file_name
     chattr -R directory_name

15. Check whether umask is 0022:

   for i in `cat hostlist`; do ssh $i umask; done

• If not, set it to:

     for i in `cat hostlist`; do ssh $i umask 0022; done

16. Check current version:

   cat /opt/sds/installer/VERSION

• Expected result: The pre-upgrade version is accurate.

3. Upgrade storage to 6.4.x
4. Verify the upgrade archive sha256:

   sha256sum ZSTACK-V6-installer-6.4.300.1-x86_64-aarch64-6089ea77.tar.gz

• Expected value:

   626afd2d38f6d73230b542eee04568636b34f30c5b1ec14f353ff313eb463997

2. Locate the primary database node:

   xms-manage db list

3. Upload the archive and prepare the directory:

   mkdir /root/sds_update
   mv ZSTACK-V6-installer-6.4.300.1-x86_64-aarch64-6089ea77.tar.gz /root/sds_update

4. Extract and upgrade:

   cd /root/sds_update
   tar zxvf ZSTACK-V6-installer-6.4.300.1-x86_64-aarch64-6089ea77.tar.gz
   ./upgrade.sh

• Enter the storage-platform UI administrator account and password during upgrade.
• Expected result: Upgrade successfully appears.

4. Post-upgrade checks
5. Check software version:

   cat /opt/sds/installer/VERSION

2. Check cluster state:

   ceph -s

3. Check service state:

   xms-cli --user admin --password <account password> service list

4. Check version information:

   xms-cli -v

5. Check MON and OSD versions against the ceph version in the installer:

   ceph tell mon.* version
   ceph tell osd.* version

6. Check UI modules such as Resource Management and Monitoring Management.

• Expected result: The platform and cluster are healthy after upgrade.

5. Enable XDC kernel mode
6. Modify:

   /etc/xdc/xdc.conf

Set:

   xdc_proxy_feature=false

• Expected result: The xdc proxy feature is disabled in preparation for kernel mode.

2. Restart xdc:

   systemctl stop xdc

• Wait for automatic restart.

3. Check state:

   systemctl status xdc

• Expected result: The xdc service returns to normal.

4. Process other storage nodes one by one using the same procedure.

• Expected result: All target nodes are switched.

Verification

• uname -a on every newly expanded node shows the target kernel version.
• ceph -s remains healthy.
• xms-cli service list, xms-manage -s, and Postgres replication are normal.
• VERSION, xms-cli -v, and MON/OSD versions match 6.4.x.
• xdc runs normally on every node in kernel mode.

Risks and notes

• Risk: Kernel upgrade requires a per-node restart; keep the cluster healthy before proceeding to the next node.
• Risk: Insufficient prechecks can amplify disk-space, permission, lsattr, or umask problems during upgrade.
• Risk level: low.