Overview

This document describes how to upgrade ZStack AIOS Platform to 5.5.38. The upgrade includes the following procedures:

• Upgrade Management Nodes
• Upgrade AI Model Platform
• Upgrade Container Management

Scope

• This document applies only to upgrades in a same-repository ZStack AIOS Platform environment. The following conditions must be met:

  • All Hosts and Management Nodes use the same operating system. For example, all nodes use the h84r operating system.
  • The target ISO and the current Hosts and Management Nodes use the same operating system. For example, if all current Hosts and Management Nodes use h84r, use the h84r ISO for the upgrade.

  To change the operating system or upgrade an environment that contains multiple repositories, contact technical support.

• This document applies when you upgrade from ZStack AIOS Platform 5.3.40 or later to 5.5.38.

Considerations

Before you upgrade ZStack AIOS Platform, review the following considerations:

• During the upgrade, Management Nodes are inaccessible, and you cannot request resources such as VM Instances or Volumes. Perform the upgrade during off-peak hours or a maintenance window to avoid resource requests or other critical service changes.
• After the Management Node upgrade is complete, the platform reconnects all Hosts, Primary Storage, Image Storage, and routers, and automatically upgrades the related resources. VM Instances, Volumes, and other related resources cannot be operated during reconnection.
• If an error interrupts the upgrade or affects services, contact technical support immediately.

Important Feature Changes

Preparations

Before you upgrade ZStack AIOS Platform, prepare the following software packages:

Software Packages Required to Upgrade the ZStack AIOS Platform Management Node

Software Packages Required to Upgrade AI Model Platform

• To upgrade from version 5.4.2 or later, prepare the AI Model Platform upgrade package, business gateway image, preinstalled inference template package, AI inference image for VM Instances, and Qwen3-ASR-0.6B preinstalled model package. To enable dGPU mode, you also need to prepare the dGPU dependency toolkit.
• To upgrade from a version earlier than 5.4.2, prepare the AI Model Platform upgrade package, business gateway image, preinstalled inference template package, AI inference image for VM Instances, and complete preinstalled model package. To enable dGPU mode, you also need to prepare the dGPU dependency toolkit.

Software Packages Required to Upgrade Container Management

Management Node Upgrade

ZStack AIOS Platform supports two deployment modes: single Management Node deployment and dual Management Node deployment. The Management Node upgrade process varies by deployment mode.

• For a single Management Node deployment, see Single Management Node Upgrade.
• For a dual Management Node deployment, see Dual Management Node Upgrade.

Single Management Node Upgrade

If your environment uses a single Management Node, follow this procedure to upgrade it.

1. Get the upgrade installation package.

   According to Preparations, obtain the latest version of the ZStack AIOS Platform ISO and ZStack AIOS Platform installation package, and store them in the directory of Management Node /root/.

2. Disable global HA for VM Instances.

   Log in to the ZStack AIOS Platform UI and choose Settings > Platform Settings > HA Policy. On the HA Policy page, click the Disable button in the upper right corner.

3. Back up the database and upgrade scripts.

   Before the upgrade, back up the database and upgrade script for recovery.

   • Run the following command on the Management Node to back up the database to /var/lib/zstack/mysql-backup/:

     [root@localhost ~]# zstack-ctl dump_mysql --file-name zstack-db-backup

   • Run the following command on the Management Node to back up the upgrade script to /root/:

     [root@localhost ~]# cp /usr/local/bin/zstack-upgrade /root/zstack-upgrade-bk

4. Upgrade the Management Node.

   Run the following command on the Management Node to perform an offline upgrade:

   #Supports two offline upgrade methods; the first is recommended:
   # First method: Upgrade the local repository and management service
   [root@zstack-1 opt]# zstack-upgrade ZStack-Cloud-x86_64-DVD-5.5.38-h84r.iso
   # Second method: Upgrade the local repository first, then upgrade the management service:
   [root@zstack-1 opt]# zstack-upgrade -r ZStack-Cloud-x86_64-DVD-5.5.38-h84r.iso
   [root@zstack-1 opt]# bash ZStack-Cloud-installer-5.5.38.bin -u -P MYSQL_ROOT_PASSWORD

   • If the database root password uses the system’s default password -P MYSQL_ROOT_PASSWORD parameter can be omitted.
   • Before running the zstack-upgrade command, ensure the prepared ISO is based on the h84r version to avoid overwriting the local source with another ISO.
   • The upgrade script updates automatically when you upgrade the local repository. After the script update is complete, follow the prompts to run the preceding command again.

5. Manually clear the browser cache.

   To ensure the new features work properly, log in to ZStack AIOS Platform after upgrading. On the UI, press Ctrl+F5 or Ctrl+Shift+R to manually clear the browser cache.

6. Enable global HA for VM Instances.

   On the ZStack AIOS Platform UI, choose Settings > Platform Settings > HA Policy. On the HA Policy page, click the Enable button in the upper right corner.

The single Management Node is upgraded. The platform automatically reconnects Hosts, Primary Storage, Image Storage, VPC vRouters, and other resources, and upgrades the related nodes. Continue with the Container Management and AI Model Platform upgrades.

Dual Management Node Upgrade

If your environment uses dual Management Nodes, follow this procedure to upgrade them.

1. Get the upgrade installation package.

   Based on Preparations, obtain the latest versions of the ZStack AIOS Platform ISO, ZStack AIOS Platform installation package, and high availability suite, and store them separately in the /root/ directory on each Management Node.

2. Disable the global VM HA switch.

   Log in to the ZStack AIOS Platform UI, choose Settings > Platform Settings > HA Policy. Then, the HA Policy page is displayed, and click Disable in the upper-right corner.

3. Back up the database and upgrade scripts.

   Before the upgrade, back up the database and upgrade script for recovery.

   • Run the following command on each Management Node to back up the database to /var/lib/zstack/mysql-backup/:

     [root@localhost ~]# zstack-ctl dump_mysql --file-name zstack-db-backup

   • Run the following command on each Management Node to back up the upgrade script to /root/:

     [root@localhost ~]# cp /usr/local/bin/zstack-upgrade /root/zstack-upgrade-bk

4. Update the local repository.

   Run the following command on each Management Node to update the local repository:

   [root@localhost ~]# zstack-upgrade -r ZStack-Cloud-x86_64-DVD-5.5.38-h84r.iso

   The upgrade script updates automatically when you upgrade the local repository. After the script update is complete, follow the prompts to run the preceding command again.

5. Get the IP address of the Management Node that owns the VIP.

   Run the following command to obtain the IP address of the Management Node where the VIP is located. The node with a “VIP” result in the output is the Management Node hosting the VIP:

   [root@localhost ~]# zsha2 status

6. Extract the high availability suite and grant execute permissions.

   Run the following commands sequentially on the VIP’s Management Node to decompress the High Availability Suite and grant executable permissions to the zsha2 and zstack-hamon files:

   [root@localhost ~]# tar zxvf ZStack-Multinode-HA-Suite-5.5.38.tar.gz #Unpack High Availability Suite
   [root@localhost ~]# chmod +x zsha2 zstack-hamon #Grant execute permissions

7. Upgrade the high availability suite.

   Run the following command on the VIP’s Management Node to complete the High Availability Suite upgrade:

   [root@localhost ~]# ./zsha2 upgrade-ha

8. Upgrade the Management Nodes.

   Run the following command on the VIP’s Management Node to upgrade both Management Nodes:

   [root@localhost ~]# zsha2 upgrade-mn -peerpass password ZStack-Cloud-installer-5.5.38.bin

   -peerpass is an optional parameter that sets the SSH login password for the Peer Management Node.

9. Manually clear the browser cache.

   To ensure the new features work properly, log in to ZStack AIOS Platform after completing the upgrade. On the UI, press Ctrl+F5 or Ctrl+Shift+R to manually clear the browser cache.

10. Enable the global VM HA switch.

    In the ZStack AIOS Platform UI, choose Settings > Platform Settings > HA Policy. Then, the HA Policy page is displayed, and click Enable in the upper-right corner.

Both Management Nodes are upgraded. The platform automatically reconnects Hosts, Primary Storage, Image Storage, VPC vRouters, and other resources, and upgrades the related nodes. Continue with the Container Management and AI Model Platform upgrades.

AI Model Platform Upgrade

After upgrading the ZStack AIOS Platform Management Node, upgrade the AI Model Platform separately to use the latest platform features.

1. Upgrade the AI Model Platform version
2. Update predefined inference templates
3. Update preinstalled model files
4. Update model metadata
5. Update the AI inference image for VM Instances
6. Install the dGPU dependency toolkit on the Host (optional)
7. Update the AI inference image for containers
8. Update the Redis node configuration
9. Confirm the VM cluster network configuration after upgrade

1. Upgrade the AI Model Platform version.

   1. Obtain the AI Model Platform upgrade package as described in Preparations.

   2. Upload the AI Model Platform upgrade package to Model Center and run the following commands to upgrade the AI Model Platform:

      [root@localhost~]# bash aios-installer-5.5.38-${Architecture}.bin

2. Update predefined inference templates.

   1. Obtain the latest preinstalled inference template package as described in Preparations.

   2. Log in to the new Model Center and run the following commands:

      Go to /root/bentoml/
      [root@localhost~]# cd /root/bentoml/
      
      Upload the latest predefined inference template package to /root/bentoml/.
      
      Unpack the preinstalled inference template package
      [root@localhost~]# tar -xzvf ModelServices_5.5.38_${Architecture}.tar.gz

3. Update preinstalled model files.

   1. Obtain the latest preinstalled model file package as described in Preparations.

      • When upgrading from version 5.4.2 or later, prepare the preinstalled model file package (Qwen3-ASR-0.6B).
      • Before upgrading to version 5.4.2 or earlier, prepare the complete preinstalled model file package.

   2. Log in to the new Model Center and run the following commands:

      Go to /root/bentoml/
      [root@localhost~]# cd /root/bentoml/
      
      Upload the latest preinstalled model file package to /root/bentoml/.
      
      Unzip the preset model file package
      ================================================
      When upgrading from versions prior to 5.4.2, run:
      [root@localhost~]# tar -xzvf Models_5.5.6.tar.gz
      =================================================
      When upgrading from version 5.4.2 or later, run:
      [root@localhost~]# tar -xzvf Model_Qwen3-ASR-0.6B_tar.gz

4. Update model metadata.

   Log in to the ZStack AIOS Platform Management Node, run the following command:

   Log in to the CLI
   [root@localhost ~]# zstack-cli // Enter the CLI
   - >>> LogInByAccount accountName=admin password=${Password} // Log in with your Admin account credentials
   
   Update Model Metadata
   admin >>> GenerateModelMetadata modelCenterUuid=2cc1f4d6b8014b44912f92bb242e9675

5. Update the AI inference image for VM Instances.

   1. Upload the latest AI inference image for VM Instances.

      Based on Preparations, obtain the latest version of the AI inference image (for VM Instances) corresponding to your GPU model. On the main menu of ZStack AIOS Platform, choose Resource Center > Cloud Resource Pool > Image. On the Image page, click Add Image and upload the AI inference image for VM Instances.

   2. Set the AI inference image for VM Instances to global sharing.

      On the Image page, locate the uploaded AI inference image (VM Instance), choose Operate > Set Sharing Mode, and set it as a global shared image.

   3. Update the default VM Instance image for system inference templates.

      Log in to the ZStack AIOS Platform Management Node, run the following command:

      [root@localhost~]# zstack-ctl aios_setup_system_services --architecture ${Architecture} --vm-image-uuid ${AIVMIMAGEUUID}
      // ${AIVMIMAGEUUID} is the AI inference image UUID for VM Instances. You can view it on the image details page. ${Architecture} is the image architecture. For x86, enter x86_64. For ARM, enter aarch64.

6. Install the dGPU dependency toolkit on the Host (optional).

   If you do not use dGPU mode, skip this step.

   Based on Preparations, download the dGPU dependency toolkit, upload it to the host where dGPU mode needs to be enabled, and log in to execute the following commands:

   [root@localhost~]# bash zstack-dgpu-toolkit.bin

   This toolkit installs the dependencies required for dGPU mode on the host. After installation, you can enable dGPU mode on the machine.

7. Update the AI inference image for containers.

   If Container Management is not deployed on ZStack AIOS Platform, skip this step.

   1. Obtain the latest AI inference image for containers that matches the GPU model, as described in Preparations.

   2. Log in to the ZStack AIOS Platform UI and choose Container Management > Artifact Repository > Local Repository. On the Local Repository page, locate the ai-model-registry repository and click to enter its details page. In the repository details page, click Image to view the old version of the AI inference image. Delete the container and upload the 5.5.38 version of the AI inference image (container).

      Supports three upload methods: online upload, file upload, and command upload

      • If using Upload Online, select the image Architecture. Multiple selections are supported.

      • If using Upload Command, specify the image architecture in the command, for example:

        • nerdctl push --platform=linux/arm64
        • nerdctl push --platform=linux/amd64
        • nerdctl push --platform=linux/arm64 --platform=linux/amd64

      If using the Ascend 910B GPU: Please prepare and upload the aiworker-npu-910b-3.0.0-800I-A2-py311-openeuler24.03-lts.tar, vllm-ascend-v0.26.0rc1.tar, and aiworker-npu-llamafactory-0.9.5.tar images.

   3. Specify the default container image for the model fine-tuning service.

      • You can skip this step by fine-tuning the model with an NVIDIA GPU.

      • Use Hygon K100-AI to fine-tune the model. Log in to the ZStack AIOS Platform Management Node and run the following command:

        [root@localhost~]# zstack-ctl aios_setup_system_services --type FineTune --architecture x86_64 --docker-image-name aiworker-dcu:vllm0.9.2

      • Use Ascend 910B for model fine-tuning. Log in to the ZStack AIOS Platform Management Node and run the following command:

        [root@localhost~]# zstack-ctl aios_setup_system_services --type FineTune --architecture aarch64 --docker-image-name aiworker-npu-llamafactory:latest

8. Update the Redis node configuration.

   You can skip this step when upgrading from version 5.3.52 or later to 5.5.38.

   1. View Redis node information.

      Log in to the ZStack AIOS Platform UI and choose Operational Management > Application Market > Default Application > Deployed. On the Default Application – Deployed page, locate the deployed ZStack AI Model Platform and choose … > View VM Instance to view Redis node information, including name and IP address.

      

   2. Check whether the root volume of your Redis VM Instance has reached 40 GB. If not, expand it to 40 GB or more.

   3. Log in to three Redis nodes respectively and run the following commands:

      [root@localhost~]# lsblk // View Disk Name
      
      If the disk name is vda, run the following command
      [root@localhost~]# sudo bash -c "echo -e 'n\np\n\n\n\nt\n\n8e\nw' | fdisk /dev/vda";
      [root@localhost~]# sudo pvcreate /dev/vda4 && sudo vgextend ubuntu-vg /dev/vda4 && sudo lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
      
      If the disk name is sda, run the following command
      [root@localhost~]# sudo bash -c "echo -e 'n\np\n\n\n\nt\n\n8e\nw' | fdisk /dev/sda";
      [root@localhost~]# sudo pvcreate /dev/sda4 && sudo vgextend ubuntu-vg /dev/sda4 && sudo lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
      
      Run the configuration modification command
      [root@localhost~]# sudo systemctl stop unattended-upgrades; sudo systemctl disable unattended-upgrades;
      [root@localhost~]# redis-cli -p 6379 -a zstack.redis.password -h 127.0.0.1 CONFIG SET appendonly yes;
      [root@localhost~]# redis-cli -p 6379 -a zstack.redis.password -h 127.0.0.1 CONFIG SET save "900 1 300 10 60 10000";
      [root@localhost~]# redis-cli -p 6379 -a zstack.redis.password -h 127.0.0.1 CONFIG REWRITE;

9. Confirm the VM cluster network configuration after upgrade.

   When upgrading from a version earlier than 5.5.32, the system may detect VM clusters that still use the default networks configured before the upgrade. VM clusters with completed business gateway configuration do not need to be configured again.

   1. Upload the business gateway image.

      Obtain the business gateway image that matches the target version and the CPU architecture of the environment as described in Preparations. In the ZStack AIOS Platform main menu, choose Resource Center > Cloud Resources > Image. On the Image page, click Add Image and upload the business gateway image.

   2. View the VM clusters marked Pending.

      Log in to the ZStack AIOS Platform UI. On the main menu, choose AI Model Platform > Statistic and Management > Model Platform Config. On the Model Platform Config page, click Resource Configuration. The system displays a prompt about the network configuration after the upgrade and marks the affected VM clusters Pending.

      

   3. Confirm and complete the business gateway configuration.

      On the card of the target cluster, click Configure to open the Configure Business Gateway for VM Cluster page. The target VM cluster cannot be changed. The system prefills the default service network configured before the upgrade that remains available to the cluster for Business Network and Developer Access Network, and prefills the default storage network that remains available to the cluster for Storage Network. For Image, select the business gateway image that matches the CPU architecture of the environment. Select an Instance Offering and a Management Network. Confirm or reselect the Business Network and Developer Access Network. Confirm the Storage Network configuration. Then click OK.

      When Global Model Cache is disabled, Storage Network is required. When Global Model Cache is enabled, Storage Network is optional.

      

      After the configuration is complete, new inference services use the reconfigured business gateway. Existing inference services continue to use their original access addresses.

The AI Model Platform service update is now complete. If Container Management is deployed in the current environment, continue upgrading the Container Management module.

Container Management Upgrade

If ZStack AIOS Platform deploys the Container Management Module, upgrade this module separately after completing the Management Node upgrade to use the latest container features.

1. Check the Container Management Node status
2. Upgrade Container Management to version 3.5.0
3. Upgrade Container Management to version 3.6.0
4. Upgrade Container Management to version 3.7.0
5. Upgrade Container Management to version 3.8.0
6. Upgrade Container Management to version 3.9.0
7. Upgrade Container Management to version 3.10.0
8. Update Container Management Cluster

1. Check the Container Management Node status.

   1. Verify that all nodes in the Container Management management cluster and service clusters are In Service.

   2. Check the GPU nodes in the service cluster to ensure all nodes have installed GPU devices. If a node does not have a GPU device and will no longer be used as a GPU node, log in to the third Container Management cluster node (Zaku-3) and run the following command:

      [root@localhost ~]# kubectl label node ${NodeName} gpu-

      To view the Zaku-3 node, log in to the ZStack AIOS Platform UI and choose Operational Management > Application Market > Default Application > Installed. On the Installed page, locate ZStack Zaku Container Management, choose … > View VM Instance in the upper right corner, and check the VM Instance named Zaku-3.

      

2. Upgrade Container Management to version 3.5.0.

   If your Container Management is version 3.5.0 or later, skip this step.

   1. Prepare the Container Management 3.5.0 upgrade package based on Preparations.

   2. Log in to the Zaku-3 VM Instance and save the Container Management 3.5.0 upgrade package in the /opt/zstack-edge directory.

   3. Run the following command on the Zaku-3 VM Instance to upgrade the management cluster to version 3.5.0:

      Compression Container Management 3.5.0 Upgrade Package
      [root@localhost ~]# cd /opt/zstack-edge
      [root@localhost ~]# tar -xzvf zaku-1.24-3.5.0-all-in-one-upgrade.tgz
      
      Copy hosts.yaml, open the configuration file, and verify that the machine configuration information for the current environment is correct.
      Command 1: If the Container Management is upgrading directly from 3.4.0 to 3.5.0, run this command to copy
      [root@localhost ~]# cp /opt/zstack-edge/kubespray/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.5.0/inventory/sample/hosts.yaml
      (Command 2: If the Container Management system is first upgraded from version 3.3.0 or earlier to 3.4.0, then to 3.5.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/upgrade-zaku-1.24-3.4.0/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.5.0/inventory/sample/hosts.yaml
      
      Run the upgrade script.
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.5.0
      [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
      
      Confirm the status of management cluster Pods: if the ze-db-migration Pod is completed and all other Pods are running, it indicates the management cluster upgrade is complete.
      [root@localhost ~]# kubectl get pod -A

   4. Run the following command on the Zaku-3 VM Instance to upgrade the service cluster to version 3.5.0:

      Run the upgrade script
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.5.0
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh // Enter the name of the service cluster to upgrade as prompted. If there are multiple service clusters, upgrade them in order
      
      Confirm whether the service cluster is upgraded
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh

3. Upgrade Container Management to version 3.6.0.

   If your Container Management is version 3.6.0 or later, skip this step.

   1. Prepare the Container Management 3.6.0 upgrade package based on Preparations.

   2. Log in to the Zaku-3 VM Instance and save the Container Management 3.6.0 upgrade package in the /opt/zstack-edge directory.

   3. Run the following command on the Zaku-3 VM Instance to upgrade the management cluster to version 3.6.0:

      Compression Container Management 3.6.0 Upgrade Package
      [root@localhost ~]# cd /opt/zstack-edge
      [root@localhost ~]# tar -xzvf zaku-1.24-3.6.0-all-in-one-upgrade.tgz
      
      Copy hosts.yaml, open the configuration file, and verify that the machine configuration information for the current environment is correct.
      Command 1: If the Container Management is upgrading directly from 3.5.0 to 3.6.0, run this command to copy
      [root@localhost ~]# cp /opt/zstack-edge/kubespray/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.6.0/inventory/sample/hosts.yaml
      (Command 2: If the Container Management system is first upgraded from version 3.4.0 or earlier to 3.5.0, then to 3.6.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/upgrade-zaku-1.24-3.5.0/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.6.0/inventory/sample/hosts.yaml
      
      Run the upgrade script.
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.6.0
      [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
      
      Confirm the status of management cluster Pods: if the ze-db-migration Pod is completed and all other Pods are running, it indicates the management cluster upgrade is complete.
      [root@localhost ~]# kubectl get pod -A

   4. Check whether the service cluster is using Huawei NPU devices.

      • Check whether the service cluster is using Huawei NPU devices; if not, you can skip this step.

      • If using a Huawei NPU device, log in to the Container Management VIP and execute the following command to check whether the configuration fields in the chipName template match: Table 1: Chipname Template Field Table mate :

        [root@localhost ~]# kubectl -n kube-system edit configmap hami-scheduler-device

      chipName	commonWord	resourceName	resourceMemoryName
      910A	Ascend910A	huawei.com/Ascend910A	huawei.com/Ascend910A-memory
      910B2	Ascend910B2	huawei.com/Ascend910B2	huawei.com/Ascend910B2-memory
      910B3	Ascend910B3	huawei.com/Ascend910B3	huawei.com/Ascend910B3-memory
      910B4	Ascend910B4	huawei.com/Ascend910B4	huawei.com/Ascend910B4-memory
      910B4-1	Ascend910B4-1	huawei.com/Ascend910B4-1	huawei.com/Ascend910B4-1-memory
      910ProB	Ascend910ProB	huawei.com/Ascend910ProB	huawei.com/Ascend910ProB-memory
      310P3	Ascend310P	huawei.com/Ascend310P	huawei.com/Ascend310P-memory

   5. Stop Huawei NPU-related workloads.

      If the service cluster does not use a Huawei NPU or if the configuration field in the chipName template matches Table 1: Chipname Template Field Table, this step can be skipped.

      On the main menu of ZStack AIOS Platform, choose Container Management > Container Orchestration > Application Management > Workload. Open each workload related to Huawei NPU and choose Operate > Autoscaling. Set the workload replica count to 0.

      Record the current replica count so you can restore it after the upgrade.

   6. Upgrade the service cluster to 3.6.0.

      Run the following command on the Zaku-3 VM Instance to upgrade the service cluster to version 3.6.0

      Run the upgrade script
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.6.0
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh // Enter the name of the service cluster to upgrade as prompted. If there are multiple service clusters, upgrade them in order
      
      Confirm whether the service cluster is upgraded
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh

   7. Edit the chipName template field.

      If the service cluster does not use a Huawei NPU or if the configuration field in the chipName template matches Table 1: Chipname Template Field Table, this step can be skipped.

      Log in to Container Management VIP and execute the following command to set resourceName and resourceMemoryName in the Huawei NPU-related workload configuration: Table 1: Chipname Template Field Table corresponding :

      [root@localhost ~]# kubectl -n kube-system edit configmap hami-scheduler-device

   8. Start Huawei NPU-related workloads.

      If the service cluster does not use a Huawei NPU or if the configuration field in the chipName template matches Table 1: Chipname Template Field Table, this step can be skipped.

      Log in to the ZStack AIOS Platform UI and choose Container Management > Container Orchestration > Application Management > Workload. Then sequentially choose Operate > Autoscaling after each Huawei NPU-related workload, and restore the replica count to its pre-upgrade value.

4. Upgrade Container Management to version 3.7.0.

   1. Prepare the Container Management 3.7.0 upgrade package as described in Preparations.

   2. Log in to the Zaku-3 VM Instance and save the Container Management 3.7.0 upgrade package in the /opt/zstack-edge directory.

   3. Run the following command on the Zaku-3 VM Instance to upgrade the management cluster to version 3.7.0:

      Compression Container Management 3.7.0 Upgrade Package
      [root@localhost ~]# cd /opt/zstack-edge
      [root@localhost ~]# tar -xzvf zaku-1.24-3.7.0-all-in-one-upgrade.tgz
      
      Copy hosts.yaml, open the configuration file, and verify that the machine configuration information for the current environment is correct.
      (Command 1: If the Container Management is upgrading directly from 3.6.0 to 3.7.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/kubespray/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.7.0/inventory/sample/hosts.yaml
      (Command 2: If the Container Management system is first upgraded from version 3.5.0 or earlier to 3.6.0, then to 3.7.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/upgrade-zaku-1.24-3.6.0/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.7.0/inventory/sample/hosts.yaml
      
      Run the upgrade script.
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.7.0
      [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
      
      Confirm the status of management cluster Pods: if the ze-db-migration Pod is completed and all other Pods are running, it indicates the management cluster upgrade is complete.
      [root@localhost ~]# kubectl get pod -A

   4. Check whether the service cluster is using NVIDIA GPUs, Huawei NPUs, or Hygon DCU devices.

   5. Stop workloads related to NVIDIA GPUs, Huawei NPUs, and Hygon DCUs.

      If the service cluster does not use the above GPU devices, you can skip this step.

      Log in to the ZStack AIOS Platform UI and choose Container Management > Container Orchestration > Application Management > Workload. Open each related workload and choose Operate > Autoscaling. Change the number of workload replicas to 0.

      Record the current number of copies to restore after upgrading.

   6. Upgrade the service cluster to version 3.7.0.

      Run the following command on the Zaku-3 VM Instance to upgrade the service cluster to version 3.7.0

      Run the upgrade script
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.7.0
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh // Enter the name of the service cluster to upgrade as prompted. If there are multiple service clusters, upgrade them in order
      
      Confirm whether the service cluster is upgraded
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh

   7. Start workloads related to NVIDIA GPUs, Huawei NPUs, or Hygon DCUs.

      If the service cluster does not use NVIDIA GPUs, Huawei NPUs, or Hygon DCUs, skip this step.

      Log in to the ZStack AIOS Platform UI and choose Container Management > Container Orchestration > Application Management > Workload. Then, choose Operate > Autoscaling after each relevant workload to restore the replica count to its pre-upgrade value.

5. Upgrade Container Management to version 3.8.0.

   If your Container Management is version 3.8.0 or later, skip this step.

   1. Prepare the Container Management 3.8.0 upgrade package based on Preparations.

   2. Log in to the Zaku-3 VM Instance and save the Container Management 3.8.0 upgrade package in the /opt/zstack-edge directory.

   3. Run the following command on the Zaku-3 VM Instance to upgrade the management cluster to version 3.8.0:

      Compression Container Management 3.8.0 Upgrade Package
      [root@localhost ~]# cd /opt/zstack-edge
      [root@localhost ~]# tar -xzvf zaku-1.24-3.8.0-all-in-one-upgrade.tgz
      
      Copy hosts.yaml, open the configuration file, and verify that the machine configuration information for the current environment is correct.
      (Command 1: If the Container Management is upgrading directly from 3.7.0 to 3.8.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/kubespray/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.8.0/inventory/sample/hosts.yaml
      (Command 2: If the Container Management system is first upgraded from version 3.6.0 or earlier to 3.7.0, then to 3.8.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/upgrade-zaku-1.24-3.7.0/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.8.0/inventory/sample/hosts.yaml
      
      Run the upgrade script.
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.8.0
      [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
      
      Confirm the status of management cluster Pods: if the ze-db-migration Pod is completed and all other Pods are running, it indicates the management cluster upgrade is complete.
      [root@localhost ~]# kubectl get pod -A

   4. Run the following command on the Zaku-3 VM Instance to upgrade the service cluster to version 3.8.0:

      Run the upgrade script
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.8.0
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh // Enter the name of the service cluster to upgrade as prompted. If there are multiple service clusters, upgrade them in order
      
      Confirm whether the service cluster is upgraded
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh

   5. Check the ZStack AIOS Platform Nginx proxy configuration.

      After the upgrade is complete, log in to the ZStack AIOS Platform Management Node and check whether zaku.server.nginx.conf includes the /zaku-core-shell/mf-manifest.json location configuration. If the configuration is missing, add the configuration shown in the following example. In a dual-Management-Node environment, log in to both Management Nodes for verification:

      [root@localhost ~]# vi /usr/local/zstack/zstack-ui/configs/zaku.server.nginx.conf // Add the following after the /zaku-ui location configuration.
      ===================================================
      location = /zaku-core-shell/mf-manifest.json {
      rewrite ^/zaku-core-shell/(.*) /ze/$1 break;
      expires -1;
      add_header Cache-Control "no-cache, no-store, must-revalidate";
      add_header Pragma "no-cache";
      proxy_pass http://${ZakuVip}; // ${ZakuVip} must be replaced with the Container Management VIP address.
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
      ===================================================
      [root@localhost ~]# /usr/sbin/nginx -t -c /usr/local/zstack/zstack-ui/configs/nginx.conf -p /usr/local/zstack/zstack-ui //Check if the configuration is correct
      [root@localhost ~]# /usr/sbin/nginx -s reload -c /usr/local/zstack/zstack-ui/configs/nginx.conf -p /usr/local/zstack/zstack-ui // Reload the Nginx configuration

6. Upgrade Container Management to version 3.9.0.

   If your Container Management is version 3.9.0 or later, skip this step.

   1. Prepare the Container Management 3.9.0 upgrade package based on Preparations.

   2. Log in to the Zaku-3 VM Instance and save the Container Management 3.9.0 upgrade package in the /opt/zstack-edge directory.

   3. Run the following command on the Zaku-3 VM Instance to upgrade the management cluster to version 3.9.0:

      Compression Container Management 3.9.0 Upgrade Package
      [root@localhost ~]# cd /opt/zstack-edge
      [root@localhost ~]# tar -xzvf zaku-1.24-3.9.0-all-in-one-upgrade.tgz
      
      Copy hosts.yaml, open the configuration file, and verify that the machine configuration information for the current environment is correct.
      (Command 1: If the Container Management is upgrading directly from 3.8.0 to 3.9.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/kubespray/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.9.0/inventory/sample/hosts.yaml
      (Command 2: If the Container Management system is first upgraded from a version earlier than 3.8.0 to 3.8.0, then to 3.9.0, run this command to copy)
      [root@localhost ~]# cp /opt/zstack-edge/upgrade-zaku-1.24-3.8.0/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.9.0/inventory/sample/hosts.yaml
      
      Run the upgrade script.
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.9.0
      [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
      
      Confirm the status of management cluster Pods: if the ze-db-migration Pod is completed and all other Pods are running, it indicates the management cluster upgrade is complete.
      [root@localhost ~]# kubectl get pod -A

   4. Run the following command on the Zaku-3 VM Instance to upgrade the service cluster to version 3.9.0:

      Run the upgrade script
      [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.9.0
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh // Enter the name of the service cluster to upgrade as prompted. If there are multiple service clusters, upgrade them in order
      
      Confirm whether the service cluster is upgraded
      [root@localhost ~]# bash zaku-upgrade/upgrade.sh

7. Upgrade Container Management to version 3.10.0.

   • If your Container Management is version 3.10.0 or later, skip this step.
   • Container Management 3.10.0 supports a hot upgrade of service clusters from K8s 1.30 to K8s 1.35.

   1. Prepare the Container Management 3.10.0 upgrade package for the current management cluster K8s version. For package details, see Preparations.

   2. Log in to the Zaku-3 VM Instance and save the Container Management 3.10.0 upgrade package in the /opt/zstack-edge directory.

   3. On the Zaku-3 VM Instance, run the commands for the current K8s version to upgrade the management cluster to version 3.10.0:

      • K8s 1.24:

        Decompress the Container Management 3.10.0 upgrade package
        [root@localhost ~]# cd /opt/zstack-edge
        [root@localhost ~]# tar -xzvf zaku-1.24-3.10.0-all-in-one-upgrade.tgz
        
        Copy hosts.yaml and verify that the machine configuration for the current environment is correct
        [root@localhost ~]# cp /opt/zstack-edge/upgrade-zaku-1.24-3.9.0/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.24-3.10.0/inventory/sample/hosts.yaml
        
        Run the upgrade script
        [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.10.0
        [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
        
        Check the management cluster Pod status. The upgrade is complete when the ze-db-migration Pod is Completed and all other Pods are Running
        [root@localhost ~]# kubectl get pod -A

      • K8s 1.35:

        When running the upgrade script, enter the gateway and subnet mask based on the actual network configuration of the current environment. Incorrect values might cause network issues.

        Decompress the Container Management 3.10.0 upgrade package
        [root@localhost ~]# cd /opt/zstack-edge
        [root@localhost ~]# tar -xzvf zaku-1.35-3.10.0-all-in-one-upgrade.tgz
        
        Copy hosts.yaml and verify that the machine configuration for the current environment is correct
        [root@localhost ~]# cp /opt/zstack-edge/kubespray/inventory/sample/hosts.yaml /opt/zstack-edge/upgrade-zaku-1.35-3.10.0/inventory/sample/hosts.yaml
        
        Run the upgrade script and enter the gateway and subnet mask of the current environment as prompted
        [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.35-3.10.0
        [root@localhost ~]# source /opt/zstack-edge/ansible-env
        [root@localhost ~]# ansible-playbook -i inventory/sample/hosts.yaml zaku-management-upgrade.yml
        
        Check the management cluster Pod status. The upgrade is complete when the ze-db-migration Pod is Completed and all other Pods are Running
        [root@localhost ~]# kubectl get pod -A

   4. After upgrading the management cluster, run the commands for the current K8s version on the Zaku-3 VM Instance to upgrade the service clusters to version 3.10.0 one by one:

      • K8s 1.24:

        [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.24-3.10.0
        [root@localhost ~]# bash zaku-upgrade/upgrade.sh

      • K8s 1.35:

        [root@localhost ~]# cd /opt/zstack-edge/upgrade-zaku-1.35-3.10.0
        [root@localhost ~]# bash zaku-upgrade/upgrade.sh

      Enter the name of the service cluster to upgrade as prompted. If there are multiple service clusters, upgrade them one by one.

   5. If the upgrade fails, check the upgrade logs in the directory that matches the current management cluster K8s version:

      • K8s 1.24: /opt/zstack-edge/upgrade-zaku-1.24-3.10.0/zaku-upgrade/log
      • K8s 1.35: /opt/zstack-edge/upgrade-zaku-1.35-3.10.0/zaku-upgrade/log

8. Update the Container Management cluster.

   Log in to the ZStack AIOS Platform Management Node and run the following commands to update the Container Management service cluster and clear dirty data:

   Log in to the CLI
   [root@localhost ~]# zstack-cli // Enter the CLI
   - >>> LogInByAccount accountName=admin password=${Password} // Log in with your Admin account credentials
   admin >>> QueryContainerManagementEndpoint // Query the Container Management Service Cluster Management Node UUID
   admin >>> QueryZone // Query the zone UUID
   admin >>> DeleteContainerResourceFromEndpoint uuid=${ContainerManagementEndpointUuid}
   admin >>> SyncContainerManagementEndpoint uuid="${ContainerManagementEndpointUuid}" zoneUuid=${ZoneUuid}

The Container Management upgrade is now complete.