• ZStack Cloud ISO.
  • ZStack Cloud ISO:
    • h84r: ZStack-Cloud-x86_64-DVD-4.8.38-h84r.iso
    • Download address: Click here
  • ZStack Cloud installation package:
    • Software: ZStack-Cloud-installer-4.8.38.bin
    • Download address: Click here
• Multi-MN HA suite
  • Software: ZStack-Multinode-HA-Suite-4.8.38.tar.gz
  • Download address: Click here

Note: After you download the required software, confirm the integrity of the file by using the MD5 checksum tool.

Note: The virtual IP address (VIP) is used to log in to the UI. Do not use the VIP to log in to a MN through SSH.

• The data provided above is sample data. The administrator can change the data in the actual deployment environment.
• The gateway must be provided by a physical network device. Meanwhile, the gateway is used to detect the network status.

# Create bond0.
[root@localhost ~]# zs-bond-lacp -c bond0

# Attach eth0 and eth1 to bond0.
[root@localhost ~]# zs-nic-to-bond -a bond0 eth0
[root@localhost ~]# zs-nic-to-bond -a bond0 eth1

# After the preceding bonds are configured, configure LACP aggregation on the corresponding switch port.

# Create br_bond0, and configure the IP address, netmask, and gateway.
[root@localhost ~]# zs-network-setting -b bond0 192.168.195.200 255.255.0.0 192.168.0.1

# Check whether bond0 is successfully created.
[root@localhost ~]# zs-show-network
...
--------------------------------------------------------------------
| Bond Name | SLAVE(s)| BONDING_OPTS                                |
--------------------------------------------------------------------
| bond0     | eth0    | miimon=100 mode=4 xmit_hash_policy=layer2+3 |
|           | eth1    |                                             |
--------------------------------------------------------------------

# Create bond1.
[root@localhost ~]# zs-bond-lacp -c bond1

# Attach em1 and em2 to bond1.
[root@localhost ~]# zs-nic-to-bond -a bond1 em1
[root@localhost ~]# zs-nic-to-bond -a bond1 em2

# After the preceding bonds are configured, configure LACP aggregation on the corresponding switch port.

# You do not need to create a bridge for the VM data network.

# Check whether bond1 is successfully created.
[root@localhost ~]# zs-show-network
...
--------------------------------------------------------------------
| Bond Name | SLAVE(s)| BONDING_OPTS                                |
--------------------------------------------------------------------
| bond1     | em1     | miimon=100 mode=4 xmit_hash_policy=layer2+3 |
|           | em2     |                                             |
--------------------------------------------------------------------

• Install the HA suite directly by using the command line.
• Install the HA suite by writing configuration files.

Note: With the same configurations, the command line method has higher priority than the configuration file method.

1. Complete the preparations before the upgrade.
2. Upgrade the HA suite.
3. Upgrade the MNs.

1. Disable the VM HA functionality globally to avoid accidentally triggering VM HA and affecting the upgrade. Method: On the main menu of the MN UI, choose Settings > Global Setting. On the Global Setting page, disable the VM HA option. You can manually enable this functionality after the upgrade is completed.
2. Run the following command on each MN to back up your database:

   [root@localhost ~]#zstack-ctl dump_mysql --file-name zstack-db-backup

3. Make sure that the zstack-upgrade script, the corresponding ISO, the installation and upgrade package, and the HA suite are downloaded for these two MNs.
4. Run the following command on each MN to update the local repo by using the latest ISO you downloaded before:

   [root@localhost ~]#cd /root/
   #bash /root/zstack-upgrade -r ZStack-Cloud-x86_64-DVD-4.8.38-h84r.iso

1. Run the following command on any MN to check on which node the HA VIP is located. After you execute zsha2, the node whose "Owns virtual address" is yes in the returned result is the MN where the VIP is located.

   [root@localhost ~]#zsha2 status

2. Log in to the MN where the VIP is located through IPMI, and run the following command to decompress the dual-MN HA suite:

   [root@localhost ~]#tar zxvf ZStack-Multinode-HA-Suite-4.8.38.tar.gz

3. Run the following command to grant executable permissions to the decompressed zsha2 and zstack-hamon:

   [root@localhost ~]#chmod +x zsha2 zstack-hamon

4. Run the following command in the MN where the VIP is located to complete the upgrade of the HA suite:

   [root@localhost ~]# ./zsha2 upgrade-ha

1. Make sure that:
   • The VIP is reachable.
   • The current gateway is reachable.
   • The standby MN is reachable.
   • The database is synchronized.
   You can view the MN status by running zsha2 status.

   [root@localhost ~]# zsha2 status
   Status report from 192.168.195.200
   =================================
   Owns virtual address:            yes //MN1 has obtained a VIP. Only one MN can obtain VIP at the same time.
   Self 192.168.195.200 reachable:   yes //MN1 is reachable.
   Gateway 192.168.0.1 reachable:    yes //The current gateway is reachable.
   VIP 192.168.199.151 reachable:    yes //The VIP is reachable.
   Peer 192.168.196.125 reachable:   yes //MN2 is reachable.
   Keepalived status:               active //The Keepalived service is in the running state.
   ZStack HA Monitor:               active //The HA monitoring service is in the running state.
   MySQL status:                    mysqld is alive //The database is running properly.
   MN status: Running [PID:6500] //The MN is running properly.
   UI status: Running [PID:9785] http://192.168.195.200:5000 //The UI is running properly.
   
   Slave Status:
   -------------
                Slave_IO_Running: Yes //Slave I/O is running properly.
               Slave_SQL_Running: Yes //Slave SQL is running properly.
                      Last_Error:
           Seconds_Behind_Master: 0
                   Last_IO_Error:
                  Last_SQL_Error:
   
   Warning: Permanently added '192.168.196.125' (ECDSA) to the list of known hosts.
   Status report from 192.168.196.125 //View the status of MN2.
   ================================
   Owns virtual address:            no
   Self 192.168.196.125 reachable:    yes
   Gateway 192.168.0.1 reachable:    yes
   VIP 192.168.199.151 reachable:     yes
   Peer 192.168.195.200 reachable:   yes
   Keepalived status:               active
   ZStack HA Monitor:               active
   MySQL status:                    mysqld is alive
   
   Slave Status:
   -------------
                Slave_IO_Running: Yes
               Slave_SQL_Running: Yes
                      Last_Error:
           Seconds_Behind_Master: 0
                   Last_IO_Error:
                  Last_SQL_Error:
   
   Note: visit ZStack UI with http://192.168.199.151:5000

2. Prepare the following necessary packages:

   • ZStack Cloud ISO:
     • h84r: ZStack-Cloud-x86_64-DVD-4.8.38-h84r.iso
     • Download address: Click here
   • ZStack Cloud installation package:
     • Software: ZStack-Cloud-installer-4.8.38.bin
     • Download address: Click here

   • Upgrade script
     • Software: zstack-upgrade
     • Download address: Click here
     Note: The version of the upgrade script must be consistent with the version of the custom ISO and the bin package.

   Note: After you download the software, check the authentication code by using MD5 and verify that the code conforms with the release information.
3. Select a proper upgrade method. You can upgrade from the bin package or from ISO.
   If you upgrade the MN from the bin package, follow these steps:

   1. Import the new version of ZStack Cloud custom ISO to each MN and run the following command on each MN to upgrade the local repo to the latest version:

      [root@localhost ~]# bash zstack-upgrade -r ZStack-Cloud-installer-4.8.38.bin

   2. Run the following command on either MN. Then, both MNs will be upgraded.

      [root@localhost ~]# zsha2 upgrade-mn -peerpass password ZStack-Cloud-installer-4.8.38.bin

   If you choose to upgrade the MN from ISO, follow this step:

   1. Run the following command on either MN. Then, both MNs will be upgraded.

      [root@localhost ~]# zsha2 upgrade-mn -peerpass password ZStack-Cloud-x86_64-DVD-4.8.38-h84r.iso

      Note: In the preceding command, the -peerpass parameter is optional. You can use this parameter to set the SSH login password for the peer MN.

[root@localhost ~]# zsha2 collect-log
Collecting logs ...
Collected log: zsha2-log-2018-09-17T154358+0800.tgz

# Decompress the log package.
[root@localhost ~]# tar zxvf zsha2-log-2021-01-17T154358+0800.tgz
tmp/zsha2-log588815976/
tmp/zsha2-log588815976/zsha2-status.log
tmp/zsha2-log588815976/zstack-ha.log
tmp/zsha2-log588815976/keepalived.data
tmp/zsha2-log588815976/zs-vip-192.168.199.151.log
tmp/zsha2-log588815976/keepalived_status.log

1. On each MN, run the zsha2 stop-node command to stop the zsha2 service.
2. Shut down each MN.
3. Maintain each MN after shutdown.
4. After powering on each MN, start the server manually or through IPMI.
5. Wait for each MN to start and successfully boot the operating system.
6. On each MN, run the zsha2 start-node command to start the zsha2 service.
7. On each MN, run the zsha2 status command to check whether the zsha2 service is running properly.
8. On each MN, run the zstack-ctl status command to check whether the MN services and the UI are running properly.

1. On the failed MN, run the zsha2 stop-node command to stop the zsha2 service.
2. Try to recover the failed node. If the recovery fails, use ZStack Cloud installation package of the same version to repair the node or install a new node.
3. If you need to install a new node, follow these steps:
   1. Provision a backup server to make the hardware specifications more similar to those of the failed node.
   2. Install the basic operating system. After the installation is completed, configure the root password and network information, which must be consistent with that of the failed node. For more information, see Installation and Deployment.
   3. Install an HA suite for the new node. For more information, see Installation and Deployment.
   4. On the new node, run the zsha2 status command to check whether the zsha2 service is running properly.
   5. In the new node, run the zstack-ctl status command to check whether the MN service and the UI are running properly.

1. Try to recover the failed nodes. If the recovery fails, use ZStack Cloud of the same version to repair the nodes or install two new nodes.
2. If you need to install two new nodes, follow these steps:
   1. Provision two backup servers to make the hardware specifications more similar to those of the failed nodes.
   2. Install the basic operating system on each backup server. After the installation is completed, configure the root password and network information, which must be consistent with that of the failed nodes. For more information, see Installation and Deployment.
   3. Select a MN that is running properly, SSH to the MN via its IP address, and run the following command to recover the database on this MN:

      [root@localhost ~]# zstack-ctl restore_mysql -f /var/lib/zstack/mysql-backup/xxx.gz --mysql-root-password MYSQL_PASSWORD

      • In the preceding command, /var/lib/zstack/mysql-backup/xxx.gz is the file path and name of the backup database.
      • MYSQL_PASSWORD is the password of the database. Default password: zstack.mysql.password.
   4. On this node, run the install_ha command to reinstall an HA suite. For more information, see Use the Command Line.
   5. On the new node, run the zsha2 status command to check whether the zsha2 service is running properly.
   6. In the new node, run the zstack-ctl status command to check whether the MN service and the UI are running properly.