This document describes solutions to common issues encountered in management and
maintenance operations. The intended audience are as follows:
Technical support engineers
Maintenance engineers
Product consulting engineers
People who are interested in ZStack Cloud
Overview
ZStack Cloud is the next-generation IaaS software
designed mainly for future-oriented, smart data centers. Also, it manages multiple
compute, storage, and network resources in data centers by providing flexible and
comprehensive APIs. You can quickly set up your own smart cloud data center by using
ZStack Cloud, or construct flexible cloud usage scenarios.Figure 1. ZStack Cloud Framework
Management Node Maintenance
This section mainly describes how to maintain management nodes (MNs) in ZStack Cloud.
Considerations
To perform the maintenance of MNs, note the following:
Before you perform any maintenance like upgrade or reboot , disable the VM
HA globally. Method: On the main menu of the UI, choose Settings > Platform Setting > HA Policy. On the HA Policy page, disable the HA
Policy option. You can manually enable this functionality after the
maintenance is completed.
MN HA
Set Up a Dual-MN HA Environment
Overview
ZStack Cloud provides high availability (HA) for
dual-management node by using separate HA suites. When any one of the MNs is
disconnected, the HA will be triggered in seconds to ensure the business
continuity.
HA Mechanism: Each MN runs a zsha2 HA process. This process monitors the
critical services, including the MN service, UI service, and database service, on
each MN in real time. When any critical service is down, the Cloud immediately
triggers virtual IP (VIP) migration through Keepalived, and then attempts to restore
the downtime service.
Taking the dual-MN HA scenario as an example, this section introduces how to install,
deploy, and upgrade a dual-MN HA environment and other related considerations. For
detailed information, see Dual-MN HA Tutorial.
Installation and Deployment
Switch a Single-MN Scenario to a Dual-MN HA Scenario
Download the latest software packages, including ZStack Cloud ISO, ZStack Cloud
installation package and Multi-MN HA Suite.
Upgrade ZStack Cloud to the latest version on the
local MN (MN1).
Install the latest ZStack Cloud on the new MN
(MN2).
Install the license for MN2. In this scenario, the license type of two
MNs must be the same.
Import the HA suite to MN1. Then decompress and install the suite.
Deploy a New Dual-MN HA Scenario
Download the latest software packages, including ZStack Cloud ISO, ZStack Cloud
installation package and Multi-MN HA Suite.
Install the latest ZStack Cloud on the two new
MNs.
Install the license for the two new MNs. In this scenario, the license
type of two MNs also must be the same.
Import the HA suite to either MN. Then decompress and install the
suite.
Upgrade
Upgrade the HA Suite
After you obtain the new HA suite, you can use the suite
to upgrade the existing zsha2 service.
If the
databases of the two MNs are automatically synchronized, import the HA suite
to the active MN and decompress it. Run the following command on the active
MN to complete the upgrade of the HA
suite:
[root@localhost ~]# ./zsha2 upgrade-ha
Upgrade the MNs
In the dual-MA HA scenario, you must upgrade the HA suite
before upgrading the MNs.
Run the following commands on either MN.
Then, both MNs will be upgraded:
If you upgrade the MN from the bin package, run the following
command:
Note: The
-peerpass parameter is optional. You can
use this parameter to set the SSH login password for the peer
MN.
Considerations
MNs must meet the configuration requirements: 8 CPUs plus 12GB memory or
higher.
We recommend that you use a 10-Gigabit or above bandwidth for the management
network. If the management network is deployed independently, a 1-Gigabit
bandwidth is allowed.
We recommend that you create a static bridge via
zs-network-setting.
The multi-MN HA suite requires and must be paired with ZStack Cloud of 2.6.0 version and above. For example, HA suite
2.6.0 must be paired with ZStack Cloud 2.6.0.
The ZStack Cloud version and the license type of two MNs
must be the same.
To install the HA suite, you need to put zsha2 and zstack-hamon in
the same directory. During installation, zsha2 will automatically deploy
zstack-hamon and other related configuration files.
During installation, the database of the standby MN will be overwritten by that
of the active MN. Please exercise caution.
To ensure the real-time synchronization of monitoring data between two MNs, we
recommend that you configure SSH password-free login for the two MNs.
The Chrome browser used to access MNs requires version 49 or later.
MN Disaster Recovery
During operation, if an MN becomes inaccessible to the system due to hardware failures or
system crashes, you need to prepare two servers and set up an HA environment for the MN.
By using the remote database backup files, you can restore the MN service in ZStack Cloud and achieve disaster recovery.
Note:
Prepare a new server of the same model and configurations as the original MN
server.
If an OS failure occurs on the original MN, you can continue to use the
original MN server to reinstall ZStack Cloud.
Preparations
Local Database Backup
ZStack Cloud automatically performs daily backups at 12:30 AM
and 12:30 PM and retains backups for up to seven days by default. Backup files are
stored in the /var/lib/zstack/mysql-backup/ directory on the MN and
can be verified by using the crontab -l
command.
For the MNs under sustained peak loads, optimize backup schedules by using the
crontab -e command: set Backup Cycle to Backup by Hour and Data
Retention Policy to by count. A maximum of 200 local backup can be
reserved.
In case that MNs have no access to the system due to failures, we recommend that you
make remote database backups.
Assuming the MN IP is 172.20.12.16 and the remote backup server IP is
172.20.12.75, follow the steps on the MN
(172.20.12.16):
# Generate an SSH key on the MN
ssh-keygen
# Copy the SSH key to the remote backup server. First-time login requires manual verification by typing yes, followed by the root password.
ssh-copy-id root@172.20.12.75
Remote Database Redundancy
If two or more servers support data redundancy, you can make remote backup on the
remote backup server.
On the remote backup server (172.20.12.75), run crontab -e to
change crontab. You can schedule remote backups by using the
rsync command. This command copies the database directory
where the MN stores.
# At midnight and midnoon, back up the MN database to the /var/lib/zstack/from-zstack-remote-backup/ path. No more than 24 databases are allowed.
0 0,12 * * * zstack-ctl dump_mysql --host root@172.20.12.75 --d --keep-amount 24
MN Disaster Recovery
About this task
Assume that MNs have no access to the system due to
hardware or software failures, follow the steps below to achieve disaster recovery.
Procedure
Install the operating system (OS) for the MNs.
Install the OS according to ZStack Cloud ISO. You can
either reinstall the OS for the original MN, or select a new server as a new
MN.
Note: You have to install the working version prior to failures. For example, if
you use ZStack Cloud 3.8.0 before the failures occur,
redeploy the same version to ensure stability.
Update licenses.
When the installation of ZStack Cloud is completed,
you need to update the license before performing other operations.
Recover databases.
Select the latest database backup file on the remote backup server and name
it
like:
# To tell that the MN(172.20.1.123) backs up databases at 14:43:43 on May 7, 2022.
172.20.1.123-zstack-backup-db-2022-05-07_14-43-43.gz
Copy the file to the newly-installed ZStack Cloud.
Assuming the file is stored in the /root/ directory,
you can run the following command to restore databases. Before restoration,
ZStack Cloud will back up the current database.
All associated MN services will temporarily unavailable until the
restoration completes.
zstack-ctl restore_mysql -f /root/
172.20.1.123-zstack-backup-db-2022-05-07_14-43-43.gz --mysql-root-password zstack.mysql.password
// -f introduces the database path to be restored, that is,root/172.20.1.123-zstack-backup-db-2022-05-07_14-43-43.gz.
// --mysql-root-password zstack.mysql.password means using "zstack.mysql.password" as the root password of mysql.
Note:
If a host serves as a node with management and compute roles before
restoration, then the node is non-recoverable and must be manually
removed.
If you have set consoleProxyOverriddenIp before restoration, you
have to reset the parameter of the following file on the new MN
to restore the original
settings.
If a MN's database backup is not synchronized to the remote backup
server before a disaster occurs, such backup cannot be
recovered.
However, if you follow the steps above to recover the VM instances
created between the last backup cycle and a disaster, they may
become unmanaged after recovery. Whereas VM instances under other
circumstances will be in sync with their original counterpart.
Change MN IP Address
When you use ZStack Cloud, if the IP address of the server where
the MN is located changes, you need to change the IP address of the MN manually. This
involves two scenarios: single-MN scenario and dual-MN HA scenario.
Single-MN Scenario
If the IP address of the server where the MN is located changes,
follow these steps to change the IP address of the MN:
On the MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
Run the following command to manually change the NIC addresses of all servers
(Assume that bond0 is the management
network):
[root@localhost ~]# zs-network-setting -i br_bond0 172.31.250.100 255.255.255.0 172.31.250.1 #Enter the address as needed.
Run the following commands in sequence to change the IP address of the
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
Further Details
If the server where the MN is located is added as another node at the same time,
update the IP address of the corresponding node in the UI by following these
steps.
Note: Changing the IP address of a host could cause the host to be
disconnected. Please exercise caution.
Scenario One: The management node, compute node, and image storage (of the
ImageStore type) share the same server.
Log in to the UI. On the Hardware page,
find the target host and choose Action > Enter Maintenance Mode. Then disable the corresponding image
storage.
On the MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
Run the following commands in sequence to manually change the
NIC addresses of all servers (Assume that bond0 is the
management
network):
[root@localhost ~]# zs-restore-network-setting -i br_bond0 #Clear the IP address on the bridge.
[root@localhost ~]# zs-show-network #Check if the IP was successfully cleared.
[root@localhost ~]# zs-network-setting -i br_bond0 172.31.250.100 255.255.255.0 172.31.250.1 #Enter the address as needed.
Run the following commands in sequence to change the IP address
of the
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
Log in to the UI and change the IP address of the host and image
storage, respectively.
Note:
Changing the IP address of a host could cause the
host to be disconnected. Please exercise
caution.
After you change the IP address of an image storage,
you cannot perform operations on resources on the
image storage during the reconnection.
Scenario Two: The management node and storage node share the same server.
Here, the storage node uses the Ceph distributed deployment method.
On the MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
Change the management IP address of the Ceph storage.
Run the following commands in sequence to change the IP address
of the
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
Scenario Three: The management node, compute node, storage node, and image
storage share the same server. Here, the storage node uses the Ceph
distributed deployment method.
On the MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
Log in to the UI. On the Hardware page, find
the target host and choose Action > Enter Maintenance Mode. Then disable the corresponding image storage.
Change the management IP address of the Ceph storage.
Run the following commands in sequence to change the IP of the
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
Log in to the UI and change the host IP, image storage IP, and
monitor node IP of the Ceph primary storage in sequence.
Note:
Changing the IP address of a host could cause the host
to be disconnected. Please exercise caution.
After you change the IP address of an image storage, you
cannot perform operations on resources on the image
storage during the reconnection.
If you cannot change the monitor node IP of a Ceph
primary storage or image storage, contact the official
technical support for assistance.
Dual-MN HA Scenario
If the IP address of the server where the MN is located changes, follow these steps to
change the IP address of the MN:
On either MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
On each MN, run the following commands in sequence to manually change the NIC
addresses of all servers (Assume that bond0 is the management
network):
[root@localhost ~]# zs-restore-network-setting -i br_bond0 #Clear the IP address on the bridge.
[root@localhost ~]# zs-show-network #Check if the IP was successfully cleared.
[root@localhost ~]# zs-network-setting -i br_bond0 172.31.250.100 255.255.255.0 172.31.250.1 #Enter the address as needed.
On each MN, run the following commands in sequence to change the IP address of
each
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
On the MN of the backup database, run the install_ha command
to reinstall the HA suite.
Further Details
If the server where the MN is located is added as another node at the same time,
update the IP address of the corresponding node in the UI by following these
steps:
Scenario One: The management node, compute node, and image storage share the
same server.
Log in to the user interface (UI). On the
Hardware page, find the target host and
choose Action > Enter Maintenance Mode. Then disable the corresponding image
storage.
On either MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
On each MN, run the following commands in sequence to manually
change the NIC addresses of all servers (Assume that bond0 is
the management
network):
[root@localhost ~]# zs-restore-network-setting -i br_bond0 #Clear the IP address on the bridge.
[root@localhost ~]# zs-show-network #Check if the IP was successfully cleared.
[root@localhost ~]# zs-network-setting -i br_bond0 172.31.250.100 255.255.255.0 172.31.250.1 #Enter the address as needed.
On each MN, run the following commands in sequence to change the
IP address of each
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
Log in to the UI by using the MN IP address of the backup
database. Then change the IP address of the host and image
storage, respectively.
On the MN of the backup database, run the
install_ha command to reinstall the HA
suite.
Scenario Two: The management node and storage node share the same server.
Here, the storage node uses the Ceph distributed deployment method.
On either MN, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
Log in to the UI by using a virtual IP address (VIP). On the
Hardware page, find the target host and
choose Action > Enter Maintenance Mode. Then disable the corresponding image storage and
primary storage. On the Resource Pool page,
disable the VM instance.
Change the management IP address of the Ceph storage.
On each MN, run the following commands in sequence to change the
IP address of each
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
On the MN of the backup database, run the
install_ha command to reinstall the HA
suite.
Note: Modifying an IP address in Ceph scenarios is a heavy operation.
Please contact the official technical support for assistance.
Scenario Three:The management node, compute node, storage node, and image
storage share the same server. Here, the storage node uses the Ceph
distributed deployment method.
On either node, run the following command to back up the
database:
[root@localhost ~]# zstack-ctl dump_mysql
Log in to the UI by using a VIP. On the
Hardware page, find the target host and
choose Action > Enter Maintenance Mode. Then disable the corresponding image storage and
primary storage. On the Resource Pool page,
disable the VM instance.
Change the management IP address of the Ceph storage.
Note: Modifying
an IP address in Ceph scenarios is a heavy operation. Please
contact the official technical support for
assistance.
On each MN, run the following commands in sequence to change the IP
address of each
MN:
[root@localhost ~]# zstack-ctl change_ip --ip 172.31.250.110 #Here, 172.31.250.110 is the new IP address of the MN.
[root@localhost ~]# zstack-ctl config_ui --db-url jdbc:mysql://172.31.250.110:3306 #Modify UI database address.
[root@localhost ~]# zstack-ctl stop && zstack-ctl start #Restart the MN and UI service process for configurations to take effect.
Log in to the UI by using the MN IP address of the backup database.
Then change the host IP, image storage IP, and monitor node IP of
the Ceph primary storage in sequence.
Note: If the type of the image
storage is Ceph, change the monitor IP address of the Ceph image
storage. For more information, contact the official technical
support.
On the MN of the backup database, run the
install_ha command to reinstall the HA
suite.
Log in to the UI by using a VIP. On the
Hardware page, enable the host,
corresponding image storage and primary storage. On the
Resource Pool page, enable the VM
instance.
Switch from MN1 to MN2
To switch from the original MN (MN1) to a new MN (MN2), follow these steps:
Log in to MN1. Run the following command to back up the MN database and copy the
backup file to the /root/
directory:
[root@localhost ~]# zstack-ctl dump_mysql #Back up the database, assuming the file located at /var/lib/zstack/mysql-backup/172.20.10.2-zstack-backup-db-2018-12-13_17-57-52.gz is the backup file.
[root@localhost ~]# cp /var/lib/zstack/mysql-backup/172.20.10.2-zstack-backup-db-2018-12-13_17-57-52.gz /root #Copy the backup file to the /root/ directory.
On MN2, install a ZStack Cloud of the same version
with that on the original MN. Then add license to it.
On MN1, run the following command to stop its service:
[root@localhost ~]# zstack-ctl stop
On MN1, run the following command to copy its backup files to
MN2:
On MN2, run the following command to restore the
database:
[root@localhost ~]# zstack-ctl restore_mysql -f /root/172.20.10.2-zstack-backup-db-2018-12-13_17-57-52.gz --mysql-root-password zstack.mysql.password #zstack.mysql.password is the database password.
On MN2, run the following command to start
it:
[root@localhost ~]# zstack-ctl start
Optional: Given data security, if MN2 runs properly, you can run the following
command to remove the relevant directories and all files within on
MN1:
[root@localhost ~]# rm –rf /user/local/zstack
Upgrade a Compute Node to a Management Node
If you want to upgrade a compute node to a management node, follow these steps:
On the compute node scheduled for an upgrade, run the following commands to
verify if any processes are
running:
[root@localhost ~]# lsof +d /usr/local/zstack/ # List all open files accessing this directory
[root@localhost ~]# ps aux|grep usr.local.zstack # Verify if any active processes are using this file
Disable the host as the compute node scheduled for an upgrade.
Run the following command to disable the original
MN:
[root@localhost ~]# zstack-ctl stop
On the compute node scheduled for an upgrade, run the following commands in
sequence to stop the agent
service:
Run the following commands in sequence to upgrade the compute node to a
management
node:
[root@localhost ~]# mv /usr/local/zstack/ /mnt/ # mv /usr/local/zstack/ /mnt/
[root@localhost ~]# bash ZStack-Cloud-install.bin -E #Upgrade the node to a management node
[root@localhost ~]# cp -r /mnt/zstack/* /usr/local/zstack/ #Copy the previously backed-up files (from /mnt/zstack/) back to /usr/local/zstack/.
Enable the host and subsequent optional operations (for example, replace the
original management node with this one, or deploy a dual-MN environment).
Basic Platform Maintenance
This chapter provides solutions for errors encountered during fundamental operations
in ZStack Cloud.
Installation and Deployment
If you fail to install, deploy, or upgrade ZStack Cloud,
investigate the following factors:
Assuming you choose the management node mode when installing the operating
system (OS). When the installation of OS is completed, ZStack Cloud is not installed at all.
The causes could be as
follows:
The minimum hardware requirement is not satisfied: 4-core CPU and
8GB memory.
No IP address is set when the OS is installed. The management node
in ZStack Cloud requires the IP address.
Solution:
In this case, you need to run the bash /opt/zstack-installer.bin -E command to
install ZStack Cloud;
Alternatively, you can run the bash
/opt/zstack-installer.bin command to install ZStack Cloud Community.
The error message below presents itself when installing ZStack Cloud.
Error
message:
Fail Reason: /usr/local/zstack is existing. Please delete it manually before installing a new ZStack
Cause:
Now,
a ZStack Cloud already exists in the system. Repeated
installation is not allowed.
Solution:
To upgrade ZStack Cloud, simply use the
-u parameter.
To reinstall ZStack Cloud, run the rm
-rf /usr/local/zstack; bash zstack-installer.bin -D command.
If the deployment fails, error details will be output on the screen.
Solution:
Check the
/tmp/zstack_installation.log path. Then try to
solve this problem according to error details.
If an ISO version mismatch is encountered during the upgrade, an error message
similar to the following will be
displayed.
mount: /dev/loop0 is write-protected, mounting read-only
/root
Current release h79c not matched the new ISO h84r,
Use zstack-upgrade -r xxx.iso to update current repo,
Use zstack-upgrade -a/--add_repo xxx.iso to add a new repo
Solution:
You
need to follow the steps below:
Download the latest ISO by using the following
command:
Run the following command to perform the upgrade. This will
synchronize the yum repository from the ISO to the local system and
directly upgrade the management
node.
If an upgrade fails due to custom software development or database
modifications, you wish to restore ZStack Cloud to the
pre-upgrade state.
Prerequisite for restoration:
When upgrading ZStack Cloud, the current core files and databases are
backed up to the corresponding directory:
Core files in ZStack Cloud have been backed
up in the/usr/local/zstack/upgrade/ directory
by default. The backup files constitue a folder named the time when
the upgrade occurs, for example, 2017-11-09-15-41-52.
The database has been backed up in the
/usr/local/zstack/db_backup/directory by
default. The backup files constitue a folder named the time when
the upgrade occurs, for example, 2017-11-09-15-42-43, while
the files end with backup.sql.
Procedures for restoration:
To back up the core files in ZStack Cloud of
the previous version, run the following
command:
Start the management node service in ZStack Cloud. Then ZStack Cloud has been successfully restored to the
pre-upgrade
state.
zstack-ctl start
If you adopt all-in-one installation, we recommend you configure Reserved
Memory of Host after initialization. The method is as follows:
For global setting:
On the main menu of ZStack Cloud, choose Settings > Platform Setting > Global Setting > Basic Settings. Set Reserved Memory of Host.
The default is 1GB.
Note: If you set this parameter on the details
page of a cluster, this global setting does not take effect on
the cluster.
For a cluster:
On the main menu of ZStack Cloud, choose Resource Center > Hardware > Cluster. Select the target cluster and enter its details
page. Click Advanced Settings on the top row
and you can configure Reserved Memory of
Host. The default is 1GB.
Note: If you set this
parameter for certain host, this advanced setting of cluster does
not take effect on the host.。
For a host:
Currently, we support configuring Reserved
Memory of Host by using CLI command
line:
UpdateResourceConfig category=kvm name=reservedMemory resourceUuid=e2f4836723a2490095768fe611c540b1 value=10G
# resourceUuid is the host UUID, while value is the reserved memory size of the host.
Start Services
ZStack Cloud services will automatically start after
the management node (MN) is restarted.
If ZStack Cloud services fail to start, check the following:
The database should run properly:
You can check its status by
using the systemctl
status/start/stop/restart mariadb
command.
Troubleshooting case 1:
Error message
1:
ERROR: failed to connect to the mysql server[hostname:10.0.0.18, port:3306, user:zstack, password:zstack.password]
Solution
1:
Mae sure that the IP address, username and
password related to the database are correct and
accessible in the
/usr/local/zstack/apache-tomcat/webapps/zstack/WEB-INF/classes/zstack.properties
path. You can use the mysql
command to access it.
For example, you can connect to the mysql
server by typing mysql -u zstack
-pzstack.password -h 10.0.0.18.
Troubleshooting case 2:
Error message
2:
8080 is occupied by some process. Please use netstat to find out and stop it
Solution
2:
Run the netstat -anp|grep
8080 command to find out the occupied
8080 and stop it. Then restart ZStack Cloud.
Troubleshooting case 3:
Error message
3:
MN status: Unknown, the management node seems to become zombie as it stops responding APIs
Solution
3:
Insufficient memory may cause such
messaging failures. To resolve this issue, allocate
more memory and avoid reusing the management node
for compute tasks.
zstack-ctl command
ZStack Cloud provides zstack-ctl command to
control services and resources.
Common zstack-ctl commands are as follows:
Check the status:zstack-ctl status
Control the status of the management node or the UI service: zstack-ctl
stop/start/stop_ui/start_ui/start_node/stop_node/restart_node/
Install the license of ZStack Cloud: zstack-ctl install_license -f license_key.txt
Reset the password of the admin in ZStack Cloud:
zstack-ctl reset_password
Change the password of the MySQL database: zstack-ctl change_mysql_password --root-password zstack.mysql.password
--user-name zstack --new-password password
Note:
The zstack-ctl command has multiple subcommands. For detailed
information about what these subcommands are and how to use them, see CTL Command
Manual.
zstack-cli command
ZStack Cloud provides zstack-cli , a command
line tool, to control the internal resources in ZStack Cloud.
The
admin account uses a default login password: password. If you change the
password, log in with the correct password.
Keywords in zstack-cli Command:
Keywords of resources:
zone cluster host vm primarystorage image L2 L3 backupstorage volume offering
Keywords for operations:
add reconnect start stop destroy delete create get set update attach detach
Tips for using zstack-cli commands:
Enter keyword(s) of resource(s) and keyword(s) for operation(s).
Utilize the Tab key. For example, when you type the keyword "cluster", press
Tab twice and all operations related to cluster will be listed, as shown
below:
To
create another cluster, enter CreateCluster based on
the operation list above. Again press Tab twice. The CLI will display
the parameters for CreateCluster.
If
the name contains the word "test", only three uuids of three VMs will be
displayed.
Note:
For detailed information about how to use zstack-cli, see
CLI Command Manual.
VM-Related Error
VM Instance Creation Procedures and Failure Analysis
To create a VM instance, follow the steps below:
Associate the selected image with its image storage. Ensure that the image
storage is in the Connected status.
Select an available cluster based on the image storage. And then select a
compatible host according to instance offering and the cluster. Ensure that
the target host is in the Connected status.
Select an available primary storage of the capacity not smaller than that of
the selected image. The primary storage must be Connected with
sufficient available capacity.
Configure network settings (IP/MAC address allocation). Ensure the network
availability as well as sufficient resources in the IP address pool.
Download images from the image storage to the primary storage. And create a
root volume.
Assign IP addresses to the VM instance.
Enable the VM on the host.
If you fail to create a VM instance, failure causes could be as follows:
The image storage is in the Disconnected status. Make sure the image
storage is in the Enabled state and in the Connected status.
The primary storage is not attached to any cluster. In this case, you cannot
find an available host. Remember to attach a cluster to the primary
storage.
No hosts in the cluster meet the requirements of the instance offering.
Verify whether there is an enabled host with a sufficient available quantity
of CPU and Memory.
The capacity of the primary storage does not match the image capacity.
Ensure sufficient primary storage capacity.
The cluster is not attached to any network or NIC. Attach the corresponding
NIC or network to the cluster.
IP addresses are insufficient within the range.
Download images from the primary storage to the image storage. Ensure a
correct transfer direction.
Ensure that the built-in dnsmasq software and DHCP service provided by the
host work properly so as to normally assign IP addresses.
Ensure that the Libvirtd service provided by the host run properly so as to
perform virtualization management tasks.
Resolve VM-Related Errors
Error Message
Cause and Solution
“code”:”HOST.1005”,”description”:”Failed
to start vm on hypervisor”, “details”:”failed to start ….,
Libvirt error: internal error no supported architecture for os
type ‘hvm’”or "details": "failed to start vm on kvm host,
because unable to start vm, libvirt error: invalid argument:
could not find capabilities for domaintype=kvm
"
Possible Causes:
Hosts must support and have hardware
virtualization (such as VMX or SVM) enabled. This error
typically occurs when CPU virtualization is disabled in
the BIOS.
Solutions:
Reboot the system and enter the host BIOS to
enable CPU virtualization support.
“details”: "cannot find either 'vmx' or
'svm' in /proc/cpuinfo, please make sure you have enabled
virtualization in your BIOS setting"
Possible Causes:
Hosts must support and have hardware
virtualization (such as VMX or SVM) enabled. This error
typically occurs when you add a host on which CPU
virtualization is not supported.
Furthermore, the
host you add is a VM instance. You need to enable the
Nested Virtualization feature inside the VM.
Solutions:
To know how to enable Nested Virtualization,
please contact the VM supplier for assistance.
"details": "the local primary storage has
no hosts with enough disk capacity[xxx bytes] required by the
vm[uuid:xxx]"
Possible causes:
There is no enough primary storage
capacity (disk space) available in the current system
(or on any single host).
When allocating VM
instances and volumes, the thin clone mode is used,
meaning only the amount of space actually used by the VM
is allocated. However, when creating a VM instance, the
primary storage capacity is deducted a maximum number
required for VM usage. Such setting prevents the VM from
eventually using more space than the available space in
the system.
Additionally, this error also occurs when the local
primary storage has no hosts with enough remaining
capacity to create a new VM instance. Because a VM's
disk cannot deliver storage service across different
hosts.
Solutions:
Delete some unnecessary VM instances, or
adjust the instance offering used by the VM
instance.
If you understand what Primary Storage
Overcommitment is and how to use this parameter, you can
also increase the overcommitment ratio to obtain more
overcommitment space.
"details": "unable to allocate hosts; due
to pagination is enabled, there might be several allocation
failures happened before; the error list is [{no host having
cpu[x HZ], memory [xxx bytes] found}]"
Possible Causes:
There is no enough CPUs or memories in
the current system (or on any single host).
Solutions:
Delete some unnecessary VM instances, or adjust
the instance offering used by the VM instance. If you
understand what Memory Overcommitment is and how to use
this parameter, you can also increase the overcommitment
ratio to obtain more overcommitment capacity.
"details": "failed to start vm[uuid:xxx
name:xxx] on kvm host[uuid:xxx, ip:x.x.x.x], because unable to
start vm[uuid:xxx, name:xxx], libvirt error: internal error:
early end of file from monitor: possible problem: Cannot set up
guest memory 'pc.ram': Cannot allocate memory" set up guest
memory 'pc.ram':Cannot allocate memory"
Possible causes:
In the current system, available physical
memory is not enough to create a new VM instance. When
this error occurs, you cannot increase the memory size
by increasing the memory overcommitment ratio
instead.
Solutions:
Decrease the instance offering of the VM
instance and increase the memory size or the number of
swap partitions (lower system performance).
"failed to migrate vm[uuid:xxx] from kvm
host[uuid:xxx, ip:xxx] to dest host[ip:xxx], unable to migrate
vm[uuid:xxx] to qemu+tcp://xxx/system, Unsafe migration:
Migration may lead to data corruption if disks use cache !=
none"
Possible causes:
You select writethrough or writeback for
Cache Mode in the Global Setting. The system considers
such migration unsafe.
Solutions:
You need to select the value None for Cache
Mode in the Global Setting before migration.
"description": "Failed to start vm on
hypervisor",libvirt error: Unable to create tap device vnic15.0:
Device or resource busy
Possible causes:
The compute node vnic15.0 is added again
on the condition that a VM has existed on the host.
Hence a device conflict.
Solutions:
After adding a host, we recommend you maintain
a clean environment or delete the running VM on it.
You do not grant the Read/Write
permission to the VM or data volume you have
created.
Solutions:
Grant Read/Write permission on the NFS
server.
internal error: unable to execute QEMU
agent command 'guest-set-user-password': failed to set password:
The user name could not be found.
Possible causes:
You specify a non-existent user name when
setting the VM password.
Solutions:
Specify an existing user name.
internal error: unable to execute QEMU
agent command 'guest-set-user-password': The command
guest-set-user-password has not been found
Possible causes:
You do not install qemu-ga or the agent
you have installed need to be updated when setting the
VM password.
Solutions:
You can run the qemu-ga -V
command to check the version.
CentOS7.2: version 2.3.0
Windows Server 2012 R2: version 0.12.1
Ubuntu 14.04: version 2.5 (Default: version 2.0.
You have to manually update the version.)（
"libvirt error: error from service:
ListActivatableNames: Connection is closed" or "libvirt error:
Activation of org.freedesktop.machine1 timed
out"
Possible causes:
This error typically occurs when the system DBUS process crashes or malfunctions.
Solutions:
Method One: Restart the host on ZStack Cloud, and DBUS will
automatically recover.
Method Two: Run the following commands on the
host:
service systemd-machined
restart
service libvirtd restart
Re-connect the host through UI or CLI.
{ "error": { "code": "HOST.1009",
"description": "Failed to migrate vm on hypervisor", "details":
"failed to migrate vm[uuid:90bb61f55e774f5f89c9e1ea46db7661]
from kvm host[uuid:e5105c61114a4efe8bcc025f744226bd,
ip:10.0.247.244] to dest host[ip:10.0.197.238], unable to
migrate vm[uuid:90bb61f55e774f5f89c9e1ea46db7661] to
qemu+tcp://10.0.197.238/system, internal error: Attempt to
migrate guest to the same host localhost.domain.com" },
"createdTime": 1488170137857, "type": { "_name":
"key.event.API.API_EVENT" }, "id":
"e6bcd7c33ad9446b919e11cd647fc8dc" }
Possible causes:
This error occurs when two hosts have the
same name. KVM migration identifies different hosts
according to their names. Two hosts with the same name
are identified the same one.
Solutions:
Run the hostnamectl set-hostname
your_new_hostname to set a new host name
that is different from other host names.
all mons failed to execute http
call[/ceph/primarystorageolume/clone
Solutions:
Reconnect Image Storage and Local Backup
server.
failed to start
vm[uuid:4ef5a9e4ae6441c7b046c384c3ae6f7f name:stack_local] on
kvm host[uuid:d0d9800a09c34da4891e0cc624a0d349,
ip:192.168.1.166], because Traceback (most recent call last):
File
"/var/lib/zstack/virtualenv/kvm/lib/python2.7/site-packages/kvmagent/plugins/vm_plugin.py",
line 2721, in start_vm self._start_vm(cmd) File
"/var/lib/zstack/virtualenv/kvm/lib/python2.7/site-packages/kvmagent/plugins/vm_plugin.py",
line 2664, in _start_vm 'unable to start vm[uuid:%s, name:%s],
libvirt error: %s' % (cmd.vmInstanceUuid, cmd.vmName, str(e)))
KvmError: unable to start
vm[uuid:4ef5a9e4ae6441c7b046c384c3ae6f7f, name:stack_local],
libvirt error: internal error: early end of file from monitor,
possible problem: qemu-kvm:
/root/rpmbuild/BUILD/qemu-2.6.0/target-i386/kvm.c:1736:
kvm_put_msrs: Assertion `ret == n' failed.
Possible causes:
This error occurs when you select
host-passthrough for CPU Mode.
Solutions:
Select another option for CPU Mode.
”code”:”SYS.1006”,description:An
operation failed, details:failed to check physical network
interfaces on lvm
Possible causes:
You fail to check the physical NIC on the
host.
Solutions:
The physical NIC has been attached to the
cluster where the host is located. Verify whether this
NIC is present on the host when adding a host.
description": "A message or a operation
timeout","details": "[Async Http Timeout] url:
http://10.10.10.253:7272/init, timeout after
300000[MILLISECONDS], command:
{\"uuid\":\"78183b0a46094e30bc8a6128b30ee8cb\",\"restartDnsmasqAfterNumberOfSIGUSER1\":0}"
Possible causes:
The management node's IP address likely
lacks external network connectivity.
This commonly
occurs on multi-NIC servers where the management service
was installed using the first NIC configured with the
default route, which may not have proper external
access.
Solutions:
Open the file at
/usr/local/zstack/apache-tomcat/webapps/zstack/WEB-INF/classes/zstack.properties.
Modify
management.server.ip =
xxx.xxx.xxx.xxx to make it have external
network connectivity.
failed：Error in connection establishment:
net::ERR_CONNECTION_REFUSED
Possible causes:
You do not set a console proxy
address.
Solutions:
Log in to user interface (UI), choose Operational Management > Access Control > Console Proxy. On the Console
Proxy page, set the console proxy
address.
Could not access KVM kernel module: No
such file or directory failed to initialize KVM: No such file or
directory.
When errors occur in daily operations and maintenance (O&M), you can follow the steps
below to analyze the errors:
Analyze UI error messages first:
Example: If VM creation fails because no host found,
possible causes include: No available hosts match the selected image
storage, cluster, and network.
Check whether there are any available
hosts in the cluster as well as any L2 networks and primary
storages attached to the cluster.
Investigate management node logs at
/usr/local/zstack/apache-tomcat/logs/management-server.log.
Read error messages and conduct context-specific analyses of failed
workflows.
Also, you can track API calls and analyze them. For example, if you use
APICreateVmInstance to create a VM, you can search
this API on the management node log and analyze the workflow step by
step so as to find the failure cause.
Review agent logs for error details:
Example: Check /var/log/zstack/zstack-kvmagent.log
on hosts for error logs when performing operations that contrl the VM
life-cycle.
Port Occupancy
The list of port occupancy on ZStack Cloud is as follow:
ZStack Cloud ISO provides the following network scripts to
simplify network configurations:
Run the command zs-show-network to check network connection and
the status of Link
Aggregation.
[root@localhost ~]# zs-show-network
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: vmnic0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:84:ba:34:92:00 brd ff:ff:ff:ff:ff:ff
inet 172.20.54.194/16 brd 172.20.255.255 scope global vmnic0
valid_lft forever preferred_lft forever
inet6 fe80::f884:baff:fe34:9200/64 scope link
valid_lft forever preferred_lft forever
----------------------------------------------------------------
| Bond Name | SLAVE | BONDING_OPTS |
----------------------------------------------------------------
Run the command zs-network-setting to configure physical NIC
network settings. This command supports two usage modes:
Mode
One:
# The parameter -i, meaning interface, helps to set network IP address.
zs-network-setting -i [interface] [ipaddress] [netmask] [gateway]
Interface IP Adderss Netmask Gateway
# Example 1: zs-network-setting -i eth0 192.168.1.10 255.255.255.0 192.168.1.1
# Example 2: zs-network-setting -i eth0.10 192.168.1.10 255.255.255.0 192.168.1.1
Mode
Two:
# The parameter -b, meaning bridge, helps to create a bridge based on the interface and set the network IP address.
zs-network-setting -b [interface] [ipaddress] [netmask] [gateway]
Interface IP Adderss Netmask Gateway
# Example 1: zs-network-setting -b eth0 192.168.1.10 255.255.255.0 192.168.1.1
# Example 2: zs-network-setting -b eth0.10 192.168.1.10 255.255.255.0 192.168.1.1
The command zs-change-nic helps to change NIC names. After you
change the NIC name, the relevant script path will be set at
/etc/udev/rules.d/70-persistent-net.rules. If you
reboot the host, the NIC name remains to be new NIC
name.
# The parameter -c means changing the NIC name.
zs-change-nic -c [old-nic-name] [new-nic-name]
Old NIC Name New NIC Name
# Example: zs-change-nic -c eth0 em01
The command zs-vlan helps to set VLAN interface. This command
supports two usage modes:
Mode One: Create
VLAN
# The parameter -c means creating VLAN interface.
zs-vlan -c [nic-name] [vlan]
NIC Name VLAN
# Example: zs-vlan -c zsnic0 10
Mode Two: Delete
VLAN
# The parameter -d means deleting VLAN interface.
zs-vlan -d [nic-name] [vlan]
NIC Name VLAN
# Example 1: zs-vlan -d zsnic0 10
# Example 2: zs-change-nic -c eth0 em01
Note:
After creating a VLAN interface, the system generates corresponding
network configuration files in
/etc/sysconfig/network-scripts/. These
configurations also take effect after host reboots. Deleting the
VLAN interface removes these configuration files.
When using this VLAN interface for cross-device communication, you
need to configure your network equipment according to the vendor's
VLAN implementation guide to enable traffic flow for this VLAN
interface.
For port-group configurations on network equipment, you can refer
to:
The command zs-bond-lacp helps to create and
delete Bonding Link Aggregation. This command supports two usage modes:
Mode One: Create Link
Aggregation
# The parameter -c means creating Link Aggregation interface.
zs-bond-lacp -c [bond-name]
Bonding interface
# Example: zs-bond-lacp -c bond0
Mode Two: Delete Link
Aggregation
# The parameter -d means deleting Link Aggregaion interface.
zs-bond-lacp -d [bond-name]
Bonding inerface
# Example: zs-bond-lacp -d bond0
Note:
This command creates an interface based on the IEEE 802.3ad Dynamic Link
Aggregation Protocol (LACP), which requires you to configure a port
group of the network switch on the host access layer.
The command zs-nic-to-bond helps to configure Link
Aggregation.
# The parameter -a means attaching a physical interface to a Link Aggregation interface.
zs-nic-to-bond -a [bond-name] [nic-name]
Bonding Interface Physical Interface
# Example: zs-nic-to-bond -a bond0 em1
# The parameter -d means detaching a physical interface from a bonsing interface.
zs-nic-to-bond -d [bond-name] [nic-name]
Bonding Interface Physical Interface
# Example: zs-nic-to-bond -d bond0 em1
Note:
After running this command, the execution interface will show itself,
displaying the relationship between the bonding interface and the
physical interface. For
example:
In this example, bond0 represents the bonding interface, while enp1s0f0 and enp1s0f1 are
the physical interfaces, showing their binding
relationship.
Additionally, the physical ports of the network device corresponding to enp1s0f0 and
enp1s0f1 form a port group. You must configure the port group
based on the specific network device model.
For port-group configurations on network equipment, you can
refer to:
Delete is a high-risk operation on ZStack Cloud for this operation
directly impacts data plane. Improper operation may lead to severe consequences such as
data loss. This section summarizes high-risk delete scenarios.
Action
Consideration
Delete Zone
Deleting a zone will delete all of its
sub-resources, for example, clusters, hosts, networks, primary
storages, vCenters and so on. Please exercise caution.
Delete Cluster
Deleting a cluster also deletes all the hosts in the
cluster.
If the cluster has a Local Storage attached, you will also
delete the VM instances, volumes, and snapshots on the hosts
of the cluster. Please exercise caution.
Delete Host
Deleting a host also deletes all the VM instances on the
host.
If the cluster to which a host belongs has attached local
storage, this operation will delete all VM instances and
volumes on the host. Please exercise caution.
If a host is designated as the migration server, this
operation will delete the migration server and the migration
task on the migration server will be cancelled. Please
exercise caution.
Delete Primary Storage
Deleting a primary storage deletes all resources
on it, for example, VM instances, volumes, snapshots and so on.
Please exercise caution.
Delete Image Storage
Deleting an image storage will delete the image
storage and image records on the Cloud. Please exercise
caution.
Delete VXLAN Pool
Deleting a VXLAN pool also deletes the VXLAN
networks in the pool and the associated L3 networks.
Detach L2 Network
Deleting an L2 network deletes associated L3
networks and VPC vRouters, and detaches associated VM NICs. Please
exercise caution.
Delete Public Network
Deleting a public network also detaches the VM
NICs that are using this network, and deletes the associated VPC
vRouters, network services, and vRouter offerings. Please exercise
caution.
Delete System Network
Deleting a system network deletes the associated
VPC vRouters.
Delete Private Network
Deleting a private network detaches the VM NICs
that are using this network.
Delete vCenter
Deleting vCenter deletes all resources associated with vCenter.
Please exercise caution.
Delete Deployment Server
Deleting deployment servers will expunge the
deploying baremetal instances, while the consoles of the deployed
baremetal instances cannot be accessed. Please exercise
caution.
Delete VPC vRouter
Delete a VPC vRouter may render network services
for associated VMs unavailable. You can only restore network
services by rebooting the VM instance. Please exercise
caution.
Delete vRouter Image
Deleting a vRouter image may lead to the
inability to create vRouters. Please exercise cation.
Delete vRouter Offering
Deleting a vRouter offering may lead to the
inability to create vRouters. Please exercise caution.
Delete VM Instance
After deletion, all resources associated with the VM
instance will be deleted as well.
After deletion, the VM's status will be marked as deleted.
The system will expunge the VM in 24 hours by default.
If you choose to delete a VM, all resources associated with
the VM will be expunged irrecoverably. Please exercise
caution.
When deleting a VM, if you select the check-box of detaching
volumes attached synchronously, then all data volumes
(except for shared volumes) attached to the VM will be
deleted.
Delete Snapshot
Snapshots of Local Storage, NFS primary storage, Shared
Mount Point (SMP), and Shared Block primary storage are
displayed in a hierarchical tree. Deleting the current
snapshot also deletes snapshots on its branch. Please
exercise caution.
Snapshots created in Ceph primary storages are independent
of each other. Deleting a snapshot does not affect other
snapshots.
In terms of shared volumes on Shared Block primary
storage, you cannot create snapshots.
Delete Drive
Deleting a drive will detach the existing ISO
within the drive.
Delete Auto-Scaling Group
Deleting auto-scaling groups will also delete all
VM instances in the groups. Please exercise caution.
Delete Security Group
Deleting a security group also deletes the
security rules created in the group. Please exercise
caution.
Delete VIP
Deleting a VIP also deletes all services the VIP
provides.
Delete EIP
Deleting an EIP will automatically remove the
provided EIP service. To delete the corresponding VIP, select the
"Delete VIP" option.
Delete Port Forwarding
Deleting a port forwarding rule will
automatically remove the provided port forwarding service. The
corresponding VIP and other services bound to it remain
unaffected.
Delete Load Balancer
Deleting a load balancer will automatically
remove all listeners and associated load balancing services. The
corresponding VIP and other bound services remain
unaffected.
Delete Listener
Deleting a listener will automatically remove the
provided load balancing service.
Delete IPsec Tunnel
Deleting an IPsec Tunnel will automatically
remove the provided IPsec tunnel service. The corresponding VIP and
other bound services remain unaffected.
Delete Alarm
After you delete an alarm, resources related to
this alarm will be removed and no more alarm messages will be
generated. Please exercise caution.
Delete Scheduler
Deleting a scheduler causes the scheduled job on
it to be detached. You can attach scheduled job to other running
schedulers.
Delete Resource Stack
Deleting a resource stack expunges all created
resources orchestrated by the resource stack, such as VM instances
and volumes. Please exercise caution.
Delete Pricing List
Deleting historical pricing lists does not affect existing
bills.
Deleting the pricing list in use triggers immediate
generation of the bill (billing period: last bill generation
time to deletion time). Billing will permanently stop
afterward.
Delete AccessKey
Deleting an AccessKey immediately disables all
API access authorized by it and the console cannot be accessed.
Please exercise caution.
Delete Account
If you set Deletion Policy to Direct, VMs and
volumes under this account will be expunged as soon as you apply
this policy. If you set Deletion Policy to Delay, VMs and volumes
under this account will be marked as deleted. Resource ownership
transfers to admin.
Delete User in Tenant Management
If a user is part of a ticket flow, deleting this
user will disable this ticket flow, and all tickets associated with
this flow will be withdrawn.
Detach Resources
Detach is a high-risk operation on ZStack Cloud. This section
summarizes all high-risk detach scenarios.
Action
Consideration
Detach Cluster from Primary Storage
This operation has the following impacts, please
exercise caution:
This operation may shut down all VM instances and VPC
vRouters on the primary storage.
All volumes on the primary storage are unable to work
properly.
Detach L2 Network from Cluster
Detaching an L2 network also detaches the
corresponding VM NIC. Please exercise caution.
Detach Primary Storage from Cluster
This operation has the following impacts, please
exercise caution:
This operation may shut down all VM instances and VPC
vRouters on the primary storage.
All volumes on the primary storage are unable to work
properly.
Detach Baremetal Cluster
Detaching baremetal clusters will expunge the
deploying baremetal instances, while the consoles of the deployed
baremetal instances cannot be accessed. Please exercise
caution.
Detach Cluster from VXLAN Pool
Detaching a cluster from a VXLAN pool also
detaches the associated VM NICs. Please exercise caution.
Detach Cluster from L2 Network
Detaching a cluster from an L2 network also
detaches the associated VM NICs. Please exercise caution.
Detach VPC vRouter from VPC Network
To detach VPC vRouter from VPC network, ensure
that the VPC network is not accessed by any VM instance.
Detach Volume from VM Instance
Detach the added volumes from the VM in the running or
stopped state.
Detaching volumes from a running VM may affect the business
continuity. Please exercise caution.
Detach NIC from VM Instance
You can dynamically detach NICs (detach network) from the
VM.
Detaching a NIC dissociates other associated network
services. Please exercise caution.
Detach ISO
Detach the added ISO image from the VM instance in the
running or stopped state. You can perform bulk action.
Detaching ISO from a running VM instance may affect the
business continuity. Please exercise caution.
Detach LUN
You can detach LUNs from a VM instance in the
running and stopped state.
Detach pGPU Device
Hot plugging is supported when you detach pGPU devices from
a VM instance.
Detaching a pGPU from a running VM instance may lead to blue
screen or system pause. We recommend you power off the VM
before detaching pGPU devices.
Detach vGPU Device
Detach vGPU devices from a VM instance. You must
ensure that the VM instance is in the stopped state before attaching
or detaching vGPUs.
Detach USB Device
You can detach USB devices from a VM instance in
the running and stopped state.
Other Platform Maintenance
Upgrade Shared Volumes on SharedBlock Primary Storage
To upgrade shared volumes on SharedBlock primary storage, follow these steps:
Run the following command to check whether there are any volumes need to be
converted into the correct format among the detached shared volumes to be
upgraded:
Ensure the data integrity after format conversion. Then run the following
command to delete those shared volumes which were originally in qcow2
format:
This chapter provides operational guidelines for physical infrastructure maintenance. For
operations in production environment, you must follow the standardized procedures
documented here to avoid unmanageable risks.
Power Maintenance in Equipment Room
Planned Shutdown
To perform planned shutdown in an equipment room, follow these steps:
Stop all running businesses and power off all VM instances.
Make all primary storages enter maintenance mode.
Make all hosts enter maintenance mode.
Stop the management node (MN). Methods of stopping MNs vary depending on
deployment scenarios:
In a single-MN scenario, run the zstack-ctl stop command to stop the MN.
In a dual-MN HA scenario, run the zsha2 stop-node
command on each MN to stop MNS and the zsha2 service.
Verify that no I/O operation is performed in the primary storages of the
Cloud, and run the poweroff command on each host to
normally shut down all hosts.
Disable the storages such as NFS, Ceph, SAN, or SMP.
Power off the switches and other hardware facilities
Planned Power-On
To perform planned power-on in an equipment room, follow these steps:
Power on the switches.
Enable the storages such as NFS, Ceph, SAN, or SMP.
Power on the servers.
Check the status of the MN to make sure that the MN service starts
successfully. Methods of starting MN service vary depending on deployment
scenarios:
In a single-MN scenario, run the zstack-ctl status command to view its status.
In a dual-MN HA scenario, on either MN, run the zsha2
status command to view the HA mode and the MN
status.
Log in to ZStack Cloud and enable all primary
storages and hosts. Make sure that all hosts and primary storages are in the
connected state.
Enable the VM instances and resume the businesses.
Recovery from Unexpected Power Outage
The steps to recover from unexpected power outage in an equipment room are the same
as those in Planned Power-On:
Power on the switches.
Enable the storages such as NFS, Ceph, SAN, or SMP.
Power on the servers.
Check the status of the MN to make sure that the MN service starts
successfully. Methods of starting MN service vary depending on deployment
scenarios:
In a single-MN scenario, run the zstack-ctl status command to view its status.
In a dual-MN HA scenario, on either MN, run the zsha2
status command to view the HA mode and the MN
status.
Log in to ZStack Cloud and enable all primary
storages and hosts. Make sure that all hosts and primary storages are in the
connected state.
Enable the VM instances and resume the businesses.
Equipment Room Relocation
To move the equipment room to another location, please follow the steps below:
Power off all involved facilities under normal conditions in sequence by
referring to Planned Shutdown.
Mark the access ports of all switches and hosts.
Pack up the servers. Move the servers to the new equipment room and reconnect
them.
Power on all involved facilities in sequence by referring to Planned
Power-On.
Check the recovery status of the network, Cloud, and VM instances.
Switch Maintenance
Management Network Switch Maintenance
The maintenance of management network switches may affect business continuity.
Please exercise caution. If this kind of maintenance is necessary, we
recommend you schedule this operating during off-peak business hours. The
maintenance procedures are as follows:
Disable HA Policy in the Platform Setting.
Stop management nodes (MNs). Methods of stopping MNs vary depending on
deployment scenarios:
In a single-MN scenario, run the zstack-ctl stop command to stop the MN.
In a dual-MN HA scenario, run the zsha2 stop-node
command on each MN to stop MNS and the zsha2 service.
Adjust or reboot the related management switches.
Verify the IP connectivity of all hardware resources at the access end of
the switch (including host, primary storage, and image storage) to ensure
that all the management networks communicate with each other
properly.
Note: If a node fails to connect the MN, troubleshoot the issue
before you proceed with subsequent operations.
Enable the MN and check its status to ensure that the MN service starts
successfully. Methods of starting MN service vary depending on deployment
scenarios:
In a single-MN scenario, run the zstack-ctl start command to start the MN service and run
the command zstack-ctl status
to view its status.
In a dual-MN HA scenario, run the zsha2 start-node
command on each MN respectively to start MN services. On either MN,
run the zsha2 status command to view the HA mode
and the MN status.
Ensure that all hosts and primary storages are in the connected status, and
HA Policy is enabled in the Platform Setting.
Business Network Switch Maintenance
We recommend you schedule the maintenance of business network switches during
off-peak business hours so as to avoid affecting business continuity. The
maintenance procedures are as follows:
Make the host connected to the switch enter maintenance mode.
Reboot the related switch.
Enable any host in the maintenance mode and check its connectivity. For
example, supposing you create a test VM instance and then check its
connectivity with other business VMs. If the connectivity succeeds, the
switch interface configurations corresponding to the host is correct and
available. If the connectivity fails, you have to continue to check the
connectivity.
Repeat Step 3 and check the connectivity in turn to ensure that all business
networks are available.
Storage Network Switch Maintenance
The maintenance of storage network switches will directly affect all business, so you
must stop all business before you perform the maintenance. The maintenance
procedures are as follows:
Make all primary storages associated with the storage switch on the Cloud
enter maintenance mode.
Log in to the user interface (UI) of Ceph Enterprise and choose Setting > Disable Data Recovery > Ban.
Adjust or reboot the related storage switches.
Ensure that the storage networks of all storage nodes are interconnected and
accessible.
Log in to the UI of Ceph Enterprise and choose Setting > Enable Data Recovery > Enable.
Enable the primary storages that are in the maintenance mode and make sure
that the primary storages are in the connected state.
Note: The