Set Up Management Node HA

Management node HA improves the availability of the ZSphere management plane. After management node HA is configured, if the active management node becomes unavailable, the standby management node can continue to provide management services.

Before setting up management node HA, prepare two servers on which the management node service has been installed, and plan the management network, business network, and VIP in advance. You can complete the setup from the user interface, or follow the CLI workflow to install the operating system, configure networks, and install the HA suite in sequence.

Set Up with the UI

Before you begin

You must install management node services on the selected two servers. For installation instructions, see Install the Management Node.
Ensure both management nodes have gateway addresses on the same network segment and have consistent NIC names.
Ensure both management nodes are running the same version.
Ensure the operating system type and architecture are supported and both management nodes run compatible operating system types.
Ensure both management nodes use the same license type.

Procedure

Log in to the UI management interface of either management node.
For a new environment that has not been initialized:
1. In the Welcome to Initialization Wizard dialog, click Next. The system automatically detects the management node HA status.
2. In the Set Up MN HA dialog, click Go to MN Ops.
3. A new browser tab opens displaying the MN Ops page.
Figure 1. Go to MN Ops from the Initialization Page
For an existing environment:
1. In the navigation pane, choose Reliability > MN Monitoring.
2. On the MN Monitoring page, click Go to MN Ops.
3. A new browser tab opens displaying the MN Ops page.
Figure 2. Go to MN Ops from the MN Monitoring Page
On the MN Monitoring page in MN Ops, click Set Up MN HA.

Figure 3. MN Monitoring
In the Set Up MN HA dialog, complete configuring MN, reviewing configuration, and setting up MN HA.
1. For Configure MN, set the following parameters:
  - VIP: Configure the VIP for accessing the management interface of the management node environment.
    Note: Ensure the VIP is in the same network segment as the active management node IP.
  - Active MN IP: Display the IP address of the active management node. Specify the SSH username and password for the active management node.
  - Standby MN IP: Specify the IP address, SSH username, and SSH password for the standby management node.
    Note:
    
    Ensure the standby management node IP is in the same network segment as the active management node IP.
    
    When setting up management node HA, the management node password can contain English letters, numbers, and these special characters: -=[];,./~!@#$%^&*()_+|{}:<>?`. If your password does not meet this rule, update it before setting up management node HA.
  - Time Sync Server: Uses the active management node IP as the time synchronization server by default.
  - Force Sync: Disabled by default. If the databases between active and standby management nodes cannot synchronize automatically, run the zsha2 installation command forcibly on the active management node.
  - Database Password: If left blank, the initial database password will be used by default. If the password has been changed, enter the new password here.
  Figure 4. Configure MN
2. Click Next.
  The system automatically checks if the requirements for addition are met.
3. In the Review Configuration step, check the resource information of both active and standby management nodes.
  Note:
  1. Upon confirmation, the system will automatically back up databases from active and standby management nodes before proceeding.
  2. During this process, the active and standby MNs will shut down and the standby MN's database will be overwritten by the active MN's database. Proceed with caution.
  3. If you find that the standby management node contains more resources during the resource comparison, and you want to preserve these resources, we recommend that you select the Switch Active/Standby MNs checkbox. When selected, this option switches the roles of active and standby management nodes, using the resource-rich standby management node as the active management node to set up management node HA.
  Figure 5. Review Configuration
4. Enter the confirmation information. Then, click OK.
5. In the Set Up MN HA step, view the addition progress and task results.
  
  Figure 6. Set Up MN HA
After the addition completes successfully, click Go to VIP to access the platform management interface through the VIP.

Figure 7. Go to VIP

Set Up with the CLI

If you cannot set up management node HA from the user interface, or if you need finer-grained configuration during server deployment, you can set up management node HA from the CLI.

To set up management node HA from the CLI, complete the following operations in sequence:

Install the operating system on both selected servers.
Configure the management network.
Configure the business network.
Install the HA suite.

After completing these operations, log in to ZSphere through the VIP and check the management node HA service status.

Install the Operating System

In this scenario, you need to install the management node mode on both selected servers.

Procedure

Select the boot option.
Enter the ISO boot interface and choose the default option to start the operating system installation. You can select based on your actual situation, but we recommend using the graphical user interface (GUI) for installation. If the server does not have a VGA port and only supports serial connections, you can use either VNC or text mode installation methods.
- GUI method
- VNC method
- Text mode method
Figure 8. System Boot
Select the installation mode.
1. On the INSTALLATION SUMMARY page, click Software Selection.
2. On the SOFTWARE SELECTION page, choose the installation mode.
  Set the servers for management nodes to Management Node mode. Set the other servers to Compute Node mode.
3. After selecting the installation mode, click Done.
Figure 9. Select Installation Mode
Configure the disk partitions.
1. On the INSTALLATION SUMMARY page, click Installation Destination to enter the INSTALLATION DESTINATION page.
2. For Device Selection, we recommend that you only configure the system disk. After the system is installed, you can configure other disks.
3. For Storage Configuration, we recommend selecting Automatic to automatically configure the disk partitions.
  If you need to manually configure disk partitions, refer to the following guidelines based on the BIOS boot mode:
  - UEFI Mode:
    - /boot: This directory stores the core files needed for Linux boot. We recommend allocating 1 GB of space.
    - /boot/efi: This directory stores the UEFI boot files. We recommend allocating 500 MB.
    - swap: This is the swap area. We recommend allocating 32 GB.
    - /: This is the root directory for the Linux system. We recommend allocating all remaining space.
  - Legacy Mode:
    - /boot: This directory stores the core files needed for Linux boot. We recommend allocating 1 GB of space.
    - swap: This is the swap area. We recommend allocating 32 GB.
    - /: This is the root directory for the Linux system. We recommend allocating all remaining space.
  Note:
  - The above values represent the recommended partition sizes for ZSphere (total disk capacity should be greater than 300 GB).
  - In Legacy mode, if the system disk capacity exceeds 2 TB, you need to configure a BIOS boot partition to support GPT partitioning. UEFI mode does not have this limitation and supports GPT partitioning.
4. Review the configuration and click Done.
Figure 10. Configure Disk Partitions
On the INSTALLATION SUMMARY page, click Root Password to set the root password for the operating system.
On the INSTALLATION SUMMARY page, click Begin Installation to begin installing the operating system.

Configure the Management Network

About this task

The following tables list the network information for the management nodes and the VIP settings for Keepalived communication in this scenario.

Server	NIC 1	NIC 2	Bond	Bridge	IP Address	Netmask	Gateway
MN 1	eth0	eth1	bond0	br_bond0	192.168.195.200	255.255.0.0	192.168.0.1
MN 2	eth0	eth1	bond0	br_bond0	192.168.196.125	255.255.0.0	192.168.0.1

-	IP Address	Netmask
VIP	192.168.199.151	255.255.0.0

Note:

The VIP is used to log in to the UI of the management node. Avoid using the VIP for SSH logins to the management nodes.
The above data is for example only. You need to modify it based on your actual deployment environment.
The gateway must be provided by physical network devices and will also serve as network status arbitration detection.

Procedure

# Create the bonded NIC bond0
[root@localhost ~]# zs-bond-lacp -c bond0

# Add NICs eth0 and eth1 to bond0
[root@localhost ~]# zs-nic-to-bond -a bond0 eth0
[root@localhost ~]# zs-nic-to-bond -a bond0 eth1

# After configuring the link aggregation, you need to set up LACP aggregation on the corresponding switch ports.

# Create the bridge br_bond0 and specify network IP, netmask, and gateway
[root@localhost ~]# zs-network-setting -b bond0 192.168.195.200 255.255.0.0 192.168.0.1

# Check if the aggregated port bond0 was created successfully
[root@localhost ~]# zs-show-network
...
----------------------------------------------------------------------------------
| Bond Name  | SLAVE(s)        | BONDING_OPTS                                     |
----------------------------------------------------------------------------------
| bond0      | eth0            | miimon=100 mode=4 xmit_hash_policy=layer2+3        |
|            | eth1            |                                                    |
---------------------------------------------------------------------------------

Log in to the MN 2 operating system and run the similar configuration commands.
Note:
- After adding eth0 and eth1 to bond0, you need to configure LACP aggregation on the corresponding switch ports. Otherwise, network communication will be disrupted.
- After creating the bridge through bond0, the bridge will be named as br_bond0 to provide management network services.
- You need to configure the bridge's IP address, netmask, and gateway according to your actual network environment.
- Once you complete the management network configuration, use the ping command to test it. If configured correctly, the management network IP addresses of the two management nodes should be able to ping each other.
- A 10 Gigabit or higher bandwidth is recommended for the management network. A 1 Gigabit bandwidth is acceptable if the network is deployed independently.

Configure the Business Network

About this task

The following table lists the configuration for the business network in this scenario.

Server	NIC 1	NIC 2	Bond	Bridge	IP Address	Netmask	Gateway
MN 1	em1	em2	bond1	-	-	-	-
MN 2	em1	em2	bond1	-	-	-	-

Procedure

# Create the bonded NIC bond1
[root@localhost ~]# zs-bond-lacp -c bond1

# Add NICs em1 and em2 to bond1
[root@localhost ~]# zs-nic-to-bond -a bond1 em1
[root@localhost ~]# zs-nic-to-bond -a bond1 em2

# After configuring the link aggregation, you need to set up LACP aggregation on the corresponding switch ports.


# You do not need to create a bridge for business network

# Check if the aggregated port bond1 was created successfully
[root@localhost ~]# zs-show-network
...
----------------------------------------------------------------------------------
| Bond Name  | SLAVE(s)        | BONDING_OPTS                                     |
----------------------------------------------------------------------------------
| bond1      | em1            | miimon=100 mode=4 xmit_hash_policy=layer2+3        |
|            | em2            |                                                    |
---------------------------------------------------------------------------------

Log in to the MN 2 operating system and run the similar configuration commands.

Note: After adding em1 and em2 to bond1, you need to configure LACP aggregation on the corresponding switch ports. Otherwise, network communication will be disrupted.

Install the HA Suite

After the operating system, management network, and business network are configured, install the HA suite to form a high availability environment with the two management nodes.

The HA suite supports the following installation methods:

Note: When installing the HA suite with the same configuration, the CLI method takes precedence over the configuration file method.

Install from the CLI

In this scenario, you have installed both servers as the ZSphere management nodes. To enable high availability for both nodes, you only need to install the HA suite on one of the nodes. If you install the HA suite on MN 1, MN 1 becomes the active management node and MN 2 becomes the standby management node.

Procedure

Import the HA suite.

# Use scp to import HA suite to MN 1
[root@localhost ~]# ls
ZStack-ZSphere-Multinode-HA-Suite.tar.gz

# Unzip the suite to get two executable files: zsha2 and zstack-hamon
[root@localhost ~]# tar zxvf ZStack-ZSphere-Multinode-HA-Suite.tar.gz
zsha2 //Installation and management program for management node high availability
zstack-hamon //Monitoring program for management node high availability

Initialize HA.
Run the following command to install the HA suite on MN 1:
```
[root@localhost ~]# chmod +x zsha2 zstack-hamon
[root@localhost ~]# ./zsha2 install-ha -nic br_bond0 -gateway 192.168.0.1 -slave "root:password@192.168.196.125" \
-vip 192.168.199.151 -myip 192.168.195.200 -db-root-pw zstack.mysql.password -time-server 192.168.196.125 -cidr 192.168.0.0/16 -yes
```
Note:
- After executing the installation command, the system will automatically back up the databases of the active and standby management nodes before proceeding with the installation.
- To install the high availability suite, ensure that zsha2 and zstack-hamon are in the same directory. During the installation, zsha2 will automatically deploy zstack-hamon and the related configuration files.

Check the management nodes status.

After initializing the HA suite, run the following command to check the status of the management nodes:

# Check the status of Management Node 1
[root@localhost ~]# zsha2 status
Status report from 192.168.195.200
=================================
Owns virtual address:            yes  // MN 1 has acquired the VIP. Only one management node can acquire the VIP at any given time.
Self 192.168.195.200 reachable:   yes // MN1 is reachable.
Gateway 192.168.0.1 reachable:    yes // Current gateway is reachable.
VIP 192.168.199.151 reachable:    yes // VIP is reachable.
Peer 192.168.196.125 reachable:   yes // MN 2 is reachable.
Keepalived status:               active // Keepalived service is active.
ZStack HA Monitor:               active // HA monitoring service is active.
MySQL status:                    mysqld is alive // Database is functioning normally.
MN status: Running [PID:6500] // Management node is operating normally.
UI status: Running [PID:9785] https://192.168.195.200:443 // UI is functioning normally.

Slave Status:
-------------
             Slave_IO_Running: Yes // Slave IO is running normally.
            Slave_SQL_Running: Yes // Slave SQL is running normally.
                   Last_Error:
        Seconds_Behind_Master: 0
                Last_IO_Error:
               Last_SQL_Error:

Warning: Permanently added '192.168.196.125' (ECDSA) to the list of known hosts.
Status report from 192.168.196.125 // Check the status of Management Node 2
================================
Owns virtual address:            no
Self 192.168.196.125 reachable:    yes
Gateway 192.168.0.1 reachable:    yes
VIP 192.168.199.151 reachable:     yes
Peer 192.168.195.200 reachable:   yes
Keepalived status:               active
ZStack HA Monitor:               active
MySQL status:                    mysqld is alive

Slave Status:
-------------
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
                   Last_Error:
        Seconds_Behind_Master: 0
                Last_IO_Error:
               Last_SQL_Error:

Note: visit ZStack UI with https://192.168.199.151:443

Note: During the installation of the HA suite, SSH password-free login has been automatically configured for both management nodes.

Install from a Configuration File

Procedure

Import the HA suite.

# Use scp to import HA suite to MN 1
[root@localhost ~]# ls
ZStack-ZSphere-Multinode-HA-Suite.tar.gz

# Unzip the suite to get two executable files: zsha2 and zstack-hamon
[root@localhost ~]# tar zxvf ZStack-ZSphere-Multinode-HA-Suite.tar.gz
zsha2 //Installation and management program for management node high availability
zstack-hamon //Monitoring program for management node high availability

Create the configuration file.

Run the following commands to create the initialization configuration file for the HA suite:

[root@localhost ~]# chmod +x zsha2 zstack-hamon
[root@localhost ~]# ./zsha2 sample-config > zs-install.config
[root@localhost ~]# cat zs-install.config
{
  "gateway": "192.168.0.1", // Arbiter gateway for active and standby management nodes
  "virtualIp": "192.168.199.151", // The VIP for Keepalived communication
  "myIp": "192.168.195.200", // Specify the local IP
  "peerIp": "192.168.196.125", // Specify the Peer management node IP
  "peerSshUser": "root", // Specify the SSH username for the Peer management node
  "peerSshPass": "password", // Specify the SSH password for the Peer management node
  "peerSshPort": 22, // Specify the SSH port for the Peer management node
  "dbRootPass": "zstack.mysql.password", // Specify the root password for the database on both management nodes (must be the same)
  "interface": "br_bond0", //Name of the physical device for configuring the VIP. Typically a management network bridge in production environments
  "timeServer": "192.168.196.125" //Specify the time synchronization server for unified time synchronization
}

Initialize HA.
Run the following command to install the HA suite:
```
[root@localhost ~]# ./zsha2 install-ha -config zs-install.config
```
Note:
- After executing the installation command, the system will automatically back up the databases of the active and standby management nodes before proceeding with the installation.
- To install the high availability suite, ensure that zsha2 and zstack-hamon are in the same directory. During the installation, zsha2 will automatically deploy zstack-hamon and the related configuration files.

Check the management nodes status.

After initializing the HA suite, run the following command to check the status of the management nodes:

# Check the status of Management Node 1
[root@localhost ~]# zsha2 status
Status report from 192.168.195.200
=================================
Owns virtual address:            yes  // MN 1 has acquired the VIP. Only one management node can acquire the VIP at any given time.
Self 192.168.195.200 reachable:   yes // MN1 is reachable.
Gateway 192.168.0.1 reachable:    yes // Current gateway is reachable.
VIP 192.168.199.151 reachable:    yes // VIP is reachable.
Peer 192.168.196.125 reachable:   yes // MN 2 is reachable.
Keepalived status:               active // Keepalived service is active.
ZStack HA Monitor:               active // HA monitoring service is active.
MySQL status:                    mysqld is alive // Database is functioning normally.
MN status: Running [PID:6500] // Management node is operating normally.
UI status: Running [PID:9785] https://192.168.195.200:443 // UI is functioning normally.

Slave Status:
-------------
             Slave_IO_Running: Yes // Slave IO is running normally.
            Slave_SQL_Running: Yes // Slave SQL is running normally.
                   Last_Error:
        Seconds_Behind_Master: 0
                Last_IO_Error:
               Last_SQL_Error:

Warning: Permanently added '192.168.196.125' (ECDSA) to the list of known hosts.
Status report from 192.168.196.125 // Check the status of Management Node 2
================================
Owns virtual address:            no
Self 192.168.196.125 reachable:    yes
Gateway 192.168.0.1 reachable:    yes
VIP 192.168.199.151 reachable:     yes
Peer 192.168.195.200 reachable:   yes
Keepalived status:               active
ZStack HA Monitor:               active
MySQL status:                    mysqld is alive

Slave Status:
-------------
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
                   Last_Error:
        Seconds_Behind_Master: 0
                Last_IO_Error:
               Last_SQL_Error:

Note: visit ZStack UI with https://192.168.199.151:443

Note: During the installation of the HA suite, SSH password-free login has been automatically configured for both management nodes.

PreviousInstall the Management Node NextManage ZSphere Services