Overview
HA Policy is a mechanism that ensures sustained and stable running of the business if VM instances are unexpectedly stopped or are errored because of errors occurring to compute, network, or storage resources associated with the VM instances. By enabling this feature, you can customize VM HA policies to ensure your business continuity and stability.
Concepts
- HA mode: Specifies whether to enable auto restart if VM instances are
unexpectedly stopped or are errored because of errors occurring to compute,
network, or storage resources associated with the VM instances. None and
NeverStop are supported:
- None: VM instances are not auto restarted no matter it is planned to be stopped or unexpectedly stopped.
- NeverStop:
- Unexpectedly stopped VM instances are auto restarted on another host depending on the failover strategy you configure for them.
- The VM instance does not reboot automatically if you stop it
manually, including:
- Manually perform the stopping VM instance, force stopping VM instance, and powering off VM instance actions on the UI.
- Manually run the
shutdown,poweroff, andhaltcommands in the VM OS. - Create a scheduled job to trigger the VM shutdown as planned.
- VM Failover Strategy: Specifies whether to migrate a VM instance to another
host if errors occur to the compute resource, storage resource, or network
resource associated with the VM instance.The VM failover mechanism inspects the following resource status:
- Management Network Connectivity Status:
- Management network connectivity status indicates the status of the network that connects the management node and the host where VM instances reside.
- This status may turn Abnormal if errors occur to the management node or to the management network.
- Storage Network Connectivity Status:
- Detects the connectivity status of the network that VM instances use to access the primary storage where the root volumes of these VM instances reside.
- This status may turn Abnormal if errors occur to the primary storage or to the storage network.
- Business NIC Status:
- Business NIC status may turn Abnormal if errors occur to the host business NIC or the switch port directly connecting to the host business NIC that is associated with the L2 network of VM instances.
- Business network group: Supports adding redundant
networks that carry the same service to a business
network group and setting the minimum number of alive
networks. A VM business NIC fault is detected only when
the number of available networks in the group is less
than this value.
Note:
- If a VM uses networks in a business network group, the number of attached networks from the same group should be no less than the minimum number of alive networks configured for that group, to avoid unexpected HA triggered by a misjudged business network exception.
- Only public networks and flat networks can be added to a business network group.
- Business network groups can be configured only when failover upon business NIC fault is enabled.
Based on the resource status inspection, the Cloud allows you to configure failover strategies for 4 fault scenarios:Fault Scenario Management Network Connectivity Status Storage Network Connectivity Status Business NIC Status Fail Over Scenario A: Business NIC Fault Normal Normal Abnormal Enable/Disable Scenario B: Storage Network Fault Normal Abnormal Normal Enable/Disable Scenarios C: Storage Network and Business NIC Fault Normal Abnormal Abnormal Set as false if both the scenario A and B have the failover policy set as false. Set as true if either of the scenario A or B has the failover policy set as true. Scenario D: Management Network Fault Abnormal Normal Normal Disable. The failover cannot be enabled in this scenario. Note: The failover policies take effect on VM instances whose HA
modes are set as NeverStop only. - Management Network Connectivity Status:
Fundamentals
- The Cloud polls the running status of VM instances. If a VM instance is
unexpectedly stopped, its HA mode is checked. If the HA mode of the VM
instance is NeverStop, then the VM instance is restarted on the current host
or another host.
Figure 1. VM HA Started After Unexpectedly Stopped 
- The Cloud polls the status of the hosts where VM instances reside. Either of
the management network connectivity status, storage network connectivity
status, and business NIC status of the host turns abnormal, the
corresponding VM failover strategy and VM HA mode are checked. If the
corresponding failover strategy is Yes and VM HA mode is NeverStop, then
related VM instances are migrated to another host.
Figure 2. VM HA Started After Host Business NIC Turns Down 
Characteristics
- Comprehensive & Powerful: Covers all mainstream HA scenarios, including various failures, and ensures the stability and continuity of your business.
- Flexible & Visualized: Provides a simple table that allows you to configure VM failover strategies with one click. This table functions together with the HA Mode that can be configured on all and individual VM instances, thus greatly improving the flexibility of your business HA configuration.
Scenarios
The following describes the scenarios of the HA Policy feature.
- Host Business NIC Turns Down:If a host business NIC turns down, to ensure high availability of business, all VM instances associated with this NIC are expected to migrate to other hosts.
- For example, your business VM instances are running MySQL database service which is required to achieve high availability. In this case, you can set the HA mode of these VM instances to NeverStop and turn on the switch corresponding to Abnormal Business NIC Status. Then as long as host resources are sufficient, in case that a host business NIC associated with these VM instances turns down, these VM instances will be auto started on other hosts.
- VM Unexpectedly Stops:If a VM instance is unexpectedly stopped, it is expected to auto HA start.
- For example, your VM instances are running important business applications. To ensure business auto-recovery in case of VM stops due to reasons such as host powered-offs or business overloads, you can set the HA mode of these VM instances to NeverStop. Then if these VM instances are stopped, they are auto started.
Preparations
Before getting started with the HA policy, make the following preparations:
- Install or upgrade to the latest version of ZStack Cloud.
- On the main menu, choose and enable the HA Policy.
- Enter the Advanced Settings tab, and set Default HA Mode When Creating VM Instance to NeverStop.
Limitations
- If the management network connectivity status turns abnormal, you cannot set the failover strategy.
- Inspection of the storage network connectivity status only applies to VM
instances that reside on shared storage, not on local storage.
- If the storage type is SharedBlock, VM instances will automatically migrate when the storage network connection fails, regardless of whether the failover strategy is set to Yes or not.
- For business VM instances using a VXLAN L2 network or L2 network that employs SR-IOV / smart NIC network acceleration mode, HA migration is not supported when errors occur to the host business NIC or the switch port directly connecting to the host business NIC that is associated with the L2 network.
- When both the storage network connectivity status and the business NIC status
turn abnormal, the failover strategy is determined by the strategies that are
applied to the two abnormal status cases:
- If you set either of the failover strategies to Yes, this failover strategy will be Yes.
- Only when you set both the failover strategies to No, this failover strategy will be No.
- The NeverStop mode triggers a VM automatic reboot only after an unexpected stopping because of a VM errors faults of compute, storage, and network it is residing on. The NeverStop mode does not trigger a VM automatic reboot if the VM instance is stopped manually or through a scheduled job.
Typical Scenarios
Business NIC Failure
In this scenario, we assume you deploy four business VM instances on a host named Host A to run MySQL database services. If a failure occurs to the business NIC of Host A, all four VM instances are expected to migrate to another host, guaranteeing the high availability of your business. To accomplish this, you need to set the HA mode of these VM instances to NeverStop and enable the failover strategy that is applied when the business NIC status turns abnormal. Before you start, make sure there are enough hosts on the Cloud.
VM HA Policy Configuration
- Ensure that the HA mode of the four VM instances is set to NeverStop.
Figure 3. VM HA Mode 
- Modify the VM failover strategy: Enable the failover of the Scenario A, and
the failover of the Scenairo C is enabled automatically.
Figure 4. Modify VM Failover Strategy 
VM HA Migration
After you enable the VM failover strategy, if the business NIC of Host A, zsn3 in this scenario, fails, the four VM instances on Host A will immediately migrate to Host B and start.
Disconnection between Host and Storage Network
In this scenario, we assume you deploy three VM instances on a host named Host A to run important business applications and use Ceph primary storage for data storage. If Host A loses connection with the Ceph storage network, all three VM instances are expected to migrate to another host, guaranteeing the high availability of your business. To accomplish this, you need to set the HA mode of these VM instances to NeverStop and enable the failover strategy that is applied when the storage network connectivity status turns abnormal. Before you start, make sure there are enough hosts on the Cloud.
VM HA Policy Configuration
- Ensure that the HA mode of the three VM instances is set to
NeverStop.
Figure 7. VM HA Mode 
- Modify VM failover strategy: Enable the failover strategy of the Scenario B.
And the failover strategy of the Scenario C is enabled automatically.
Figure 8. Modify VM Failover Strategy 
VM HA Migration
After you enable the VM failover strategy, if Host A loses connection with the Ceph storage network, the three VM instances on Host A will immediately migrate to Host B and start.
Unexpected Shutdown of VM Instances
In this scenario, we assume you deploy four business VM instances to run important business applications, with two residing on SharedBlock storage and two on local storage. If any VM instance is unexpectedly stopped due to host powered-offs, business overloads, or other factors, it is expected to auto restart, guaranteeing your business continuity.
VM HA Policy Configuration
- Ensure that the HA mode of the four VM instances is set to
NeverStop.
Figure 10. VM HA Mode 
VM Instances Residing on SharedBlock Storage HA Start
VM Instances Residing on Local Storage HA Start after Unexpected Shutdown
