HA Policy

What is HA Policy?

HA Policy is a mechanism that ensures sustained and stable running of the business if VM instances are unexpectedly or scheduled stopped or are errored because of errors occurring to compute, network, or storage resources associated with the VM instances. By enabling this feature, you can customize VM HA policies to ensure your business continuity and stability.

Concepts

The HA Policy feature involves the following key concepts:

HA mode: Specifies whether to enable auto restart if VM instances are unexpectedly stopped or are errored because of errors occurring to compute, network, or storage resources associated with the VM instances. None and NeverStop are supported:
- None: VM instances are not auto restarted no matter it is planned to be stopped or unexpectedly stopped.
- NeverStop:
  - Unexpectedly stopped VM instances are auto restarted on another host depending on the failover strategy you configure for them.
  - VM instances do not restart automatically after they are stopped manually or through scheduled jobs.

VM Failover Strategy: Specifies whether to migrate a VM instance to another host if errors occur to the compute resource, storage resource, or network resource associated with the VM instance.

The VM failover mechanism inspects the following resource status:

Management Network Connectivity Status:
- Management network connectivity status indicates the status of the network that connects the management node and the host where VM instances reside.
- This status may turn Abnormal if errors occur to the management node or to the management network.

Storage Network Connectivity Status:
- Detects the connectivity status of the network that VM instances use to access the primary storage where the root volumes of these VM instances reside.
- This status may turn Abnormal if errors occur to the primary storage or to the storage network.
Business NIC Status:
- Business NIC status may turn Abnormal if errors occur to the host business NIC or the switch port directly connecting to the host business NIC that is associated with the L2 network of VM instances.

Based on the resource status inspection, the Cloud provides the following truth table for configuring VM failover strategies:

Management Network Connectivity Status	Storage Network Connectivity Status	Business NIC Status	Fail Over
Normal	Normal	Abnormal	Yes/No
Normal	Abnormal	Normal	Yes/No
Normal	Abnormal	Abnormal	Yes/No
Abnormal	Normal	Normal	No

Fundamentals

ZStack Cloud HA Policy has the following mechanisms：

The Cloud polls the running status of VM instances. If a VM instance is unexpectedly stopped, its HA mode is checked. If the HA mode of the VM instance is NeverStop, then the VM instance is restarted on the current host or another host.

Figure 1. VM HA Started After Unexpectedly Stopped
The Cloud polls the status of the hosts where VM instances reside. Either of the management network connectivity status, storage network connectivity status, and business NIC status of the host turns abnormal, the corresponding VM failover strategy and VM HA mode are checked. If the corresponding failover strategy is Yes and VM HA mode is NeverStop, then related VM instances are migrated to another host.

Figure 2. VM HA Started After Host Business NIC Turns Down

Characteristics

HA Policy has the following characteristics:

Comprehensive & Powerful: Covers all mainstream HA scenarios, including various failures, and ensures the stability and continuity of your business.
Flexible & Visualized: Provides a simple table that allows you to configure VM failover strategies with one click. This table functions together with the HA Mode that can be configured on all and individual VM instances, thus greatly improving the flexibility of your business HA configuration.

Scenarios

The following describes the scenarios of the HA Policy feature.

Host Business NIC Turns Down:
If a host business NIC turns down, to ensure high availability of business, all VM instances associated with this NIC are expected to migrate to other hosts.
- For example, your business VM instances are running MySQL database service which is required to achieve high availability. In this case, you can set the HA mode of these VM instances to NeverStop and turn on the switch corresponding to Abnormal Business NIC Status. Then as long as host resources are sufficient, in case that a host business NIC associated with these VM instances turns down, these VM instances will be auto started on other hosts.
VM Unexpectedly Stops:
If a VM instance is unexpectedly stopped, it is expected to auto HA start.
- For example, your VM instances are running important business applications. To ensure business auto-recovery in case of VM stops due to reasons such as host powered-offs or business overloads, you can set the HA mode of these VM instances to NeverStop. Then if these VM instances are stopped, they are auto started.

Manage HA Policy

On the main menu of ZStack Cloud, choose Settings > Platform Setting > HA Policy. Then, the HA Policy page is displayed.

HA Policy supports the following actions:

Action	Description
Enable HA Policy	Enables the HA Policy feature.
Disable HA Policy	Disables the HA Policy feature. Note: If you disable HA Policy, VM instances will not be auto restarted if they are stopped. This may cause business interruptions. Proceed with caution.

HA Policy|Failover Policy

On the Enable HA Policy page or the Overview page of HA Policy, you can modify the following true table to configure failover policies for VM instances.

Management Network Connectivity Status	Storage Network Connectivity Status	Business NIC Status	Fail Over
Normal	Normal	Abnormal	Yes/No
Normal	Abnormal	Normal	Yes/No Note: If the storage type is SharedBlock and this status is Abnormal, VM instances will auto fail over regardless of this configuration.
Normal	Abnormal	Abnormal	Yes/No Note: The failover policy of this scenario follows the preceding two failover policies of this table. If you set both the preceding two policies to No, then this failover policy is set to No. If you set either of the two to Yes, then this failover policy is set to Yes.
Abnormal	Normal	Normal	No Note: If the management network is in Abnormal status, you cannot set this failover policy.

Note:

For Storage Network Connectivity Status, only shared storage is detected. Local storage is not supported.
If an L2 network of a VM instance is of the VXLAN type or the L2 network applies the SR-IOV or Smart NIC, and errors occur to the host business NIC associated with this L2 network or occur to the switch port directly connecting to the host business NIC, this VM instance will not fail over.

On the Enable HA Policy page or the Overview page of HA Policy, you can modify the following host error inspection settings to modify the inspection intervals of the preceding failover policy.

Name	Description
Host Self-Inspection Interval	The interval at which a host inspects its own status. Default: 5 seconds. If this interval is too long, detection of host abnormal or recovered status may be delayed. We recommend that you keep this value consistent with Abnormal Host Check Interval in Advanced Settings.
Maximum Host Self-Inspection Attempts	The maximum number of attempts that a host inspects its own status. If the self-inspection of a host fails by the maximum attempts, it is determined that network errors occur with the host. Default: 6.

HA Policy|Advanced Settings

On the Enable HA Policy page or the Overview page of HA Policy, you can modify the advanced settings of HA Policy. They can be classified into the following two categories:

Category	Name	Description
VM Instance	VM Cross-Cluster HA	Specifies whether to enable VM migration across clusters to achieve high availability. Default: false. If set to true, hosts across clusters can be detected to achieve VM high availability. Note: Before you enable this feature, make sure that clusters are well connected.
	Maximum GC Retry Interval of NeverStop VM	The maximum interval of garbage collection (GC) attempts to start up NeverStop VM instances that are stopped unexpectedly. Default: 300. Unit: second.
	Delay of NeverStop VM Startup Attempt	The delay of another retry to start up a NeverStop VM instance after the last startup attempt fails. Default: 60. Unit: second.
	NeverStop VM Scanning Interval	The interval of scanning NeverStop VM instances that fail to start up. Default: 60. Unit: second.
	Sync Speed of HA VM State Update	The synchronization speed of the state of highly available VM instances on the UI. Default: 1. Valid values: -1 to 5, integer. A higher value indicates a lower synchronization speed. However, a higher value lowers system loads because outdated status update notifications are ignored. The value -1 indicates the state of HA VM instances on the UI does not automatically change.
	VM HA Mode Default Value	Sets the default value of HA mode when creating VM instances. Valid values: None and NeverStop. None: The HA mode of a newly created VM instance is set as None by default. If an unexpected stopping occurs, the VM instance does not reboot automatically. NeverStop: The HA mode of a newly created VM instance is set as NeverStop by default. The VM instance attempts to reboot automatically when it is unexpectedly stopped because of its own errors. The VM instance attempts to migrate to another host automatically if errors occur to the compute, network, or storage resource it is residing on. The VM instance does not reboot automatically if it is stopped manually or through a scheduled job. Note: If you choose NeverStop, make sure that you have enabled HA Policy in Platform Setting. Otherwise, the NeverStop mode does not take effect. You can modify the HA mode of a VM instance individually during or after the creation. After the modification, the VM instance uses the HA mode you specified for it and is not affected by this default value.
Host	Abnormal Host Check Interval	The interval at which the system detects and updates status changes of abnormal hosts. Default: 5 seconds. If this interval is too long, updates of abnormal host status may be delayed. We recommend that you keep this value consistent with Host Self-Inspection Interval on the Overview page.
	Maximum Attempts to Determine Host Disconnection	The maximum number of failed connections that are required to determine that a host is disconnected. Default: 12.
	Host Successful Connection Period	The time period of a successful connection to a host. Default: 5. Unit: second. If a connection request is responded within the specified time, the connection succeeds.
	Host Successful Connection Possibility	The possibility of successful connections in contrast to failed connections that determine whether a host is successfully connected. Default: 50. Unit: %.
	Minimum Attempts to Determine Successful Host Connection	The minimum number of successful connections that are required to determine that a host is successfully connected. Default: 5.
	Timeout Period of Primary Storage Inspection by Host	The timeout period that a host checks its connection with primary storages. Default: 5. Unit: second.

HA Log

On the main menu of ZStack Cloud, choose Settings > Platform Setting > HA Policy. Then, the HA Policy page is displayed. If HA policy is enabled and the HA mechanism is triggered, then HA logs are generated.

This page displays all VM HA logs in the Cloud. You can view the log information such as task result, VM name, VM owner, host information, and start and end time. These logs can be applied in O&M and audit.

You can select a time span to view HA logs. Available time spans: recent 7 days and recent 1 month. By default, logs generated in recent 7 days are displayed.
You can customize a time span to view the HA logs in the specified time span.
You can search for HA logs by VM name or VM owner.
You can filter HA logs by task result. The task results include succeeded and failed.
You can sort HA logs by creation or completion time.
You can export the HA logs in CSV format.
You can adjust the number of HA logs displayed on each page. Optional values: 10, 20, 50, and 100.

PreviousSettings