Reliability Design
The reliability design outlines high-availability and data protection mechanisms across ZCF's capability domains. Specifically, the cloud platform ensures reliability for workloads such as VM instances and Network Service; storage services provide data redundancy and self-healing from failures; and Container Service ensures high availability of container clusters—jointly minimizing service disruption and data loss under failure scenarios.
Workload High Availability
Business Load High Availability covers two types of workloads: VM instances and container clusters. VM instance high availability restores business operations upon host or VM instance failure, while container cluster high availability ensures continuous availability of Kubernetes control plane nodes and critical components.
VM Instance High Availability
The cloud platform employs KVM-based hardware virtualization to abstract a group of physical servers into a logical resource pool, managed in clusters. The platform continuously monitors the health of all hosts and VM instances within each cluster. If a host fails, the management node detects the outage and automatically restarts all affected VM instances on another healthy host in the same cluster, ensuring business continuity. This compute high availability requires no dedicated standby hardware or third-party software integration, minimizing downtime and IT service disruption while avoiding the cost and complexity associated with OS- or application-specific failover solutions.
Container Cluster High Availability
The management plane of the Kubernetes cluster underlying the Container Service adopts Kubernetes' official high-availability (HA) architecture: three control-plane nodes constitute the Container Service cluster. Each control-plane node hosts three core components—kube-apiserver, kube-scheduler, and kube-controller-manager—and also runs a dedicated etcd instance. The three etcd instances collectively form an etcd cluster serving as Kubernetes' metadata store, ensuring high reliability and consistency of cluster data.
Network High Availability
Network high availability addresses failure scenarios involving physical links, foundational network services, and network service instances, helping business networks continue running. It mainly includes host network high availability and network service high availability.
Host Network High Availability
Host network high availability is implemented through switch-side link redundancy and host-side NIC bonding. The switch side provides link and device redundancy, while the host side binds multiple NICs into one logical interface. Together, they reduce the impact of a single NIC, link, or switch failure on business networks.
Switch-side High Availability: Stacking/M-LAG
Switch-side high availability is primarily implemented through stacking and M-LAG. Stacking virtualizes multiple switches into one logical device. M-LAG establishes cross-device link aggregation between independently running switches to provide link redundancy. Stacking or M-LAG can achieve the following goals:
- Switch high availability: If any switch fails, traffic can fail over to another switch.
- Loop-free links: Link aggregation is managed based on LACP, reducing loop risks caused by multi-link forwarding.
- Higher bandwidth: Multiple physical links are aggregated to increase total network bandwidth.
Host-side High Availability: NIC Bonding
Host-side high availability is primarily implemented through NIC bonding. Bonding binds multiple NICs on a host into one logical interface and works with switch-side stacking or M-LAG to provide link redundancy and bandwidth aggregation. Common modes include mode 1 (active-backup) and mode 4 (802.3ad).
- Mode 1 (active-backup): Only one NIC is active at a time, and the other NICs remain standby. If the active NIC or active link fails, traffic can fail over to a standby link.
- Mode 4 (802.3ad): Multiple NICs form a logical link aggregation group (LAG) through LACP. Switches and host NIC bonding work together to provide link redundancy and bandwidth aggregation.
Network Service High Availability
Network services include basic network services (e.g., DHCP) and additional services provided by VPC vRouters or load balancer instances (e.g., load balancing, port forwarding).
Distributed DHCP
L3 networks include: flat network, public network, and VPC network.
A flat network assigns private network addresses to VM instances, while allowing VM instances to access the public network through distributed elastic IPs (EIPs). Flat networks support network services such as DHCP, User Data, security groups, EIPs, and port mirroring.
A VPC network is a customizable network space that enables tenants to build an isolated, self-managed virtual network within the cloud platform. It offers greater control over configuration and policy, enhancing the security of cloud resources. VPC vRouters provide VPC networks and their services. A single VPC vRouter can host multiple isolated VPC networks, providing VM instances with network services such as DHCP, DNS, SNAT, routing tables, security groups, VPC firewalls, EIPs, port forwarding, load balancing, IPsec, NetFlow, OSPF, and multicast routing.
VPC vRouter High Availability
ZStack Cloud supports VPC vRouter high availability (HA) groups. You can deploy a pair of active-standby VPC vRouters in a VPC vRouter HA group. If the active VPC vRouter fails, an HA failover is triggered within seconds, automatically switching to the standby VPC vRouter to ensure continuous and stable business operation.
Load Balancer Instance High Availability
Dedicated-performance load balancers support HA. When the active instance fails, the standby instance can seamlessly switch over and quickly take over the service, which greatly reduces the risk of business interruption. ZStack Cloud supports an LB instance configuration detection mechanism, which triggers automatic synchronization when the active and standby instance configurations are inconsistent. You can also perform manual synchronization to ensure configuration consistency if automatic synchronization fails.
Storage Reliability
Storage reliability encompasses capabilities such as data redundancy, fault-domain isolation, consistency checking, fault detection, and data reconstruction—ensuring continuous data accessibility under hardware failures, node anomalies, and capacity changes. This topic uses ZStack ZStone as an example to illustrate the reliability design of distributed storage within the ZCF architecture.
Data Redundancy
Overview
Data redundancy is the cornerstone for storage systems to achieve high availability, data protection, and business continuity. Faced with increasing data volumes, complex application scenarios, and potential risks such as hardware failures, network interruptions, and human errors, appropriate data redundancy policies provide you with a robust data protection barrier, ensuring business continuity and maximizing data value. This topic briefly compares the characteristics of redundancy techniques in centralized storage and distributed storage, and then uses ZStack ZStone as an example to describe two core data protection policies: replication and erasure coding (EC).
Centralized Storage vs. Distributed Storage
Centralized Storage
Traditional centralized storage uses controllers and disk enclosures to provide data management and read/write capabilities. It typically employs dual controllers for redundancy, while some high-end storage uses multiple controllers. Storage space is provided through the controller's built-in drive bays or externally connected expansion enclosures. Traditional centralized storage commonly uses RAID to protect data, such as RAID 5, RAID 6, or RAID 10.
Distributed Storage
Distributed storage adopts a decentralized architecture where each storage node provides computing and storage resources, enabling more flexible scalability and larger storage capacity. Storage nodes are interconnected via standard Ethernet switches and managed by the distributed storage software to provide a unified storage resource pool to upper-layer services. Furthermore, distributed storage supports horizontal scaling. A single cluster can expand to thousands of nodes to provide EB-level capacity, making it suitable for massive data storage scenarios.
Centralized Storage vs. Distributed Storage
- Cross-Node Redundancy: Distributed storage supports redundancy across multiple nodes. For example, a three-replica policy can tolerate two simultaneous node failures without data loss, whereas RAID only provides redundancy within a single node.
- Global Hot Spares and Data Recovery: Unlike RAID, which relies on the dedicated hot spare disk, distributed storage uses all available disks for data recovery, significantly improving efficiency. Additionally, distributed storage requires no additional hardware support, whereas RAID typically needs a dedicated RAID card.
Replication
Definition
Replication is a data protection technique that achieves data redundancy and high availability by storing the copies of the same data across different nodes. If a node fails, data can be recovered from the replicas on other nodes. You can configure 2 to 6 replicas, with 3 replicas recommended for production environments.
- Read/Write in Normal State
Taking a three-replica policy at the server level as an example: During data write, the system copies the data into three identical replicas and stores each on data disks in three different servers. During data read, the system reads the data from any one of the servers and returns it to the user.
Figure 6. Read/Write in Normal State with Three Replicas 
- Read/Write Under Failures
Taking a three-replica policy at the server level as an example: If server C fails, the system stores replicas on the remaining two servers. During data read, the system reads one replica from either of the remaining two servers and returns it to the user.
Figure 7. Read/Write Under Failures with Three Replicas 
Erasure Coding (EC)
Overview
- Standard EC (K+M): K indicates the number of data blocks, and M indicates the number of parity blocks. This configuration tolerates simultaneous failures of M fault domains without affecting data availability.
- Folded EC (K+M:B): K indicates the number of data blocks, and M:B indicates the number of parity blocks. This configuration tolerates simultaneous failures of M hard disks or B fault domains without affecting data availability.
Policy Types
| EC Policy | Storage Efficiency | |
|---|---|---|
| Recommended Values | 2+1 | 66.67% |
| 4+2 | 66.67% | |
| 8+3 | 72.73% | |
| 4+2:1 | 66.67% | |
| 8+2:1 | 80.00% | |
| 16+2:1 | 88.89% | |
| Custom | K/(K+M) | |
Standard EC
- Read/Write in Normal State
Standard EC (K+M): Taking a server-level 4+2 EC policy as an example: During data write, the system splits the data into 4 equally sized data blocks and generates 2 parity blocks of the same size through the parity algorithm. The system randomly stores these 6 blocks across 6 servers. If any 2 servers fail, data remains accessible. During data read, the system reads data blocks from different data disks on 4 servers, assembles the 4 data blocks into complete data, and returns it to the user.
Figure 8. Data Write in Normal State with EC (4+2) 
Figure 9. Data Read in Normal State with EC (4+2) 
- Read/Write Under Failures
Standard EC (K+M): Taking a server-level 4+2 EC policy as an example: If the number of available servers drops below K+M due to failures, the system stores newly written data on the remaining servers before the recovery. This ensures I/O continuity without reducing the reliability. Once the failed servers recover, the data redundancy policy reverts to K+M. During data read, the system reads data from the remaining healthy servers and recovers the data using the parity algorithm before returning it to the user.
Figure 10. Data Write Under Failures with EC (4+2) 
Figure 11. Data Read Under Failures with EC (4+2) 
Folded EC
Folded EC, also known as sub-node EC, is another common data redundancy technique. Unlike the standard K+M EC configuration, the folded EC typically follows a K+M:B format, where B is usually set to 1. The folded EC maintains high data reliability while delivering improved storage efficiency.
For example, because a standard EC uses the storage node as its smallest fault domain, it requires at least 6 nodes. A folded EC with a 4+2:1 configuration can achieve data redundancy with as few as 3 storage nodes.
Furthermore, folded EC supports scaling in and out just like standard EC. You can convert a folded EC to a standard EC as long as the fault domain requirements are met.
- Read/Write in Normal State
Folded EC (K+M:B): Taking a server-level 4+2:1 EC policy as an example: During data write, the system splits the data into 4 equally sized data blocks and generates 2 parity blocks of the same size through the parity algorithm. The system randomly stores these 6 blocks across 5 servers. If any one server fails, data remains accessible. During data read, the system reads data blocks from different data disks on 3 servers, assembles the 4 data blocks into complete data, and returns it to the user.
Figure 12. Data Write in Normal State with EC (4+2:1) 
Figure 13. Data Read in Normal State with EC (4+2:1) 
- Read/Write Under Failures
Folded EC (K+M:B): Taking a server-level 4+2:1 EC policy as an example: If one server or M hard disks fail, the system continues writing data and parity blocks to the remaining healthy servers according to the K+M configuration. During data read, the system reads data from other healthy servers and recovers the data using the parity algorithm before returning it to the user.
Figure 14. Data Read Under Failures with EC (4+2:1) 
Replication vs. EC
- Storage Efficiency: EC holds a significant advantage. For example, a 4+2 EC policy offers approximately 66% effective storage, while a three-replica policy achieves only 33.3%.
- Read/Write Performance: Performance varies significantly in small I/O scenarios. The gap narrows in large I/O scenarios. EC involves data validation during writes, which may introduce write amplification. During reads, performance can be impacted if any of the involved nodes encounter high latency. In contrast, replication only needs to read one complete copy without any data reassembly.
- Rebuild Performance: Replication generally outperforms EC in rebuild speed. Replication involves simple data copying without validation, resulting in faster rebuilds. EC rebuild requires reverse parity calculation, demanding more data I/O and higher CPU consumption.
- Fault Tolerance: Both policies have strengths and weaknesses. Replication allows up to (number of replicas - 1) simultaneous non-monitoring node failures without data loss. For EC, a 4+2 policy allows two simultaneous non-monitoring node failures without data loss.
Fault Domain Isolation
- Server-Level: Each server in the cluster acts as a fault domain. Different replicas or blocks of data are stored on different servers.
- Rack-Level: Each rack in the cluster acts as a fault domain. Different replicas or blocks of data are stored in different racks. Recommended for clusters of larger scale with more racks.
- Room-Level: Each room in the cluster acts as a fault domain. Different replicas or blocks of data are stored in different rooms. Recommended for very large clusters spanning multiple rooms.
Fault domain isolation helps contain the impact of failures to a specific scope, preventing a domino effect and thereby enhancing business continuity.
By leveraging fault domain-aware scaling along with rational storage policies, newly added nodes can form an independent disk pool without requiring data migration. This enables seamless capacity expansion, shielding applications from underlying storage changes and eliminating the need for application-level adjustments traditionally associated with storage updates. As a result, both operational and administrative workloads are significantly reduced, while system reliability and performance are enhanced.
Data Consistency Check
- Scrub Check: Focuses on metadata. It completes quickly and runs frequently. It is recommended to perform Scrub daily. You can customize the schedule.
- Deep-Scrub Check: Focuses on data. It takes longer to complete and may impact I/O performance. It is recommended to run during off-peak hours. An alert is triggered if no Deep-Scrub has been completed within 30 days.
Fault Detection and Self-Healing
Storage services support automatic failure detection and alerting. This mechanism monitors the storage system and individual storage servers. Upon detecting a failure, the system automatically sends alert messages to the platform. You can also add email endpoints to receive alarms, enabling timely response and recovery.
When a failure occurs, the system supports automatic service restart and data migration. This maximizes data reliability and availability, forming a highly reliable and highly available distributed storage system.
Data Rebalancing
Distributed storage supports data rebalancing to evenly distribute data across data disks under all storage servers in the cluster. This enhances storage system performance and reliability.
- Based on storage pool configurations and storage server load, the system automatically migrates data from overloaded nodes to those with lower load to achieve load balancing.
- When a server fails or a new server is added, the system automatically migrates data to maintain consistency and reliability.
Manual Data Rebalancing:
The system also supports manual data rebalancing. You can manually initiate rebalancing operations based on the actual data distribution.
Data Protection and Recovery
Data protection and recovery cover capabilities including snapshots, disaster recovery, continuous data protection (CDP), and system configuration backups—preserving recoverable data states in scenarios involving human error, system failure, or disasters.
Snapshot Management
- The snapshot mechanism for centralized storage: Local storage, NFS, Shared Mount Point, and SharedBlock use QCOW2 external snapshot, which is a type of ROW snapshot mechanism.
- The snapshot mechanism for distributed storage: Ceph enterprise edition, Vhost, and CBD use ROW snapshots. ZStack ZStone uses COW snapshots.
Centralized Storage Snapshot Mechanism
This topic introduces QCOW2 external snapshots.
- Snapshot Chain and Snapshot Tree
Typically, a single disk corresponds to one snapshot chain. ZStack Cloud supports creating a snapshot tree for a disk, where each branch of the tree represents a distinct snapshot chain.
Figure 18. Snapshot Tree 
A snapshot tree includes the following information:- Snapshot Chain: A relational chain composed of a set of snapshots for a disk. Each branch of the snapshot tree is a snapshot chain.
- Snapshot Node: An individual node in a snapshot chain, representing a single snapshot of the disk.
- Snapshot Size: The storage space consumed by a snapshot. You can view the total size of all snapshots in the snapshot tree and the size of an individual snapshot node.
Note:
- For non-Ceph storage, the system allows a maximum of 128 nodes per snapshot chain by default. You can modify Maximum Incremental Volume Snapshot in the global settings to set the maximum length of the snapshot chain. For Ceph storage, the maximum number of snapshots per disk is 32, including both manual and automatic snapshots.
- When a snapshot chain reaches its maximum length:
- If you continue to create automatic snapshots, the system automatically deletes the earliest automatic snapshot.
- If you continue to create manual snapshots, you must manually delete unneeded snapshots.
- In production environments, we recommend that you keep the number of snapshots for a single disk below 5. Excessive snapshots can affect the I/O performance of VM instances or volumes, data security, and primary storage capacity. For long-term backup, use the backup service.
- Create Snapshots
When an external snapshot is created, a new empty QCOW2 file is generated. This new file has its backing file pointed to the original QCOW2 file. The original QCOW2 file is set to read-only, effectively turning it into a snapshot itself. Subsequent data writes are directed only to the new QCOW2 file. The creation of an external snapshot involves creating a new blank qcow2 file.
- Create a single snapshot chain based on a backing file.
Figure 19. Create Snapshot - Single Chain 
Assume there is an original base image (Base). A VM instance 1 is created using this base image as a template. Then, snapshot 1A and 1B are created sequentially for VM instance 1.- Base Image: A pre-made disk image file containing a complete operating system and bootloader, serving as the Base (read-only).
- VM Instance 1: A new empty file, Overlay-1, is created. Its backing file points to Base. Base remains read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-1.
- Snapshot 1A: A new empty file, Overlay-1A, is created. Its backing file points to Overlay-1. Overlay-1 is set to read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-1A.
- Snapshot 1B: A new empty file, Overlay-1B, is created. Its backing file points to Overlay-1A. Overlay-1A is set to read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-1B. VM instance 1 uses the disk file corresponding to the last snapshot 1B in the snapshot chain, and snapshot 1B is Active.
- Create multiple snapshot chains based on backing files.
Figure 20. Create Snapshot - Multiple Chains 
Assume there is an original base image (Base). VM instance 1, VM instance 2, and VM instance 3 are created using this base image as a template. Then, snapshot 1A and snapshot 1B are created sequentially for VM 1, snapshot 2A is created for VM 2, and snapshot 3A is created for VM 3.- Base Image: A pre-made disk image file containing a complete operating system and bootloader, serving as the Base (read-only).
- Snapshot Chain 1:
- VM Instance 1: A new empty file, Overlay-1, is created. Its backing file points to Base. Base remains read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-1.
- Snapshot 1A: A new empty file, Overlay-1A, is created. Its backing file points to Overlay-1. Overlay-1 is set to read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-1A.
- Snapshot 1B: A new empty file, Overlay-1B, is created. Its backing file points to Overlay-1A. Overlay-1A is set to read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-1B. VM 1 uses the disk file corresponding to the last snapshot 1B in snapshot chain 1, and snapshot 1B is Active.
- Snapshot Chain 2:
- VM Instance 2: A new empty file, Overlay-2, is created. Its backing file points to Base. Base remains read-only. Subsequent data is written only to Overlay-2.
- Snapshot 2A: A new empty file, Overlay-2A, is created. Its backing file points to Overlay-2. Overlay-2 is set to read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-2A. VM 2 uses the disk file corresponding to the last snapshot 2A in snapshot chain 2, and snapshot 2A is Active.
- Snapshot Chain 3:
- VM Instance 3: A new empty file, Overlay-3, is created. Its backing file points to Base. Base remains read-only. Subsequent data is written only to Overlay-3.
- Snapshot 3A: A new empty file, Overlay-3A, is created. Its backing file points to Overlay-3. Overlay-3 is set to read-only, thus becoming a snapshot. Subsequent data is written only to Overlay-3A. VM 3 uses the disk file corresponding to the last snapshot 3A in snapshot chain 3, and snapshot 3A is Active.
- Create a single snapshot chain based on a backing file.
- Merge Snapshots
External snapshots are interdependent, where each overlay file depends on its backing file. Each snapshot preserves its corresponding data, preventing the direct deletion of an individual snapshot to shorten the chain length. The chain length of external snapshots can be reduced through two methods: downward merging (Blockcommit) or upward merging (Blockpull).
- Blockcommit
Within the same snapshot chain, you can merge overlays to backing files.
Figure 21. Blockcommit 
Assume there is an original base image (Base). VM instance 1 is created based on Base, and three interdependent external snapshots are created for VM instance 1: Snapshot 1A, Snapshot 1B, and Snapshot 1C. Now, Snapshot 1A and Snapshot 1B are merged downward into VM instance 1. As a result, the backing file of Snapshot 1C (Active) points directly to VM instance 1, so the snapshot chain is shortened. Snapshots 1A and 1B are no longer useful and can be deleted.
- Blockpull
Within the same snapshot chain, you can merge backing files to overlays.
Figure 22. Blockpull 
Assume there is an original base image (Base). VM instance 1 is created based on Base, and three interdependent external snapshots are created for VM instance 1: Snapshot 1A, Snapshot 1B, and Snapshot 1C. Now, Snapshot 1A and Snapshot 1B are merged upward into Snapshot 1C (Active). As a result, the backing file of Snapshot 1C (Active) points directly to VM instance 1, so the snapshot chain is shortened. Snapshots 1A and 1B are no longer useful and can be deleted.
- Blockcommit
Distributed Storage Snapshot Mechanism
Ceph enterprise edition uses ROW snapshot technology. ZStack ZStone uses COW snapshots. For more information, see Volume Snapshot Protection.
Backup Service
The backup service is business-centric and integrates various backup technologies such as scheduled incremental and full backup into the platform. The backup service supports multiple disaster recovery solutions, including local, remote, and public cloud disaster recovery. You can choose an appropriate backup method based on your business characteristics. The backup service is provided as a separate functional module.
Typical Scenarios
- You can use a locally deployed ImageStore as a local backup server to store scheduled backups of local VM instances, volumes, and management node databases (hereinafter referred to as databases). The local backup server supports seamless active-standby failover, effectively ensuring business continuity.
- If local data is accidentally deleted or data in the local primary storage is damaged, you can restore the data from the local backup server to the local environment.
- If a disaster occurs in the local data center, you can rely entirely on the local backup server to rebuild the data center and restore services.
- You can use a storage server in a remote data center as a remote backup server to store scheduled backups of local VM instances, volumes, and databases. The backup data must be synchronized from a local backup server to the remote backup server.
- If local data is accidentally deleted or data in the local primary storage is damaged, you can restore the data from the remote backup server to the local environment.
- If a disaster occurs in the local data center, you can rely entirely on the remote backup server to rebuild the data center and restore services.
- You can use a storage server on a public cloud as a public cloud backup server to store scheduled backups of local VM instances, volumes, and databases. The backup data must be synchronized from a local backup server to the public cloud backup server.
- If local data is accidentally deleted or data in the local primary storage is damaged, you can restore the data from the public cloud backup server to the local environment.
- If a disaster occurs in the local data center, you can rely entirely on the public cloud backup server to rebuild the data center and restore services.
Data Backup
Disaster recovery services support data backup at the QEMU block device layer, enabling backup for VM instances on all types of primary storage. Backup types include full backup and incremental backup. A full backup contains the complete dataset, while an incremental backup contains only data modified since the last backup. Both full and incremental backups store only actual (non-zero) data.
By default, the backup policy triggers a full backup after every 63 incremental backups following the initial full backup. This is because incremental backups depend on each other; prior incremental backups can be safely deleted only after a new full backup is completed. In practice, the system employs more intelligent and adaptive internal logic to dynamically select the optimal backup method, ensuring data safety and reliability.
The data backup process consists of three parts: data replication, data transfer, and data storage.
Data Replication
The Backup Service module utilizes the dirty data tracking functionality (Dirty Bitmap) at the QEMU block device layer to track and export backup data.
Locations in the VM instance's disk where data changes occur are called dirty data locations. The Dirty Bitmap records all such locations in the virtual disk file that have been modified since the last backup. Based on these records, all data modified since the last backup (incremental backup data) can be exported. Ultimately, the full backup file and subsequent incremental backup files form a complete backup chain to preserve the entire data set.
ZStack Cloud provides an adaptive backup export policy. Depending on the scenario, the backend decides whether to export incremental or full backup data. Since the Dirty Bitmap resides in the memory of the QEMU process, the tracking information is lost after a VM reboot. Therefore, ZStack Cloud automatically exports full backup data after a VM reboot.
Data Transfer
Two different implementation schemes are supported for different virtualization component versions, differing primarily in how backup data is transferred.
Scheme 1: Data Over SSHFS. This scheme uses SSHFS to mount the backup directory of the remote backup server on the compute node and imports the backup data into the backup server. SSHFS is a simple FUSE Over SSH solution. The data channel is encrypted by the SSH session, and each backup job uses a dedicated SSHFS link.
Scheme 2: Data Over NBD. This scheme uses the NBD module on the backup server to export a backup disk. Then, the compute node utilizes a QEMU block device job (Block-job) to write backup data directly to this disk.
Data Storage
The backup server supports various storage media types, including SAN, NAS, disk arrays, and tape libraries.
The backup server stores data in a chunked and deduplicated format. The backup data is split into 64MB chunks. A hash is calculated for each chunk, and an index is created. Data chunks with identical hash values are stored only once.
Data Recovery
When restoring a VM instance or volume from a local backup, the chunked data on the backup server is merged and imported into the primary storage.
For non-Ceph primary storage, the restored backup data is stored as a disk chain. For Ceph primary storage, the disk chain is merged into a single disk file for storage.
For a new recovery, the disk data restored to the primary storage serves as an image cache to create a new VM instance or volume. For an overwrite recovery, the system updates the database record of the current VM instance or volume with the new disk path on the primary storage and then deletes the old VM instance or volume files.
CDP Service
The CDP service delivers second-level, granular continuous data protection for critical business systems running on VM instances. The CDP service enables you to restore VM data to a specific point in time or retrieve files without performing a full system restoration. CDP recovery supports two strategies: creating a new VM instance and restoring to the original VM instance. You can choose an appropriate recovery method based on your business requirements. The CDP service is provided as a separate functional module.
Typical Scenarios
- Supports using a locally deployed ImageStore as a local backup server to store local VM instance data.
- Supports creating CDP tasks for multiple VM instances to provide unified CDP protection for VM instances. When creating a CDP task, you can set the RPO in seconds or minutes. During important business adjustments, you can mark and lock recovery points to preserve important recovery point data for a long time.
- In scenarios involving accidental local data deletion or data damage caused by sudden failures, due to hardware-based licensing requirements of the business application, to quickly validate service availability, you can locate a locked recovery point and restore the data to the original VM instance to check if the application functions correctly. Recovery to the original VM instance by creating new volumes is supported. And the volumes before recovery can all be retained and reattached to the VM instance, maximizing data security.
- During CDP recovery, the VM instance is rapidly restored with an RTO that can be as low as one second, effectively ensuring business continuity.
- Supports using a locally deployed ImageStore as a local backup server to store local VM instance data.
- Supports creating CDP tasks for multiple VM instances to provide unified CDP protection for VM instances. When creating a CDP task, you can set the RPO in seconds or minutes.
- During important recovery tests, you can create a new VM instance based on the selected recovery point without affecting the normal operation of the current VM instance. After confirming that the data is correct, you can then restore to the original environment.
- During CDP recovery, the VM instance is rapidly restored with an RTO that can be as low as one second, effectively ensuring business continuity.
Advantages
- This service is software-defined, hardware-independent, and scalable.
- Supports previewing and downloading backed up files without performing a full system restoration.
- Guided workflows with intelligent parameter recommendations minimize operational complexity and error rates.
- No agent installation is required for VM instances, eliminating OS dependencies and imposing no performance overhead on VM instances.
- Delivers second-level, granular continuous data protection for VM instances with an RPO that can be as low as one second.
- Instant VM instance recovery from any recovery point ensures business continuity with an RTO that can be as low as one second.
- Streamlined backup intelligently identifies disk partitions and valid data, backing up valid data with a smaller consumed storage space and at a faster speed.
- Supports flexible settings such as backup frequency, RPO, and retention policy to meet different needs.
- Supports multiple recovery levels, such as full VM recovery and file-level recovery.
- Full VM recovery supports multiple strategies, including creating a new VM instance and restoring to the original VM instance.
- Not limited by the type of primary storage, meeting CDP requirements in different storage scenarios.
- The UI provides a CDP overview, supporting unified viewing of CDP status and related alerts.
- Data recovery supports retaining current volume data, maximizing data security and facilitating post-failure analysis.
- Supports creating a new VM instance based on the selected recovery point and restoring to the original environment after confirming the data is correct, meeting recovery test requirements.
- Supports marking and locking recovery points for long-term preservation of important recovery point data.
- Provides a recovery task list and supports viewing recovery records and progress, facilitating subsequent auditing and tracing.
Data Backup
CDP (Continuous Data Protection) backup continuously tracks data changes on VM instances and synchronizes changed data to the backup server, generating point-in-time recoverable states.
Data Replication
The CDP module uses the dirty data tracking capability of the QEMU block layer, Dirty Bitmap, and Drive-mirror to track and export backup data.
The changed data locations in a VM disk file are called dirty data locations. Dirty Bitmap records all dirty data locations generated on the virtual disk file since the last backup. Based on these records, the system can export all data modified since the last backup, which forms incremental backup data.
The cloud platform provides an adaptive backup export policy. The backend selects incremental export or full export according to different conditions. Dirty Bitmap exists in the memory of the QEMU process and is lost after the VM instance restarts. Therefore, after a VM instance restarts, the cloud platform automatically exports full backup data.
The exported backup data is imported by Drive-mirror into a blank QCOW2 disk file stored on the CDP backup server. The blank disk file is created in advance when the CDP task is created, and is then exported as a network-accessible block device through NBD. In this way, the CDP backup task can continuously import VM volume data to the CDP backup server.
Recovery Point
After VM instance data is transferred to the CDP backup server, it is stored as QCOW2 disk files. For QCOW2 disk files, a QEMU storage service provides data recovery points under different protection policies.
Recovery points of a CDP task consist of BP points and RP points. BP points are coarse-grained external snapshots generated periodically, every 20 minutes by default. RP points are generated based on the configured CDP protection policy and can record I/O changes as frequently as once per second.
The CDP backup server first performs a full copy of VM instance data to generate a base BP point. Then QEMU continuously captures I/O data changes, adds timestamps to each I/O change, and saves them as RP points. Recovery points are generated on the CDP backup server and do not affect the original VM instance.
Data Recovery
CDP data recovery supports restoring VM instance data or retrieving individual files from a specified recovery point, helping users verify and restore data in scenarios such as accidental operations, service anomalies, or recovery drills.
File Retrieval
When restoring a VM instance or volume from a local backup, the chunked data on the backup server is merged and imported into the primary storage.
For non-Ceph primary storage, the restored backup data is stored as a disk chain. For Ceph primary storage, the disk chain is merged into a single disk file for storage.
For a new recovery, the disk data restored to the primary storage serves as an image cache to create a new VM instance or volume. For an overwrite recovery, the system updates the database record of the current VM instance or volume with the new disk path on the primary storage and then deletes the old VM instance or volume files.
Instant Recovery
When a VM service fails or is affected by a virus and requires full-machine recovery, CDP backup data supports second-level instant recovery to meet the requirement for rapid service recovery.
Instant recovery mainly includes two steps: quickly starting the VM instance and migrating VM instance data.
On the CDP backup server, VM backup data is stored in QCOW2 disk format. To quickly start the VM instance, the backed-up QCOW2 disk is first mapped as a network block device through NBD. The VM instance then accesses the backup disk on the backup server through NBD from the compute node. At this point, the VM instance can already provide services normally, and users can read and write data on the VM instance.
Because the VM disk has not yet been stored on primary storage, a storage migration task is started in the background to synchronize disk data from the backup server to primary storage. During this process, all writes to the VM instance are recorded as dirty pages and synchronized to the target disk on primary storage.
When the synchronization task detects that all data has been copied, the VM instance dynamically switches the disk path to the primary storage path by default. The background migration process synchronizes all VM instance data to primary storage. The entire process is transparent to users and does not affect VM services.
Backup Reliability Metrics
Two key metrics for evaluating disaster recovery system reliability are RPO and RTO. RPO (Recovery Point Objective) defines the maximum tolerable data loss—i.e., how much data may be lost after a disaster. RTO (Recovery Time Objective) defines the maximum tolerable time required to restore operations after a disaster.
Under low VM instance load conditions, the CDP module achieves both RPO and RTO as low as 1 second.
System Configuration Backup
System configuration backup is critical for the cloud platform. When the cloud platform encounters anomalies or configuration data is lost, recovery can be performed using the backed-up system configuration data.
The cloud platform provides a backup service module that supports multiple disaster recovery solutions, including local disaster recovery, remote disaster recovery, and public cloud disaster recovery. For details, see Backup Service.