ZStack Cloud runs on ZCF to deliver foundational cloud platform capabilities, abstracting server compute, storage connectivity, and network connectivity into unified, on-demand cloud resources—enabling efficient resource provisioning and business workload execution.

Compute virtualization abstracts physical server resources into logical resources via virtualization technologies, transforming a single physical server into multiple isolated virtual servers. Hardware resources—including CPU, memory, disks, and I/O devices—are pooled into virtualized resource pools for unified, dynamic management—improving resource utilization, reducing system management overhead, and enhancing IT agility in response to business changes.

The cloud platform supports hardware virtualization based on KVM. KVM is a Linux kernel module that turns the Linux kernel into a hypervisor. Within Linux, KVM runs as a process scheduled by the standard Linux scheduler, enabling it to leverage existing Linux kernel features—such as memory management and CPU scheduling. However, KVM itself provides only CPU and memory virtualization; I/O device virtualization requires integration with QEMU. QEMU is a userspace device emulator that provides virtual device models for VM instances, handling creation, invocation, and management of various virtual devices.

In the x86 architecture, CPUs generally have four privilege levels (RING0 to RING3) for operating systems and applications to access hardware. Linux uses only two of these levels: RING0 (kernel mode) and RING3 (user mode).

VMX Root Mode and VMX Non-Root Mode: For hardware-assisted virtualization, CPUs introduce two operation modes to enable VM Instances to run without modifying the operating system: VMX root mode and VMX non-root mode. The host runs in root mode, with its kernel in RING0 and user-mode programs in RING3. The VM instance runs in non-root mode, with its kernel in RING0 and user-mode programs in RING3.

VM Exit and VM Entry: A switch from non-root mode to root mode occurs when a VM instance in non-root mode encounters external interrupts, page faults, or actively executes the VMCALL instruction to invoke VMM services. This entire process is called VM Exit. Conversely, when the VMM explicitly executes either VMLAUNCH or VMRESUME instruction to switch to non-root Mode, the hardware automatically loads the VM instance context and executes VM instructions. This transition is called VM Entry.

After a VM instance triggers a VM Exit from non-root mode to root mode, KVM takes further action based on the exit reason. If the exit is due to an I/O operation, KVM delegates the processing to QEMU. For non-I/O exits, KVM handles them directly. After processing, KVM initiates a VM Entry to switch back to non-root mode for VM instance execution.

The Virtual Machine Monitor (VMM) manages and allocates physical memory for each VM instance. The guest OS sees a virtualized Guest Physical Address (GPA) space. The OS memory management module maps Guest Virtual Addresses (GVAs) to GPAs. The target address in instructions is also a GPA. In a non-virtualized environment, such an address would be the actual physical address. However, in a virtualized scenario, this address cannot be used directly. Instead, the VMM must first translate the GPA into a Host Physical Address (HPA), which is then executed by the physical processor.

Hardware-assisted memory virtualization uses Extended Page Tables (EPT) technology to translate GVAs into HPAs at the hardware level.

Memory Isolation: On the host's operating system, VM instance processes require a virtual Memory Management Unit (vMMU) and EPT for address translation. Although the VM instance process exists as a regular process on the host, it maintains two sets of page tables: the EPT and the host's standard page tables. When allocating memory to a VM instance, the hypervisor first allocates memory from the host in the form of a regular process. It then assigns this memory to the VM instance through the EPT. This approach ensures: unified allocation of HPAs, memory isolation between different VM instances, and a mapping relationship between the EPT and the host's page tables. From the VM's perspective, the contiguous memory may actually map to multiple non-contiguous address ranges on the host. The hypervisor dynamically allocates this memory on demand. This design achieves effective isolation between different types of processes, enables resource management, and maintains system flexibility and efficiency.

Complete Memory Release: During VM instance memory allocation, the initial allocation does not involve actual physical memory. When a VM instance genuinely uses the memory, control is transferred to the host's hypervisor for actual allocation. From the hypervisor's perspective, allocating memory to a VM instance means allocating memory space for the VM instance's processes. This space corresponds to Host Virtual Addresses (HVA). The host's memory management subsystem maps these HVAs to specific HPAs. This subsystem uniformly manages all physical memory on the host. During allocation, the subsystem marks the memory to ensure different processes receive different HPAs. When HPA resources are insufficient, the subsystem may reclaim some memory previously allocated to VM instances and mark the corresponding pages as invalid in the EPT. If a VM instance later accesses the reclaimed memory, the subsystem reallocates a new HPA space. This mechanism prevents physical memory from being repeatedly allocated to different VM instances, maintaining memory isolation and security.

There are three main approaches to device virtualization: device emulation, paravirtualized devices, and device passthrough.

Device Emulation

Device emulation uses the device models provided by QEMU to fully emulate interfaces identical to those of physical devices. Consequently, the VM operating system can use these devices with its native drivers. However, device emulation can only replicate devices with basic functionalities and does not support complex features or models. While fully emulated devices offer good compatibility, they suffer from lower performance since they are purely software-based emulation.

Paravirtualized Devices

Paravirtualized devices implement frontend and backend drivers. Utilizing a frontend driver in the VM instance, requests are directly sent to a backend driver on the host through a transaction-based communication mechanism, significantly reducing context switching overhead and improving performance over full device emulation. However, since the Virtio backend driver is still implemented in QEMU, the I/O processing path involves multiple switches between user space and kernel space. To further enhance performance, the functionality of the Virtio backend driver can be moved into the kernel space. This implementation is known as the Vhost-kernel backend. With this change, data transmission requires only a single switch from user space to kernel space, thereby improving performance.

As technology evolves, moving data processing to user space achieves greater flexibility. Consequently, modifications to the original Vhost architecture led to the Vhost-user backend, which works with relevant user-space libraries from DPDK and SPDK to further boost performance.

Device Passthrough

Device passthrough uses hardware-assisted device virtualization technology to directly map a physical PCI/PCIe device to a VM instance's address space. The VM instance can use the native device driver to directly operate the devices, achieving performance nearly identical to that of physical devices. Once a physical device is passed through, it is dedicated to that VM instance and cannot be shared with other VMs.

SR-IOV is an extension to the PCIe specification defined by the PCI-SIG. Its purpose is to provide a standardized specification for granting VM instances independent memory spaces, interrupts, and DMA data streams. SR-IOV enables a single physical PCIe device (Physical Function, PF) to be virtualized into multiple virtual PCIe devices (Virtual Function, VF). These VFs can then be passed through directly to VM instances using device passthrough technology. This allows a single physical PCIe device to support multiple VM instances.

GPUs and heterogeneous compute resources support high-performance computing scenarios such as graphics processing, AI inference, and model training. The cloud platform delivers GPU compute capacity primarily through GPU passthrough, vGPU partitioning and passthrough, and container-level GPU memory partitioning.

GPU Passthrough

Supports direct mapping of a physical GPU card into a VM instance's address space. Within the VM instance, native device drivers can be used directly to access the GPU, achieving near-bare-metal performance. Once passed through, the GPU is exclusively dedicated to that VM instance and cannot be shared with other VM instances.

vGPU Partitioning and Passthrough

Supports partitioning a physical GPU card into finer-grained vGPUs to form a vGPU resource pool, and enables direct mapping of vGPUs into a VM instance's address space. Once passed through, the vGPU is exclusively dedicated to that VM instance and cannot be shared with other VM instances.

Container GPU Memory Partitioning

Container GPU memory partitioning primarily uses the software-based vCUDA solution: the native CUDA driver is rewritten and mounted into Pods as a replacement, while CUDA API calls are intercepted within the driver to achieve resource isolation and quota enforcement.

Storage virtualization pools server storage resources, enabling unified integration, management, and scheduling of storage capacity, and exposes multiple storage interfaces to upper-layer services so that VM instances can flexibly allocate storage space from the pool based on their requirements. In the ZCF architecture, centralized and distributed storage capabilities are carried by Storage Service. The cloud platform organizes these storage resources into deliverable cloud resources and exposes data access capabilities to VM instances and other workloads. For details, see Storage Service.

The cloud platform achieves network virtualization either natively or via ZStack ZNS Network Service, providing network connectivity, isolation, access control, and common network services for VM instances, bare metal servers, and elastic bare metal instances. For details, see Network Service.

Container Service extends the cloud platform's resource scope to container clusters and cloud-native applications, delivering container cluster management, containerized application orchestration, and application lifecycle operations capabilities. For details, see Container Service.

Cloud resource scheduling and management are built upon compute virtualization, storage connectivity, and network connectivity, and are responsible for integrating VM instances, bare metal servers, and elastic bare metal servers into a unified lifecycle and scheduling system. This capability supports resource creation, runtime operations, migration, scaling, and orchestration—enabling business workloads to dynamically acquire compute, storage, and network resources on demand.

A VM instance is the foundational cloud resource type used by the cloud platform to host business workloads. The cloud platform provides full lifecycle management for VM instances—including creation, start, stop, restart, migration, cloning, and deletion—and defines their runtime environment through combinations of compute specifications, images, volumes, networks, and security policies.

In the ZCF architecture, VM instance management remains the responsibility of ZStack Cloud; ZCF delivers consistent access paths and operational visibility for VM instance environments via a Unified Portal, Unified Authentication, and Observability-based operations and maintenance capabilities.

VM scheduling policies determine the host scope available when a VM instance is created, started, or migrated. The scheduling process considers resource capacity, cluster boundaries, storage and network reachability, instance offerings, and user-defined scheduling policies to select an eligible target node.

With affinity and anti-affinity scheduling policies, the cloud platform can place related VM instances in the same resource group when possible, or distribute critical service instances across different hosts to balance performance, availability, and fault isolation. For VM instances loaded with dedicated devices such as GPUs, scheduling also considers device availability and GPU allocation policies before selecting the final node.

ZStack Cloud provides three VM instance clone methods to meet different service needs: linked clone, full clone, and instant full clone.

VM hot migration refers to migrating a running VM instance from one host to another without user awareness or service interruption. This technology is widely applied in cloud computing and data center management to achieve resource optimization, load balancing, maintenance upgrades, and disaster recovery. ZStack Cloud supports two types of VM hot migration: between hosts with the same storage and between hosts with different storage.

To understand how VM hot migration works, focus on the transmission mechanisms of the VM's core resources: CPU, memory, disk, and network.

Hot Migration Between Hosts with the Same Storage

Two hosts use the same storage, and the VM instance is hot migrated from one host to the other.

How are the core resources of the source VM instance synchronized to the target VM instance during hot migration?

First is the network. The network requests of a VM instance are commonly stateless, so it is only necessary to synchronize the network configuration of the source VM to the target VM. Next is the disk. Both the source and target VMs reside on shared storage, and the source and target hosts have an identical view of this shared storage. Consequently, there is no need to migrate the disk.

The critical component of hot migration involves transferring the CPU and memory. Throughout the migration, the source VM instance continues running, while the target VM instance is not yet started. During migration, the CPU persistently generates new data, which is written to memory. Memory pages modified by the CPU during this period are designated as dirty pages. The hot migration process initiates by copying the entire memory content from the source to the target VM instance. Subsequent iteration cycles transfer only the dirty pages that accumulated during the preceding copy operation, rather than retransmitting the complete memory image. This iterative process continues recursively. Migration progresses until the volume of dirty pages on the source host diminishes below a predefined minimal threshold—typically resulting in transfer times measured in milliseconds. At this point, the process enters the stop-and-copy phase: the source VM instance is suspended to stop dirty page generation, the final set of dirty pages is transferred, CPU execution state and peripheral device states are migrated, and the target VM instance is activated. Upon completion, the target VM instance assumes full operational control of the source workload. The minimal duration of the stop-and-copy phase ensures user-unaware migration with continuous service availability.

Hot Migration Between Hosts with Different Storage

Two hosts use different storage, and the VM instance is hot migrated from one host to the other.

During a hot migration, because the source and target hosts use different storage, disk migration is required. Data managed by the disk falls into two categories: data that has been written to disk and data not yet written to disk. Data not yet written to disk (data in memory) is referred to as dirty data. Disk migration first creates an identical target disk on the destination storage, then establishes a mirroring relationship between the target disk and the source disk. QEMU receives the specified source and target disk devices and creates a mirror job, which runs continuously in the background until all data replication is complete.

Full Sync: After the mirror job initiates, a full data synchronization is first performed. QEMU sequentially iterates through all data blocks of the source disk, reading and writing data block-by-block to the target disk. During this period, if the VM instance generates new write operations, QEMU temporarily stashes these writes and synchronizes the incremental data to the target disk immediately after the initial synchronization is completed.

Incremental Sync: Throughout the full data synchronization process, the VM might still write to the source disk. To ensure data consistency between the source and target disks, QEMU intercepts write operations to the source disk and copies the incremental data blocks to the target disk.

Write operation interception: QEMU's block layer is responsible for intercepting disk I/O requests and logging write operations. These writes are stashed, and metadata such as the operation timestamp and block offset are recorded to ensure accurate incremental copying. QEMU synchronizes new write operations to the target disk via Incremental Sync, maintaining consistency between the target disk and source disk.

Mirror completion and switchover: Upon completion of the mirror job, QEMU performs a final data synchronization to copy any remaining unsynchronized data blocks to the target disk. Then, you can choose to switch the VM's I/O from the source disk to the target disk. At this point, the data from the source disk is fully synchronized to the target disk. The entire VM hot migration finishes after the subsequent completion of the memory and CPU hot migration.

The core processes of baremetal management are hardware information retrieval and unattended deployment of baremetal chassis.

Elastic baremetal management provides dedicated physical servers for applications, ensuring high performance and stability for core applications. By combining these servers with ZStack Cloud's resource elasticity, this management model enables flexible provisioning and on-demand usage. Elastic baremetal management integrates the respective benefits of hosts and VM instances. Applications can benefit from not only the robust and stable computing power of hosts but also various resources within ZStack Cloud, such as primary storage and L3 networks. This approach avoids virtualization overhead while bridging the boundary between cloud and physical resources to improve the availability of cloud resources. Elastic baremetal is particularly suitable for deploying traditional non-virtualized applications.

Elastic bare metal supports two cluster types: gateway-proxy clusters and DPU-accelerated clusters—each with distinct architectural designs.

To retrieve the hardware information of a baremetal node, the management node uses IPMI to instruct the baremetal node to boot from the network. During the boot process, the baremetal node loads a LiveCD system located on the gateway node via PXE. It then uses pre-configured scripts within that system to collect its own hardware information and returns that data to the management node.

Auto scaling automatically adjusts the number of VM instances based on service load changes. Administrators can define scaling groups from VM templates and configure scaling rules, trigger conditions, and cooldown periods so that the cloud platform scales out under higher load and scales in when load decreases.

Auto scaling is typically used with monitoring alarms and load balancing. After monitoring data or alarm conditions trigger a scaling activity, the cloud platform creates or removes VM instances based on the scaling group template and uses the cooldown period to prevent frequent repeated scaling activities, maintaining a balance between resource utilization and service capacity.

CloudFormation uses templates to define a set of cloud resources and their dependencies, and allows the cloud platform to automatically create, configure, and manage the lifecycle of these resources. A template can describe resource objects such as VM instances, volumes, and networks, as well as references, dependencies, and output information between resources.

CloudFormation manages a group of resources as a resource stack. When a resource stack is created, the cloud platform parses the resource definitions and dependencies in the template and creates resources in the correct order. After the resource stack is running, users can view resources, events, and template content, and retain or delete resources in the stack according to the template policy.

Visual orchestration provides a graphical orchestration method. Users can drag resources on the canvas and establish dependencies, and then generate a stack template or create a resource stack directly. This reduces the orchestration cost of complex resource combinations and helps reuse standardized delivery solutions.