In the ZCF architecture, storage services are provided by storage components and exposed to upper-layer workloads through the cloud platform. The cloud platform organizes storage resources into types such as primary storage and backup storage to support VM instances, volumes, images, and container workloads.

The cloud platform supports integration with both centralized and distributed storage. Centralized storage is suitable for scenarios requiring stable storage resources delivered via shared storage devices; distributed storage is suitable for scenarios requiring highly available and scalable storage resource pools built through horizontal expansion across multiple nodes.

Centralized storage refers to storing data on a central node composed of one or more servers. All services are centrally deployed on this node, which uniformly manages data for subsidiary nodes. Data access is achieved through a single controller. Centralized storage falls into three categories: DAS, NAS, and SAN. You can select the appropriate type based on your data storage needs.

On architecture: The storage controller is the core component of a centralized storage architecture. Typically, it contains two controllers operating in an active-standby mode to prevent the system-wide failures due to hardware issues. The storage controller features frontend and backend ports. Frontend ports provide storage services to servers, while backend ports expand the storage system capacity. Through backend ports, the storage controller connects to more additional storage devices, forming a large storage pool.

Shared Block

In centralized storage scenarios, the cloud platform can connect to SAN storage through the Shared Block primary storage type. This method directly uses LUN devices allocated by users on the SAN storage as storage pools, which are then provided to business VM instances. Unlike Shared Mount Point (SMP), another file system-based primary storage type, Shared Block offers advantages such as simplified deployment, flexible scalability, and superior performance. Tests show that Shared Block can deliver the full performance of underlying physical disks. Currently, Shared Block supports shared access protocols including iSCSI, FC, and NVMe-oF.

Working principles:

• Block-Level Access: Shared Block divides data into blocks, each with a unique identifier. Servers read from or write to specific data blocks using these identifiers.
• Parallel Access: Multiple nodes can simultaneously access different blocks within Shared Block, enabling parallel read and write operations.
• Shared Access: Multiple nodes can share the same Shared Block, allowing concurrent access to the same data blocks. This is crucial for scenarios requiring data sharing in distributed systems.
• Fault Tolerance: Shared Block typically provides data redundancy and failure recovery mechanisms to ensure data safety and reliability.
• Scalability: Shared Block can scale capacity and performance by adding more storage nodes.
• Performance Optimization: Shared Block usually uses various techniques, such as caching and load balancing, to optimize performance.

On architecture: In centralized storage, a volume is mapped to a host. Depending on the storage controller's connection method, the multipath service on the host automatically aggregates multiple SCSI devices sharing the same WWID into a multipath device. The cloud platform detects multipath devices with the same WWID across all hosts in the cluster and creates a shared volume group (VG) for the user-selected volume. Logical volumes (LV) are provisioned from this VG, corresponding to volumes, snapshots, and other resources in the cloud platform. To maintain storage cluster consistency, the cloud platform implements a shared storage lock mechanism (Sanlock) for maintaining storage heartbeats, node management, and metadata management.

Sanlock + Lvmlockd

Sanlock (Shared storage lock manager) is essentially a lease management mechanism built upon Lamport's algorithms: Delta Paxos and Disk Paxos. This system handles cluster membership management, heartbeat maintenance, arbitration, and message passing. The next step involves translating this lease mechanism into locks usable by LVM: Lvmlockd.

With Sanlock and Lvmlockd, most LVM commands from servers are first sent to Lvmlockd, which translates LVM operations into lock operations recognizable by Sanlock.

Different resource operations require different lock permissions. Resource operations proceed only after obtaining the appropriate locks, ensuring the security of metadata and data. This entire process requires no intermediate node coordination or centralized operations, forming a fully distributed architecture.

In ZCF Storage Service, distributed storage capabilities are provided by composable storage components and exposed to upper-layer workloads through the cloud platform. The cloud platform supports integration with various distributed storage systems. This topic uses ZStack ZStone as an example to describe the capability composition and working mechanisms of distributed storage in the ZCF architecture.

As a composable storage component of ZCF, ZStack ZStone can build a distributed storage system on general-purpose servers and provide resource management capabilities for servers, hard disks, data disks, block storage volumes, storage pools, buckets, and more. These capabilities allow users to create storage pools with different data redundancy types and configure buckets with specified storage policies and access permissions, providing block, file, and object storage services for the cloud platform and workloads.

Using ZStack ZStone as an example, the distributed storage system follows a master-worker architecture and consists of multiple Master and Node components. The system can be divided into the control plane and the Nodes managed by the control plane.

ZCF supports distributed block storage capabilities, which are exposed through the cloud platform as virtual block devices for VM instances, volumes, and container workloads, supporting storage scenarios such as root volumes, data volumes, and persistent volumes. When ZStack ZStone is used as the distributed storage component, block storage capabilities are carried by ZStack ZStone.

Block storage capabilities support integration with third-party block storage volumes. Block storage volumes created on third-party platforms can be automatically synchronized into unified management and displayed separately from local volumes.

Block storage capabilities support cloning multiple block storage volumes from a single volume snapshot, offering two clone types: linked clone and independent clone.

Linked clones leverage snapshot layering and Copy-on-Write (COW) mechanisms. First, the system creates a read-only snapshot of the original block storage volume, then uses COW to copy original data to multiple clone volumes. These clone volumes share the same physical storage space, copying data only when modifications occur—thus significantly saving storage capacity.

Linked clone volumes maintain a dependency relationship with their source snapshot. The system supports breaking this dependency chain to convert a linked clone into an independent volume.

ZCF supports distributed file storage capabilities, providing shared file access to workloads through a distributed file system and supporting scenarios such as multi-node data sharing, application configuration sharing, and file collaboration. When ZStack ZStone is used as the distributed storage component, file storage capabilities are carried by ZStack ZStone.

File storage capabilities support sharing file directories via the SMB and NFS protocols.

ZCF supports distributed object storage capabilities, providing object data access through an S3-compatible API and supporting scenarios such as unstructured data, backup archiving, and application object data. When ZStack ZStone is used as the distributed storage component, object storage capabilities are carried by ZStack ZStone.

Object storage capabilities provide S3-compatible RESTful APIs to deliver object storage access capabilities to the cluster.

Each API type corresponds to a primary management service; a primary management service can further register multiple sub-management services, so each API may be associated with multiple management services. Each management service maintains the Resources it supports—each management service can be viewed as a Resource type. Each Resource may contain multiple Handlers, and each Handler contains multiple OPs. The diagram below illustrates the op_delete operation using the S3 protocol API as an example.

Object storage capabilities ensure data transmission and access security through three-tiered permissions: key pairs, user permissions, and bucket permissions.

Object storage supports multipart upload, enabling large objects to be split into multiple parts for independent upload. Each part represents a contiguous segment of the object's data and can be uploaded in any order. If a part fails to upload, only that part needs re-uploading—other parts remain unaffected. After all parts are successfully uploaded, the system assembles them sequentially into a complete object.

Note: During multipart upload, each part consumes one object quota; after merging, the final object consumes only one quota.

A storage policy defines rules governing resource allocation sources and data storage formats for buckets, managing the mapping between buckets and storage pools to route data of different storage classes to corresponding pools. Object storage capabilities support up to seven storage classes per storage policy (storageclass_0 through storageclass_6), where storageclass_0 is the system-provided default. Users can add storage classes either during policy creation or via the storage policy detail page. Each storage class maps to a dedicated data pool, and specifying a storage class directs object data to its corresponding pool.

Data compression technology uses compression algorithms to perform lossless or lossy compression on data, reducing storage space requirements, lowering storage costs, and improving storage efficiency. Data compression employs the Snappy algorithm and supports enabling online compression via the web interface.

ZCF supports capacity and performance management for distributed storage to improve storage utilization, optimize data access performance, and control the impact of recovery tasks and business I/O on cluster resources. When ZStack ZStone is used as the distributed storage component, capacity and performance management capabilities are carried by ZStack ZStone.

Cache acceleration capabilities use SSD-based caching devices to accelerate data access for HDD data disks. These capabilities use high-speed SSDs through the ZAS cache module to provide I/O caching for traditional HDD devices. The ZAS cache module caches frequently accessed hot data in the high-speed SSD devices and returns it to the application, significantly improving I/O performance, especially in scenarios characterized by hot data access patterns.

The ZAS cache module supports two caching policies: Writethrough and Writeback. In Writethrough mode, data is written simultaneously to the cache and the backend storage device, ensuring data consistency. In Writeback mode, most of the cache is used to buffer write data, ensuring that dirty data is written sequentially to the backend storage device. The system disables the Writeback policy by default but you can switch caching policies at runtime.

Thin provisioning provides upper-layer applications with more virtual storage space than is physically available in the storage cluster before data is written to the logical volume. It improves storage utilization and provides more flexible capacity scaling capabilities.

Storage pool recovery QoS controls the impact of recovery tasks on I/O resources and helps balance resource allocation between fault recovery and business workloads.

Within data nodes, an op_shardedwq queue handles various I/O requests from the upper level. This is a composite queue, typically containing several sub-queues. After I/O requests are dequeued, they interact with the disk via the ObjectStore interface. I/O types fall into two main categories: business read/write I/O requests from clients, and I/O generated by internal activities of the storage system, including I/O requests between data nodes, SnapTrim, Scrub, and Recovery.

This capability uses a weighted priority queue (wpq) to categorize and store the aforementioned I/O types into corresponding sub-queues. Each priority (prior) queue is created when its first request is enqueued. During dequeue, a weighted probability method determines the prior level. The priority (prior) of each queue serves as its weight. The probability of a prior queue being selected equals the ratio of its priority weight to the total weight of all queues. Even if selected, a prior queue is not guaranteed to dequeue a request; this also depends on the size of the request about to be dequeued.

Volume service QoS controls business I/O for block storage volumes, including maximum IOPS and maximum read/write bandwidth. By setting a business QoS, you can control the performance of different block storage volumes to meet diverse business performance requirements.

ZCF supports distributed storage O&M capabilities, including multi-resource pool management, physical disk LED indication, hard disk S.M.A.R.T. monitoring, data disk maintenance mode, data rebalancing, automatic fault detection, and alerting to support daily maintenance and fault handling of storage resources. When ZStack ZStone is used as the distributed storage component, O&M capabilities are carried by ZStack ZStone.

Multiple resource pool capabilities support independent use of storage media with different performance characteristics and enable fault isolation.

Each resource pool has distinct attributes and performance, such as the number of replicas, data redundancy level, and storage media. You can flexibly allocate and manage resources based on actual requirements to improve storage efficiency and performance.

Resource pools are isolated from each other. You can implement data isolation management across multiple pools. Besides, failures in a single resource pool does not affect other resource pools, effectively safeguarding data security and storage reliability.

Disk locate capabilities provide a visual interface to turn on the disk LED for rapid locating. When you need to maintain or replace a disk, click Disk Light in the UI. The corresponding disk's LED indicator lights up in the physical environment, guiding you to quickly and accurately identify the device, thereby improving O&M efficiency.

Hard disk S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) monitoring tracks the health status, temperature, firmware, and total bytes written of disks. Upper-layer services trigger relevant alerts based on I/O errors and disk status information returned by the S.M.A.R.T. data.

Data disk maintenance mode is designed for server or disk maintenance scenarios. To maintain servers or disks, you can put the corresponding disks into maintenance mode on the UI. A disk in this mode stops all services and data access, and the data on the disk does not undergo rebalancing.

Automatic fault detection and alerting monitors the storage system and individual storage servers. Upon detecting a failure, the system automatically sends alert messages to the platform. You can also add email endpoints to receive alarms, enabling timely response and recovery.

When a failure occurs, the system supports automatic service restart and data migration. This maximizes data reliability and availability, forming a highly reliable and highly available distributed storage system.

Data rebalancing evenly distributes data across data disks under all storage servers in the cluster. This enhances storage system performance and reliability.

Manual Data Rebalancing:

Data rebalancing also supports manual triggering. You can manually initiate rebalancing operations based on the actual data distribution.