Primary Storage

What is Primary Storage?

A primary storage is one or more servers that store volume files of VM instances. These files include root volume snapshots, data volume snapshots, image caches, root volumes, and data volumes.

The Cloud supports the following types of primary storage:

Local Storage: This type of primary storage uses the hard disks to store disk files.
Network Shared Storage: This type of primary storage supports NFS, Shared Mount Point (SMP), Ceph, SharedBlock, Vhost, CBD, AliyunNAS, and AliyunEBS.
- NFS primary storage uses the Network File System (NFS) to store files.
- SMP primary storage supports network shared storage provided by commonly used distributed file systems, such as MooseFS, GlusterFS, OCFS2, and GFS2.
- Ceph primary storage uses distributed block storage to store files.
- SharedBlock primary storage uses shared block storage to store files.
- Vhost primary storage uses vhost-user mode to connect with high-performance SSD distributed storage.
- CBD connects high-performance distributed block storage through the CBD interface.
- AliyunNAS primary storage uses distributed files to store files.
- AliyunEBS primary storage uses distributed block storage to store files.

The following table lists the relationship between primary storage and a cluster.

Primary Storage	Cluster
LocalStorage	A cluster can have one or more LocalStorage primary storage attached.
NFS	A cluster can have one or more NFS primary storage attached.
SharedBlock	A cluster can have one or more SharedBlock primary storage attached.
SMP	A cluster can have one SMP primary storage attached.
Ceph	A cluster can have only one Ceph primary storage attached.
Vhost	A cluster can have only one Vhost primary storage attached.
CBD	A cluster can have only one CBD primary storage attached.
AliyunNAS	A cluster can have only one AliyunNAS primary storage attached.
AliyunEBS	A cluster can have only one AliyunEBS primary storage attached.
LocalStorage + NFS	A cluster can have one LocalStorage and one NFS primary storage attached.
LocalStorage + SMP	A cluster can have one LocalStorage and one SMP primary storage attached.
LocalStorage + SharedBlock	A cluster can have multiple LocalStorage and multiple SharedBlock primary storage attached.
Ceph + LocalStorage	A cluster can have both a Ceph and a maximum of three LocalStorage primary storage attached.
Ceph + SharedBlock	A cluster can have one Ceph and one SharedBlock primary storage attached. A cluster can have one Ceph and multiple SharedBlock primary storage attached.
NFS + SharedBlock	A cluster can have both multiple NFS and multiple SharedBlock primary storage attached.

Add a Primary Storage

On the main menu of ZStack Cube Ultimate, choose Resource Center > Hardware > Storage Facility > Primary Storage. On the Primary Storage page, click Add Primary Storage. Then, the Add Primary Storage page is displayed.

The following lists the primary storage that you can add:

Add a LocalStorage primary storage.
Add an NFS primary storage.
Add a SharedMountPoint (SMP) primary storage.
Add a Ceph primary storage.
Add a SharedBlock primary storage.
Add a Vhost primary storage.
Add a CBD primary storage.

Add a LocalStorage Primary Storage

Set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select LocalStorage.
Note:
- If you select LocalStorage, the Cloud uses the hard disks of each host as a primary storage. LocalStorage primary storage can work with ImageStore and SFTP image storage. The total capacity of a LocalStorage primary storage is the sum of each host directory capacity.
- If you attach multiple LocalStorage primary storage, make sure that each LocalStorage primary storage is deployed on an exclusive logical volume or physical disk.
Mount Path: Enter the directory for mounting the LocalStorage primary storage.
Note:
- If the directory you entered does not exist, the Cloud automatically creates one.
- The following system directories cannot be used. Otherwise, the hosts might fail to work properly.
  - /
  - /dev
  - /proc
  - /sys
  - /usr/bin
  - /bin
Cluster: Select a cluster to which the LocalStorage primary storage is attached.

Figure 1. Add LocalStorage Primary Storage

Add an NFS Primary Storage

Set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select NFS.
Note: If you select NFS, ZStack Cube Ultimate automatically mounts the same NFS shared directory on all hosts as a primary storage. NFS primary storage can work with ImageStore and SFTP image storage, and can automatically mount the directory on all hosts.
Mount Path: Enter the shared directory of the NFS server.
Note:
- Format: NFS_Server_IP:/NFS_Share_folder. For example, 192.168.0.1:/nfs_root.
- You need to set the access permissions of the corresponding directories on the NFS server in advance.
- To ensure security control on the NFS server, we recommend that you configure corresponding security rules for access control.
- You can check the shared directory of the NFS server by using the showmount -e command on NFS server in advance.
- The following system directories cannot be used. Otherwise, the hosts might fail to work properly.
  - /
  - /dev
  - /proc
  - /sys
  - /usr/bin
  - /bin
Mount Option: Optional. To add mount options, make sure that these options are supported by the NFS server.
Note:
- The options are separated by commas (,). For example, nfsvers=3,sec=sys,tcp,intr,timeo=5. The preceding example means that the NFS server version is 3, the standard UNIX authentication mechanism is used, TCP is used as the transmission protocol, an NFS call can be interrupted in case of an exception, and the timeout is 0.5 seconds (5/10).
- To specify the mount options, you can refer to the content in the -o option of mount.
- You can set the options according to the mount command on commonly used clients. If the configured options conflict with the NFS server, the server shall prevail.
Storage Network: Enter the storage network specified for the shared storage. The storage network can be the management network of the management node.
Note:
- If you have a dedicated storage network, enter its CIDR.
- We recommend that you plan an independent storage network in advance to avoid potential risks. If you do not have an independent storage network, enter the network address according to your actual needs.
- The Cloud uses the storage network to check the health status of VM instances.
Cluster: Select a cluster to which the NFS primary storage is attached.

Add a SMP Primary Storage

Before you add a SMP primary storage, configure the corresponding distributed file system in advance, and mount the shared file system to the same file path on each host according to the client configuration of different storage systems.

The following uses MooseFS as an example to configure the primary storage:

Download and install the client tool mfsmount of MooseFS, and create a directory as the mount node.
Assuming that the IP address of the MooseFS Master Server is 172.20.12.19, create /mnt/mfs as the mount point, and use the mfsmount command to mount the MooseFS system.
You can also use the mfssetgoal command to set the number of file copies to be saved as needed.

[root@localhost ~]#mkdir /mnt/mfs 
[root@localhost ~]#mfsmount /mnt/mfs -H 172.20.12.19
[root@localhost ~]#mkdir /mnt/mfs/zstack
[root@localhost ~]#mfssetgoal -r 2 /mnt/mfs/zstack/
#The preceding commands mount the files in the /mnt/mfs/zstack/ directory to 172.20.12.19, and save two copies to the MooseFS storage server.

To add a SMP primary storage, set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select SharedMountPoint.
Note:
- For SMP primary storage, ZStack Cube Ultimate supports network shared storage provided by commonly used distributed file systems such as MooseFS, GlusterFS, OCFS2, and GFS2.
- SMP primary storage can work with ImageStore and SFTP image storage.
Mount Path: Enter the directory of the shared storage mounted by the host.
Note:
- The following system directories cannot be used. Otherwise, the hosts might fail to work properly.
  - /
  - /dev
  - /proc
  - /sys
  - /usr/bin
  - /bin
Storage Network: Enter the storage network specified for the shared storage. The storage network can be the management network of the management node.
Note:
- If you have a dedicated storage network, enter its CIDR.
- We recommend that you plan an independent storage network in advance to avoid potential risks. If you do not have an independent storage network, enter the network address according to your actual needs.
- The Cloud uses the storage network to check the health status of VM instances.
Cluster: Select a cluster to which the SMP primary storage is attached.

Add a Ceph Primary Storage

ZStack Cube Ultimate supports Ceph block storage. To add a Ceph primary storage, you need to add a Ceph or an ImageStore image storage, and configure the Ceph distributed storage in advance.

To add a Ceph primary storage, set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select Ceph.
Note: ZStack Cube Ultimate primary storage can work with the following Ceph editions:
- Ceph open-source edition: Jewel series, Luminous series, and Nautilus series.
- ZStack Ceph Enterprise: All released ZStack Ceph Enterprise. If you are concerned more about data security and I/O performance, ZStack Ceph Enterprise is recommended. For more information, contact our official technical support.
Cephx: Optional. Determine whether to enable Ceph authentication.
Note:
- By default, Ceph authentication is enabled.
- If the network of the storage node and the compute node is relatively safe, you can disable Cephx to avoid Ceph authentication failure.
- Make sure that the key authentication of the Ceph storage is consistent with this option. If Cephx is not disabled for the Ceph storage, enabling Ceph authentication here may cause VM creation failure.
Mon Node: Enter the IP address, SSH port, user name, and password of the Ceph monitor.
- Management IP: Enter the IP address of the Ceph monitor.
- SSH Port: Enter the SSH port of the Ceph monitor. Default: 22.
- User Name: Enter the user name of the Ceph monitor.
- Password: Enter the password of the Ceph monitor.
You can click Add Mon Node to add more Ceph monitors.
Image Cache Pool: Optional. Enter the name of an image cache storage pool.
Note:
- You can specify a storage pool for image caches. If you do not specify a storage pool, the Cloud creates one automatically.
- If you specify a storage pool, make sure that a storage pool is already available in the Ceph cluster. Then you can specify the UUID of a storage pool.
Data Volume Pool: Optional. Enter the name of a data volume storage pool.
Note:
- You can specify a storage pool for data volumes. If you do not specify a storage pool, the Cloud creates one automatically.
- If you specify a storage pool, make sure that a storage pool is already available in the Ceph cluster. Then you can specify the UUID of a storage pool.
Root Volume Pool: Optional. Enter the name of a root volume storage pool.
Note:
- You can specify a storage pool for root volumes. If you do not specify a storage pool, the Cloud creates one automatically.
- If you specify a storage pool, make sure that a storage pool is already available in the Ceph cluster. Then you can specify the UUID of a storage pool.
Storage Network: Enter the storage network specified for the shared storage. The storage network can be the management network of the management node.
Note:
- If you have a dedicated storage network, enter its CIDR.
- We recommend that you plan an independent storage network in advance to avoid potential risks. If you do not have an independent storage network, enter the network address according to your actual needs.
- The Cloud uses the storage network to check the health status of VM instances.
Cold Migration Network: A network dedicated for VM cold migrations across Ceph primary storage. If not set, the system uses the management network for VM cold migrations across Ceph primary storage by default.
Note: To avoid high workloads on the management network, we recommend that you set a Dedicated network for VM cold migrations and ensures the cold migration network connectivity between Ceph primary storage.
Ceph Enterprise Access Token: Optional. If you add the access token, the ZStack Ceph Enterprise storage is managed by the token.
Note: Make sure that your ZStack Ceph Enterprise license is valid.
Cluster: Select a cluster to which the Ceph primary storage is attached.

Add a SharedBlock Primary Storage

Set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select SharedBlock.
Note:
- SharedBlock primary storage uses LUN devices for storage and can work with ImageStore image storage.
- You can add LUN devices online.
- Currently, SharedBlock primary storage supports two shared access protocols: iSCSI, FC, and NVMe-oF.
Default Storage Space Allocation Strategy: Select a storage space allocation strategy, including thick provisioning and thin provisioning.
- Thick Provisioning: Allocates the required storage space in advance to provide sufficient storage capacities and to ensure storage performance.
- Thin Provisioning: Allocates storage spaces as needed to achieve a higher storage utilization.
Storage Network: Enter the storage network specified for the shared storage. The storage network can be the management network of the management node. You need to specify a storage network if you choose iSCSI or FC protocol.
Note:
- If you have a dedicated storage network, enter its CIDR.
- We recommend that you plan an independent storage network in advance to avoid potential risks. If you do not have an independent storage network, enter the network address according to your actual needs.
- The Cloud uses the storage network to check the health status of VM instances.
Cluster: Select a cluster to which the SharedBlock primary storage is attached.
LUN: Select one or more LUN devices as needed. Here you need to enter the disk unique identifier.
Note:
- LUNs are provided by iSCSI, FC, or NVMe storage. Make sure that the compute node is connected with the iSCSI, FC, or NVMe storage properly, and is added to the Cloud.
- To use LUNs provided by an iSCSI storage, attach the iSCSI storage to the cluster you select for the primary storage in advance.
Cleanse LUN: Optional. Determine whether to cleanse LUN devices. By default, LUN devices are not cleansed.
- If you choose to cleanse LUN devices, the residual data, such as file systems, RAID, or signatures of partition tables in the LUN devices will be forced to clean up.
- If data exists in your LUN devices, and you do not cleanse the devices, you will fail to add LUN devices or attach primary storage.
- The LUN devices to be added cannot have partitions. Otherwise, you will fail to add the devices.

Figure 5. Add SharedBlock Primary Storage

Add a Vhost Primary Storage

Set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select Vhost.
Note: Vhost primary storage uses vhost-user mode to connect with high-performance SSD distributed storage, and can work with ImageStore image storage.
Vendor: Select the vendor of the Vhost storage. Currently, ZHPS and XEBS-XINFINI are supported
Cluster: Optional. Select a cluster to which the Vhost primary storage is attached.
Storage Device Info: Enter storage device information. The parameters that you need to set depend on the storage vendor.
- If the vendor is ZHPS, set the following parameters:
  - IP Address: Enter the Vhost management IP address.
  - Port: Enter the port of the Vhost management IP address.
  - Username: Enter the username of the Vhost management IP address.
  - Password: Enter the password corresponding to the username.
  - Test Connection: Test the connectivity of the IP address and obtain the storage pool information.
  - Storage Pool: Specify a storage pool. Make sure that a storage pool is already available in the storage cluster.
- If the vendor is XEBS-XINFINI, set the following parameters:
  - IP Address: Enter the Vhost management IP address and port. You can add multiple IP addresses. To ensure security, we recommend that you add 2 or more addresses.
  - Token: Enter the Vhost token.
  - Test Connection: Test the connectivity of the IP address and obtain the storage pool information.
  - Storage Pool: Specify a storage pool. Make sure that a storage pool is already available in the storage cluster.

Add a CBD Primary Storage

Set the following parameters:

Zone: By default, the current zone is displayed.
Name: Enter a name for the primary storage.
The name must be 1 to 128 characters in length and can contain Chinese characters, letters, digits, spaces, hyphens (-), underscores (_), periods (.), parenthesis (), colons (:), and plus signs (+) and cannot begin or end with spaces.
Description: Optional. Enter a description for the primary storage.
Type: Select CBD.
Cluster: Select a cluster to which the primary storage is attached.
MDS Node:
- MDS Node MN IP: Enter the management IP address of the MDS node.
- SSH Port: Enter the port of the MDS management IP address. Default: 22.
- Username: Enter the username.
- Password: Enter the password corresponding to the username.
Storage Pool: Enter the pool name after creating the storage pool on the storage side in advance.

Manage a Primary Storage

On the main menu of ZStack Cube Ultimate, choose Resource Center > Hardware > Storage Facility > Primary Storage. Then, the Primary Storage page is displayed.

The following table lists the actions that you can perform on a primary storage.

Action	Description
Add Primary Storage	Add a primary storage to the Cloud.
Edit Primary Storage	Edit the name and description of the primary storage.
Enable Primary Storage	Enable the disabled primary storage.
Disable Primary Storage	Disable the primary storage. After the primary storage is disabled, you cannot use it to create VM instances, volumes, or snapshots. However, existing resources are not affected.
Reconnect Primary Storage	Reconnect the primary storage. This action will update the storage information about the primary storage. Note: If any host can connect properly to the primary storage, the status of the primary storage will be Connected.
Enter Maintenance Mode	Place the primary storage in maintenance mode. Note: Placing a primary storage in maintenance mode stops resources such as VM instances on the primary storage. After a primary storage enters into maintenance mode, volumes on the primary storage cannot be used.
Set Ceph Enterprise Access Token	Add or modify an access token for the Ceph Enterprise storage.
Create Volume	Create a volume on the primary storage. Then, this volume will be an instantiated volume. Note: Shared volumes now support only Ceph and SharedBlock primary storage.
Attach Cluster	Attach the primary storage to a cluster. Note that one cluster can have multiple primary storage attached. The following are primary storage of the same type that a cluster can have: A cluster can have one or more LocalStorage primary storage attached. A cluster can have one or more NFS primary storage attached. A cluster can have one or more SharedBlock primary storage attached. A cluster can have one SharedMountPoint (SMP) primary storage attached. A cluster can have only one Ceph primary storage attached. A cluster can have only one Vhost primary storage attached. A cluster can have only one CBD primary storage attached. A cluster can have only one AliyunNAS primary storage attached. A cluster can have only one AliyunEBS primary storage attached. The following are combinations of primary storages that a cluster can have: A cluster can have both a LocalStorage and an NFS primary storage attached. A cluster can have both a LocalStorage and an SMP primary storage attached. A cluster can have both multiple LocalStorage and multiple SharedBlock primary storage attached. A cluster can have both a Ceph and a maximum of three LocalStorage primary storage attached. A cluster can have both a Ceph and a SharedBlock primary storage attached. A cluster can have both a Ceph and more than one SharedBlock primary storage attached. A cluster can have both multiple NFS and multiple SharedBlock primary storage attached.
Detach Cluster	Detach the primary storage from a cluster. Note: If you detach a primary storage from a cluster, note the following: This operation might shut down VM instances on the primary storage and associated with the selected cluster, thus affecting the service running. Please exercise caution. This operation might delete VPC vRouters on the primary storage and associated with the selected cluster, thus affecting the corresponding network service. Please exercise caution. This operation might cause the volumes on the primary storage and associated with the selected cluster to fail to work properly. Please exercise caution. If you detach a primary storage from an elastic baremetal cluster, note the following: This operation might shut down VM instances on the primary storage and associated with the selected elastic baremetal cluster, thus affecting the service running. Please exercise caution. This operation might cause the volumes on the primary storage and associated with the selected elastic baremetal cluster to fail to work properly. Please exercise caution.
Delete Primary Storage	Delete the primary storage. Note: When you delete a primary storage, note the following: Before you can delete the primary storage, detach it from all clusters. Deleting a primary storage also deletes all the resources on the primary storage, such as VM instances, elastic baremetal instances, volumes, and snapshots. Even if you add this primary storage again, the original files cannot be automatically detected.
Clean up Data	Clean up original data reserved during the migrations across Ceph primary storage, the migrations across NFS primary storage, and the migrations across SharedBlock primary storage. To perform this action, go to the Data Cleanup tab of the primary storage details page, and click Cleanup. These data cannot be recovered after cleanup. Please exercise caution.
Clean Up Storage Trash	Clean up the trash data save in the recycle bin of a Ceph primary storage. Data deleted from Ceph primary storage will be retained in a recycle bin for a period before expunged. You can specify this period by setting Retention Period of Data Deleted from Ceph Primary Storage in Global Setting. During this period, you can recover deleted data from the recycle bin. After this period, the data is expired. The expired data will be expunged by the system regularly and cannot be recovered. You can also manually expunge data in the recycle bin by cleaning up storage trash. The data that is cleaned up cannot be recovered. Please exercise caution.

SharedBlock Details

SharedBlock LUN

On the main menu of ZStack Cube Ultimate, choose Resource Center > Hardware > Storage Facility > Primary Storage. On the Primary Storage page, click the name of a SharedBlock primary storage and enter its details page. Then, click LUN to enter the LUN tab.

View LUN Multipath

On the LUN tab, click the name of a LUN device. Then, the Resource Details page is displayed. You can view the path information of the LUN on associated hosts, including runtime name, path status, health status, and target identifier.

You can specify a host to display the path information of the LUN on that host.

View LUN Monitoring Data

On the LUN tab, click the name of a LUN device. Then, the Resource Details page is displayed. On the Resource Details page, click Monitoring. You can view the monitoring data of the LUN on different hosts, including disk read/write, IOPS, and latency.

PreviousHost NextImage Storage