DocsKnowledge BaseSecurityAPI Explorer
Documentation›ZCF›ZStack Zaku›User Guide
ZCF

Container O&M

One-Click Inspection

General

One-click inspection performs an all-round one-click health check on the key resources and services of the platform and provides optimization suggestions for the risk items, which ensures that the platform resources and services are in the best condition and facilitates efficient O&M.

How It Works

  • Inspection categories and inspection items:

    One-click inspection provides three categories of inspection items, basic services, computing and network, for the management cluster and the business clusters, and supports inspecting key platform resources and services such as cluster nodes, image servers and network components:

    • Basic services: detects the running status of the basic services and components of a cluster.
    • Computing: detects the usage and the running status of computing resources.
    • Network: detects the network configuration and status.

    You can select the clusters and the inspection items for a one-click inspection as needed. After the inspection is started, the platform performs a health check on the resources or services involved in the selected inspection items. For the detailed information of the inspection items, see Inspection Item Overview.

  • Inspection results:

    One-click inspection provides four inspection results for the inspected resources or services: Normal, Warning, Fault and Failure.

    • Normal: the inspected resource or service is in the normal state, which is marked with a green icon.
    • Warning: the inspected resource or service is in a poor state, which may affect the performance and stability of the related resources and services to some extent. It is marked with a yellow icon.
    • Fault: the inspected resource or service is in a very dangerous state, which may seriously affect the running of the business. It is marked with a red icon.
    • Failure: the inspection of the resource or service failed, which may seriously affect the running of the business. It is marked with a gray icon.
  • Health scoring:

    One-click inspection has a built-in health scoring mechanism, which supports quantitative scoring of the health status of the inspected resources or services and helps you grasp the overall running status of the platform intuitively and accurately.

    • Inspection resource/service score: scored according to the inspection result of the resource or service.
      • If the inspection result of a resource or service is Normal, the score is 100 points.
      • If the inspection result of a resource or service is Warning, the score is 50 points.
      • If the inspection result of a resource or service is Fault or Failure, the score is 0 points.
    • Inspection item score: calculated according to the scores of the resources or services involved in the inspection item. The details are as follows:
      • Scoring mechanism: (score of resource 1 + score of resource 2 + ... + score of resource N)/(N*100)*100.
      • Example: suppose an inspection item involves 3 resources whose inspection results are Normal, Warning and Fault/Failure and whose scores are 100, 50 and 0 respectively. Then the score of the inspection item is: (100 + 50 + 0)/(3*100)*100 = 50 points.
    • Cluster score: calculated comprehensively according to the scores of the inspection items under the cluster and the weights of the inspection items. The details are as follows:
      • Scoring mechanism: (score of inspection item 1*weight of inspection item 1 + score of inspection item 2*weight of inspection item 2 + ... + score of inspection item N*weight of inspection item N)/(100*weight of inspection item 1+100*weight of inspection item 2+......100*weight of inspection item N)*100
      • Example: suppose 3 inspection items under a cluster are selected for a one-click inspection, their scores are 100, 50 and 0 and their weights are 1, 2 and 3 respectively. Then the score of the cluster is: (100*1 + 50*2 + 0*3)/(100*1+100*2+100*3)*100 = 33 points.
    • Overall platform score: the average of the scores of all the clusters.
      • Scoring mechanism: (score of cluster 1 + score of cluster 2 + ... + score of cluster N)/N
      • Example: suppose 3 clusters are selected for a one-click inspection and their scores are 100, 50 and 0. Then the overall platform score is: (100 + 50 + 0)/3 = 50 points.
  • Inspection suggestions:

    For the inspected resources and services in the Warning and Fault states, one-click inspection analyzes the potential risks they face and the impact on the related resources and services, and provides targeted repair suggestions. For the detailed inspection suggestions, see Inspection Item Overview.

  • Inspection report:

    One-click inspection supports exporting an inspection report in WORD format, which displays the overview of the inspection information and the result statistics, summarizes the details of all the abnormal inspection items in the order of the abnormality level, and provides an inspection suggestion for each abnormal inspection item.

Advantages

ZStack Zaku one-click inspection has the following advantages:

  • All-round, efficient and customizable inspection: the three categories of inspection items cover all the key resources and services on the platform, customized selection of inspections is supported, and the results are delivered within minutes.
  • Multi-level scoring mechanism: the built-in four-level scoring mechanism (resource/service, inspection item, cluster and platform) helps you grasp the running status of the platform from the micro level to the macro level.
  • Intelligent risk troubleshooting suggestions: resource-level risk analysis and countermeasures are pushed intelligently, which facilitates accurate and efficient O&M.
  • Support for inspection without login: a one-click inspection can still be performed when the platform services are abnormal, which helps locate problems quickly.

Managing One-Click Inspection

In the ZStack Zaku main menu, click Container O&M > One-Click Inspection to enter the One-Click Inspection page.

One-click inspection supports the following operations:
Operation Description
Start One-Click Inspection After the clusters and the inspection items to be inspected are selected as needed, perform a one-click inspection on the selected inspection items.
Cancel One-Click Inspection When a one-click inspection is in progress, cancel the one-click inspection of the selected inspection items.
Retest After a one-click inspection is completed, inspect the inspection items selected in the last one-click inspection again.
Export Inspection Report Export an inspection report in WORD format.
Set Auto Inspection Automatic inspection can be set. The platform inspects all the inspection items automatically according to the inspection start time and the inspection cycle that are set. After the inspection is completed, you can enter the One-Click Inspection page to view or export the details of the inspection report.
  • Automatic inspection is enabled by default and the inspection cycle is 8 hours by default. It can be disabled as needed.
  • The start time and the inspection cycle of automatic inspection can be set as needed.
  • Email notification can be enabled as needed. If email notification is enabled, an email notification is sent to the specified endpoints when an abnormal inspection item is found.

Viewing Inspection Results

In the ZStack Zaku main menu, click Container O&M > One-Click Inspection to enter the One-Click Inspection page. After selecting the clusters to be inspected and their inspection items, click Start One-Click Inspection. After the inspection is completed, you can view the inspection report.

The inspection results display the overall health status of the selected inspection items by quantitative scoring, and classify and summarize the selected inspection items by status, which helps you quickly find the inspection items in an abnormal state. Meanwhile, the information and the status of the resources involved in each inspection item are clearly displayed in a list, and the corresponding inspection suggestions are provided.

  1. Overall health check results

    The upper part of the inspection results page summarizes the total number of the inspection items of the current inspection and the number of the inspection items in an abnormal state, and scores the inspection items of the current inspection as a whole according to the health scoring mechanism, which intuitively displays the overall health status of the inspection items. Meanwhile, the inspection results record the duration and the completion time of the current one-click inspection, which helps you schedule the next inspection. For the detailed health scoring mechanism, see Health Scoring.

  2. Classified display of the inspection item statuses
    The lower left part of the inspection results page summarizes all the clusters and their inspection items of the current inspection, and classifies all the inspection items as abnormal or normal according to the inspection results of the resources they involve. You can switch between the Normal and Abnormal pages freely to quickly search for and locate an inspection item. The specific classification rules are as follows:
    • If all the inspection results of the resources or services involved in an inspection item are Normal, the inspection item is marked with a green icon and classified as normal.
    • If all the inspection results of the resources or services involved in an inspection item are Warning, or the inspection results of some resources or services are Warning and the rest are Normal, the inspection item is marked with a yellow icon and classified as abnormal.
    • If all the inspection results of the resources or services involved in an inspection item are Fault, or the inspection results of some resources or services are Fault and the rest are Normal or Warning, the inspection item is marked with a red icon and classified as abnormal.
    • If all the inspection results of the resources or services involved in an inspection item are Failure, or the inspection results of some resources or services are Failure and the rest are Normal, Warning or Fault, the inspection item is marked with a gray icon and classified as abnormal.
  3. Inspection resource information, status and inspection suggestions
    You can click any inspection item in the lower left part of the inspection report page, and the lower right part displays the basic information and the inspection results of the inspected resources involved in the inspection item in a list, and provides inspection suggestions for the inspected resources whose inspection results are Warning and Fault.
    • List information:
      • Basic information: one-click inspection displays different basic information for different inspected resources and services. For example, the Node Time Synchronization Check inspection item can display the Node IP, the Node Time and the chronyd Service Status.
      • Inspection result: the corresponding inspection result is displayed according to the health status of the inspected resource or service, including Failure, Fault, Warning and Normal.
        • Failure: the resource or service cannot be inspected. It is marked with a gray icon.
        • Fault: the inspected resource or service is in a very dangerous state, which may seriously affect the running of the business. It is marked with a red icon.
        • Warning: the inspected resource or service is in a poor state, which may affect the performance and stability of the related resources and services to some extent, but does not seriously affect the running of the business. It is marked with a yellow icon.
        • Normal: the inspected resource or service is in the normal state. It is marked with a green icon.
      • Inspection suggestion: for the inspected resources and services in the Warning and Fault states, one-click inspection analyzes the potential risks they face and the impact on the related resources and services, and provides targeted repair suggestions, which facilitates efficient O&M. For the detailed inspection suggestions, see Inspection Item Overview.

Performing One-Click Inspection Without Login

When the platform services are abnormal and login is unavailable, a one-click inspection of the management cluster can be performed without login, which helps locate problems quickly.

You can enter the login-free one-click inspection in the following two ways:
  • Click One-Click Inspection on the Login page.
  • Access the login-free one-click inspection address directly. The login-free one-click inspection address is: http://${ZStack Zaku IP or domain name}/ze/cs/inspection.
Note: After you access the login-free one-click inspection address, the inspection password must be verified before an inspection can be performed. You can log in to a management cluster node and use the kubectl get secret inspection-secret -n zstack -ojsonpath={.data.password} |base64 -d - command to obtain the inspection password.

Inspection Item Overview

Inspection Type Inspection Item Meaning of the Inspection Item Inspection Suggestion Weight
Basic services (detects the running status of the basic services and components of a cluster) etcd status check Checks whether the status of the etcd service is normal. The status of the etcd service is abnormal. Execute the systemctl status etcd command to check the status of the etcd service. If the etcd service is not started, execute systemctl start etcd to start the etcd service. 3
kubelet status check Checks whether the status of the kubelet service is normal. The status of the kubelet service is abnormal. Execute systemctl status kubelet to check the status of the kubelet service. If the kubelet service is not started, execute systemctl start kubelet to start the kubelet service. 3
Container runtime status check Checks whether the status of the container runtime is normal. The status of the container runtime is abnormal. Execute systemctl status containerd to check the status of the containerd service. If the containerd service is not started, execute systemctl start containerd to start the containerd service. 3
kube-apiserver status check Checks the connectivity of kube-apiserver. The status of kube-apiserver is abnormal. Execute kubectl get pod -n kube-system to confirm the pod status of kube-apiserver. If an abnormal pod is found, restart the abnormal pod. 3
kube-controller-manager status check Checks whether the status of kube-controller-manager is normal. The status of kube-controller-manager is abnormal. Execute kubectl get pod -n kube-system to confirm the pod status of kube-apiserver and kube-controller-manager. If an abnormal pod is found, restart the abnormal pod. 3
kube-scheduler status check Checks whether the status of kube-scheduler is normal. The status of kube-scheduler is abnormal. Execute kubectl get pod -n kube-system to confirm the pod status of kube-apiserver and kube-scheduler. If an abnormal pod is found, restart the abnormal pod. 3
PostgreSQL status check Checks whether the status of PostgreSQL is healthy.
  • If the status of PostgreSQL is detected as Data Verification, it may be because the pod of PostgreSQL was abnormal and data verification is in progress. Usually, data verification is completed within 1 hour. Wait for a while and then perform the check again.
  • If another abnormal status of PostgreSQL is detected, execute the kubectl get pod -n zstack-middleware |grep postgres command to confirm the pod status of PostgreSQL. If an abnormal pod is found, restart the abnormal pod.
 3
Redis status check Checks whether the status of Redis is healthy. The status of Redis is abnormal. Execute kubectl -n zstack-middleware get svc -lapp.kubernetes.io/name=redis -l app.kubernetes.io/component=sentinel to obtain the redis sentinel service IP, and execute kubectl -n zstack-middleware get pod -l app.kubernetes.io/name=redis -l app.kubernetes.io/component=sentinel to obtain the pod of redis sentinel. Then use the kubectl -n zstack-middleware exec [redis sentinel pod name] -c sentinel -- redis-cli -h [redis sentinel service ip] -p 26379 info sentinel | grep status command to confirm the pod status of each Redis respectively. If an abnormal pod is found, restart the abnormal pod. 3
Container image repository check Checks whether the container image repository is normal. The container image repository is abnormal. Execute kubectl get pod -n harbor to check whether the harbor service is normal. 3
Container image repository connectivity check Checks whether the container image repository can be connected normally. Failed to pull an image from the container image repository. Check whether the network from the cluster nodes to the container image repository is normal. 3
Docker Registry status check Checks whether the status of Docker Registry on the control plane nodes is normal. The status of Docker Registry on some control plane nodes is abnormal. Log in to each control plane node, execute systemctl status registry.service to check the status of Docker Registry on the node, view the Docker Registry logs by the journalctl -u registry.service command, and troubleshoot according to the log content. 3
ChartMuseum status check Checks whether the status of ChartMuseum on the control plane nodes is normal. The status of ChartMuseum on some control plane nodes is abnormal. Log in to each control plane node, execute systemctl status chartmuseum.service to check the status of ChartMuseum on the node, view the ChartMuseum logs by the journalctl -u chartmuseum.service command, and troubleshoot according to the log content. 3
zstack-edge-nginx status check Checks whether the status of zstack-edge-nginx on the control plane nodes is normal. The status of zstack-edge-nginx on some control plane nodes is abnormal. Log in to each control plane node, execute systemctl status zstack-edge-nginx.service to check the status of zstack-edge-nginx, view the zstack-edge-nginx logs by the journalctl -u zstack-edge-nginx.service command, and troubleshoot according to the log content. 3
Prometheus status check Checks whether the status of the Prometheus pod is Running. The status of Prometheus is abnormal. Execute kubectl get pod -n monitoring to confirm the pod status of Prometheus. If an abnormal pod is found, restart the abnormal pod. 1
Loki status check Checks whether the status of Loki is normal. The status of Loki is abnormal. Execute kubectl get pod -n monitoring to confirm the pod status of Loki. If an abnormal pod is found, restart the abnormal pod. 1
Monitoring data disk usage check Checks the usage and the used capacity of the monitoring data disk. The usage of the monitoring data disk exceeds 80%. Check whether the retention period of the monitoring data is reasonable. It is recommended to shorten the retention period as needed. If necessary, it is recommended to expand the monitoring data disk. 1
Log data disk usage check Checks the usage and the used capacity of the log data disk. The usage of the log data disk exceeds 80%. Check whether the retention period of the logs is reasonable. It is recommended to shorten the retention period as needed. If necessary, it is recommended to expand the log data disk. 1
Container image repository data disk usage check Checks the usage and the used capacity of the data disk of the container image repository. The usage of the data disk of the container image repository exceeds 80%. It is recommended to expand the data disk of the container image repository. 1
Computing (detects the usage and the running status of computing resources) Node CPU usage check Checks the CPU usage of the nodes. The CPU usage of some nodes exceeds 80%. Log in to the nodes, check whether there are business exceptions, and optimize the running business as needed. If necessary, it is recommended to scale out the cluster. 1
Node memory usage check Checks the memory usage of the nodes. The memory usage of some nodes exceeds 80%. Log in to the nodes, check whether there are business exceptions, and optimize the running business as needed. If necessary, it is recommended to scale out the cluster. 1
Node system disk usage check Checks the usage and the used capacity of the system disks of the nodes. The usage of the system disks of some nodes exceeds 75%. Log in to the nodes immediately, and check and clean up the data that has no impact on the business. When the usage of a system disk exceeds 85%, the cluster evicts pods automatically, which affects the normal access to the business. 1
Node time synchronization check Checks whether the chronyd service is normal and whether the time difference between the other nodes and the first node is less than or equal to 2 seconds. The chronyd service is abnormal or the node time is not synchronized. Log in to the corresponding node to check. 3
Node SWAP partition check Checks whether the SWAP partitions of the nodes are disabled. The SWAP partitions of some nodes are not disabled, which may cause cluster exceptions. Log in to the corresponding nodes immediately to disable the SWAP partitions. 3
Node file descriptor check Checks the usage of the system file descriptors of the nodes. The usage of the system file descriptors of some physical machines is greater than 50%. Check the following items in turn:
  • Whether a process occupies a large number of file descriptors, and whether the application corresponding to the process is abnormal.
  • Whether the fs.file-max value in the /etc/sysctl.conf file is set too small. Increase the total number of file descriptors appropriately, and execute sysctl -p to make the setting take effect.
 3
Node CPU request rate check Checks the percentage of the sum of the CPU requests of all the pods on a node in the total CPU. The CPU request rate exceeds 80%. Log in to the cluster to check whether the resource allocation of the nodes is reasonable. If necessary, it is recommended to add nodes. 1
Node memory request rate check Checks the percentage of the sum of the memory requests of all the pods on a node in the total memory. The memory request rate exceeds 80%. Log in to the cluster to check whether the resource allocation of the nodes is reasonable. If necessary, it is recommended to expand the memory. 1
Node status check Checks whether the status of the cluster nodes is normal. The status of some nodes is abnormal. Log in to the nodes, execute systemctl status kubelet to check whether the kubelet service is normal, and execute systemctl status docker or systemctl status containerd to check whether the container runtime is normal. 3
Network (detects the network configuration and status) Node to management cluster connectivity check Checks the connectivity from the nodes to the management cluster and whether there is packet loss. There is packet loss or connection failure from some nodes to the management cluster. Check whether the network from the nodes to the management cluster is normal. 3
Inter-node network check Checks the IP connectivity between the nodes in a cluster and whether there is packet loss. There is packet loss or connection failure on the network of some nodes. Check whether the system and the network of the corresponding nodes are normal. 3
Calico status check Checks whether the status of the Calico pods is Running. The status of Calico is abnormal. Execute kubectl get pod -n kube-system |grep calico to confirm the pod status of Calico. If an abnormal pod is found, restart the abnormal pod. 3
CoreDNS status check Checks whether the status of the CoreDNS component is normal. The status of CoreDNS is abnormal. Execute kubectl get pod -n kube-system |grep coredns to check the pod status, and restart the abnormal pod. 3
NodeLocalDNS status check Checks whether the status of the NodeLocalDNS component is normal. The status of NodeLocalDNS is abnormal. Execute kubectl get pod -n kube-system |grep nodelocaldns to check the pod status, and restart the abnormal pod. 3
ResolvConf file status check Checks whether the status of the ResolvConf file on the nodes is normal. The status of the ResolvConf file on a node is detected to be abnormal. Check the following items:
  • Whether the ResolvConf file exists.
  • Whether a DNS service is configured in the ResolvConf file.
 1

Monitoring Panel

General

The monitoring panel provides the monitoring data of various resources such as CPU, memory, storage and network from multiple dimensions, including cluster, node, namespace, pod, network, PVC and control plane, which helps you quickly grasp the resource usage of clusters, nodes and pods and improves the O&M efficiency.

The monitoring panel organizes the monitoring views into the following two categories:
  • Resource monitoring: includes 6 menus, cluster, node, namespace, pod, network and PVC, where the usage of various resources can be viewed.
  • Platform monitoring: includes 1 menu, control plane, where the health status and the performance metrics of the Kubernetes control plane components can be viewed.
Note:
  • The monitoring panel is supported only by self-built clusters and the management cluster, and is not supported by integrated clusters.
  • The admin and administrator users have the permissions of all the menus. Common users can view only the 4 menus under resource monitoring, namespace, pod, network and PVC, and can view only the monitoring data of the namespaces on which they have permissions.
  • The time range and automatic refresh can be set in the upper right corner of each monitoring view. A full-screen button is provided on the right of the filter bar, with which a monitoring view can be enlarged to full screen or exit from full screen.
  • When no cluster can be selected or the monitoring service is abnormal, the monitoring view displays No Data.

Viewing Monitoring Data

In the ZStack Zaku main menu, click Container O&M > Monitoring Panel to enter the Monitoring Panel page. The Resource Overview tab of the Cluster menu is displayed by default. Select the monitoring menu to be viewed in the navigation on the left, and select the resource object to be viewed through the filter bar under the menu. The details of each monitoring menu are as follows:

  • Cluster: view the overall CPU and memory resource usage and the performance health status of a cluster. It is filtered by cluster and includes the following tabs:
    • Resource Overview: view the used amount, the request rate and the limit rate of the CPU and memory of the cluster, as well as the resource usage of each namespace.
    • Performance Diagnosis (USE): view the overall performance health status of the cluster by the utilization, saturation and errors method.
  • Node: view the resource usage details of a single node and the resource distribution of the pods on it. It is filtered by cluster and node, and includes the following tabs:
    • Node Resource: view the usage of the CPU, memory, disk and network resources of the node.
    • Pod Resource: view the CPU and memory resource usage of each pod on the node.
    • Performance Diagnosis (USE): view the performance health status of the single node by the USE method.
  • Namespace: view the resource usage and the quota of pods and workloads by namespace. It is filtered by cluster and namespace, and includes the following tabs:
    • Pod Resource: view the resource usage of each pod under the namespace.
    • Workload Resource: view the resource usage of each workload under the namespace. Filtering by workload type is supported.
  • Pod: view the CPU, memory, network and storage usage details of each container in a single pod. It is filtered by cluster, namespace and pod.
  • Network: view the network traffic and the bandwidth by cluster, namespace or pod to analyze the network load. The time granularity (30s/5m/1h) can be selected. It includes the following tabs:
    • Cluster Network: view the overall network traffic of the cluster. This tab is visible only to the admin and administrator users.
    • Namespace Network: view the network traffic at the namespace level.
    • Pod Network: view the network traffic of a single pod.
  • PVC: view the capacity usage and the read/write I/O of persistent volumes, which is used for storage capacity and performance analysis. It is filtered by cluster, namespace and PVC.
  • Control Plane: view the health status and the performance metrics of the API Server and etcd. It is filtered by cluster and includes the following tabs:
    • API Server: view the performance metrics of the cluster API Server, such as the availability and the request latency.
    • etcd Cluster: view the health status and the performance metrics of the cluster etcd.
Note: The lower-level options in the filter bar depend on the higher-level options. To view the monitoring data of a pod, select the cluster, the namespace and the pod in turn. After a higher-level option is changed, the lower-level options are reloaded.

Log & Events

Container Log

In the ZStack Zaku main menu, click Container O&M > Log and Event > Container Log to enter the Container Log page.

This page displays all the container logs in the clusters in a centralized way.

On this page, you can search and filter the logs and manage them in a unified way:
  • A time period can be selected to view the container logs of the selected time period. The available time periods include the last 5 minutes, the last 15 minutes, the last 30 minutes, the last 1 hour, the last 6 hours, the last 12 hours, the last 24 hours, the last 3 days, the last 7 days and custom. The logs of the last 5 minutes are displayed by default.
  • The logs can be searched by keyword.
  • The logs can be searched by cluster, namespace, workload, pod and container.
  • The context of a log can be viewed. You can search the context by keyword, and the keyword is highlighted in the search results.
  • The logs and the log contexts can be exported.

Event

In the ZStack Zaku main menu, click Container O&M > Log and Event > Event to enter the Event page.

This page displays all the Kubernetes events in each namespace of each cluster in a centralized way.

On this page, you can search and filter the Kubernetes events and manage them in a unified way:
  • Kubernetes events can be stored persistently, and the retention period can be set as needed.
  • A time period can be selected to view the Kubernetes events of the selected time period. The available time periods include the last 30 minutes, the last 1 hour, the last 6 hours, the last 1 day, the last 1 week and custom. The events of the last 30 minutes are displayed by default.
  • The events can be searched by resource name.
  • The events can be filtered by event type, reason and resource type.
  • The events can be sorted by count, earliest occurrence time and latest occurrence time.

Alarm Service

Alarm Message

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Alarm Message to enter the Alarm Message page.

This page displays all the alarm messages and alarm recovery messages in a centralized way, including the message content, the alarmed resource, the alarm level, the confirmation status and the occurrence time of an alarm message.

On this page, you can filter and search the alarm messages and manage them in a unified way:
  • A time period can be selected to view the alarm messages of the selected time period. The available time periods include the last 12 hours, the last 1 day, the last 3 days, the last 7 days and custom. The messages of the last 7 days are displayed by default.
  • The alarm messages can be searched by resource name, resource type and confirmation status. The unconfirmed alarm messages of all the resources within 7 days are displayed by default.
  • The details of an alarm message can be viewed.
  • The status of an alarm message that you have known can be changed to confirmed.
  • The number of the alarm messages displayed on each page can be adjusted. The available values are 10, 20, 50 and 100, and page turning is supported.

Alarm

General

An alarm is used to monitor and respond to the status changes of time series data, and pushes alarm messages to the specified endpoints through the notification service. You can create alarms to monitor resources.

Creating an Alarm

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Alarm to enter the Alarm page. Click Create Alarm and the Create Alarm page is displayed.

You can refer to the following example to enter the required information:
  • Name: set the name of the alarm. Naming rules: the length is limited to 1 to 128 characters, and the input can contain only Chinese characters, English letters, digits and the following 7 special characters: (-) (_) (.) (() ()) (:) (+).
  • Description: optional. Note the information about the alarm.
  • Resource Type: select the alarm resource type: node, pod, pvc, cluster, or GPU.
  • Alarm Item: select an alarm item according to the selected resource type.
  • Resource Scope: set the resource scope to which the alarm applies. The alarm can be applied to all the resources or specified resources of the selected resource type:
    • If an alarm is created for a single custom resource, the alarm monitors only this single resource, and an alarm is triggered when this resource meets the alarm condition.
    • If an alarm is created for multiple custom resources, the alarm monitors these resources, and an alarm is triggered when any of them meets the alarm condition.
    • If an alarm is created for all the resources, the alarm monitors all the resources of this resource type on the platform, and an alarm is triggered when any of them meets the alarm condition.
    Note:
    • For the pod resource type, you can select resources by pod or by workload. Selecting by pod monitors the selected pods. Selecting by workload monitors the pods of the selected workloads.
    • For the GPU resource type, first select a cluster and a GPU manufacturer. Then choose whether to monitor all GPUs within that scope or only specified GPUs.
  • Alarm Trigger Rule: set the alarm trigger rule.
  • Alarm Level: set the alarm level. Three levels are supported, Emergency, Critical and Alert. Alarms of different levels send alarm messages of different levels.
  • Recurring Alarm: set whether alarm messages are allowed to be sent repeatedly when the alarmed resource is not recovered in time after the first alarm.
    Note:
    • Recurring alarm enabled:
      • If a single resource is attached to the alarm and the resource continues to meet the alarm condition after an alarm is triggered, the alarm sends an alarm every 3 hours.
      • If multiple resources are attached to the alarm and one of them continues to meet the alarm condition after an alarm is triggered, the alarm sends an alarm for this resource every 3 hours. If another resource meets the alarm condition within the 3-hour interval, an alarm is sent for that resource directly.
    • Recurring alarm disabled:
      • If a single resource is attached to the alarm and the resource continues to meet the alarm condition after an alarm is triggered, the alarm no longer sends an alarm. If the resource returns to normal and then meets the alarm condition again, an alarm is triggered again.
      • If multiple resources are attached to the alarm and one of them continues to meet the alarm condition after an alarm is triggered, the alarm no longer sends an alarm for this resource. If another resource meets the alarm condition, an alarm is sent for that resource once.
  • Alarm Recovery Notification: optional. It is enabled by default. If it is enabled, the system sends a recovery notification once any resource in the alarm returns from the alarm state to the normal state.
  • Endpoint: optional. The default is the system alarm endpoint. The alarm messages are sent to the specified endpoints.

Managing Alarms

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Alarm to enter the Alarm page.

Alarms support the following operations:
Operation Description
Create Alarm Create an alarm.
Edit Modify the alarm, including the description, the resource scope, the alarm trigger rule, the alarm level, the recurring alarm, the alarm recovery notification and the endpoints.
Delete Delete the alarm.
Note: After the alarm is deleted, it no longer monitors the resources or sends alarm messages. Perform this operation with caution.
Enable Enable a disabled alarm.
Disable Disable an enabled alarm.
Note: After the alarm is disabled, the status of the alarm is still monitored, but no alarm message is generated.

Endpoint

General

An endpoint is the way for you to obtain the information of a subscribed topic. The endpoint types include system, Email, WeCom, DingTalk and Webhook.

Creating an Endpoint

Creating an endpoint includes the following four scenarios:
  • Creating an endpoint of the Email type
  • Creating an endpoint of the WeCom type
  • Creating an endpoint of the DingTalk type
  • Creating an endpoint of the Webhook type

Creating an Endpoint of the Email Type

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Endpoint to enter the Endpoint page, and click Create Endpoint to enter the Create Endpoint page.

You can refer to the following example to enter the required information:
  • Name: set the name of the endpoint. Naming rules: the length is limited to 1 to 128 characters, and the input can contain only Chinese characters, English letters, digits and the following 7 special characters: (-) (_) (.) (() ()) (:) (+).
  • Description: optional. Note the information about the endpoint.
  • Type: select Email.
  • Email Address: enter the email addresses. Multiple email addresses can be added, up to 100.
  • Notification Language: set the language used for the messages sent to the endpoint. Simplified Chinese and English are supported. The default is the same as the current platform language.

Creating an Endpoint of the WeCom Type

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Endpoint to enter the Endpoint page, and click Create Endpoint to enter the Create Endpoint page.

You can refer to the following example to enter the required information:
  • Name: set the name of the endpoint. Naming rules: the length is limited to 1 to 128 characters, and the input can contain only Chinese characters, English letters, digits and the following 7 special characters: (-) (_) (.) (() ()) (:) (+).
  • Description: optional. Note the information about the endpoint.
  • Type: select WeCom.
  • Group Bot Address: create a group bot in the WeCom group, and enter the Webhook address of the group bot here. After the creation is completed, the platform alarm messages are pushed to the WeCom group through the group bot.
  • Notification Language: set the language used for the messages sent to the endpoint. Simplified Chinese and English are supported. The default is the same as the current platform language.

Creating an Endpoint of the DingTalk Type

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Endpoint to enter the Endpoint page, and click Create Endpoint to enter the Create Endpoint page.

You can refer to the following example to enter the required information:
  • Name: set the name of the endpoint. Naming rules: the length is limited to 1 to 128 characters, and the input can contain only Chinese characters, English letters, digits and the following 7 special characters: (-) (_) (.) (() ()) (:) (+).
  • Description: optional. Note the information about the endpoint.
  • Type: select DingTalk.
  • Group Bot Address: create a custom bot in the DingTalk group, and enter the Webhook address of the bot here. After the creation is completed, the platform alarm messages are pushed to the DingTalk group through the bot.
  • Signing Secret: if the DingTalk version in use requires security settings for the group bot, select the signing method and enter the obtained signing secret here.
  • Notification Language: set the language used for the messages sent to the endpoint. Simplified Chinese and English are supported. The default is the same as the current platform language.

Creating an Endpoint of the Webhook Type

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Endpoint to enter the Endpoint page, and click Create Endpoint to enter the Create Endpoint page.

You can refer to the following example to enter the required information:
  • Name: set the name of the endpoint. Naming rules: the length is limited to 1 to 128 characters, and the input can contain only Chinese characters, English letters, digits and the following 7 special characters: (-) (_) (.) (() ()) (:) (+).
  • Description: optional. Note the information about the endpoint.
  • Type: select Webhook.
  • Webhook Address: enter the Webhook address. If the authentication method is Token, carry the token in the address.
  • Username: optional. If the specified Webhook can be accessed only with a username and a password, enter the actual username.
  • Password: optional. If the specified Webhook can be accessed only with a username and a password, enter the actual password.
  • Notification Language: set the language used for the messages sent to the endpoint. Simplified Chinese and English are supported. The default is the same as the current platform language.
  • Alarm Message Template: you can specify an alarm message template so that the alarm messages are sent to the endpoint in the specified format. If no template is specified, the default template of the current project is used. If no default template is created for the project, the built-in template of the system is used.

Managing Endpoints

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Endpoint to enter the Endpoint page.

Endpoints support the following operations:
Operation Description
Create Endpoint Create an endpoint.
Edit Modify the endpoint. All the content except the endpoint type can be modified.
Delete Delete the endpoint. After the endpoint is deleted, the related subscription messages are no longer sent to it. Perform this operation with caution.

Message Template

General

A message template is a standardized text template used when an alarm sends messages to an endpoint. The format and the content can be customized, which flexibly adapts to the requirements of different notification scenarios.

The system matches a template automatically in the following order of priority:
  1. If a dedicated template is specified for the endpoint, the specified template is used.
  2. If no template is specified, the default template of the project is used.
  3. If the project has no default template, the built-in template of the system is used automatically.

Creating a Message Template

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Message Template to enter the Message Template page. Click Create Message Template and the Create Message Template page is displayed.

You can refer to the following example to enter the required information:
  • Name: set the name of the message template. Naming rules: the length is limited to 1 to 128 characters, and the input can contain only Chinese characters, English letters, digits and the following 7 special characters: (-) (_) (.) (() ()) (:) (+).
  • Description: optional. Note the related information.
  • Type: currently only the Webhook type is supported.
  • Alarm Message Text: set the content template of the alarm messages. Enter the alarm message text in valid JSON format, which is used to define the content structure of the Webhook notifications. The system provides an example template, and you can refer to it and fill in the content as needed:
    {
  "facts": [
    {
      "name": "Alarm Message Details",
      "value": null
    },
    {
      "name": "Resource Type",
      "value": "${ALARM_RESOURCE_TYPE}"
    },
   {
      "name": "Alarm Resource",
      "value": "${ALARM_RESOURCE}"
    },
   {
      "name": "Cluster",
      "value": "${CLUSTER_NAME}"
    },
    {
      "name": "Node IP",
      "value": "${NODE_IP}"
    },
    {
      "name": "Namespace",
      "value": "${NAMESPACE}"
    },
    {
      "name": "Alarm",
      "value": "${ALARM_NAME}"
    },
    {
      "name": "Alarm Level",
      "value": "${EMERGENCY_LEVEL}"
    },
    {
      "name": "Alarm Trigger Condition",
      "value": "${ALARM_METRIC} ${COMPARISON_OPERATOR} ${ALARM_THRESHOLD}"
    },
    {
      "name": "Current Value",
      "value": "${CURRENT_VALUE}"
    },
    {
      "name": "Alarm Time",
      "value": "${ALARM_TIME}"
    },
    {
      "name": "Project",
      "value": "${PROJECT_NAME}"
    },
  ]
}
    Note: The variables wrapped in ${} in the text are automatically replaced with the actual alarm data when a message is sent, so as to generate dynamic alarm content. If a variable wrapped in ${} is not supported by the system, it is sent as a string as it is. The variables supported by the system are as follows:
    • ${ALARM_RESOURCE_TYPE}: the resource type
    • ${ALARM_RESOURCE}: the alarmed resource
    • ${CLUSTER_NAME}: the cluster name
    • ${NODE_IP}: the node IP address
    • ${NAMESPACE}: the namespace
    • ${ALARM_NAME}: the alarm name
    • ${EMERGENCY_LEVEL}: the alarm level
    • ${ALARM_METRIC}: the alarm item
    • ${COMPARISON_OPERATOR}: the alarm comparison operator
    • ${ALARM_THRESHOLD}: the alarm threshold
    • ${CURRENT_VALUE}: the current value of the metric
    • ${ALARM_TIME}: the alarm trigger time
    • ${PROJECT_NAME}: the project name
  • Alarm Recovery Text: set the content template of the alarm recovery messages. Enter the recovery message text in valid JSON format, which is used to define the content structure of the Webhook notifications. The system provides an example template, and you can refer to it and fill in the content as needed:
    {
  "facts": [
    {
      "name": "Alarm Recovery Details",
      "value": null
    },
    {
      "name": "Resource Type",
      "value": "${ALARM_RESOURCE_TYPE}"
    },
   {
      "name": "Alarm Resource",
      "value": "${ALARM_RESOURCE}"
    },
   {
      "name": "Cluster",
      "value": "${CLUSTER_NAME}"
    },
    {
      "name": "Node IP",
      "value": "${NODE_IP}"
    },
    {
      "name": "Namespace",
      "value": "${NAMESPACE}"
    },
    {
      "name": "Alarm",
      "value": "${ALARM_NAME}"
    },
    {
      "name": "Alarm Level",
      "value": "${EMERGENCY_LEVEL}"
    },
    {
      "name": "Recovery Trigger Condition",
      "value": "${ALARM_METRIC} ${COMPARISON_OPERATOR_RECOVERY} ${ALARM_THRESHOLD}"
    },
    {
      "name": "Earliest Alarm Time",
      "value": "${FIRST_ALARM_TIME}"
    },
   {
      "name": "Alarm Recovery Time",
      "value": "${ALARM_RECOVERY_TIME}"
    },
    {
      "name": "Project",
      "value": "${PROJECT_NAME}"
    },
  ]
}
    Note: The variables wrapped in ${} in the text are automatically replaced with the actual alarm recovery data when a message is sent, so as to generate dynamic recovery message content. If a variable wrapped in ${} is not supported by the system, it is sent as a string as it is. The variables supported by the system are as follows:
    • ${ALARM_RESOURCE_TYPE}: the resource type
    • ${ALARM_RESOURCE}: the alarmed resource
    • ${CLUSTER_NAME}: the cluster name
    • ${NODE_IP}: the node IP address
    • ${NAMESPACE}: the namespace
    • ${ALARM_NAME}: the alarm name
    • ${EMERGENCY_LEVEL}: the alarm level
    • ${ALARM_METRIC}: the alarm item
    • ${COMPARISON_OPERATOR_RECOVERY}: the recovery comparison operator
    • ${ALARM_THRESHOLD}: the alarm threshold
    • ${FIRST_ALARM_TIME}: the earliest alarm time
    • ${ALARM_RECOVERY_TIME}: the alarm recovery time
    • ${PROJECT_NAME}: the project name
  • Default Template: select whether to set the template as the default template. After it is set as the default template, the messages of the endpoints of the Webhook type for which no message template is configured under the current project are sent according to this template.

Managing Message Templates

In the ZStack Zaku main menu, click Container O&M > Alarm Service > Message Template to enter the Message Template page.

Message templates support the following operations:
Operation Description
Create Message Template Create a message template.
Edit Modify the message template. All the content except the type can be modified.
Set as Default Set the message template as the default template. There can be only one default template of the same type in the same project. After it is set as the default template, the endpoints of the Webhook type for which no message template is configured under the current project use this template to send alarm messages.
Delete Delete the message template.
Note:
  • After the message template is deleted, the Webhook endpoints that used it switch to the default template of the current project automatically to send alarm messages.
  • If the deleted template is the default message template, the Webhook endpoints for which no template is configured use the built-in template of the system automatically to send alarms.

Operation Log

Current Task

In the ZStack Zaku main menu, click Container O&M > Operation Record > Current Task to enter the Current Task page.

This page displays the information of the operations in progress, including the operation description, the operated resource, the number of resources, the task progress, the operator and the creation time.

On this page, you can search and filter the current tasks and manage them in a unified way:
  • The current tasks can be searched by operation description, operated resource and operator.
  • The details of a task can be viewed.
  • The number of the tasks in progress displayed on each page can be adjusted. The available values are 10, 20, 50 and 100, and page turning is supported.

Operation History

Operation history displays the completed operations and provides centralized viewing and management.

In the ZStack Zaku main menu, click Container O&M > Operation Record > Operation History to enter the Operation History page.

This page displays the information of all the completed operations, including the operation description, the operated resource, the number of resources, the task result, the operator, the creation time and the completion time.

On this page, you can search and filter the historical tasks and manage them in a centralized way:
  • A time period can be selected to view the logs of the operations completed in the selected time period. The available time periods include the last 3 days, the last 7 days, the last 1 month and custom. The logs of the last 3 days are displayed by default.
  • The historical logs can be searched by operation description, operated resource, operator and task result.
  • The completed operations can be sorted by creation time or completion time.
  • The number of the logs of the completed operations displayed on each page can be adjusted. The available values are 10, 20, 50 and 100, and page turning is supported.

Audit Log

Audit logs record platform resource operations, authentication activities, and Kubernetes resource operations to help you trace operations and troubleshoot issues.

In the ZStack Zaku main menu, click Container O&M > operation log > Audit Log to enter the Audit Log page.

Log Types

Audit logs include the following three types:

  • Resource Operations: records operations such as resource management and permission assignment. The admin user and administrators can view all records. Regular users can view records for projects they have permission to access, excluding records whose resource type is project, user, or namespace.
  • Identity & Authentication: records operations related to login and account credentials. The admin user and administrators can view all records. Regular users can view only their own operation records.
  • Kubernetes Audit: records the Kubernetes resource operations collected by the platform. Only the admin user and administrators can view these records. Before using this feature, go to Global Settings and enable Kubernetes Audit Log. Logs can be collected only from self-built clusters, including the management cluster. Integrated clusters are not supported.
Note: Resource Operations and Identity & Authentication logs are retained for 365 days by default. The retention period for Kubernetes audit logs is configured in Global Settings and defaults to 30 days. For more information, see Global Settings.

Querying and Exporting Logs

You can query and export audit logs on this page:

  • Select a log type and time range to view the corresponding operation records. Available time ranges are the last day, the last three days, the last seven days, and a custom range. By default, records for the last day are displayed. The last day covers the period from midnight today to the current time. The last three and seven days are calculated in the same way.
  • For Resource Operations and Identity & Authentication, you can find records by event name, resource type, target resource, or operator. Resource Operations also supports filtering by cluster, namespace, and project. Kubernetes Audit supports filtering by cluster, resource type, and operation, and searching by target resource or operator.
  • Click an event name to view the operation details. For Resource Operations and Identity & Authentication, you can view and copy operation results, related resources, and request and response details. For Kubernetes Audit, you can view the log in YAML format.
    Note: Sensitive information, such as passwords and access keys, is not displayed in plain text in Identity & Authentication logs.
  • The admin user and administrators can export the records on the current page or all records that match the selected criteria in CSV format.
Note: The list displays up to 3,000 Kubernetes audit logs, or up to 500 when the queried resource type is custom resource definition. To obtain more complete results, shorten the query time range or add filter criteria.
PreviousArtifact RepositoryNextCluster Management