Issue/Introduction

This article describes how to troubleshoot a blank ZCenter monitoring report page, a Grafana panel without a graph, or a resource monitoring API that returns no data. The blank state can originate in Grafana, Prometheus, a monitoring synchronization task, target resource labels, or the cloud platform metrics API. Do not assume that Grafana is the cause.

Symptoms

• The monitoring report page is blank, or a panel displays No data.
• The monitoring page opens, but the cloud environment or resource drop-down list is empty.
• 1 resource has no data while other resources work, or no resource has data.
• A monitoring API returns an empty array, and a Prometheus query matches no time series.
• The VM service log might contain MetricDataApiImpl has no executable configuration info, the mo didn't get any metric data!, sync cloud metric error!, or sync metric data error!.

Environment

• Product: ZCenter
• Version: 5.1.0
• Component: VM service, Prometheus, Grafana, and monitoring synchronization tasks
• Resource types: VMs, physical hosts, bare metal instances, or storage resources

Possible causes

• Grafana is inaccessible, a data source is missing, or the Prometheus data source URL differs from the ZCenter configuration.
• Prometheus is not running, a scrape target is abnormal, or the target lacks required labels such as moUuid and nodeType.
• syncMetricDataTask is paused, was not created, has no configuration, or failed.
• The cloud platform monitoring plugin returns no metrics, or the target resource has no available monitoring data.
• The query uses an incorrect moUuid, cloud type, or resource type.
• Grafana queries only the last 30 minutes by default, and the data is outside the current time range.

Resolution

1. Confirm the scope

• Record the resource's ZCenter moUuid, cloud account, cloud type, resource type, page URL, selected time range, and occurrence time.
• Determine whether the issue affects 1 resource, 1 cloud environment, or all resources.
• Note: Grafana queries use the ZCenter moUuid. Do not directly use the cloud platform originalId as the moUuid.

2. Check Grafana

• Command:

  curl -sS -i http://<grafana-host>:3000/api/health

• Check: Query /api/datasources from the Grafana administration page or with an authentication method approved for the deployment. Do not put the Grafana administrator password directly in shell history or a ticket.
• Expected result: Grafana is accessible, and the authenticated data-source list contains a data source with the current Prometheus address. A 401 from an unauthenticated request proves only that the HTTP service is reachable; it does not prove a data-source failure.
• If abnormal: If Grafana is unreachable, check the process, listen address, reverse proxy, and firewall. If the data source is missing or its URL differs, correct it according to the deployment specification.

3. Check Prometheus and target status

• Commands:

  curl -sS -i http://<prometheus-host>:9090/-/healthy
  curl -sS http://<prometheus-host>:9090/api/v1/targets
  curl -sS --get http://<prometheus-host>:9090/api/v1/query --data-urlencode 'query=up'

• Expected result: Prometheus is healthy, targets are available, and the target resource has labels such as moUuid, originalId, nodeType, and cloudType.
• If abnormal: If Prometheus is unreachable, check the service and port. If a target is unhealthy, check the cloud platform metrics API, network, and target configuration. If labels are missing, continue checking ZCenter monitoring synchronization and resource mapping.

4. Query the actual time series for the target resource

• Command:

  curl -sS --get http://<prometheus-host>:9090/api/v1/query \
    --data-urlencode 'query=cpu_util{moUuid="<MO_UUID>"}'

• Expected result: Metric data for the target resource is returned.
• If abnormal: If the result is empty, confirm that moUuid is correct and the time range covers when the data was generated, and search the VM service log for metric synchronization errors. The built-in panels default to approximately the last 30 minutes.

5. Check the monitoring task and cloud-level configuration

• UI path: Open Task Management and check monitoring metric synchronization task task_5 and the monitoring configuration for the corresponding cloud environment.
• Check: Task state, execution time, Cron configuration, cloud-type monitoring configuration, start and end times, and execution interval.
• Expected result: task_5 is running, the corresponding cloud environment has an executable monitoring configuration, and the latest execution time continues to update.
• If abnormal: If the task is paused or was not created, restore it through the supported task-management process. Do not directly modify the task table or manually insert a monitoring configuration.

6. Check ZCenter monitoring and variable APIs

• Resource monitoring API example: GET /api/vm/v4/resource/QueryVmMonitorData?moUuid=<MO_UUID>.
• Grafana variable API examples: GET /api/portal/v4/pub/monitor/grafana/getCloudType and GET /api/portal/v4/pub/monitor/grafana/getVm?cloudType=<CLOUD_TYPE>.
• Expected result: The APIs return the target resource and metric data, and the cloud-type and resource drop-down lists are not empty.
• If abnormal: If the APIs return no data, check resource management state, cloud account state, and the corresponding cloud platform monitoring plugin. If an API has data but Grafana does not, focus on the data source, dashboard variables, and Prometheus labels.

7. Collect support information

• Grafana, Prometheus, and VM service logs, and the target resource moUuid.
• Prometheus /api/v1/targets output and target metric query result.
• Monitoring task state, latest execution time, cloud type, and time range.
• Page request URL, HTTP status code, and occurrence time.

Handling

1. Restore Grafana or Prometheus, and correct the data source address, reverse proxy, and service port.
2. Correct the Prometheus target configuration or cloud platform metric connection so that the target is healthy and includes the resource labels required by ZCenter.
3. Restore the monitoring synchronization task through the supported task-management process. Wait 1 synchronization cycle before querying the data again.
4. Correct the resource, cloud type, or time-range selection. For a single-resource issue, investigate that resource first. Do not rebuild every dashboard.
5. Do not directly delete Prometheus history, purge monitoring tables in bulk, or manually modify Prometheus time-series data.

Verification

• Grafana: /api/health and /api/datasources return successful responses, and dashboard variables load.
• Prometheus: The target is healthy, and queries such as cpu_util{moUuid="<MO_UUID>"} return data.
• Task: task_5 continues to run, and the monitoring configuration update time continues to change.
• API: Resource monitoring APIs such as QueryVmMonitorData return metric data.
• Page: After the dashboard is refreshed, the cloud environment and resource drop-down lists and monitoring graphs display normally.

Risks and rollback

• Record the original Grafana data source, Prometheus target, or task configuration before changing it to avoid affecting other cloud environments.
• When restoring a task, do not simultaneously change the collection interval, time range, and data source, so that the effect of the fix remains identifiable.
• If a configuration change fails, restore the original configuration. Do not roll back by deleting monitoring data or rebuilding every dashboard.

Additional information

• The VM service syncMetricDataTask synchronizes monitoring data. Use the task configuration in the target environment for the actual execution interval.