Issue/Introduction

This article describes how to troubleshoot HTTP 403 and requestTimeTooSkewed returned by ZSphere when ZCenter adds, tests, or reconnects a ZSphere cloud account. The error means that ZSphere determined that the difference between the REST request Date header and the current management node time exceeded the allowed range. The system clocks might be unsynchronized, or ZCenter and ZSphere might use different time zones while the time-zone validation switch uses a comparison method unsuitable for the environment.

Symptoms

• Adding a ZSphere cloud environment, testing a cloud account connection, or reconnecting an account in ZCenter fails.
• The page might display ZSphere account validation failed, with requestTimeTooSkewed in the cause.
• ZSphere returns HTTP 403 before reaching the normal account, license, or resource query flow.
• The cloud account status might remain or change to off, preventing subsequent resource synchronization.
• ZCenter and ZSphere might display local times several hours apart, for example, when one uses UTC and the other uses UTC+8.

Environment

• Product: ZCenter
• Version: 5.1.0
• Integrated platform: ZSphere 5.1.x and versions with the same time-zone validation mechanism
• Component: ZCenter ZSphere connection component, ZSphere REST API, and ZSphere management nodes
• Authentication: AK/SK-signed calls to the ZSphere REST API

Possible causes

When ZCenter calls the ZSphere REST API, it adds a time-zone-aware Date header and includes the value in the AK/SK signature. Before subsequent authentication, ZSphere checks the request time. The allowed difference is about plus or minus 15 minutes. If the difference exceeds the range, ZSphere returns HTTP 403 and requestTimeTooSkewed.

The ZSphere rest.checkTimeZone setting determines how the times are compared:

• true: Honor the time zone in the request and compare absolute times. The systems can use different time zones, but their actual system times must be synchronized.
• false: Ignore the time-zone offset in the request and compare displayed local times. Even at the same absolute time, systems in different time zones might be considered several hours apart.

Therefore, displayed time differences alone cannot identify this issue. Compare both epoch and the time-zone configuration.

Resolution

1. Confirm the error keyword and affected API

• Record the error time and the ZSphere cloud environment and cloud account UUIDs. Confirm that the response contains the exact case-sensitive string requestTimeTooSkewed.
• Search the ZCenter connection component and Portal logs for requestTimeTooSkewed, meta-data/opensource, /licenses, /zones, or ZSphere account validation failure messages.
• Integration validation might access ZSphere metadata, license, and zone APIs in sequence. Use the target-environment logs to identify the API that failed.

2. Compare the actual time on ZCenter and ZSphere

• Run the following commands on the ZCenter host, the process or container that actually sends ZSphere requests, and every active ZSphere management node:

  date '+local=%Y-%m-%dT%H:%M:%S %z %Z epoch=%s'
  date -u '+utc=%Y-%m-%dT%H:%M:%S %z %Z epoch=%s'
  timedatectl status

• If Chrony is used, also check:

  chronyc tracking
  chronyc sources -v

• Expected result: The epoch values on all nodes are approximately equal, and the system time synchronization service is healthy.
• Decision rules:
• An epoch difference greater than 900 seconds is a real system-time difference. Correct time synchronization first.
• If epoch is approximately equal but displayed local times differ by a fixed time-zone offset, such as about 8 hours, inconsistent time-zone handling is highly likely.

3. Check the time zone of the process that sends the request

• The ZSphere connection component in ZCenter generates the Date header. Check that process or container, not only the host that displays the Portal page.
• Docker example:

  docker exec <zsphere-connector-container> date '+local=%Y-%m-%dT%H:%M:%S %z %Z epoch=%s'
  docker exec <zsphere-connector-container> date -u '+utc=%Y-%m-%dT%H:%M:%S %z %Z epoch=%s'
  docker exec <zsphere-connector-container> java -XshowSettings:properties -version 2>&1 | grep -E 'user.timezone|user.country'

• Expected result: The connection component's system time and JVM time zone match the deployment design.
• If abnormal: If the container time zone differs from the host or the JVM uses an unexpected time zone, standardize the time zone according to the deployment specification and redeploy the connection component.

4. Check the ZSphere time-zone validation switch

• UI path: System Management > Global Settings > Platform Policy > Management Node Policy.
• Setting: Check Time Zone in REST API Calls.
• Backend setting: rest.checkTimeZone.
• Behavior:
• When enabled, ZSphere honors the request time zone and compares absolute times.
• When disabled, ZSphere ignores the time-zone offset and compares only local times.
• Therefore, if the systems use different time zones but have the same epoch, disabling the switch might incorrectly report a difference of several hours.

5. Rule out proxy modification of the request header

• If ZCenter accesses ZSphere through a proxy or load balancer, check whether the intermediary modifies, removes, or regenerates the Date header.
• Record redacted request time, response time, and proxy logs. Do not expose the AK, signature, or complete Authorization header in logs or tickets.

Handling

1. Temporary workaround

In ZSphere, change Check Time Zone in REST API Calls to the opposite of its current state:

• If the switch is enabled when the integration fails, disable it and test again.
• If the switch is disabled when the integration fails, enable it and test again.

After saving the setting, test or reconnect the ZSphere cloud account again in ZCenter. This operation quickly confirms and temporarily works around a mismatched time comparison method. It does not replace system time synchronization.

2. Permanent resolution

1. Configure the same NTP/Chrony sources for the ZCenter host, ZSphere connection component, and all ZSphere management nodes.
2. Confirm that the epoch difference among the nodes is within the range allowed by ZSphere and, preferably, within normal time-synchronization error.
3. Standardize or explicitly configure the operating system, container, and JVM time zones according to the deployment specification to prevent a container from inheriting an unexpected time zone.
4. Set rest.checkTimeZone according to the standardized environment and security policy. Do not rely on repeatedly toggling the switch to maintain connectivity.
5. If ZSphere is accessed through a proxy, correct the proxy's handling of the Date header so that it remains unchanged.

Verification

• Time: The epoch values on the ZCenter connection component and all ZSphere management nodes are approximately equal, with no difference greater than 900 seconds.
• Configuration: Record the final state of Check Time Zone in REST API Calls and confirm that it matches the current time-zone deployment.
• Connection: Run POST /api/portal/v4/cloud-account/CheckAccount in ZCenter and confirm that it no longer returns requestTimeTooSkewed.
• Account: Reconnect an existing account and confirm that its status changes to on.
• Business: Wait for or trigger 1 controlled resource synchronization and confirm that ZSphere zones and resources can be queried.
• Logs: Subsequent scheduled connection checks no longer report requestTimeTooSkewed or an HTTP 403 time-skew error.

Risks and rollback

• Check Time Zone in REST API Calls is a global ZSphere REST API policy. Changing it might affect other clients that use AK/SK to call ZSphere. Assess the impact and record the original value before changing it.
• Changing operating system time, time zone, or NTP configuration might affect log timestamps, scheduled tasks, certificate validation, and cluster components. Perform the change in a maintenance window according to platform specifications.
• If changing the switch does not restore the connection, return it to its original value and continue checking the actual clock difference, connection component JVM time zone, and proxy request headers.
• Do not work around the issue by increasing the allowed time difference, modifying the signature algorithm, or leaving time synchronization disabled.

Additional information

• The English description of the ZSphere setting is Check Time Zone in REST API Calls. Determine its default and final recommended value from the target ZSphere version and the environment's security policy.
• For general troubleshooting of cloud environment and cloud account connection failures, see How do I troubleshoot a failure to add a cloud environment or test a cloud account connection?.