Issue/Introduction

This article troubleshoots 408 Request Timeout from backup, snapshot, or host-management requests, especially an MN-to-KVM Agent HTTP request that fails after about 15 seconds. The related development item provides adjacent socket-timeout evidence from large-scale host reconnection, not direct proof of cause or fix for this backup/snapshot scenario.

Symptoms

• Management-node logs show Async POST request ... resulted in 408 or 408 Request Timeout: [no body].
• Veeam/Vinchin backup fails during takesnapshot, SharedBlock checks, or host-capacity requests.
• KVM Agent business logs might lack an entry for the URI while other requests still run.
• Multiple hosts and URIs return 408 after about 15 seconds; one snapshot failure alone does not prove an LVM-command failure.

Environment

• Product: ZVF ZSphere
• Version: 4.10.19.3 (site build in the incident bundle; code checks from 4.10.19 through 4.10.28 only confirm unchanged defaults and do not expand scope)
• Components: ZSphere management node, KVM Agent HTTP service, and host management network
• Deployment: The management node accesses KVM Agent through TCP 7070 while concurrent backup, snapshot, or capacity queries might run

Possible causes

1. CherryPy server.socket_timeout=15 returns an empty-body 408 before the business handler.
2. The default Agent thread pool is 10; queued requests or management-network fluctuation prevents timely reading.
3. The MN-to-Agent network path is abnormal, a separate branch from thread-pool contention that requires packet capture.
4. A slow business storage operation later succeeds; exceeding 15 seconds does not itself mean command failure.

Resolution

1. Identify who returned 408

• Check: URI, destination host, start/return times, and response body in management-server.log.
• Location: Query in order; do not change Agent parameters after searching only for 408.

1. Record task apiId, time, VM UUID, and host, then run on the management node:

     KB_API_UUID='<apiId-from-task-details>'
     cd /usr/local/zstack/apache-tomcat/logs || exit 1
     ls -lh management-server.log*
     zgrep -a -n -C 3 "$KB_API_UUID" management-server.log*
     zgrep -a -n "$KB_API_UUID" management-server.log* | grep -E 'json POST|http response|408 Request Timeout'

Record URI, KVM Agent IP, taskuuid, request/response time difference, and whether the response is [no body]. If absent, verify node, UUID, date, and relevant .gz archives.

2. On the ZSphere host in the request URL, query current and archived Agent logs:

     KB_TASK_UUID='<taskuuid-from-management-server-log>'
     cd /var/log/zstack || exit 1
     ls -lh zstack-kvmagent.log*
     zgrep -a -n -C 5 "$KB_TASK_UUID" zstack-kvmagent.log*

Absence from Agent logs supports a pre-handler timeout only after host IP, time window, and archives are verified.

3. If no apiId exists, locate the request by VM UUID and minute-precise failure time, then use that request's taskuuid; do not combine another VM or retry response.
4. For context, use vim -R management-server.log or vim -R /var/log/zstack/zstack-kvmagent.log, search with /, use n, and quit with :q!.

• Expected result: A business response reaches the management node or the error maps to a specific KVM Agent handler.
• Exception handling: An empty-body HTTP 408 after about 15 seconds follows the Agent/CherryPy branch. Connection refusal, read timeout, or I/O exception follows the management-network/client-timeout branch.

2. Compare the current KVM Agent HTTP configuration

• Check: zstacklib/zstacklib/utils/http.py from the exact installed package.
• Location: Read-only inspection of the HTTP server construction. The analyzed 4.10.19 package contains:

  site_config['server.thread_pool'] = int(os.getenv('POOLSIZE', '10'))
  site_config['server.socket_timeout'] = 15

• Expected result: The code explains the 15-second signature but cannot distinguish pool saturation from packet loss.
• Exception handling: If site values differ, record the exact package/version and use its values.

3. Use a short packet capture to distinguish queueing from network path

• Check: TCP 7070 traffic on an affected host. With authorization, run:

  KB_MN_IP='<management-node-ip-from-management-server-log>'
  timeout --signal=INT 3600 tcpdump -i any -nn -s0 -U -w /tmp/kvmagent-7070.pcap "tcp port 7070 and host $KB_MN_IP"
  ls -lh /tmp/kvmagent-7070.pcap

• It stops after 3600 seconds; use Ctrl+C after early reproduction. On an authorized analysis system with tshark:

  tshark -r /tmp/kvmagent-7070.pcap -Y 'http.response.code == 408' -T fields -e frame.number -e frame.time_relative -e tcp.stream
  tshark -r /tmp/kvmagent-7070.pcap -Y 'tcp.analysis.retransmission' -T fields -e frame.number -e frame.time_relative -e ip.src -e ip.dst -e tcp.stream
  KB_TCP_STREAM='<tcp-stream-number-from-previous-408-record>'
  tshark -r /tmp/kvmagent-7070.pcap -Y "tcp.stream == $KB_TCP_STREAM" -T fields -e frame.time_relative -e ip.src -e ip.dst -e _ws.col.Info

• Expected result: POST reaches business handling and returns without a fixed approximately 15-second empty-body 408.
• Exception handling: If the complete POST arrives and a 408 is returned only after about 15 seconds, prioritize checking Agent request queueing. If the POST does not arrive completely or the same stream contains clear TCP retransmissions, prioritize comparing the switch, rack, and uplink paths from the management node to affected and unaffected hosts. A pcap contains management-network data; provide it only to authorized personnel and retain and clean it up according to company requirements.

4. Check thread-pool saturation with a low-concurrency comparison

Workaround

• Applicability: 408 aligns with 7070 connection/pool saturation or packet loss, and KVM Agent exit or host disconnection is excluded. Do not change concurrency without repeated samples.
• Check KVM Agent threads, the number of established connections on port 7070, and threads in the D state. With authorization, run on the affected host:

  KVMAGENT_PID="$(pgrep -o -f 'kvmagent')"
  ps -L -p "$KVMAGENT_PID" -o tid,stat,wchan:25,comm | grep -c 'CP Server'
  ss -tn state established '( sport = :7070 )' | wc -l
  ps -L -p "$KVMAGENT_PID" -o tid,stat,wchan:30 | awk '$2 ~ /D/'

• Expected result: Threads/connections do not remain at pool capacity, no thread remains in D state, and lowering concurrent backup/snapshot requests eliminates 408.
• Temporary action: Record schedules/concurrency, pause new tasks in the batch, wait for running tasks to end, then back up one VM while monitoring 7070, threads, and management-node 408. Do not kill tasks or Agent.
• Branch handling: Retain low concurrency and escalate if the pool remains saturated; repair confirmed packet loss/retransmission first on the network branch. Do not change POOLSIZE directly; it lacks a stable entry and can be lost on redeployment. Parameter testing requires version-specific location, original value, rollback, and metrics from ZStack Support.
• Verification: The same backup/snapshot succeeds with no empty-body 408 for the same taskUuid, and capture has no fixed 15-second 408.
• Rollback: Low concurrency changes only scheduling and does not modify host parameters. If one VM still fails, stop retries and keep the batch schedule paused. After the root cause is fixed and verified, restore the original schedule and concurrency according to the recorded values. If Technical Support separately changes a parameter, restore it using the original value and rollback procedure they provide; do not use this article as a basis for parameter tuning.

Additional information

• Support information must include the exact time window, resource/task UUIDs, complete error, and relevant component logs.
• Do not extrapolate a single site to all versions. Stop self-service work when cause remains open or operations might affect service/data, and submit the listed evidence.