Problem symptom

The UI service on one management node cannot be started.

How to confirm

• The Redis-backed UI service cannot come up on the affected node, breaking login and management.

Applicable environment

• Product: ZCF
• Version: 4.8.x.
• Component: ZStack Cloud
• Deployment form: multi-management-node (zsha2).

Possible causes

• /var/log has been moved, deleted, or lost its execute (search) permission, which causes the Redis unit to fail to open its log files.

Solution

1. Rule out platform-level Redis hardening

• Check: confirm that no 6379-related security hardening was performed on the affected node.
• Command/UI:

  iptables -nvL | grep 6379
  iptables -nvL -t mangle

Review the output of both commands for any DROP or REJECT rule that matches port 6379, in either the filter table or the mangle table. Also review platform change records for any recent 6379-related hardening change.

• Expected result: no DROP/REJECT rule matching port 6379 in either table, and no recent 6379 hardening change in the change records.
• If abnormal: roll back the hardening change (or remove the matching rule) first.

2. Try to start Redis manually

• Check: run redis-server manually to see whether it starts outside of systemd, optionally with a copy of the working node's redis.conf.

• Command/UI:

  # Basic: start redis-server with its built-in default config
  redis-server

  # start redis-server with a copy of the healthy node's redis.conf
  redis-server /etc/redis.conf

For the Custom Config form, first copy /etc/redis.conf (or the platform's default path) from the healthy node to the affected node.

• Expected result: Redis starts successfully with either form.

• If abnormal: continue to Step 3.

3. Check system resources and existing Redis processes

• Check: confirm memory is sufficient and no zombie process is starving the system.
• Command/UI:

  free -h
  top
  ps -ef | grep -i redis

• Expected result: memory is sufficient, no zombie processes, no stray redis processes holding state.
• If abnormal: free memory or kill stray processes; otherwise continue to Step 4.

4. Inspect the zstack-ui log

• Check: confirm the failure mode in the zstack-ui log.
• Command/UI: tail the zstack-ui log at /usr/local/zstack/apache-tomcat/logs/zstack-ui.log; look for connection failures to port 6379.
• Expected result: the log shows connection refused or a similar "cannot connect to 6379" error.
• If abnormal: continue to Step 5.

5. Inspect the systemd journal for Redis

• Check: capture the exact startup error for the Redis unit.
• Command/UI:

  journalctl -xe -u redis

• Expected result: lines such as "unable to open file ... Permission denied" point to a filesystem/permission problem.
• If abnormal: continue to Step 6.

6. Verify /var/log directory state

• Check: confirm whether /var/log exists, whether expected subdirectories exist, and whether the directory has been moved.
• Command/UI:

  ls -ld /var/log
  ls /var/log
  ll /var/log

• Expected result: /var/log exists with the same set of subdirectories and permissions as the other management node.
• If abnormal: identify when /var/log was relocated or had its permissions changed (check history, change records).

7. Restore /var/log and fix permissions

• Check: move /var/log back to its original location (if it was relocated) and grant the +x permission on /var/log so Redis can traverse it.
• Command/UI:

  # If /var/log was relocated to a non-standard mount,
  # confirm the target path with the customer's change records first, then restore.
  # Replace <new-log-path> with the actual relocated path.
  rsync -a <new-log-path>/ /var/log/   # copy contents back
  chmod +x /var/log
  systemctl start redis
  systemctl start zstack-ui

• Expected result: Redis starts cleanly; zstack-ui connects to 6379 successfully and the UI becomes available on this management node.

Verification

• systemctl status redis reports active (running) on the affected node.
• systemctl status zstack-ui reports active (running).

Collection of supporting information

• Log bundle: journalctl -xe -u redis output and zstack-ui log from the affected node.
• Timestamps: the date /var/log was relocated (from history), the time Redis started failing, and the time the UI service failed.
• Scope and impacted objects: which management node failed, the host IPs of both management nodes, and the global HA status at the time of failure.

Additional information

• Best practice: do not relocate /var/log to other directories on management nodes. If space is tight, free space by removing old collected logs and monitoring logs.