Environment

• Product: ZStack Cloud
• Version: 4.2.x~4.8.x.
• Component: VPC network, DHCP service, zstack-cli, and the MariaDB UsedIpVO table
• Trigger/Condition: A VPC network was created without DHCP service, so existing VMs cannot obtain an IP address, or the DHCP service IP must be changed to a different address in the IP range.

Symptoms

• A VM fails to attach to the VPC network, or it does not obtain an IP address from the VPC network after a reboot.
• The L3 network was created without the DHCP network service attached.

Diagnosis

1. In the UI, open the affected VPC L3 network and check whether DHCP is in the attached network services.
2. Use zstack-cli to query the network service providers and the L3 network's current services.
3. From a VM on the L3 network, confirm that no DHCP offer is received when the NIC is brought up.

Cause

• The VPC L3 network has no DHCP service attached, so the VPC vRouter does not provide DHCP for that network.
• The previously assigned DHCP service IP must be migrated to a new IP address in the same L3 network range.

Resolution

1. Prerequisites

• Confirm the L3 network UUID and the target Flat Network Service Provider UUID before issuing the AttachNetworkServiceToL3Network call.
• For an IP change, take a database backup before modifying UsedIpVO.

2. Procedure

A. Enable DHCP service on a VPC L3 network

1. Log in to zstack-cli:

   # zstack-cli
   admin >>> LogInByAccount accountName=admin password=password

2. Query the network service providers and capture their UUIDs:

   admin >>> QueryNetworkServiceProvider fields=name,uuid
   {
   "inventories": [
     {
       "name": "Flat Network Service Provider",
       "uuid": "e3b79d7832a541398a094ab2d7f60dfa"
     },
     {
       "name": "SecurityGroup",
       "uuid": "4b57f5625f354379a179ae2a5e1583f7"
     },
     {
       "name": "VirtualRouter",
       "uuid": "eaf2159b6a114d2ca24eb8510d675bf1"
     },
     {
       "name": "vrouter",
       "uuid": "49f657c414c64941a94e2829970ca589"
     }
   ],
   "success": true
   }

Notes:

• FLAT (Flat Network Service Provider) is used by flat networks.
• vrouter is used by VPC networks.
• VirtualRouter is no longer used.
• For VPC networks, use the first provider, which is the distributed DHCP service. The fourth provider is the centralized DHCP service.
• After enabling DHCP, an IP is automatically picked at random from the address pool and assigned as the DHCP service IP.

3. Attach the DHCP service to the L3 network by using the first provider for distributed DHCP:

   admin >>> AttachNetworkServiceToL3Network l3NetworkUuid="39cc5ccb7fec48118c4bd47aedd193a2" networkServices="{'e3b79d7832a541398a094ab2d7f60dfa':['DHCP']}"

B. Change the DHCP service IP on an L3 network

1. Query the current DHCP service IP via SystemTag:

   admin >>> QuerySystemTag tag~=xxx.xxx.119.2
   {
   "inventories": [
     {
       "createDate": "Dec 25, 2023 5:51:00 PM",
       "inherent": true,
       "lastOpDate": "Dec 25, 2023 5:51:00 PM",
       "resourceType": "L3NetworkVO",
       "resourceUuid": "39cc5ccb7fec48118c4bd47aedd193a2",
       "tag": "flatNetwork::DhcpServer::xxx.xxx.119.2::ipUuid::ca64804fd169344796821ccc96459bb2",
       "type": "System",
       "uuid": "8aba217b49ec31e5822616d98729c81e"
     }
   ],
   "success": true
   }

2. Update the SystemTag so the DHCP service IP points to the new IP (the new IP must be inside the same IP range and currently free):

   UpdateSystemTag uuid=8aba217b49ec31e5822616d98729c81e tag="flatNetwork::DhcpServer::xxx.xxx.119.250::ipUuid::ca64804fd169344796821ccc96459bb2"

3. Modify UsedIpVO in MariaDB:

a. Check IP availability via CLI first:

   admin >>> CheckIpAvailability ip=xxx.xxx.119.2 l3NetworkUuid=39cc5ccb7fec48118c4bd47aedd193a2
   {
     "available": false,
     "reason": "it is used",
     "success": true
   }

   admin >>> CheckIpAvailability ip=xxx.xxx.119.250 l3NetworkUuid=39cc5ccb7fec48118c4bd47aedd193a2
   {
     "available": true,
     "success": true
   }

b. Enter MariaDB and update the IP:

   # mysql -uroot -pzstack.mysql.password zstack
   MariaDB [zstack]> select * from UsedIpVO where ip="xxx.xxx.119.2";
   ...
   | ca64804fd169344796821ccc96459bb2 | 72b4271e4fd2471aa4f31b8292a7772e | 39cc5ccb7fec48118c4bd47aedd193a2 | xxx.xxx.119.2 | 173569794 | xxx.xxx.119.254 | xxx.xxx.255.0 | NULL | NULL | 2023-12-25 17:51:00 | 2023-12-25 17:51:00 | 4 | NULL |

   MariaDB [zstack]> update UsedIpVO set ip='xxx.xxx.119.250' where uuid='ca64804fd169344796821ccc96459bb2';

c. Verify:

   MariaDB [zstack]> select * from UsedIpVO where ip="xxx.xxx.119.2";
   Empty set (0.00 sec)

   MariaDB [zstack]> select * from UsedIpVO where ip="xxx.xxx.119.250";
   ...
   | ca64804fd169344796821ccc96459bb2 | 72b4271e4fd2471aa4f31b8292a7772e | 39cc5ccb7fec48118c4bd47aedd193a2 | xxx.xxx.119.250 | 173569794 | xxx.xxx.119.254 | xxx.xxx.255.0 | NULL | NULL | 2023-12-25 18:58:41 | 2023-12-25 17:51:00 | 4 | NULL |

d. Re-check IP availability via CLI:

   admin >>> CheckIpAvailability ip=xxx.xxx.119.2 l3NetworkUuid=39cc5ccb7fec48118c4bd47aedd193a2
   {
     "available": true,
     "success": true
   }

   admin >>> CheckIpAvailability ip=xxx.xxx.119.250 l3NetworkUuid=39cc5ccb7fec48118c4bd47aedd193a2
   {
     "available": false,
     "reason": "it is used",
     "success": true
   }

4. Reconnect the host. Otherwise the DHCP namespace inside the host will still hold the old IP.

Verification

• Newly created VMs on the L3 network obtain an IP address from DHCP.
• After changing the DHCP service IP, CheckIpAvailability reports the new IP as it is used and the old IP as available.
• VMs that depend on the DHCP namespace pick up the new IP after host reconnect.

Risks and rollback

• Risk: directly modifying UsedIpVO is a database-level operation; take a MariaDB backup first.
• Risk: Existing VMs created before the DHCP service was enabled were not configured for DHCP. If their operating systems use DHCP, a reboot assigns them different IP addresses. Configure a static IP address in each guest OS before rebooting.
• Risk: Enabling DHCP later may interfere with live migration because the platform assigns a DHCP IP address to new VMs while older VMs retain their static configurations.
• Rollback: detach the DHCP network service via DetachNetworkServiceFromL3Network, and revert the UsedIpVO row from the backup.

Best practice

• Decide during network planning whether the L3 network will use DHCP, and keep that configuration consistent. Repeatedly changing the DHCP state of an existing L3 network may cause IP address changes and unexpected behavior during live migration.