VM not picking assigned IP through VR

[rbaweja]

problem

A VM deployed on Apache CloudStack (ACS) is unable to acquire a DHCP-assigned IP address from the Virtual Router (VR). The DHCP request is visible in the VR’s tcpdump, but the VM does not receive an IP. However, manually assigning an IP to the VM allows it to reach both the VR and the internet.

versions

Environment Details
• CloudStack Version: 4.20
• Hypervisor: KVM
• Virtual Router IP: 10.0.0.1/24
• Network Setup: Advanced network with VR acting as DHCP server

The steps to reproduce the bug

1. VM sends a DHCP renew request.
2. VR receives the request, as confirmed via tcpdump -i eth2 port 67 or port 68 -n.
3. VR responds with a DHCP reply to the VM.
4. The VM does not acquire the IP.
5. Manually assigning an IP allows the VM to reach both VR and the internet.

What to do about it?

No response

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

This may be same as #10436

[rbaweja]

This may be same as #10436

i saw this post earlier, but didn't get solution by that method..

Do you suggest that i post it there or continue here?

[DaanHoogland]

@rbaweja , so the dhcp reply is sent. Can you check on the client logs if anything goes wrong there? I think in the case of #10436 the lease is not acquired at all, so this is a different issue it seems.

[rbaweja]

On VM:

New VR Logs:

Tue Mar 18 09:43:11 PM UTC 2025 Starting guest services for kvm
Tue Mar 18 09:43:13 PM UTC 2025 acpiphp and pci_hotplug module already compiled in
Tue Mar 18 09:43:14 PM UTC 2025 Received a new non-empty cmdline file from qemu-guest-agent
Tue Mar 18 09:43:14 PM UTC 2025 Booting from cloudstack, remove old configuration files in /etc/cloudstack/
Tue Mar 18 09:43:14 PM UTC 2025 Applying iptables rules
Tue Mar 18 09:43:14 PM UTC 2025 Setting up interface: eth0
Tue Mar 18 09:43:14 PM UTC 2025 Set up route for management network: via local gateway: for device eth0 for hypervisor: kvm
Tue Mar 18 09:43:16 PM UTC 2025 Executing cloud-early-config
Tue Mar 18 09:43:16 PM UTC 2025 Scripts checksum detected: oldmd5=0e40239dbb8573201b0a80a411b03b12 newmd5=0e40239dbb8573201b0a80a411b03b12
Tue Mar 18 09:43:18 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:20 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:22 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:24 PM UTC 2025 Could not find patch file, retrying
Tue Mar 18 09:43:24 PM UTC 2025 Scripts checksum detected: oldmd5=0e40239dbb8573201b0a80a411b03b12 newmd5=82d3395ea6108bf4a1ceaefdffb7a3f9
Tue Mar 18 09:43:24 PM UTC 2025 Patched scripts using /var/cache/cloud/cloud-scripts.tgz
Tue Mar 18 09:43:24 PM UTC 2025 Bootstrapping systemvm appliance
...skipping...
2025-03-24 14:07:48,627 INFO Executing: ip link show eth2 | grep ' state '
2025-03-24 14:07:48,631 INFO Executing2: arping -c 1 -I eth2 -A -U -s 10.10.10.1 10.10.10.1
2025-03-24 14:07:48,632 INFO Adding route: dev eth2 table: Table_eth2 network: 10.10.10.0/24 if not present
2025-03-24 14:07:48,632 INFO Executing: ip route show 10.10.10.0/24 table Table_eth2 proto static
2025-03-24 14:07:48,636 INFO Executing: sudo ip route flush cache
2025-03-24 14:07:48,647 ERROR Not able to setup source-nat for a regular router yet
2025-03-24 14:07:48,647 INFO Not making dns publicly available
2025-03-24 14:07:48,649 INFO Executing: systemctl start cloud-password-server@10.10.10.1
2025-03-24 14:07:48,659 INFO Service cloud-password-server@10.10.10.1 start
2025-03-24 14:07:48,660 INFO Nothing to commit. The /etc/monitor.conf file did not change
2025-03-24 14:07:48,660 INFO Nothing to commit. The /etc/cron.d/process file did not change
2025-03-24 14:07:48,661 INFO Executing: systemctl stop conntrackd
2025-03-24 14:07:48,674 INFO Service conntrackd stop
2025-03-24 14:07:48,674 INFO Executing: systemctl stop keepalived
2025-03-24 14:07:48,688 INFO Service keepalived stop
2025-03-24 14:07:48,688 INFO Executing: mount

[DaanHoogland]

@rbaweja, can you check /var/log/dnsmasq.log on the router for any errors? I thought an answer went out of the VR as per you initial description but on second read it seems only the request may have come in.

[rbaweja]

@rbaweja, can you check /var/log/dnsmasq.log on the router for any errors? I thought an answer went out of the VR as per you initial description but on second read it seems only the request may have come in.

the worst part is there are 2 routers and both have same issue, i tried with default 5.5 centos template but same

Mar 24 15:58:34 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:34 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:34 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:43 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPDISCOVER(eth2) 02:01:00:de:00:01
Mar 24 15:58:59 dnsmasq-dhcp[3006]: DHCPOFFER(eth2) 10.0.0.138 02:01:00:de:00:01

[DaanHoogland]

that log looks ok,

you said

The DHCP request is visible in the VR’s tcpdump, but the VM does not receive an IP

do you mean you see the request coming in, but no response going out from the VR using tcpdump?

[rbaweja]

as per response i see that the response is also tehre but VM is not receiving the response.

tcpdump for VR:

17:15:21.881789 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.881790 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.881790 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.881790 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:21.882263 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:21.882528 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:21.882773 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:21.883042 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.882073 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882074 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882074 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882074 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:29.882559 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.882815 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.882950 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:29.883171 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.883434 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.883694 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.883694 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.883694 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:41.884023 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.884335 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.884541 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:41.884779 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.883875 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.883876 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.884042 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.884042 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 02:01:00:cc:00:16, length 300
17:15:51.884429 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.884779 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.885070 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344
17:15:51.885342 IP 10.10.10.1.67 > 10.10.10.169.68: BOOTP/DHCP, Reply, length 344

[weizhouapache]

are there firewall rules in the vm ?

are there firewall rules with the kvm host ?

[rbaweja]

are there firewall rules in the vm ?

are there firewall rules with the kvm host ?

No firewall rules. Even the fresh VM of centos 5.5 doesn’t get IP

[weizhouapache]

are there firewall rules in the vm ?

are there firewall rules with the kvm host ?

No firewall rules. Even the fresh VM of centos 5.5 doesn’t get IP

Are the vm and vr running on the same host ?

[rbaweja]

are there firewall rules in the vm ?
are there firewall rules with the kvm host ?

No firewall rules. Even the fresh VM of centos 5.5 doesn’t get IP

Are the vm and vr running on the same host ?

all on one host for now.

[weizhouapache]

all on one host for now.

@rbaweja
can you restart network with cleanup and retry ?

please use tcpdump -vvv instead of tcpdump

[rbaweja]

tcpdump -vvv instead of tcpdump

this is what i get -

09:35:47.319133 IP (tos 0xc0, ttl 64, id 6910, offset 0, flags [none], proto UDP (17), length 372)
10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xc11f8f51, secs 20, Flags [none] (0x0000)
Your-IP 10.10.10.169
Server-IP 10.10.10.1
Client-Ethernet-Address 02:01:00:cc:00:16
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 10.10.10.1
Lease-Time (51), length 4: 4294967295
BR (28), length 4: 10.10.10.255
Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49"
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 10.10.10.1
Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11
Domain-Name (15), length 17: "cs2cloud.internal"
END (255), length 0
09:35:47.319469 IP (tos 0xc0, ttl 64, id 6911, offset 0, flags [none], proto UDP (17), length 372)
10.10.10.1.67 > 10.10.10.169.68: [udp sum ok] BOOTP/DHCP, Reply, length 344, xid 0xc11f8f51, secs 20, Flags [none] (0x0000)
Your-IP 10.10.10.169
Server-IP 10.10.10.1
Client-Ethernet-Address 02:01:00:cc:00:16
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Offer
Server-ID (54), length 4: 10.10.10.1
Lease-Time (51), length 4: 4294967295
BR (28), length 4: 10.10.10.255
Hostname (12), length 39: "vm-d0d0ba5c-24e0-4acb-8cf0-785a1bef0e49"
Subnet-Mask (1), length 4: 255.255.255.0
Default-Gateway (3), length 4: 10.10.10.1
Domain-Name-Server (6), length 8: 10.10.10.1,172.27.5.11
Domain-Name (15), length 17: "cs2cloud.internal"
END (255), length 0