Fix External-DHCP APIPA/Link-local address on L2 networks by nvazquez · Pull Request #14333 · apache/cloudstack

nvazquez · 2026-10-07T10:47:53Z

Description

This PR an issue observed on L2 Networks with external DHCP server:

On an L2 guest network with an external DHCP server, CloudStack learns a Windows VM's IP address as the self-assigned (APIPA / link-local) address 169.254.149.120. It recorded that as the NIC's IP and never refreshed it. More than 30 minutes later, listVirtualMachines still reports 169.254.149.120, while the guest really has 172.25.16.236 (DHCP). The QEMU guest agent reports 172.25.16.236, and SSH on 172.25.16.236:22 is reachable.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

nvazquez · 2026-10-07T10:49:31Z

@blueorangutan package

blueorangutan · 2026-10-07T10:50:05Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-10-07T10:56:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 4.04%. Comparing base (2974af8) to head (5b8be7e).
⚠️ Report is 6 commits behind head on 4.22.

❗ There is a different number of reports uploaded between BASE (2974af8) and HEAD (5b8be7e). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (2974af8) HEAD (5b8be7e)

unittests 1 0

Additional details and impacted files

@@              Coverage Diff              @@
##               4.22   #14333       +/-   ##
=============================================
- Coverage     18.02%    4.04%   -13.99%     
=============================================
  Files          5936      449     -5487     
  Lines        535823    38283   -497540     
  Branches      65612     7082    -58530     
=============================================
- Hits          96582     1547    -95035     
+ Misses       428242    36521   -391721     
+ Partials      10999      215    -10784

Flag	Coverage Δ
uitests	`4.04% <ø> (-0.01%)`	⬇️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

weizhouapache

code lgtm

blueorangutan · 2026-10-07T11:52:43Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19442

nvazquez · 2026-10-07T12:25:47Z

@blueorangutan test

blueorangutan · 2026-10-07T12:28:06Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Copilot

Copilot review overview

🟡 Changes recommended

Legacy APIPA values can remain exposed, and malformed IPv6 zone input can throw unexpectedly.

Review effort: Balanced
Findings: 2

Open (2)

Clear stale APIPA values when KVM lookup fails · New
Handle delimiter-only input without array bounds exception · New

What changed in this PR

Fixes stale APIPA/link-local addresses reported for externally DHCP-managed L2 VM NICs.

Changes:

Adds IPv4/IPv6 link-local detection utilities.
Filters link-local addresses from KVM guest-agent results.
Retries NICs containing previously recorded APIPA addresses.

File	Description
`NetUtils.java`	Adds link-local address helpers.
`NetUtilsTest.java`	Tests link-local detection.
`UserVmManagerImpl.java`	Handles and retries APIPA NIC addresses.
`LibvirtGetVmIpAddressCommandWrapper.java`	Ignores link-local guest addresses.
`LibvirtGetVmIpAddressCommandWrapperTest.java`	Tests APIPA and DHCP parsing.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

blueorangutan · 2026-10-08T03:26:48Z

[SF] Trillian test result (tid-17089)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49968 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14333-t17089-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

Damans227 · 2026-10-13T13:45:56Z

                            nic.setIPv4Address(null);
                            _nicDao.update(nicId, nic);
                        }
+                    } else if (NetUtils.isLinkLocalIp4(vmIp)) {


on xenserver this still counts as a failed try, so after about 30 min we stop checking and a lease that comes later is never picked up. should this skip the count like kvm does now?

Damans227

thanks @nvazquez, fix looks right. LGTM

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

Setting decrementCount = false on APIPA can result in effectively unbounded retries if a guest… · New
Avoid String.format(...) inside logger.debug(...) since it formats eagerly even when debug… · New
These test fixtures are constants and should be declared private static final to prevent… · New
The ‘clear APIPA and persist’ logic is duplicated with the retry loop earlier in this class.… · New

2 resolved since last review

🧠 Review effort: Lite

+                    } else if (NetUtils.isLinkLocalIp4(vmIp)) {
+                        // a link-local (APIPA) address means the guest has not obtained a DHCP lease yet; do not record it,
+                        // clear any previously recorded link-local address and keep retrying without consuming the retry budget
+                        // (consistent with KVM, which reports this case as a failed answer), so a late DHCP lease is still picked up
+                        logger.debug("Instance [ID: {}, UUID: {}, name: {}] - ignoring link-local IP {}, DHCP lease not obtained yet",
+                                vmId, vmUuid, vmName, vmIp);
+                        decrementCount = false;
+                        if (nic != null && NetUtils.isLinkLocalIp4(nic.getIPv4Address())) {
+                            nic.setIPv4Address(null);
+                            _nicDao.update(nicId, nic);
+                        }


+            // Skip link-local addresses (e.g. Windows APIPA 169.254.0.0/16 before a DHCP lease is obtained),
+            // they are not routable and must not be reported as the instance IP
            if (ipFamily.equals("ipv4")) {
-                ipv4 = ipPart;
+                if (NetUtils.isLinkLocalIp4(ipPart)) {
+                    logger.debug(String.format("Ignoring link-local ipv4 %s with mac address %s", ipPart, macAddress));
+                } else {
+                    ipv4 = ipPart;
+                }
            } else if (ipFamily.equals("ipv6")) {
-                ipv6 = ipPart;
+                if (NetUtils.isLinkLocalIp6(ipPart)) {
+                    logger.debug(String.format("Ignoring link-local ipv6 %s with mac address %s", ipPart, macAddress));
+                } else {
+                    ipv6 = ipPart;
+                }


+    // Windows guest that has not obtained a DHCP lease yet and self-assigned an APIPA address
+    private static String VIRSH_DOMIF_OUTPUT_WINDOWS_APIPA = " Name       MAC address          Protocol     Address\n" + //
+            "-------------------------------------------------------------------------------\n" + //
+            " Ethernet   02:01:00:d0:00:87    ipv6         fe80::df84:2984:e396:8ac3%6/64\n" + //
+            " -          -                    ipv4         169.254.149.120/16\n" + //
+            " Loopback Pseudo-Interface 1                      ipv6         ::1/128\n" + //
+            " -          -                    ipv4         127.0.0.1/8\n";
+
+    // Same Windows guest once the DHCP lease has been obtained
+    private static String VIRSH_DOMIF_OUTPUT_WINDOWS_DHCP = " Name       MAC address          Protocol     Address\n" + //
+            "-------------------------------------------------------------------------------\n" + //
+            " Ethernet   02:01:00:d0:00:87    ipv6         fd00:ac19:0:1910:0:d3fc:4177:2c32/128\n" + //
+            " -          -                    ipv6         fe80::df84:2984:e396:8ac3%6/64\n" + //
+            " -          -                    ipv4         172.25.16.236/22\n" + //
+            " -          -                    ipv4         169.254.149.120/16\n" + //
+            " Loopback Pseudo-Interface 1                      ipv6         ::1/128\n" + //
+            " -          -                    ipv4         127.0.0.1/8\n";


+                    // a link-local (APIPA) address recorded before the guest obtained its DHCP lease is not a valid instance IP,
+                    // clear it so it is no longer exposed and the NIC is retried as one without an IP
+                    if (NetUtils.isLinkLocalIp4(nic.getIPv4Address())) {
+                        logger.debug("Clearing link-local IP {} recorded for NIC {}", nic.getIPv4Address(), nic);
+                        nic.setIPv4Address(null);
+                        _nicDao.update(nic.getId(), nic);
+                    }


nvazquez · 2026-10-14T00:47:11Z

@blueorangutan package

blueorangutan · 2026-10-14T00:48:05Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-10-14T01:46:47Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19542

nvazquez · 2026-10-14T02:51:15Z

@blueorangutan test

blueorangutan · 2026-10-14T02:52:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-10-14T18:56:22Z

[SF] Trillian test result (tid-17132)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 54441 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14333-t17132-kvm-ol8.zip
Smoke tests completed. 144 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestCreateVolume>:setup	`Error`	0.00	test_volumes.py
test_01_root_volume_encryption	`Error`	1.39	test_volumes.py
test_02_data_volume_encryption	`Error`	1.31	test_volumes.py
test_03_root_and_data_volume_encryption	`Error`	1.50	test_volumes.py
ContextSuite context=TestVolumes>:setup	`Error`	27.83	test_volumes.py
test_05_rvpc_multi_tiers	`Error`	8.35	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	1.71	test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure	`Error`	1.75	test_host_maintenance.py
test_01_cancel_host_maintenance_ssh_enabled_agent_connected	`Failure`	15.76	test_host_maintenance.py
test_03_cancel_host_maintenance_ssh_disabled_agent_connected	`Failure`	21.72	test_host_maintenance.py
test_04_cancel_host_maintenance_ssh_disabled_agent_disconnected	`Failure`	33.43	test_host_maintenance.py
ContextSuite context=TestHostMaintenanceAgents>:teardown	`Error`	34.58	test_host_maintenance.py

The KVM agent parsed the IPv6 address from 'virsh domifaddr --source agent' but discarded it, so nics.ip6_address was never set for external-DHCP/L2 networks. - Return the IPv6 address in a new GetVmIpAddressAnswer (IPv4 stays in details for compatibility). - Report the first global IPv6 address, falling back to the first link-local one (zone suffix removed) when there is no global address. - Treat the retrieval as successful when the guest has an IPv4 address or a global IPv6 address, so IPv6-only guests are supported. A link-local IPv6 address alone does not count as success. - Store the IPv6 address in nics.ip6_address on the management server. - Add unit tests for the IPv4/IPv6 combinations.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

weizhouapache · 2026-10-15T18:54:45Z

@blueorangutan package

blueorangutan · 2026-10-15T18:56:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-10-15T19:54:52Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19568

Fix External-DHCP IP fetch records Windows APIPA address on L2 networks

3c97325

nvazquez added this to the 4.22.2 milestone Oct 7, 2026

nvazquez requested review from NuxRo, Pearl1594, RosiKyu, kiranchavala, shwstppr, vladimirpetrov and weizhouapache October 7, 2026 10:47

nvazquez added type:bug component:networking labels Oct 7, 2026

boring-cyborg Bot added the component:kvm label Oct 7, 2026

weizhouapache approved these changes Oct 7, 2026

View reviewed changes

nvazquez requested a balanced review from Copilot October 7, 2026 14:25

Copilot started reviewing on behalf of nvazquez October 7, 2026 14:30

View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

Comment thread server/src/main/java/com/cloud/vm/UserVmManagerImpl.java Outdated

Comment thread utils/src/main/java/com/cloud/utils/net/NetUtils.java Outdated

Damans227 reviewed Oct 13, 2026

View reviewed changes

Address review comments

52c4e0d

nvazquez requested a balanced review from Copilot October 13, 2026 15:13

Damans227 approved these changes Oct 13, 2026

View reviewed changes

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Copilot stopped reviewing on behalf of nvazquez due to an error October 13, 2026 20:15
The job was not acquired by a runner even after multiple attempts.

boring-cyborg Bot added the component:api label Oct 15, 2026

weizhouapache requested a balanced review from Copilot October 15, 2026 18:54

Copilot AI reviewed Oct 15, 2026

Conversation

nvazquez commented Oct 7, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

nvazquez commented Oct 7, 2026

Uh oh!

blueorangutan commented Oct 7, 2026

Uh oh!

codecov Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Oct 7, 2026

Uh oh!

nvazquez commented Oct 7, 2026

Uh oh!

blueorangutan commented Oct 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Oct 8, 2026

Uh oh!

Damans227 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Damans227 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

nvazquez commented Oct 14, 2026

Uh oh!

blueorangutan commented Oct 14, 2026

Uh oh!

blueorangutan commented Oct 14, 2026

Uh oh!

nvazquez commented Oct 14, 2026

Uh oh!

blueorangutan commented Oct 14, 2026

Uh oh!

blueorangutan commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Oct 15, 2026

Uh oh!

blueorangutan commented Oct 15, 2026

Uh oh!

blueorangutan commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Oct 7, 2026 •

edited

Loading