server: skip HA restart for VMs in Error state

[Dogface2k]

Description

Fixes #13785.

When a failed deployment leaves an HA-enabled VM in Error, HA can still enqueue restart work. The existing state/update comparison does not reject work recorded while the VM was already in Error, even though the VM lifecycle has no start transition from that state.

This change rejects Error-state VMs at the shared scheduleRestart(...) entry point, before creating HA work or calling VM orchestration. It also completes previously queued HA restart work through the existing Step.Done and DAO-update path:

• processWork(...) checks the VM before the HA-disabled retry path.
• restart(...) checks its own VM lookup before the future/running-work queries and host investigation. This covers a VM entering Error between the worker's two VM reads.

The worker-entry guard applies only to HA restart work, preserving stop and destroy cleanup. VM lifecycle transitions, database schema, configuration and other HA work types are unchanged.

The PR retains one commit against 4.20 and changes the existing HA implementation and test class.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate)

Not applicable.

How Has This Been Tested?

Local validation of 68dd95ba1160e8efc98ee5e48d475bc6545ed368:

• The 24-module server Maven reactor and Checkstyle passed.
• All 40 selected tests passed: 30 HA-manager tests, four HA-DAO tests and six KVM-fencer tests, with zero failures, errors or skips.
• The new worker regressions use real VMInstanceVO and HaWorkVO objects and execute the actual dispatch, retry/completion and cancellation/investigation code. DAO, configuration and external-service boundaries use the existing Mockito test framework; worker methods are not stubbed to succeed.
• The full local server suite ran 2,472 tests: one failure, zero errors and nine skips. The failure is StorageBrowserImplTest.testGetFormattedPaths expecting Linux path separators on Windows. The same failure was reproduced in a clean checkout of the unchanged base commit 549daae0; all HA tests passed in the full run.

Tests ran on Windows with Java 17 and Maven 3.9.16. The schema's native execution hook was skipped with -Dexec.skip=true. No live database or hypervisor validation is claimed.

How did you try to break this feature and the system with this change?

• Checked scheduling rejection for both values of investigate.
• Reproduced Error-state queued work being retried before the worker guard, including with HA disabled.
• Reproduced a VM entering Error between the worker's two reads, including work already at Investigating and the running-work retry branch. The corrected ordering completes the item before host lookup or retry.
• Checked that non-Error HA work still follows its retry/investigation paths and that Error-state Stop, CheckStop, ForceStop and Destroy work still reaches the existing cleanup methods.

[DaanHoogland]

clgtm

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.29%. Comparing base (549daae) to head (68dd95b).
⚠️ Report is 50 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...java/com/cloud/ha/HighAvailabilityManagerImpl.java	83.33%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #13811      +/-   ##
============================================
+ Coverage     16.26%   17.29%   +1.02%     
- Complexity    13434    13684     +250     
============================================
  Files          5667     5264     -403     
  Lines        500731   468560   -32171     
  Branches      60803    55042    -5761     
============================================
- Hits          81455    81044     -411     
+ Misses       410172   378444   -31728     
+ Partials       9104     9072      -32     

Flag	Coverage Δ
uitests	?
unittests	17.29% <83.33%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This pull request prevents CloudStack’s HA subsystem from attempting to restart VMs that are already in VirtualMachine.State.Error, aligning HA behavior with the VM lifecycle (no normal start transition from Error) and addressing issue #13785.

Changes:

• Add an early return in scheduleRestart(...) to avoid creating HA work and triggering side effects for Error-state VMs.
• Add an early return in restart(...) so already-queued HA work for Error-state VMs completes without proceeding into investigation/fencing/restart logic.
• Add focused regression tests in HighAvailabilityManagerImplTest covering both scheduling-time and execution-time guards.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
server/src/main/java/com/cloud/ha/HighAvailabilityManagerImpl.java	Adds Error-state guards in HA scheduling and worker execution paths to skip HA restart processing.
server/src/test/java/com/cloud/ha/HighAvailabilityManagerImplTest.java	Adds regression tests validating no HA work/side effects are triggered for Error-state VMs and normal processing continues otherwise.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.