Stop resolution changes stranding the desktop and spamming dialogs (#114, #132)

[SwatX18]

Four defects behind #114 and #132, both "Changing the resolution failed: DispChangeBadFlags". Split into two commits so the fixes can be taken without the testing convention — see "About the second commit".

There is no flags bug to find

Worth saying first, because it shapes the whole change. DISP_CHANGE_BADFLAGS means "an invalid set of flags was passed in", so that is where I looked, and ruled it out:

Candidate	Verdict
Wrong flag constants	Correct against the Win32 headers
Illegal flag combination	CDS_UPDATEREGISTRY | CDS_NORESET then a global commit is the pattern MSDN documents; both documented BADFLAGS rules are satisfied, hwnd and lParam correctly NULL
Devmode struct layout	156 bytes = sizeof(DEVMODEA) on x86; the display arm of the union is exactly the size of the printer arm, so every following member sits at the right offset
dmSize	Marshal.SizeOf gives 156, set before the call, and the API overwrites it anyway
ANSI/Unicode mismatch	No CharSet on any DllImport, so Ansi, binding the ...A entry points — consistent with CharSet.Ansi on Devmode

So -4 originates inside win32k for a reason this source cannot express — most plausibly a device name that cannot take a mode set (mirroring or indirect-display drivers, hybrid-laptop render adapters). This change therefore targets everything around the failure, which is where the user harm actually is. Afterwards a persistent -4 costs three attempts, one balloon, one log line and no dialog — and the log names the device and the code, which is the reproduction data neither issue ever provided.

1. A modal dialog from inside the foreground callback

ResolutionHelper.cs:122 calls MessageBox.Show on failure, reached via PerformResolutionChange from both proxies' OnWinEventHook — the UI thread the SetWinEventHook callback runs on. It reappears on every foreground change, forever, and blocks the hook while open. That is #132's "it keeps on saying that", and the user cannot dismiss their way out.

No MessageBox is now reachable from the hook. A failure logs once per (device, failure code), raises one tray balloon, and stops being attempted after a bound.

2. Success is read from the wrong call

DispChange returnValue = (DispChange)ChangeDisplaySettingsEx(lpszDeviceName, ref mode, IntPtr.Zero,
    (ChangeDisplaySettingsFlags.CdsUpdateregistry | ChangeDisplaySettingsFlags.CdsNoreset), IntPtr.Zero);
ChangeDisplaySettingsEx(null, IntPtr.Zero, (IntPtr)null, ChangeDisplaySettingsFlags.CdsNone, (IntPtr)null);

CDS_NORESET only stages the change. The second call applies it, and its return value is discarded — so the helper can report success for a mode that never landed.

Now CDS_TEST then CDS_UPDATEREGISTRY: one authoritative result, and a mode that fails validation is never written to the registry. That also dissolves the rollback question — there is no longer a window where the registry holds a mode that was never applied. Which matters, because CDS_UPDATEREGISTRY gets no 15-second revert-if-unconfirmed safety net.

The mode is then read back. A driver reporting success without changing anything yields AppliedUnverified rather than a plain failure — because the mode probably did change, and reporting failure would clear the flag the revert path depends on, leaving the desktop at the game's resolution while telling the user the opposite.

3. The frozen snapshot — likely #114's actual complaint

_windowsResolutionSettings is built once in the VibranceGUI constructor and never refreshed, and nothing in the codebase reacts to a display change (zero hits for SystemEvents, DisplaySettingsChanged, WM_DISPLAYCHANGE).

So: change your desktop resolution in Windows, and vibranceGUI's idea of "the Windows resolution" is still the startup one. The next time a game exits, the revert path drags the desktop back. With every API call returning DISP_CHANGE_SUCCESSFUL. That is #114's "i tried changing it back" verbatim, and no amount of investigating BADFLAGS would have found it.

It now refreshes on SystemEvents.DisplaySettingsChanged, keeping the captured Windows mode while one of ours is applied — so a game going fullscreen cannot become the mode we later restore to.

One subtlety worth flagging for review: the handler checks IsDisposed || !IsHandleCreated before InvokeRequired. Control.InvokeRequired returns false when the control has no handle, so without that check the dictionary mutation would run on the SystemEvents thread during the constructor — which is exactly when autostart-at-logon fires the event as monitors settle.

4. The retry storm

IsResolutionChangeNeeded compares via ResolutionModeWrapper.Equals, which includes DmDisplayFixedOutput — but dmFields never declared DM_DISPLAYFIXEDOUTPUT, so drivers may ignore it. A user selecting a "(Center)" mode gets a change that succeeds on the four real fields yet can never satisfy the guard, re-firing a real mode set and registry write on every foreground event.

The guard now compares only the four fields a mode change controls, and dmFields declares what it sets — OR-ed into what EnumDisplaySettings returned, never overwritten, since dropping the enumerated DM_POSITION would let a multi-monitor desktop rearrange itself.

Asymmetric failure bounds — deliberate

Three attempts to enter a game's resolution, ten to leave it. Giving up on the way in costs a feature; giving up on the way out leaves someone's desktop at 1280x720.

Removed

ChangeResolution (verified zero callers on master before deleting), the ChangeDisplaySettings P/Invoke, and the ChangeDisplaySettingsEx overload used only by the discarded global commit.

Verification

Warnings unchanged. master builds with 2 warnings (ResolutionModeWrapper CS0659, WinEventHook.cs CS0168); this branch builds with the same 2, both configurations. Neither is touched here.

About the second commit

The repo has no test project, so the second commit adds --selftest-resolution and a ResolutionChangeFixture — 51 checks behind an IDisplayModeDevice seam with a fake backed by an in-memory mode table and a recorded call log. It covers the two-call sequence, the dmFields mask, all six failure codes, the fixed-output fallback, both bounds, the revert cycle and the no-repeat-notification behaviour.

This introduces a convention that does not exist here yet. It is a separate commit specifically so you can drop it and take only the fixes.

It is real evidence rather than decoration: every one of the 15 check methods was individually broken — one production line at a time — and confirmed to produce a clean [FAIL] before being restored and hash-verified. Reintroducing CDS_NORESET breaks check 1; dropping the dmFields mask breaks check 2; bypassing the CDS_TEST gate breaks check 3; reintroducing DmDisplayFixedOutput into the guard breaks check 12 — that last one is exactly the #132 regression.

There is no hardware variant of this fixture and there must never be one. A display mode has no guaranteed undo, and a mode the panel cannot show leaves the user unable to see any dialog asking them to confirm it. That is literally #114.

Known limits

• -4 has not been reproduced on real hardware. Two of the three ranked hypotheses are environment facts invisible from here. I would not claim this makes -4 disappear — it makes it survivable and diagnosable.
• The DisplaySettingsChanged refresh has no automated test. The fixture drives ResolutionHelper; the dictionary rebuild lives in VibranceGUI behind Screen.AllScreens with no seam. Verified by reading and independently traced, not covered by a fixture.
• The .NET 4.0 constraint is verified by inspection and API scan, not by compilation — this machine has no v4.0 targeting pack, so builds used -p:TargetFrameworkVersion=v4.8. The csproj still declares v4.0 and was not edited.
• Two residual holes are deliberately deferred and documented in comments: a game that changes the mode itself with no vibranceGUI profile applied, and a monitor detached then reattached losing its captured entry.
• isResolutionChangeApplied is added to VibranceInfo — defect 3 needs to know a change is currently applied so the refresh does not capture the game's own mode as the desktop resolution.

Happy to drop the fixture commit, split this into separate PRs, or adjust anything else you would prefer.