test: keep WorkspaceService tests from bleeding state into each other through shared UserDefaults - #1602
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
✅ PR readiness gate passed. |
There was a problem hiding this comment.
Plat Ironwood, Platform Lead
⛔ Request changes — partial fix claims to close #1536, and the one measurement that would prove it is unrun
This narrows the shared state correctly, but leaves 8 of the suite's tests on the exact path the issue names, and the PR still says Closes #1536.
- Blocking:
Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift(TODO near thesetWorkspacesRoot/restoreWorkspacesRoothelpers) — 8 tests still write throughUserDefaults.standard, the shared domain the issue is about. The acceptance criterion is "the failure does not reproduce across repeated concurrent full-suite runs"; with a third of the suite still exposed, that's unproven and plausibly false. - Blocking: evidence item 1 (three concurrent
swift testruns, results pasted) ispending-ci. This is not something the ordinarybuild-and-testlane exercises — it's two fullswift testprocesses racing, per the issue's own repro. Deferring it to the standard downstream evidence job doesn't get it run; someone needs to actually execute the repro and paste the three results. - Blocking:
## Evidence Statusitem 3,CI build-and-test succeeds, isblocked, and the readiness-gate bot comment on this PR confirms the gate is failing with theblocked:evidencelabel still on. That needs to turn green before this merges regardless of the above.
Evidence: ⛔ item 1 (concurrent-run proof) unrun; item 3 (CI green) unconfirmed. Items 2 and 4 are auto-marked "blocked" but are actually answered in prose in the PR body — see below.
File:line check on items 2/4, and why the partial conversion matters
Items 2 and 4 don't need a person to look at anything — they're written statements, and I read them against the diff:
- Item 2 (name the shared state + file:line + how it crosses the boundary): the PR body correctly points at
WorkspaceService.workspacesRoot/setWorkspacesRoot/resetWorkspacesRoot, which the diff's first hunk (@@ -85,6 +85,12 @@through@@ -108,25 +114,33 @@) confirms previously read/wroteLaunchPreferences.defaultsunconditionally — that resolves toUserDefaults.standard, an on-disk cfprefsd domain keyed by executable identity, not by process. That's a correct and specific answer. - Item 4 (production-reachable or fixture-only): the PR states it's reachable from production — a live app or a
workspacesCLI invocation without the isolation env vars hits the same code path. That's the right answer and worth taking seriously: it means this was never purely a test-fixture bug.
Given item 4, the incomplete conversion is not just a test-suite tidiness gap — the production-reachable path (WorkspaceService.setWorkspacesRoot) is now fixed for the ~2/3 of tests using makeIsolatedPreferences(), but the remaining 8 tests exercise the same production method through the still-shared domain. So the concurrent-run repro in the issue is very likely to still reproduce for those 8, and the PR's own Risks section says so plainly.
Given that, Closes #1536 is premature. Either finish converting the remaining 8 tests in this PR (it's the same mechanical pattern already applied to the rest of the file — makeIsolatedPreferences() plus dropping the two old helpers), or retitle this as a partial mitigation, drop the Closes keyword, and open the follow-up as its own tracked issue rather than a TODO comment.
Code itself is clean: preferences: UserDefaults = LaunchPreferences.defaults as a trailing defaulted param on both initializers is a non-breaking, well-scoped change, and the per-test UUID-suite pattern in makeIsolatedPreferences() is a sound way to get true process isolation without touching the actor's production default.
|
@fairchild — Three evidence items need you. The factory can't verify them itself.
To clear them, edit this PR's description. In the "Evidence Status" list, change those three lines to: Then remove the For item 1, state in this PR body the command you ran and the line it printed. After that, run a fresh review: Actions → Factory Review → Run workflow → enter 1602. |
…n preferences domain Finishes the owner's part of #1602: the eight remaining tests converted, helpers removed, concurrent-suite evidence run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrSceQotsidvcP4tVgf2ud
Codex triage — head 266db38Adjudicated: taken 3, declined 0 — head 266db38 (answering plat's review on 97c1de3; the owner's part of this PR, done from a repo-steward worker tile)
Gate, bare in the worker's worktree at 266db38: Orchestrator note (workspaces) |
Codex pass — 2026-09-11, head 266db38One independent review of the whole diff ( The change injects a Plat's three blockers, judged from the code
Findings
No blocking findings. The item above does not gate the merge. Not verified
Verdict: FINDINGS Steward note (steward v9) |
|
The finding about leaked preference plists is worth fixing before merge, not after: this suite alone adds roughly 23 empty The three items still open are outside what a static review can close:
Recommend the next turn either patches the plist leak here or explicitly defers it to a filed follow-up issue referenced in the PR body's Risks section — right now the PR body describes only the crash case, which this review shows is not the only leak path. Once |
…st behind Steward pass minor on 266db38 taken. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrSceQotsidvcP4tVgf2ud
Codex triage — head d13afd8Adjudicated: taken 1, declined 0 — head d13afd8 (answering the steward pass's minor on 266db38)
Gate, bare in the worker's worktree at d13afd8: Orchestrator note (workspaces) |
Codex pass — 2026-09-11, head d13afd8Second pass ( The base fix is unchanged and still sound: all 23 root-touching tests take a UUID-named store, the other 21 never reach the preference, production still defaults to The newest commit replaces Findings
The double cleanup at line 242, deferred and explicit, is harmless and No blocking findings. None of the three gates the merge. What I measuredA standalone Swift probe outside the package, mimicking the helper: create the suite, Separately, the leak this commit fixes is visible in the on-disk record. Not verified
Verdict: FINDINGS Steward note (steward v9) |
|
Finding 2 is the one to act on before merge: drop the pre-cleanup Findings 1 and 3 don't need a code change now. Add a short comment at line 133 noting the cleanup depends on Still open before this can merge: |
Codex triage — head d13afd8Adjudicated: taken 0, declined 2, filed #1611 — head d13afd8
Head stays d13afd8. Gate on this head as in the previous comment: 46 passed bare, lint clean, no plist left after a filtered run. Remaining legs are plat's re-review on this head and readiness once Orchestrator note (workspaces) |
There was a problem hiding this comment.
Plat Ironwood, Platform Lead
🟡 Approve with follow-ups — blockers from my last pass are cleared
All three things I blocked on are answered on head d13afd8e: all 23 root-touching tests now go through makeIsolatedPreferences() (the 8 stragglers are converted, the old UserDefaults.standard helpers are gone), the concurrent-repro evidence is pasted with real numbers (rounds 5-7 clean, one unrelated flake filed as #1606), and build-and-test reports green on this head (run 34596303172, completed 12:20:57Z).
- Worth noting: the leak-fix's cleanup (
WorkspaceServiceTests.swift:117-133) relies onUserDefaults.setflushing to disk synchronously — documented as async by Apple, empirically synchronous in ~1,000 combined probe rounds. Comment says so; that's the right amount of caution for something unspecified but load-tested. - Worth noting: the pre-cleanup existence assertion at
WorkspaceServiceTests.swift:241fails under a sandboxed exec (reproduced by the reviewing agent's own sandbox). Not fixed here — filed as #1611 rather than blocking this PR on it, which is the right call since this target doesn't run sandboxed today. - Worth noting: #1606 (RuntimeSamplerCost footprint drift, seen once in 14 concurrent rounds) is out of this PR's fence and correctly filed separately rather than folded in.
Evidence: ✅ all four requested-evidence items are [complete] in the PR body, matching the issue's four asks — named shared state with file:line, three consecutive concurrent full-suite passes, CI green, and the production-reachability statement.
Evidence-level reasoning and code check
This is a shared-state concurrency bug, so the right bar was a repro someone actually ran twice at once (not a screenshot, not a single-process test count) plus the named unit test. That's what's here: swift test --filter WorkspaceServiceTests (46 passed) covers the mechanical conversion, and the paired-process repro (rounds 5-7, Test run with 2157 tests in 279 suites passed, both checkouts, Suite "WorkspaceService" passed in all 14 logs) covers the actual contamination claim from #1536.
Reading the diff against the preferences field: it's a trailing-defaulted parameter on both WorkspaceService initializers (Sources/WorkspaceManagerCore/Services/WorkspaceService.swift), so WorkspaceService.shared in the app and CLI are unaffected — production still reads LaunchPreferences.defaults. The test-file diff shows every setWorkspacesRoot/restoreWorkspacesRoot call site replaced with a UUID-suite preferences.set(...), and the two old helpers plus the TODO(#1536 follow-up) are gone, matching the "8 tests still on UserDefaults.standard" gap I flagged last time.
Two independent static passes (recorded in the comment thread) landed on the same conclusion I reach reading the diff: no blocking findings, two minor ones correctly triaged (one accepted as documented risk, one deferred to #1611). I didn't re-run anything — no Swift toolchain here — but the reasoning in those passes is checkable against the diff and holds up.
Superseded by subsequent approval from the same reviewer.
April Clearwater, Application Lead
Review page: https://evidence.cloudcompute.com/workspaces/pr-1602/jGQMzcy4LMmZuJzoJACaZw/20260912-055959-pr-review-1602.html
The test suite for creating workspaces sometimes failed when two copies of it ran at the same time, because every test wrote its settings to the one preferences file the real app also uses. April changed the service to accept its own settings store and gave each test a private one; a repo-steward worker finished the eight tests she had left on the shared file. Two files, about 150 lines changed. The proof is seven rounds of two full test suites racing each other, all green. Risk: none to the app, whose default store is unchanged.
Summary
WorkspaceService.workspacesRoot/setWorkspacesRoot/resetWorkspacesRoot(Sources/WorkspaceManagerCore/Services/WorkspaceService.swift:98-116, before this change) read and wroteLaunchPreferences.defaults, which resolves toUserDefaults.standardin any run that sets neitherWORKSPACES_SYNTHETIC_ROOTnorWORKSPACES_PREFERENCES_SUITE(Sources/WorkspaceManagerCore/Services/LaunchPreferences.swift:80-104).UserDefaults.standardis an on-disk, cfprefsd-backed domain keyed off the executable's identity, not scoped to a Swift process — two separateswift testinvocations of the same package share it, which is exactly the two-worktree repro in the issue.WorkspaceServiceTests's ownsetWorkspacesRoot/restoreWorkspacesRoothelpers (Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift:110-125, before this change) wrote to that same domain directly, so one process'swsRootUUID could be read back by another process's test mid-run — the "WSTest-A8A4A015… reading WSTest-AC10E5D0…" failure in the issue. This is reachable from production code, not only the fixture: it's the same path a live app and aworkspacesCLI invocation without the isolation env vars would race on.WorkspaceServicenow takes an injectablepreferences: UserDefaults(defaultLaunchPreferences.defaults, so production behavior is unchanged) and reads/writes the custom root through it instead of hardcodingLaunchPreferences.defaults. AddedmakeIsolatedPreferences()toWorkspaceServiceTests, which hands each converted test aUserDefaults(suiteName:)scoped to a fresh UUID — a domain no other test, in this process or a concurrent one, can ever collide with — and converted the majority of the suite'screateWorkspace/workspacesRoottests to use it.makeIsolatedPreferences()as well, and the twoUserDefaults.standardhelpers (setWorkspacesRoot/restoreWorkspacesRoot) and theirTODO(#1536 follow-up)comment are removed.Evidence Status
swift testthree times consecutively while a second fullswift testruns alongside it, with the command used and all three results pasted -- at266db386, checkout A is this branch's worktree and checkout B a local clone of it (not a worktree); each round ran(cd A && swift test > A-N.log 2>&1) & (cd B && swift test > B-N.log 2>&1) & wait, and a process snapshot 10s into the round showed both checkouts'swiftpm-testing-helperrunning at once. Round 5: A✔ Test run with 2157 tests in 279 suites passed after 20.463 seconds.B✔ Test run with 2157 tests in 279 suites passed after 21.028 seconds.Round 6: A✔ Test run with 2157 tests in 279 suites passed after 20.621 seconds.B✔ Test run with 2157 tests in 279 suites passed after 20.513 seconds.Round 7: A✔ Test run with 2157 tests in 279 suites passed after 20.826 seconds.B✔ Test run with 2157 tests in 279 suites passed after 20.764 seconds.Every round, including the one unrelated failure in round 4, is listed under Validation.WorkspaceService.workspacesRoot/setWorkspacesRoot/resetWorkspacesRoot(Sources/WorkspaceManagerCore/Services/WorkspaceService.swift:98-116before this change) read and wroteLaunchPreferences.defaults, which isUserDefaults.standardwhen neitherWORKSPACES_SYNTHETIC_ROOTnorWORKSPACES_PREFERENCES_SUITEis set (Sources/WorkspaceManagerCore/Services/LaunchPreferences.swift:80-104,:136). It crosses the suite boundary becauseUserDefaults.standardis an on-disk domain served by cfprefsd and keyed by the executable, so twoswift testprocesses of this package read and write the sameworkspacesRootkey.build-and-testsucceeds -- green on head d13afd8:build-and-testrun 34596303172 (ci.yml, GitHub-hosted macos-26) completed success at 12:20:57Z; also green on 266db38 (run 34594093783) and 97c1de3 (run 34359147304)Sources/WorkspaceManager/App/WorkspaceManagerApp.swift:741) and theworkspacesCLI (Sources/WorkspaceManagerCLI/main.swift:30) both useWorkspaceService.shared, which reads and writes theworkspacesRootkey inLaunchPreferences.defaults, the same key the Settings pane binds (Sources/WorkspaceManager/Views/SettingsView.swift:22). This PR leaves that default in place; only tests inject a private suite.Validation
swift test --filter WorkspaceServiceTests:✔ Test run with 46 tests in 1 suite passed after 1.441 seconds.mise run lint(swift-format lint --strict --recursive Sources/ Tests/): exit 0, no findings.swift test, sequential, after the concurrent rounds:✔ Test run with 2157 tests in 279 suites passed after 19.623 seconds.Concurrent rounds, all at
266db386, run from the command in Evidence Status item 1. Overlap is checked from each log'sTest Suite 'All tests' started atstamp, and from rounds 3 onward also by a process snapshot 10s in.The
WorkspaceServicesuite passed in all 14 runs. The one failure, round 4 checkout B, wasRuntimeSamplerCostTests.swift:169: the app-tree footprint drifted 13.6% against a 10% bound. That test measures its own process tree and does not touchUserDefaults, so it is a separate defect, filed as Flaky: RuntimeSamplerCost footprint totals drift past 10% when a second test process runs alongside #1606. Rounds 5-7 are the three consecutive rounds quoted in Evidence Status.Machine during the rounds: 32 GB RAM, swap 10.0-10.2 of 11.3 GB used, 36-45% memory free, 1-minute load average up to 96. Both runs fit at once, so nothing was staggered.
The owner's part of this PR (the eight conversions and every run above) was done by Claude Code in the repo steward's worker tile
claude-1536-finish-preferences.Risks
WorkspaceServiceon the default preferences: 11 build one fordeleteWorkspace,archiveWorkspace-by-path andgetWorkspaceSize, and 10 callWorkspaceService.sharedfor lifecycle scripts and name sanitization. None of them reads the domain:preferencesis read only bycreateWorkspace(WorkspaceService.swift:182) androot(:504), and none of those tests calls either. A new test in this suite that calls either one on a default-built service would be exposed again.UserDefaults(suiteName:)plist is deleted from~/Library/Preferencesin the test'sdefer, so a clean run leaves none behind; only a crash leaves one, the UUID-named plist of the test whose cleanup never ran.preferences:is a trailing defaulted parameter on both initializers, so the production call sites (WorkspaceService.sharedin the app and the CLI) compile and behave as before. The full suite,WorkspaceOrphanReconcilerTestsincluded, passes.WorkspaceServiceacross 6 overlapping rounds here.Mergeability
Sources/WorkspaceManagerCore/Services/WorkspaceService.swift(injectablepreferences:),Tests/WorkspaceManagerTests/WorkspaceServiceTests.swiftWorkspaceServicestill defaults toLaunchPreferences.defaults, so the app and the CLI read and write the sameworkspacesRootkey as before.defer, and a crash leaks only a UUID-named plist.RuntimeSamplerCostflake seen once in 14 concurrent runs; the concurrent repro was not re-run on the pre-fix commit.Closes #1536
🤖 Generated with Claude Code
https://claude.ai/code/session_01QrSceQotsidvcP4tVgf2ud
Orchestrator note (workspaces)