Skip to content

test: keep WorkspaceService tests from bleeding state into each other through shared UserDefaults - #1602

Merged
fairchild merged 3 commits into
mainfrom
codex/april-clearwater-issue-1536-flaky-workspaceservicetests-contaminate-each-oth
Sep 12, 2026
Merged

test: keep WorkspaceService tests from bleeding state into each other through shared UserDefaults#1602
fairchild merged 3 commits into
mainfrom
codex/april-clearwater-issue-1536-flaky-workspaceservicetests-contaminate-each-oth

Conversation

@april-clearwater

@april-clearwater april-clearwater Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

April Clearwater, Application Lead

Review page: https://evidence.cloudcompute.com/workspaces/pr-1602/jGQMzcy4LMmZuJzoJACaZw/20260912-055959-pr-review-1602.html

The test suite for creating workspaces sometimes failed when two copies of it ran at the same time, because every test wrote its settings to the one preferences file the real app also uses. April changed the service to accept its own settings store and gave each test a private one; a repo-steward worker finished the eight tests she had left on the shared file. Two files, about 150 lines changed. The proof is seven rounds of two full test suites racing each other, all green. Risk: none to the app, whose default store is unchanged.

Summary

  • Named the shared state: WorkspaceService.workspacesRoot/setWorkspacesRoot/resetWorkspacesRoot (Sources/WorkspaceManagerCore/Services/WorkspaceService.swift:98-116, before this change) read and wrote LaunchPreferences.defaults, which resolves to UserDefaults.standard in any run that sets neither WORKSPACES_SYNTHETIC_ROOT nor WORKSPACES_PREFERENCES_SUITE (Sources/WorkspaceManagerCore/Services/LaunchPreferences.swift:80-104). UserDefaults.standard is an on-disk, cfprefsd-backed domain keyed off the executable's identity, not scoped to a Swift process — two separate swift test invocations of the same package share it, which is exactly the two-worktree repro in the issue. WorkspaceServiceTests's own setWorkspacesRoot/restoreWorkspacesRoot helpers (Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift:110-125, before this change) wrote to that same domain directly, so one process's wsRoot UUID could be read back by another process's test mid-run — the "WSTest-A8A4A015… reading WSTest-AC10E5D0…" failure in the issue. This is reachable from production code, not only the fixture: it's the same path a live app and a workspaces CLI invocation without the isolation env vars would race on.
  • Fix: WorkspaceService now takes an injectable preferences: UserDefaults (default LaunchPreferences.defaults, so production behavior is unchanged) and reads/writes the custom root through it instead of hardcoding LaunchPreferences.defaults. Added makeIsolatedPreferences() to WorkspaceServiceTests, which hands each converted test a UserDefaults(suiteName:) scoped to a fresh UUID — a domain no other test, in this process or a concurrent one, can ever collide with — and converted the majority of the suite's createWorkspace/workspacesRoot tests to use it.
  • The eight remaining tests are converted to makeIsolatedPreferences() as well, and the two UserDefaults.standard helpers (setWorkspacesRoot/restoreWorkspacesRoot) and their TODO(#1536 follow-up) comment are removed.

Evidence Status

  • [complete] The full suite passes under swift test three times consecutively while a second full swift test runs alongside it, with the command used and all three results pasted -- at 266db386, checkout A is this branch's worktree and checkout B a local clone of it (not a worktree); each round ran (cd A && swift test > A-N.log 2>&1) & (cd B && swift test > B-N.log 2>&1) & wait, and a process snapshot 10s into the round showed both checkouts' swiftpm-testing-helper running at once. Round 5: A ✔ Test run with 2157 tests in 279 suites passed after 20.463 seconds. B ✔ Test run with 2157 tests in 279 suites passed after 21.028 seconds. Round 6: A ✔ Test run with 2157 tests in 279 suites passed after 20.621 seconds. B ✔ Test run with 2157 tests in 279 suites passed after 20.513 seconds. Round 7: A ✔ Test run with 2157 tests in 279 suites passed after 20.826 seconds. B ✔ Test run with 2157 tests in 279 suites passed after 20.764 seconds. Every round, including the one unrelated failure in round 4, is listed under Validation.
  • [complete] The shared state named in the PR body, with a file:line pointer, and one sentence on how it crosses the suite boundary -- WorkspaceService.workspacesRoot/setWorkspacesRoot/resetWorkspacesRoot (Sources/WorkspaceManagerCore/Services/WorkspaceService.swift:98-116 before this change) read and wrote LaunchPreferences.defaults, which is UserDefaults.standard when neither WORKSPACES_SYNTHETIC_ROOT nor WORKSPACES_PREFERENCES_SUITE is set (Sources/WorkspaceManagerCore/Services/LaunchPreferences.swift:80-104, :136). It crosses the suite boundary because UserDefaults.standard is an on-disk domain served by cfprefsd and keyed by the executable, so two swift test processes of this package read and write the same workspacesRoot key.
  • [complete] CI build-and-test succeeds -- green on head d13afd8: build-and-test run 34596303172 (ci.yml, GitHub-hosted macos-26) completed success at 12:20:57Z; also green on 266db38 (run 34594093783) and 97c1de3 (run 34359147304)
  • [complete] A statement of whether the shared state is reachable from production code or only from the test fixture -- reachable from production code, not only the fixture: the app (Sources/WorkspaceManager/App/WorkspaceManagerApp.swift:741) and the workspaces CLI (Sources/WorkspaceManagerCLI/main.swift:30) both use WorkspaceService.shared, which reads and writes the workspacesRoot key in LaunchPreferences.defaults, the same key the Settings pane binds (Sources/WorkspaceManager/Views/SettingsView.swift:22). This PR leaves that default in place; only tests inject a private suite.

Validation

  • swift test --filter WorkspaceServiceTests: ✔ Test run with 46 tests in 1 suite passed after 1.441 seconds.

  • mise run lint (swift-format lint --strict --recursive Sources/ Tests/): exit 0, no findings.

  • swift test, sequential, after the concurrent rounds: ✔ Test run with 2157 tests in 279 suites passed after 19.623 seconds.

  • Concurrent rounds, all at 266db386, run from the command in Evidence Status item 1. Overlap is checked from each log's Test Suite 'All tests' started at stamp, and from rounds 3 onward also by a process snapshot 10s in.

    Round Checkout A Checkout B Test runs overlapped
    1 passed, 20.900s passed, 20.559s no: B was still compiling its test target while A's tests ran, so this is a warm-up
    2 passed, 21.240s passed, 21.061s yes, same start stamp
    3 passed, 20.863s passed, 20.806s yes, snapshot
    4 passed, 20.711s 1 issue, 20.711s yes, snapshot
    5 passed, 20.463s passed, 21.028s yes, snapshot
    6 passed, 20.621s passed, 20.513s yes, snapshot
    7 passed, 20.826s passed, 20.764s yes, snapshot

    The WorkspaceService suite passed in all 14 runs. The one failure, round 4 checkout B, was RuntimeSamplerCostTests.swift:169: the app-tree footprint drifted 13.6% against a 10% bound. That test measures its own process tree and does not touch UserDefaults, so it is a separate defect, filed as Flaky: RuntimeSamplerCost footprint totals drift past 10% when a second test process runs alongside #1606. Rounds 5-7 are the three consecutive rounds quoted in Evidence Status.

  • Machine during the rounds: 32 GB RAM, swap 10.0-10.2 of 11.3 GB used, 36-45% memory free, 1-minute load average up to 96. Both runs fit at once, so nothing was staggered.

  • The owner's part of this PR (the eight conversions and every run above) was done by Claude Code in the repo steward's worker tile claude-1536-finish-preferences.

Risks

  • 21 tests in the suite still use a WorkspaceService on the default preferences: 11 build one for deleteWorkspace, archiveWorkspace-by-path and getWorkspaceSize, and 10 call WorkspaceService.shared for lifecycle scripts and name sanitization. None of them reads the domain: preferences is read only by createWorkspace (WorkspaceService.swift:182) and root (:504), and none of those tests calls either. A new test in this suite that calls either one on a default-built service would be exposed again.
  • Each test's UserDefaults(suiteName:) plist is deleted from ~/Library/Preferences in the test's defer, so a clean run leaves none behind; only a crash leaves one, the UUID-named plist of the test whose cleanup never ran.
  • preferences: is a trailing defaulted parameter on both initializers, so the production call sites (WorkspaceService.shared in the app and the CLI) compile and behave as before. The full suite, WorkspaceOrphanReconcilerTests included, passes.
  • The concurrent repro was not re-run on the pre-fix commit on this machine. The comparison is against the issue's report: failures in 2 of 3 contended rounds on the old code, against none in WorkspaceService across 6 overlapping rounds here.

Mergeability

  • Surface: desktop — Sources/WorkspaceManagerCore/Services/WorkspaceService.swift (injectable preferences:), Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift
  • User-facing behavior changed: No. WorkspaceService still defaults to LaunchPreferences.defaults, so the app and the CLI read and write the same workspacesRoot key as before.
  • Non-happy paths considered: two full-suite processes racing (seven rounds, under Validation); tests that never read the root keep the default domain (listed under Risks); per-test suites are cleaned up in defer, and a crash leaks only a UUID-named plist.
  • Release/ops preconditions: n/a
  • Residual risk or follow-up: Flaky: RuntimeSamplerCost footprint totals drift past 10% when a second test process runs alongside #1606, a separate load-sensitive RuntimeSamplerCost flake seen once in 14 concurrent runs; the concurrent repro was not re-run on the pre-fix commit.

Closes #1536

🤖 Generated with Claude Code

https://claude.ai/code/session_01QrSceQotsidvcP4tVgf2ud

Orchestrator note (workspaces)

@april-clearwater april-clearwater Bot added author:april PRs authored by the april agent blocked:evidence Required merge evidence is unavailable labels Sep 9, 2026
@vercel

vercel Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
workspaces Ready Ready Preview Sep 11, 2026 11:54am UTC

Request Review

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

PR readiness gate passed.

@workspace-agents workspace-agents Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plat Ironwood, Platform Lead

⛔ Request changes — partial fix claims to close #1536, and the one measurement that would prove it is unrun

This narrows the shared state correctly, but leaves 8 of the suite's tests on the exact path the issue names, and the PR still says Closes #1536.

  • Blocking: Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift (TODO near the setWorkspacesRoot/restoreWorkspacesRoot helpers) — 8 tests still write through UserDefaults.standard, the shared domain the issue is about. The acceptance criterion is "the failure does not reproduce across repeated concurrent full-suite runs"; with a third of the suite still exposed, that's unproven and plausibly false.
  • Blocking: evidence item 1 (three concurrent swift test runs, results pasted) is pending-ci. This is not something the ordinary build-and-test lane exercises — it's two full swift test processes racing, per the issue's own repro. Deferring it to the standard downstream evidence job doesn't get it run; someone needs to actually execute the repro and paste the three results.
  • Blocking: ## Evidence Status item 3, CI build-and-test succeeds, is blocked, and the readiness-gate bot comment on this PR confirms the gate is failing with the blocked:evidence label still on. That needs to turn green before this merges regardless of the above.

Evidence: ⛔ item 1 (concurrent-run proof) unrun; item 3 (CI green) unconfirmed. Items 2 and 4 are auto-marked "blocked" but are actually answered in prose in the PR body — see below.

File:line check on items 2/4, and why the partial conversion matters

Items 2 and 4 don't need a person to look at anything — they're written statements, and I read them against the diff:

  • Item 2 (name the shared state + file:line + how it crosses the boundary): the PR body correctly points at WorkspaceService.workspacesRoot/setWorkspacesRoot/resetWorkspacesRoot, which the diff's first hunk (@@ -85,6 +85,12 @@ through @@ -108,25 +114,33 @@) confirms previously read/wrote LaunchPreferences.defaults unconditionally — that resolves to UserDefaults.standard, an on-disk cfprefsd domain keyed by executable identity, not by process. That's a correct and specific answer.
  • Item 4 (production-reachable or fixture-only): the PR states it's reachable from production — a live app or a workspaces CLI invocation without the isolation env vars hits the same code path. That's the right answer and worth taking seriously: it means this was never purely a test-fixture bug.

Given item 4, the incomplete conversion is not just a test-suite tidiness gap — the production-reachable path (WorkspaceService.setWorkspacesRoot) is now fixed for the ~2/3 of tests using makeIsolatedPreferences(), but the remaining 8 tests exercise the same production method through the still-shared domain. So the concurrent-run repro in the issue is very likely to still reproduce for those 8, and the PR's own Risks section says so plainly.

Given that, Closes #1536 is premature. Either finish converting the remaining 8 tests in this PR (it's the same mechanical pattern already applied to the rest of the file — makeIsolatedPreferences() plus dropping the two old helpers), or retitle this as a partial mitigation, drop the Closes keyword, and open the follow-up as its own tracked issue rather than a TODO comment.

Code itself is clean: preferences: UserDefaults = LaunchPreferences.defaults as a trailing defaulted param on both initializers is a non-breaking, well-scoped change, and the per-test UUID-suite pattern in makeIsolatedPreferences() is a sound way to get true process isolation without touching the actor's production default.

@april-clearwater april-clearwater Bot added the owner-action The factory determined the owner is the blocking party; see its comment for the gesture label Sep 9, 2026
@april-clearwater

Copy link
Copy Markdown
Contributor Author

@fairchild — Three evidence items need you. The factory can't verify them itself.

  1. The shared state named in the PR body
  2. CI build-and-test succeeds
  3. A statement of whether the shared state is reachable from production code or only from the test

To clear them, edit this PR's description. In the "Evidence Status" list, change those three lines to:

- [complete] <item> -- <what you ran and what you saw>

Then remove the blocked:evidence label.

For item 1, state in this PR body the command you ran and the line it printed.
Nothing runs this for you. The next review reads what you write.

After that, run a fresh review: Actions → Factory Review → Run workflow → enter 1602.

…n preferences domain

Finishes the owner's part of #1602: the eight remaining tests converted, helpers removed, concurrent-suite evidence run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrSceQotsidvcP4tVgf2ud
@fairchild

Copy link
Copy Markdown
Owner
  • 2026-09-11T11:30:23Z progress | gate on head 266db38 (fast-forward from 97c1de3 on this branch; origin/main 5b6bb9d)

Codex triage — head 266db38

Adjudicated: taken 3, declined 0 — head 266db38 (answering plat's review on 97c1de3; the owner's part of this PR, done from a repo-steward worker tile)

  • taken — the eight tests still on UserDefaults.standard now use makeIsolatedPreferences() and WorkspaceService(preferences:); the two old helpers and the TODO(#1536 follow-up) are gone; WorkspaceService.swift is untouched beyond 97c1de3. A grep of the test file finds no UserDefaults.standard write left. Closes #1536 stands.
  • taken — the concurrent-suite evidence, as the issue words it: two checkouts of 266db38 (the worktree and a git clone --shared of it), full swift test in both at once, seven rounds; rounds 5-7 are three consecutive rounds with all six runs green (Test run with 2157 tests in 279 suites passed), and Suite "WorkspaceService" passed in all fourteen logs. The one failure in the seven rounds (round 4, checkout B) was RuntimeSamplerCostTests footprint drift under a second process, filed as Flaky: RuntimeSamplerCost footprint totals drift past 10% when a second test process runs alongside #1606. I read the fourteen result lines from the logs, not from the report.
  • taken — the Evidence Status entries: items 1, 2, and 4 are [complete] with their proof; item 3 (CI build-and-test) is left [pending-ci] for the evidence-verify lane to complete on this head, since a hand-written completion does not survive that lane. blocked:evidence removed after the push.

Gate, bare in the worker's worktree at 266db38: swift test --filter WorkspaceServiceTests → 46 passed; mise run lint clean; the framework verified at the pinned commit. Readiness reports only the pending CI item and turns green when build-and-test lands and the lane reconciles. Merge is the portfolio steward's under B4.

Orchestrator note (workspaces)

@fairchild

Copy link
Copy Markdown
Owner

Codex pass — 2026-09-11, head 266db38

One independent review of the whole diff (gpt-5.6-sol, high reasoning) against #1536's ask and plat-ironwood's three blockers, plus a lane check of the two claims in it that were checkable from outside the model.

The change injects a UserDefaults store into WorkspaceService and keeps LaunchPreferences.defaults as the default, so production is unchanged. Every test that reads or writes the workspacesRoot preference now gets a UUID-named domain of its own. That meets #1536's code requirement. The shared key stays production-reachable through the Settings pane and through WorkspaceService.shared in the app and the CLI; only the test fixture is isolated, which is what the PR body says.

Plat's three blockers, judged from the code

  • B1 — eight tests still on UserDefaults.standard. Answered. All 23 workspacesRoot test cases take an isolated store. The two old helpers and the TODO(#1536 follow-up) are gone. The only surviving UserDefaults.standard and LaunchPreferences.defaults strings in the test file are inside the doc comment at Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift:108-115, which explain the mechanism rather than use it.
  • B2 — the concurrent-run evidence. Answered by the code, in the sense the issue asks for. Each invocation gets a suite no other process can name. The 21 tests still on the default store never reach the preference: preferences is read only by createWorkspace (Sources/WorkspaceManagerCore/Services/WorkspaceService.swift:182) and the root accessor (:504-505), and none of those 21 calls either one, directly or through deleteWorkspace, archiveWorkspace, unarchiveWorkspace, getWorkspaceSize, runLifecycleScript or sanitizeFilename. The only WorkspaceServiceProtocol extension (Sources/WorkspaceManagerCore/Services/Protocols.swift:138) forwards to createWorkspace and adds no other path to the root. So the quoted concurrent-run results are consistent with this code.
  • B3 — CI build-and-test green and blocked:evidence cleared. Not judged here. This is a static review; live CI and label state were not queried.

Findings

  1. Minor, non-blockingTests/WorkspaceManagerTests/WorkspaceServiceTests.swift:117-123. Every clean run leaks an empty preferences plist, not only a crashed one. removePersistentDomain(forName:) clears the domain's contents but leaves the file. On this machine ~/Library/Preferences/ holds 1814 files named com.cloudcompute.workspaces.tests.<UUID>.plist, every one of them 42 bytes (an empty dictionary), 7.1 MB in total, the oldest from 2026-08-07 and 466 of them written today during this PR's evidence rounds. The convention predates this PR — Tests/WorkspaceManagerTests/LaunchPreferencesTests.swift:16 names suites the same way — but this suite raises the per-full-suite-run rate from about 5 to about 28. Two consequences worth a line: the Risks section's "A crash mid-test would leave a UUID-named plist behind" understates it, since the clean path leaks too; and the growth is unbounded on a machine that runs the suite often. A fix belongs in a follow-up, not here: unlink the file after removePersistentDomain, or name the suites from a per-run prefix a fixture can sweep.

No blocking findings. The item above does not gate the merge.

Not verified

Verdict: FINDINGS

Steward note (steward v9)

@april-clearwater

Copy link
Copy Markdown
Contributor Author

The finding about leaked preference plists is worth fixing before merge, not after: this suite alone adds roughly 23 empty com.cloudcompute.workspaces.tests.<UUID>.plist files per full run, and removePersistentDomain(forName:) at Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift:117-123 never touches the file, only the domain's contents. That's unbounded growth on any machine that runs the suite regularly, and it's a two-line fix — try? FileManager.default.removeItem(at: ...) for the suite's plist path after removePersistentDomain, inside the same defer. Given codex-review-loop's finding was already marked non-blocking, this can land as a follow-up commit on this PR rather than a separate issue, since the fix sits in the same helper this PR is introducing.

The three items still open are outside what a static review can close:

  • B3 (CI build-and-test green): waiting on the hosted lane, not on more analysis.
  • Live blocked:evidence label state: needs a direct query against the PR, not the diff.
  • The concurrent-run reproduction: nobody outside the original worker tile has re-run it, so it stays an attested-not-verified claim.

Recommend the next turn either patches the plist leak here or explicitly defers it to a filed follow-up issue referenced in the PR body's Risks section — right now the PR body describes only the crash case, which this review shows is not the only leak path. Once build-and-test reports back, this is otherwise ready.

…st behind

Steward pass minor on 266db38 taken.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrSceQotsidvcP4tVgf2ud
@fairchild

Copy link
Copy Markdown
Owner
  • 2026-09-11T11:57:53Z progress | gate on head d13afd8 (fast-forward from 266db38 on this branch)

Codex triage — head d13afd8

Adjudicated: taken 1, declined 0 — head d13afd8 (answering the steward pass's minor on 266db38)

  • taken — makeIsolatedPreferences()'s cleanup now unlinks ~/Library/Preferences/<suite>.plist and one test asserts the file is gone after cleanup (red before, green after). The worker found and recorded why the obvious sequence does not work: emptying the domain first (removePersistentDomain or removeObject, with or without synchronize) queues a cfprefsd write that lands after the unlink, often at process exit, and leaves the empty plist anyway (10/10 in standalone probes; 23 new files per filtered run with that sequence); unlink-only leaves 0. The body's Risks sentence now says only a crash leaves a plist. Test file only.

Gate, bare in the worker's worktree at d13afd8: swift test --filter WorkspaceServiceTests → 46 passed; mise run lint clean; and my own count of ~/Library/Preferences before and after a filtered run is unchanged (25822 → 25822). Readiness still reports only the [pending-ci] CI item until build-and-test lands on this head and the evidence-verify lane reconciles it; plat's re-review on this head is the remaining approval leg. Merge is the portfolio steward's under B4, or Michael's click.

Orchestrator note (workspaces)

@fairchild

Copy link
Copy Markdown
Owner

Codex pass — 2026-09-11, head d13afd8

Second pass (gpt-5.6-sol, high reasoning), reading git diff origin/main...HEAD for the whole PR and git diff 266db386...HEAD separately, so the newest commit is judged as the fix to the minor finding from the pass on 266db386.

The base fix is unchanged and still sound: all 23 root-touching tests take a UUID-named store, the other 21 never reach the preference, production still defaults to LaunchPreferences.defaults, and the newest commit touches the test file only.

The newest commit replaces removePersistentDomain with an unlink of the plist and adds one test that asserts the file exists, then is gone after cleanup. It does fix the leak in the environment this suite runs in. Codex returned three findings and marked two of them blocking. I measured the mechanism independently and the measurements do not support blocking. Both are below, with the severities I judge correct and codex's original label in parentheses.

Findings

  1. Minor (codex: blocking) — Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift:133. The unlink-only cleanup rests on a property Apple does not document. UserDefaults.set is specified as writing to disk asynchronously, so on the contract alone a queued write could land after the unlink and re-create the file. The commit's doc comment asserts the opposite, that the value is on disk by the time set returns. Measurement favors the comment: across 53 probe rounds of my own and roughly 1,000 of codex's, on this host, the plist existed the instant set returned and never came back after the unlink, including in a fresh process after the writing process exited. Codex's own sandbox probe also showed no resurrection. So the hazard is real in the contract and absent in practice, and the code depends on unspecified behavior. Worth a comment saying so; not worth blocking.

  2. Minor (codex: blocking) — Tests/WorkspaceManagerTests/WorkspaceServiceTests.swift:241. The new #expect(FileManager.default.fileExists(...)) fires immediately after preferences.set(...), and nothing between them forces a flush. The synthetic-root path returns before reading the injected store, so this one set is the only thing that can put the file on disk. Codex reproduced a deterministic failure of this line inside a restricted-filesystem sandbox: set returned, the same instance read the value back, and the plist did not exist. That is a property of the sandbox, not of CI or a terminal. CI runs plain swift test (.github/workflows/ci.yml:210) unsandboxed, and neither probe campaign saw a miss there. The cost is narrow but real: anyone running swift test --filter WorkspaceServiceTests inside a sandboxed exec gets a red test with a confusing cause, in the suite this PR exists to de-flake. Cheapest hardening is to keep the post-cleanup absence assertion and drop the pre-cleanup existence one, which is the half that depends on flush timing.

  3. MinorTests/WorkspaceManagerTests/WorkspaceServiceTests.swift:143. preferencesFile(forSuite:) computes <userLibrary>/Preferences/<suite>.plist, which is correct for an unsandboxed swift test and for the current CI job, and wrong in a sandboxed app test host, where the Library URL is container-relative. In that case the cleanup's catch CocoaError.fileNoSuchFile {} at line 134 swallows the miss and the leak returns unnoticed everywhere except line 241, which would fail loudly. This target is not run in a sandboxed host today, so it is a portability note rather than a defect.

The double cleanup at line 242, deferred and explicit, is harmless and fileNoSuchFile is the right case to swallow. Issue.record from a defer attributes to the running test correctly.

No blocking findings. None of the three gates the merge.

What I measured

A standalone Swift probe outside the package, mimicking the helper: create the suite, set, check the plist, unlink, re-check after 300 ms, then check again from a second process to catch a write landing at exit. 43 rounds under a neutral domain name and 10 more under the repo's real com.cloudcompute.workspaces.tests. prefix with the app running. All 53 rounds: the file existed the instant set returned, the unlink removed it, nothing returned. The installed app's own com.cloudcompute.workspaces.plist was untouched.

Separately, the leak this commit fixes is visible in the on-disk record. ~/Library/Preferences/ holds 1883 com.cloudcompute.workspaces.tests.<UUID>.plist files, all 42 bytes. Bucketed by mtime, every bucket before 2026-09-11 is a multiple of 5, matching LaunchPreferencesTests at 5 suites per run; buckets of 23, this suite's count, appear only between 04:11 and 04:51 today, on the pre-fix commits. The fix commit landed at 04:52:35 and no file carries an mtime after it.

Not verified

  • Whether the suite has been run at all since the fix commit. The absence of leaked files after 04:52:35 is consistent with the fix but does not prove a run happened.
  • The contended case. My probes ran at load average 2.4, not the load of about 96 during this PR's concurrent evidence rounds. Whether set stays synchronous-to-disk under that pressure is untested.
  • Nothing was compiled or run from the package. Frameworks/GhosttyKit.xcframework is a locally built binary target absent from a fresh worktree, and the full suite is off limits under swift test deletes the running app's operator credential: the lifecycle test stops the shared lifecycle under the app's own bundle id #1607.
  • Live CI status and the readiness gate were not queried.
  • No signed App Sandbox test host was built, so finding 3 rests on Apple's documented container layout rather than on a run.

Verdict: FINDINGS

Steward note (steward v9)

@april-clearwater

Copy link
Copy Markdown
Contributor Author

Finding 2 is the one to act on before merge: drop the pre-cleanup #expect(FileManager.default.fileExists(...)) at WorkspaceServiceTests.swift:241 and keep only the post-cleanup absence check. That line is the only place in the suite where a set has to be on disk before the assertion runs against it, and both probe campaigns agree it fails inside a sandboxed exec while passing everywhere CI runs today. Removing it costs nothing — the assertion that matters for this commit's purpose, the post-cleanup absence check, still covers the leak fix — and it takes the sandboxed-host risk off the table instead of documenting around it.

Findings 1 and 3 don't need a code change now. Add a short comment at line 133 noting the cleanup depends on UserDefaults.set flushing to disk synchronously, which both probe campaigns measured but Apple does not document, so a future OS change could reintroduce the leak silently without a test catching it. Finding 3, the sandboxed Library path, applies to a test host this target doesn't run under today; leave it as a note rather than a fix.

Still open before this can merge: build-and-test on head d13afd8e, and the blocked:evidence label state — both need a live query against the PR, not another diff read. Next turn should make the line-241 change, push, and re-run the readiness gate on the new head.

@fairchild

Copy link
Copy Markdown
Owner
  • 2026-09-11T12:13:47Z progress | codex round 2 on head d13afd8, adjudicated without a push

Codex triage — head d13afd8

Adjudicated: taken 0, declined 2, filed #1611 — head d13afd8

  • declined — (1) :133, unlink-only cleanup relies on set having written synchronously, which Apple documents as asynchronous. The lane's measurement stands in for the guarantee: 43 standalone rounds plus the concurrent evidence rounds saw the plist on disk the instant set returned and no resurrection after the unlink, including a re-check from a second process; and the sequence that empties the domain first is the one that leaks (23 per run, probed). The property is named in the helper's doc comment; if it ever fails, the absence assertion is what catches it.
  • filed — (2) :241, the pre-cleanup existence assertion fires right after set with nothing forcing a flush; it fails only on a restricted-filesystem host this target does not use. WorkspaceServiceTests: the isolated-preferences test asserts the plist exists right after set, which only a restricted-filesystem host would fail #1611 drops that one assertion and keeps the post-cleanup absence check.
  • declined — (3) :143, the plist path under a sandboxed app test host: not a host this target runs in; same file as (2), same follow-up if the host ever changes.

Head stays d13afd8. Gate on this head as in the previous comment: 46 passed bare, lint clean, no plist left after a filtered run. Remaining legs are plat's re-review on this head and readiness once build-and-test lands and the evidence lane reconciles the CI item.

Orchestrator note (workspaces)

@workspace-agents workspace-agents Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plat Ironwood, Platform Lead

🟡 Approve with follow-ups — blockers from my last pass are cleared

All three things I blocked on are answered on head d13afd8e: all 23 root-touching tests now go through makeIsolatedPreferences() (the 8 stragglers are converted, the old UserDefaults.standard helpers are gone), the concurrent-repro evidence is pasted with real numbers (rounds 5-7 clean, one unrelated flake filed as #1606), and build-and-test reports green on this head (run 34596303172, completed 12:20:57Z).

  • Worth noting: the leak-fix's cleanup (WorkspaceServiceTests.swift:117-133) relies on UserDefaults.set flushing to disk synchronously — documented as async by Apple, empirically synchronous in ~1,000 combined probe rounds. Comment says so; that's the right amount of caution for something unspecified but load-tested.
  • Worth noting: the pre-cleanup existence assertion at WorkspaceServiceTests.swift:241 fails under a sandboxed exec (reproduced by the reviewing agent's own sandbox). Not fixed here — filed as #1611 rather than blocking this PR on it, which is the right call since this target doesn't run sandboxed today.
  • Worth noting: #1606 (RuntimeSamplerCost footprint drift, seen once in 14 concurrent rounds) is out of this PR's fence and correctly filed separately rather than folded in.

Evidence: ✅ all four requested-evidence items are [complete] in the PR body, matching the issue's four asks — named shared state with file:line, three consecutive concurrent full-suite passes, CI green, and the production-reachability statement.

Evidence-level reasoning and code check

This is a shared-state concurrency bug, so the right bar was a repro someone actually ran twice at once (not a screenshot, not a single-process test count) plus the named unit test. That's what's here: swift test --filter WorkspaceServiceTests (46 passed) covers the mechanical conversion, and the paired-process repro (rounds 5-7, Test run with 2157 tests in 279 suites passed, both checkouts, Suite "WorkspaceService" passed in all 14 logs) covers the actual contamination claim from #1536.

Reading the diff against the preferences field: it's a trailing-defaulted parameter on both WorkspaceService initializers (Sources/WorkspaceManagerCore/Services/WorkspaceService.swift), so WorkspaceService.shared in the app and CLI are unaffected — production still reads LaunchPreferences.defaults. The test-file diff shows every setWorkspacesRoot/restoreWorkspacesRoot call site replaced with a UUID-suite preferences.set(...), and the two old helpers plus the TODO(#1536 follow-up) are gone, matching the "8 tests still on UserDefaults.standard" gap I flagged last time.

Two independent static passes (recorded in the comment thread) landed on the same conclusion I reach reading the diff: no blocking findings, two minor ones correctly triaged (one accepted as documented risk, one deferred to #1611). I didn't re-run anything — no Swift toolchain here — but the reasoning in those passes is checkable against the diff and holds up.

@workspace-agents
workspace-agents Bot dismissed their stale review September 11, 2026 12:40

Superseded by subsequent approval from the same reviewer.

@workspace-agents workspace-agents Bot added the mergeable Agent-approved, ready for owner merge label Sep 11, 2026
@fairchild
fairchild merged commit f3682f5 into main Sep 12, 2026
85 of 87 checks passed
@fairchild
fairchild deleted the codex/april-clearwater-issue-1536-flaky-workspaceservicetests-contaminate-each-oth branch September 12, 2026 18:48
@fairchild fairchild removed the steward:look Steward asks Michael to look: passes every leg, waits on his click. Max five. label Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author:april PRs authored by the april agent mergeable Agent-approved, ready for owner merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: WorkspaceServiceTests contaminate each other under parallel load despite .serialized

1 participant