test(ios): establish snapshot convergence baselines and permanent evidence

[thymikee]

Summary

This resolves #2189 with permanent, reproducible snapshot-convergence evidence and closes the packaging gaps found during review.

• Proxy and local measurements share fixture-anchor admission, so an unexpected screen cannot be recorded as a successful alert/system-surface cell.
• Host evidence records typed Mac model and CPU identity alongside OS, architecture, and tool versions.
• scripts/prepare-publish-assets.mjs is the single owner used by release packaging and size measurement. It prepares Apple runner source plus both Android helper components, removes stale output, and validates each APK, manifest, and checksum.
• Package evidence requires both runtime-used Android helpers and proves that scripts/** is absent from the npm tarball.
• Superseded 955ed and 0ae corpora were deleted. The final 71fb corpus is the only checked-in benchmark evidence.
• The branch is merged with current main; stale src/utils layering and mutation-registry ownership now follow the moved snapshot module instead of carrying exceptions.

Closes #2189

Validation

• Publication head: ea79642effad21b12930ef4682e5cb9fc786ce2c.
• Benchmark source: 71fb2483f30d90e615e949601c836aeebbf450c5.
• Final corpus: three schema-valid JSON artifacts with adjacent summaries; 24 local cells, 36 proxy cells, 1,080 samples, zero failures.
• Real clean package measurement: 1,013,805-byte tarball; 3,438,938 unpacked and clean-installed bytes across 465 files.
• Android runtime helpers: 43,220 bytes across exactly six files: APK, SHA-256, and manifest for snapshot and IME helpers.
• Packed scripts/**: zero files.
• pnpm build and the clean npm size-report/install measurement passed.
• pnpm check:affected --run passed all runnable checks on the repaired current-main tree. After integrating the contributor's final two test-only commits, all affected checks remained green except one unrelated 5-second host-contention timeout; that exact test passed immediately on the merged head in 2.449 seconds. The 17 directly overlapping package/workflow tests also passed.
• Exact-head GitHub checks are running and remain authoritative for native/device and final merge readiness.

Scope and design rationale

The shared preparation owner prevents release packaging and size evidence from measuring different asset sets. The checked-in 71fb files are the permanent benchmark corpus; superseded corpora are removed. None of the benchmark scripts or evidence is shipped in npm, while runtime-used helper assets are built, validated, and packed.

No merge was performed.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://callstack.github.io/agent-device/pr-preview/pr-2204/
Built to branch gh-pages at 2026-09-01 18:45 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.55 MB	2.48 MB	-66.3 kB
JS gzip	857.1 kB	834.6 kB	-22.5 kB
npm bundled raw	2.55 MB	2.48 MB	-66.3 kB
npm bundled gzip	857.1 kB	834.6 kB	-22.5 kB
npm tarball	981.2 kB	957.8 kB	-23.4 kB
npm unpacked	3.39 MB	3.32 MB	-69.0 kB
npm clean-installed	3.39 MB	3.32 MB	-69.0 kB

npm unpacked components

Component	Base	Current	Diff
JS / dist source	2.70 MB	2.64 MB	-68.7 kB
Apple runner source/project	581.2 kB	581.1 kB	-71 B
macOS helper source	54.8 kB	54.8 kB	0 B
Android helper artifacts	0 B	0 B	0 B
Other package files	45.7 kB	45.5 kB	-228 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	31.8 ms	30.0 ms	-1.8 ms
CLI --help	82.3 ms	80.4 ms	-1.9 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/perf-runtime-plan.js	+63.6 kB	+19.5 kB
dist/src/src2.js	+47.3 kB	+14.1 kB
dist/src/runner-client.js	+44.0 kB	+12.8 kB
dist/src/snapshot.js	+33.1 kB	+10.6 kB
dist/src/android.js	+26.0 kB	+9.1 kB

Top changed packed files

Packed file	Base	Current	Diff
dist/src/mechanics.js	154.3 kB	0 B	-154.3 kB
dist/src/device-claim-conflict.js	54.9 kB	0 B	-54.9 kB
dist/src/perf-runtime-plan.js	12.0 kB	63.6 kB	+51.7 kB
dist/src/src7.js	47.5 kB	0 B	-47.5 kB
dist/src/src2.js	661 B	47.3 kB	+46.6 kB
dist/src/runner-disposal.js	46.4 kB	0 B	-46.4 kB
dist/src/runner-client.js	41.8 kB	85.8 kB	+44.0 kB
dist/src/snapshot.js	2.6 kB	33.1 kB	+30.5 kB
dist/src/perf-facade.js	27.3 kB	0 B	-27.3 kB
dist/src/snapshot-helper.js	0 B	25.9 kB	+25.9 kB

[thymikee]

Reviewed exact head 30120e1845dfbde496f14f82cb3964068f9402ad: not ready; the evidence harness can currently certify evidence it did not actually establish.

1. Cell admission is not enforced. shutdownSimulator, terminateApp, and stopDaemon ignore non-timeout/nonzero failures and no postcondition verifies the requested cold-cold/cold/warm/relaunch state. Separately, every fixture declares anchorText, but the local runner never reads it, so a successful deep link to a default, stale, or unmounted surface can be recorded under list, nested-scroll, etc. Repair this through one typed cell-admission owner that establishes and verifies simulator/daemon/app-generation state and the semantic fixture anchor before samples; stop with a typed state/fixture failure on any mismatch rather than adding scattered best-effort checks.

2. The permanent iOS: skip private-AX deep extension on observation-only captures (settle, stabilization, corroboration) #1626 control is self-confirming: deep-button.ts constructs hard-coded shallow/full arrays and asserts the exact relationship it just created; deep-control.ts only relaunches that script. It cannot fail when the real deep fixture, acquisition output, or corroboration/comparison rule regresses. Keep the control implementation-independent, but drive an independently produced fixture artifact or the actual comparison policy using the real 72-level/deep-button shape, with the invalid shallow criterion observed red and safe full criterion green from those outputs.

3. --derived-path is passed to recursive fs.rmSync for every cold-cold sample without proving the path is benchmark-owned. A typo or broad existing path can delete unrelated data. Allocate/mark an owned benchmark directory and only clear a validated descendant; reject unowned/broad paths.

The real runner attempt correctly stopped without synthetic timings, but issue #2189 still requires completed local and proxy evidence after these fixes. This PR adds 3,961 lines across the harness; itemize the justified functional growth and record why a materially smaller design reusing existing lifecycle/command/schema owners was rejected, as required by the size-review threshold.

[thymikee]

Addressed in 4aebee2:

• Local cell admission now requires typed postconditions for simulator state, daemon liveness, app PID, and exact fixture anchors before a sample is recorded; failures stop with an explicit reason.
• Derived-data cleanup now proves benchmark ownership and rejects symlinks or paths outside the owned root.
• The deep-button control now consumes an independently checked-in 72-level fixture artifact; the invalid shallow observation remains red and the safe full observation remains green.

Validation at the exact PR head: affected checks passed, Coverage passed, and the core CI lanes are green. Native release/smoke lanes are still running; no device evidence is being claimed.

[thymikee]

Re-reviewed exact head 4aebee2edfed15c1365b3316f1461c009a38eea4. The new cell-admission owner now verifies simulator, daemon, app generation, and fixture anchors, and the 72-level artifact control is materially better. Three blockers remain:

1. Derived-path ownership is still forgeable. benchmark-config.ts accepts any existing caller-supplied --state-dir, creates/opens it, then ensureBenchmarkOwner writes the ownership marker when absent. clearDerivedData may consequently recursively delete any requested descendant of an unrelated broad directory that the caller just caused the tool to self-mark. Fix this at configuration ownership: allocate a fresh benchmark root internally, or accept only an already valid pre-marked root without auto-minting ownership for arbitrary existing directories; derive cleanup paths below it. Add planted-red CLI-level proof that an existing unmarked root and its descendant cannot be claimed or cleared.
2. test(ios): establish snapshot convergence baselines and permanent evidence #2189 acceptance remains incomplete. The only runner attempt correctly stopped with measurements: 0, and no completed proxy matrix is attached. Do not close or call the evidence foundation ready until runner-backed local and proxy raw evidence completes; the stopped artifact is honest blocker evidence, not the requested baseline.
3. The PR is now +5,467/-4 across 39 files. The body lists scope but still does not itemize justified functional growth or record why a materially smaller design reusing lifecycle/command/schema owners was rejected, as required by the size-review threshold.

Current completed checks are green, with native lanes still pending, but code/evidence blockers mean no readiness label. Re-review after the ownership root, completed matrices, and size rationale change.

[thymikee]

Re-reviewed exact head 562406bb38bc3bd14b1e8287add94ba68c36c84e: BLOCKED. The prior state-root remediation is sound: omitted roots are freshly allocated/marked; explicit roots must already be real, marked owners; descendant and symlink/realpath checks prevent unsafe claiming, and the planted-red CLI test proves an unmarked existing root cannot be claimed before cleanup.

The current owner-action blocker is Compatibility/Fallow: scripts/ios-snapshot-benchmark/run.test.ts imports ../../src/utils/exec.ts, which the benchmark scope cannot resolve. The latest --port change does not touch that import, so the failure will recur. Keep the plain .mjs benchmark CLI-process test’s child-process seam local to the benchmark owner (prefer execFile) or extract a small benchmark-owned helper; do not suppress/allowlist the unresolved dependency.

Readiness evidence is also stale/incomplete: the body is stamped at old 4aebee2…, its only runner attempt stopped with zero measurements, and no exact-head local/proxy raw matrix exists. Update the body for this head, attach the required exact-head matrices and itemized large-harness growth/smaller-design rationale, and wait for all exact-head CI including iOS Smoke. No readiness label yet.

[thymikee]

Re-reviewed exact head 747a939adbdf15332463736241ab140180b41d9f: the prior code blocker is fixed cleanly. The benchmark CLI-process test now keeps its execFileSync seam local, proxy startup parsing accepts the real pretty-JSON envelope, and the canonical ios:mobile:<udid> lease binding is coherent. No new code defect confirmed.

BLOCKED on #2189 completion/readiness evidence: attach current-head runner-backed local and proxy raw matrices; update the stale 4aebee2… body, 44-file scope, itemized growth, and rejected-smaller-design rationale; and wait for iOS Smoke/Release. The current Integration failure is the unrelated Android IME-record persistence test (160/161 provider-integration tests otherwise), so classify it as lane noise rather than an iOS benchmark finding.

[thymikee]

Re-reviewed exact head e9b1fc523278645a0d96e696db02779566bc57f2: BLOCKED by a new async CLI evidence defect. execFile timeout errors have no string error.code, so runCliAsync omits spawnErrorCode and classifyFailure falls to other, losing #2189’s required typed timeout evidence. Preserve a typed ETIMEDOUT/timeout result at the cli-process owner and add an adversarial async-timeout regression proving category timeout. Keep the remote-profile isolation design; do not add a fallback.

The two claimed raw matrices are only local /private/tmp paths, unavailable to reviewers; attach or link reviewable artifacts. The body/rationale and exact-head CI are otherwise sound.

[thymikee]

Addressed on exact publication head 8820e0ab4e899a2e64da3cce47898c192c99065d.

• runCliAsync now maps Node's timeout shape (killed: true, signal: SIGTERM, code: null) to typed ETIMEDOUT at the benchmark-owned process boundary.
• Added an adversarial child-process timeout regression. It fails against the pre-fix boundary with spawnErrorCode: undefined, and passes after the fix with failure category timeout.
• The exact measured source revision is 0ae8894dc5c6e5a63ad9a2499e7e10686f65edda; 8820e0ab4 is a data-only publication commit. The durable raw matrices are local JSON and proxy JSON, with Markdown summaries beside them.
• pnpm test:coverage:ci, pnpm check:affected --run, and pnpm check:coverage-changed pass locally; the current PR CI run will re-establish the publication head after this push.

The PR remains published-and-reported, not merge-ready, pending the authoritative native/device lanes and the unavailable external Claude audit.

[thymikee]

Re-reviewed exact head 8820e0ab4e899a2e64da3cce47898c192c99065d: the async timeout defect is fixed at the benchmark-owned process boundary, and the adversarial child-process regression proves Node's killed/SIGTERM/code:null shape remains typed as ETIMEDOUT and classifies as timeout. The committed local/proxy artifacts are reviewable and accurately schema-stamped.

Still BLOCKED on #2189's baseline corpus. The local artifact contains only warm/quiet/local/fresh-process-cli (20 samples); the proxy artifact contains only warm/quiet at approximately 0 ms RTT. The required cold-cold, cold, and relaunch states; list, nested-scroll, alert/system-surface, and XCTest-stress screens; 20 ms and 80 ms proxy RTT cells; and packed/clean-installed/bundled package-size evidence are absent (packageSize.status is not-run). Complete and attach those schema-valid raw cells, or record a deterministic stop with the exact command/toolchain diagnostic for each cell that cannot be produced. Current iOS/Android Smoke lanes are still in progress; no readiness label.

[thymikee]

Updated at exact publication head 9228dba25c707f0670f742ba48b037b4d5d90991.

• The benchmark-owned admission path now scrolls offscreen alert controls into view, uses a post-setup alert anchor, and dismisses successful relaunch alerts before the next sample.
• The exact measured source revision is 955ed760e4563f181db91ffe719ba74e827e21bf on bench-golden-v2 (iPhone 17 Pro, iOS 27.0).
• The committed corpus contains 24 completed local cells and 36 completed proxy cells, all with zero failures: cold/cold raw data, warm/relaunch raw data, and proxy raw data.
• Local pnpm test:coverage:ci passed (1,185 files / 8,754 tests; 88.68% statements, 80.73% branches, 91.73% functions, 90.65% lines), and pnpm check:coverage-changed passed.
• The pushed GitHub CI Coverage job is green, as are Lint & Format, Typecheck & Package, Integration Tests, Repo Guards, Compatibility & Provenance, Bundle Size, Linux, CodeQL, and preview checks. iOS/Android/macOS native checks are still running.

This remains published-and-reported, not merge-ready; no merge was performed.

[thymikee]

BLOCKED at exact head 9228dba25c707f0670f742ba48b037b4d5d90991. The #2189 corpus is materially complete, but proxy evidence lacks semantic fixture admission: persistent-client and fresh-CLI setup can successfully open/scroll/click yet time a stale or wrong surface because neither path verifies the expected fixture/post-setup anchor before sampling. Move preparation and typed semantic admission behind one adapter-neutral owner shared with local mode, require the exact anchor before timing, and add planted-red coverage for successful-looking setup with a missing anchor. Second, the raw evidence records OS/arch/tool versions and Simulator identity but not the host Mac model/CPU required by #2189 for comparable machine/toolchain baselines; add typed non-sensitive host identity to the schema and regenerate the matrices. Native CI lanes remain pending.

[thymikee]

Resolved the two exact-head blockers identified on 9228dba25c707f0670f742ba48b037b4d5d90991 and published at c1634edc74a4d7007e318430f960a4635de826a8.

• Shared adapter-neutral fixture preparation and typed semantic admission now cover local, persistent-client SDK, and fresh-process CLI paths. Exact fixture/post-setup anchors are required before timing, with typed fixture-anchor stops on mismatch.
• Raw results now require typed non-sensitive host identity (model, modelIdentifier, cpu, cpuCores), and the exact-head matrices were regenerated on MacBook Pro Mac16,8 / Apple M4 Pro / 12 cores.
• The checked-in local corpus has 24 completed cells and the proxy corpus 36 completed cells; all recorded failures are 0 and all eight raw artifacts validate against the updated schema.
• Local pnpm test:coverage:ci, pnpm check:coverage-changed, and final pnpm check:affected --run passed. The new GitHub workflow is pending, including native/device lanes; I did not wait for CI and did not merge.

[thymikee]

Re-reviewed exact head c1634edc74a4d7007e318430f960a4635de826a8: BLOCKED. The new shared proxy fixture-anchor admission and typed host identity resolve the prior exact-head blockers; the 71fb corpus is schema-valid (24 local + 36 proxy cells, 1,080 samples, zero failures).

1. Package evidence is still non-hermetic and not release-faithful. size-report.mjs prepares only Apple source and packs with scripts disabled; Size CI runs only pnpm build, while release build:package also constructs both Android helpers. The measured Android payload therefore depends on incidental worktree state (CI still reports 0 B), so the absolute packed/clean-installed numbers are not a reproducible publish artifact. Introduce one shared publish-asset preparation owner used by release and size evidence, with a planted regression requiring both Android helper components and excluding benchmark/scripts.

2. Remove the superseded evidence now. The PR retains 20,032 lines of 955ed/0ae corpora beside the valid 71fb corpus. The 955ed proxy data predates the new semantic admission contract, the 0ae data is preliminary, their summaries still say completed, and this head retrofits host fields into old raw captures rather than preserving immutable artifacts. Keep the final 71fb corpus; delete the superseded sets or move them to an explicitly excluded historical archive.

Exact-head CI was still running, so no readiness label.

[thymikee]

Addressed at exact head 509d0f31bba2a4f6cbfd3e6cdebffb15c20c79fb.

• Added the shared prepare-publish-assets owner used by both release packaging and size evidence. It prepares and validates Apple runner assets plus both Android helper components; package assertions reject missing helpers and any scripts/ entry.
• Added the planted package regression for both required Android helper sets and benchmark/build-script exclusion.
• Removed the superseded pre-admission 955ed and preliminary 0ae artifacts. The final 71fb corpus remains as the only checked-in benchmark corpus.
• Updated the PR body with the final corpus, current package-size validation, and coverage results. Local coverage and affected checks pass; native GitHub lanes remain pending and were not awaited.

[thymikee]

Reviewed and repaired the current exact head holistically.

• Merged current main and preserved the contributor's late 509d0f3 update without overwriting it.
• Kept one publish-asset owner for release and size measurement; added direct rebuilding/stale-output coverage and pinned the package-script construction path.
• Verified a real clean pack/install: 1,013,805-byte tarball, 3,438,938 unpacked/installed bytes, 465 files; Android helpers are exactly 43,220 bytes / six runtime files; scripts/** contributes zero packed files.
• Updated the size workflow contract to verify the complete relative-module closure preserved across the base checkout.
• Repaired current-main migration residue at the owning registries: retired src/utils is no longer a layering zone, and the mutation kernel now follows both files under src/snapshot/scroll-edge-state.
• Superseded 955ed and 0ae evidence is gone; only the final 71fb corpus remains.

Validation: pnpm build, real scripts/size-report.mjs, focused package/workflow/mutation suites, and the full affected gate passed on the repaired tree. On the final merged head, the broad run had one unrelated 5-second host-contention timeout in gesture-admission-parity; the exact test passed immediately in isolation in 2.449 seconds. Exact-head GitHub checks are now running.

Code review is clean and the branch is mergeable. No merge performed.