ci: run the bench Playwright suite, so the gates actually gate

[blove]

The browser tests that prove eviction works were run by nothing

grep -rn "bench" .github/workflows/     →  ZERO hits

apps/bench's test script is vitest run --environment jsdom, which collects src/, not tests/. So the 28 Playwright specs in apps/bench/tests/ — the only proof several shipped features work at all, including every eviction gate, the windowed-geometry gate, the AG Grid wrapped auto-height spec and the jsdom wrapped-scale fence — ran only when a human ran them by hand.

They were repeatedly described as "gates" during the eviction work. They gated nothing.

The job

bench-e2e — "Bench — Playwright (real Chromium)" — in ci.yml, on PRs to main and pushes to main.

Build the bench, serve vite preview on 4173 with a readiness loop, run pnpm bench:e2e -- --workers=1 with PRETABLE_BENCH_EXTERNAL_SERVER=1, upload the server log and playwright-report/ + test-results/ on failure.

It deliberately needs: nothing, and nothing needs it. The deploy jobs' needs: lists gate the website deploy; apps/bench is private and never deployed. Putting it there would mean a browser flake on main skips the production deploy and strands prod on an older commit — the exact failure deploy-prod-alarm exists to catch, and one that already happened twice this week. dev-smoke and smoke-preview sit outside those lists for the same reason.

One manual step remains: the check name Bench — Playwright (real Chromium) must be added to the required checks on main. Until then the job runs and reports but does not block a merge.

Nothing excluded, no PR/main split

The full suite is 28 tests in 16.0s at one worker. The only numeric budget in the directory is resident-cap-memory.spec.ts's 32 MB whole-page heap ceiling, measuring 12.66 MB — 2.5× under, on heap size rather than wall time. Everything else is layout geometry, DOM structure, or metric shape (bench.spec.ts asserts expect.any(Number), no timing budgets). Splitting would have cost coverage and bought nothing.

--workers=1 because ag-grid-wrap-auto-height.spec.ts polls a 15s settle budget; two workers on a 2-core runner is the only plausible way to race it. Costs ~10s.

Explicit build + external server rather than the config's webServer: enable-pre-post-scripts is on, so prepreview:bench runs the whole dependency build — 11.74s warm, measured — inside the webServer's 30s readiness budget. On a cold runner that reads as "server never came up", and a broken build reads as a timeout rather than a compile error.

A trap found and fixed rather than excluded

Setting Playwright's trace globally breaks bench.spec.ts, which drives context.tracing by hand to write its trace zip into status/traces/:

Error: tracing.start: Tracing has been already started
> 93 |   await page.context().tracing.start({

Under trace: "on-first-retry" that makes every retry of that spec a guaranteed failure, so a retry could never recover a genuinely flaky run. Fixed with a file-level test.use({ trace: "off" }) — no assertion touched — verified to override even an explicit --trace=on.

The workflow also carries a warning against adding PRETABLE_BENCH_ADAPTER/SCENARIO/SCALE/SCRIPT to the job env: resident-cap-memory.spec.ts test.skips itself when any of the four is set, so a convenience selector would silently drop a test while the job stayed green.

Proof the gate actually gates

A CI job that passes because it ran nothing is worse than no job.

Count matches. --list → Total: 28 tests in 9 files. The job's own command → Running 28 tests using 1 worker → 28 passed (15.9s).

It cannot pass on an empty collection. testDir pointed at a nonexistent directory → Error: No tests found / EXIT=1.

Break one assertion, then run the exact command the job runs with CI=1:

EXIT CODE = 1
Running 28 tests using 1 worker
  1 failed
  27 passed (18.4s)

Error: returned: the cursor is on the same row
Expected: "row-15009"
Received: "row-5010"

Restored → EXIT CODE = 0, 28 passed (17.8s). Retries and artifacts both produced trace zips and failure screenshots.

Cost

16.0s locally at one worker; ~3–5 min in CI dominated by pnpm install, playwright install and a cold bench build. Comparable to dev-smoke, fully parallel with the other jobs, no added critical path.

Adjacent hole, filed not fixed

apps/bench/tsconfig.json includes only ["src", "vite.config.ts"] and the lint script is eslint src, so those 9 specs are neither typechecked nor linted — Playwright transpiles with esbuild and never checks types. CI now runs them; nothing checks them.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
pretable	Ignored		Aug 17, 2026 2:29am

[github-actions]

Vercel preview ready

Preview: https://pretable-m6ocyf21t-cacheplane.vercel.app
Commit: a66b972af8144cb56d480d14bde5a13895a19250

_{Updated automatically by the deploy-preview job.}