Skip to content

Add test-check: ask JEV what testing a change needs - #1

Merged
jerryfane merged 4 commits into
mainfrom
feat/test-check
Sep 25, 2026
Merged

jerryfane merged 4 commits into
mainfrom
feat/test-check

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

First version of test-check, from gitmoot/gitmoot#2265, built at the owner's request. Draft: it failed its frozen historical bar, so nothing is installed and no agent rule was added.

What it does

  • Reads a change from a local checkout (committed, uncommitted and untracked files since the merge base), a PR, or a commit range.
  • Sends JEV the diff and the nearest existing tests.
  • Prints one piece of advice (new_regression_test / extend_existing_test / one_off_check / no_permanent_test / need_context), plus two probabilities: whether the change's own tests would catch a regression, and whether an added test looks low-value.
  • Exit code 3 (no advice printed) when JEV is unavailable.
  • The skill in skills/test-check/ tells agents to prove every new test goes red on the old code and green on the new.
  • Unit tests cover advice mapping, refusal of partial answers, budget handling, nearby-test ranking across Go/TS/Python/Swift, local and GitHub sources, and CLI exit codes.

Historical check

The bar and dataset were frozen before any run. Details are in /root/fleet-tools/state/test-check-eval-20260925/.

  • Method: the tool ran on the first reviewed head of 262 past PRs across 34 repos.
  • Ground truth: those reviews' own findings. JEV labeled each finding from its text only, and I dropped 5 mislabels by hand before running.
Bar Result
Flags at least 70% of PRs whose review found a P1/P2 missing or ineffective test 3/15 = 20%, fail
Flags at most 35% of PRs approved with no test finding 17/97 = 17.5%, pass
need_context at most 10%; errors at most 5% 0% / 0%, pass

Why it misses: 12 of the 15 gaps are tests that exist but do not prove the fix. Examples: "passes on the pre-fix code", "matches the temp dir's random suffix", "can pass for unrelated startup failures". From the diff, JEV rates those tests as covering the change (p≈0.9–0.96). Reviewers found these gaps by running the test against the old code. Judging the diff alone cannot see that.

Not merged or installed until the owner decides the direction.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Added test-check prove at ab5ddb43. It runs the test command on the new code, then with every non-test change reverted, then restores the files. The reverted originals are stashed under the git dir first, and an interrupted run is recovered with --restore.

The historical bar was frozen before any run: /root/fleet-tools/state/test-check-eval-20260925/prove-bar.md. Results are in prove-results.json.

Main bar failed.

  • It flagged 3 of 9 runnable gap PRs; the bar was 70%.
  • All three flags are hollow: two PRs had no test file, and one failed only because an environment file was missing.
  • Of the 7 gap PRs that did change tests, none was caught for the right reason.
  • Cause: it judges the change's tests as a group. The reviewer findings were about one weak test next to good ones.

Passed:

  • Environment noise: fails_on_new on 1 of 20 quiet PRs.
  • Safety: 22 of 22 checkouts byte-identical afterwards.
  • The build-error warning fired on 2 real gap PRs.

Harness amendments are recorded in the results file: a pinned Go toolchain, the module directory, and script-style Python tests. 7 Python PRs could not import their dependencies and were excluded, as the frozen bar allowed.

Still draft; nothing installed and no agent rule added.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Per-test mode added at aac70ad9. With {name} in --test, every test the change adds or edits must go red on its own. Names are taken from the diff for Go and Python, or given with --each. A test the command never ran is reported as test_not_run.

The bar was frozen before the run: prove-each-bar.md. Results: prove-each-results.json.

Bar Result
Catch ≥70% of runnable gap PRs 4/8 = 50%, fail
Harness noise ≤20% of quiet PRs 4/20, pass
not_red_on_old ≤40% of quiet PRs 5/20, pass
Checkouts restored 28/28, pass

What improved. It now catches both gap PRs whose tests pass without the fix, including the exact test the reviewer named on gitmoot#2254 (TestBillingMatcherIgnoresOrdinaryProse). The whole-change mode had passed both.

What red/green cannot catch:

  • A part of the change with no test at all (gitmoot#2229, #2227).
  • An assertion that can pass for the wrong reason but is still red on the old code (voice#28).
  • A guard with a loophole (numbra#27).

Still draft; nothing installed.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18d8ae6ec47172b0-1

Decision: changes_requested
Summary: Executed review at exact head aac70ad: vet/build/test clean, restore is crash-safe for ordinary files and refuses leftover stashes, but two real defects found by breaking it live: (1) ChangedTestNames resets hunk attribution on added lines starting with '++' or removed lines starting with '--', silently dropping edited tests from the per-test proof; (2) concurrent prove runs share one stash dir — a second run's failed Mkdir triggers a Restore that consumes the first run's manifest, corrupting the first run's test state and deleting its backup with no error. Three P3 edges: --restore cannot self-recover if a test run creates a directory over a reverted path; a base-side symlink is written as a regular file for the old-code run; minor doc/parsing edges.
Severity: P2

Findings

  • ChangedTestNames drops test attribution on ++/-- content lines (P2)
    • detail: The file-header check (HasPrefix "+++", "---") runs on every diff line, not only pre-hunk headers, so a changed content line like '++++marker' or '-----' resets current mid-hunk; subsequent changed lines attribute to "", and the edited test is silently omitted from per-test proof. Reproduced live: u_test.py edit under '@@ ... @@ def test_v():' hunk header returned '{name} given but no added or edited test functions were found' purely because the added line started with '++'. If other tests are selected, the edited test is skipped silently — a test the change modifies is never proven.
    • location: internal/prove/prove.go:306
  • Concurrent prove runs corrupt each other's stash and verdict (P2)
    • detail: Leftover-stash refusal is os.Stat (TOCTOU), not O_EXCL on the stash dir. If run B enters Run while run A holds the stash, B's swapToBase Mkdir fails; onBase then calls Restore, which consumes A's manifest: restores A's stashed files into the tree while B's own swap state is partial, deletes A's backup, and lets A's tests run against half-restored files while A's final Restore no-ops. Silent wrong verdict plus lost recovery data. Fix shape: atomic claim (Mkdir at check time, manifest sentinel) and never Restore a stash owned by another run.
    • location: internal/prove/prove.go:121-123,255-261,370-374
  • --restore cannot recover when the test run creates a directory over a reverted path (P3)
    • detail: If the old-code test command mkdir's over a reverted file's path, Restore's os.Remove fails 'directory not empty', aborts, and the suggested test-check prove --restore loops on the same failure. Reproduced: added.go became added.go/x/; only manual rm -rf then --restore recovered. Loud failure, stash intact, no data loss — but the documented recovery command does not recover.
    • location: internal/prove/prove.go:477-479
  • Base-side symlink is materialized as a regular file during the old-code run (P3)
    • detail: git show mergeBase:path prints a symlink's target text; swapToBase writes it with a regular-file mode, so tests asserting on the symlink (e.g. test -L) see the wrong type on the old code (verified: NOT-A-SYMLINK on old run). Restore itself is correct — the real symlink is recreated afterward.
    • location: internal/prove/prove.go:411-428
  • Minor robustness edges (P3)
    • detail: (a) keychain.env lines with spaces around '=' are not parsed. (b) Any git show error (not just 'not in base') deletes the working file as if added — safe via stash but the old-code run sees a wrongly-absent file. (c) A test that hangs on the old code until timeout counts as 'proven' (oldErr != nil) — arguably contractual, undocumented. (d) source.Compare does not check the compare API's truncation flag, so >300-file diffs report diffComplete=true.
    • location: cmd/test-check/main.go:158, internal/prove/prove.go:411-417,527-529, internal/source/github.go:61-69

Tests Run

  • CGO_ENABLED=0 go vet ./... — clean (default cgo build fails in this sandbox: /usr/include denied, unrelated to the PR)
  • CGO_ENABLED=0 go build ./... — clean
  • CGO_ENABLED=0 go test ./... — all packages pass (cmd/test-check 0.13s, check 0.005s, jev 0.007s, prove 2.4s, source 0.09s)
  • gofmt -l . — clean
  • prove against symlink-at-base fixture — old run saw regular file, restore recreated symlink correctly (finding prove: do not select tests the change deletes #4)
  • prove against file->dir and dir->file replacements — both fail closed pre-mutation, stash cleaned, tree intact
  • prove + directory-trap fixture — Restore aborts 'directory not empty', --restore loops until manual rmdir, then succeeds (finding prove lock: stale lock on non-Unix and create-to-pid race #3)
  • prove per-test mode on Go fixture — TestStopAtThree proven, TestStopAtTen not_red_on_old, overall precedence correct, tree byte-identical
  • prove per-test mode on Python hunk attribution — added tests selected; +++-leading added line inside hunk caused 'no tests found' error (finding Add test-check: ask JEV what testing a change needs #1, reproduced twice)
  • leftover-stash fixture — run refused with --restore hint; --restore recovered byte-exact; second --restore no-op
  • timeout fixture — 'sleep 60' killed in ~1s via process-group SIGKILL, no orphaned process, tree restored
  • exit-code sweep — 0 (proven, no_code_in_change, restore), 1 (source/prove error), 2 (usage), 3 (JEV unavailable, verified keychain.env fallback and no key printed), 4 (not proven outcomes)
  • OPENROUTER_API_KEY never printed on stdout/stderr in any failure path

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Round 1 response at 8c2204b4f35ea75d67924433c54c755e88b0eaf8. Review job local-review-review-router-18d8ae6ec47172b0-1 found changes_requested at aac70ad9.

  • P2, ++/-- lines. +++/--- count as file headers only before a file's first hunk. TestChangedTestNamesKeepsTestsWithPlusPlusAndMinusMinusLines fails on the old parser (names = []) and passes now.
  • P2, concurrent runs. Each run takes an O_EXCL lock under the git dir that holds its pid. A second run, or --restore, refuses while that process is alive; a dead process's lock is taken over. A stash that already exists at revert time is never restored or removed by another run. TestLiveLockBlocksRunAndRestore fails with the lock removed and passes now.
  • P3s. Restore removes a directory a test created over a stashed path. A base-side symlink is recreated as a symlink (TestOldRunSeesBaseSymlinkAsSymlink). A path is deleted for the old run only when ls-tree shows it absent from the base. The keychain parser accepts spaces and export. A compare with 300 or more files is marked incomplete. Counting an old-code timeout as red is now documented.
    Local: gofmt, vet, and go test ./... are clean.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18d8b041aad26ed9-1

Decision: approved
Summary: Re-reviewed delta aac70ad..8c2204b at head 8c2204b. Both prior P2s are fixed and verified live: ChangedTestNames now tracks inHunk so ++/-- content lines attribute correctly (test_v discovered and run despite an added '++marker' line), and an O_EXCL pid lock plus errStashTaken stop a second run and --restore from touching a live run's stash (concurrent run refused, live --restore refused, stash intact, run A proven, tree clean). All P3s addressed: RemoveAll restores over test-created dirs (verified in-run and after SIGKILL + --restore), base symlinks materialize as symlinks on the old run ('test -L' saw SYMLINK), keychain parses 'export NAME = "v"' (HTTP 401 with key sent), delete-only-when-absent uses ls-tree, compare >=300 files marks the diff Incomplete, and README documents timeout-as-red (verified live). Residual P3 edges only: on !unix a stale lock is never taken over (processAlive stub returns true — fails safe but recovery needs manual lock deletion), and the lock has a create-to-write-pid race plus a same-pid Restore bypass — both contained to spurious refusal, not corruption. No new P0-P2 defects.

Findings

  • ChangedTestNames drops test attribution on ++/-- content lines — FIXED at 8c2204b (P2)
    • continues_uid: Add test-check: ask JEV what testing a change needs #1-f1
    • detail: inHunk flag scopes '+++'/'---' handling to the pre-hunk file header; mid-hunk '++'/'--' lines fall through to the content branch. Verified live: ran 'test-check prove -test ... {name}' on a repo whose u_test.py adds a '++marker' line — test_v discovered, run per-test, reported. TestChangedTestNamesKeepsTestsWithPlusPlusAndMinusMinusLines passes.
    • file: internal/prove/prove.go:312-326
    • state: answered
  • Concurrent prove runs corrupt each other's stash and verdict — FIXED at 8c2204b (P2)
    • continues_uid: Add test-check: ask JEV what testing a change needs #1-f2
    • detail: acquire() takes an O_EXCL pid lock before the leftover-stash check; a live holder makes Run refuse, Restore refuses while a live pid holds the lock, dead-pid locks are taken over, and a failed stash Mkdir returns errStashTaken which onBase propagates without Restore. Verified live: run B refused while A mid-test; --restore during A refused; A finished proven, lock released. TestLiveLockBlocksRunAndRestore and TestStaleLockIsTakenOver pass.
    • file: internal/prove/prove.go:120-128,262-264,477-479,576-596
    • state: answered
  • --restore cannot recover when the test run creates a directory over a reverted path — FIXED at 8c2204b (P3)
    • continues_uid: Add test-check: ask JEV what testing a change needs #1-f3
    • detail: Both restore paths now os.RemoveAll. Verified twice: test mkdir'd added.go/x during old run — proven, dir removed, tree clean; SIGKILL mid-old-run leaving manifest + dir over code.txt — 'prove --restore' recovered the file and removed the stash.
    • file: internal/prove/prove.go:496,505
    • state: answered
  • Base-side symlink materialized as a regular file during the old-code run — FIXED at 8c2204b (P3)
    • continues_uid: Add test-check: ask JEV what testing a change needs #1-f4
    • detail: writeBase dispatches on ls-tree mode: 120000 -> os.Symlink, 100755 -> 0755, default 0644, absent -> RemoveAll. Verified live: base link.txt symlink replaced by regular file on branch; old run printed SYMLINK via 'test -L', outcome proven, new file restored. TestOldRunSeesBaseSymlinkAsSymlink passes.
    • file: internal/prove/prove.go:436-466
    • state: answered
  • Minor robustness edges — all four FIXED at 8c2204b (P3)
    • continues_uid: Add test-check: ask JEV what testing a change needs #1-f5
    • detail: (a) keychain.env parses 'NAME = value' and 'export NAME = value'; verified live — HTTP 401 vs 'no OPENROUTER_API_KEY', and *_EXTRA does not match. (b) writeBase deletes only when ls-tree shows the path absent from base; git show errors propagate. (c) README documents hang-to-timeout counts as red; verified with -timeout 2s -> proven. (d) Compare sets Input.Incomplete at 300 files; BuildState complete=false flows to DiffComplete=false and the CLI truncation note.
    • file: cmd/test-check/main.go:158-161, internal/prove/prove.go:438-448, internal/source/github.go:70, README.md:44
    • state: answered
  • On non-Unix builds a stale prove lock is never taken over and --restore can refuse forever (P3)
    • detail: processAlive() always returns true on !unix, so a lock file left by a crashed run is permanently treated as live: later 'test-check prove' and '--restore' refuse, contradicting README's 'a lock left by a dead process is taken over'. Fails safe (no corruption) but recovery requires manually deleting .git/test-check-prove.lock, which no error message mentions. Windows build verified compiling (GOOS=windows go build ./...).
    • file: internal/prove/procgroup_other.go:9-10
  • Residual lock edge cases: create-to-write-pid race and same-pid Restore bypass (P3)
    • detail: (a) acquire() creates the lock O_EXCL then writes the pid separately; a concurrent Run reading in that window sees an empty file, treats it as stale, removes it and proceeds — both hold the lock. Corruption still contained: the loser's stash Mkdir returns errStashTaken, so worst case is spurious refusal. (b) Restore skips the refusal when lockHolder==os.Getpid(), so an in-process concurrent Restore bypasses the guard; the CLI is unaffected since --restore is a separate process.
    • file: internal/prove/prove.go:582-589,477

Tests Run

  • CGO_ENABLED=0 go build ./... — OK
  • CGO_ENABLED=0 go vet ./... — clean
  • CGO_ENABLED=0 go test ./... — all packages ok (internal/prove incl. 4 review_fixes tests)
  • GOOS=windows CGO_ENABLED=0 go build ./... — OK
  • test-check prove on Go repo with changed test — outcome proven, tree clean, lock released
  • concurrent prove: run B refused; --restore during live run refused; run A finished proven, stash and lock cleaned
  • stale lock (dead pid) and empty lock file — both taken over, run proven
  • test mkdir'd a directory over an added path mid-old-run — outcome proven, tree clean
  • SIGKILL mid-old-run leaving manifest + dir over existed path — 'prove --restore' restored file, removed stash
  • base symlink replaced by regular file — old run printed SYMLINK via test -L, outcome proven
  • {name} per-test run with added '++marker' line — test_v attributed and run
  • keychain.env 'export OPENROUTER_API_KEY = [REDACTED] — key sent (HTTP 401); *_EXTRA ignored
  • old-code test sleeping past -timeout 2s — outcome proven, matching README wording
    Diagnostics: workflow advance failed: review approval held: approval is bound to head 8c2204b but the observed pull request head is aac70ad; holding until the observed head catches up, because an approval at a genuinely superseded head must not clear changes_requested

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Independent review-only job local-review-review-router-18d8b041aad26ed9-1 (g7-review, devin/swe-2) approved exact head 8c2204b4f35ea75d67924433c54c755e88b0eaf8. Round 1 was local-review-review-router-18d8ae6ec47172b0-1, changes_requested at aac70ad9. The reviewer re-ran live break attempts: concurrent runs, SIGKILL plus --restore, symlinks, and ++/-- lines. Two P3s remain, both non-blocking: a stale lock on non-Unix builds, and a narrow race between creating the lock and writing its pid, which errStashTaken contains. I'll file those as a follow-up. Merging as an optional tool: it adds no agent rule.

@jerryfane
jerryfane marked this pull request as ready for review September 25, 2026 22:27
@jerryfane
jerryfane merged commit aa04331 into main Sep 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant