Skip to content

prove --pieces: list changed code no test checks (advisory) - #2

Merged
jerryfane merged 2 commits into
mainfrom
feat/prove-pieces
Sep 26, 2026
Merged

jerryfane merged 2 commits into
mainfrom
feat/prove-pieces

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Adds test-check prove --pieces and --list-tests, stacked on #1. Draft: the combined check failed its frozen noise bar, so the piece check stays advisory.

  • --pieces undoes one changed piece of code at a time and runs the tests; a piece that leaves them passing is not_tested. A piece is one diff hunk, one added file, or one deleted file.
  • Comment- and whitespace-only hunks are skipped.
  • Above --max-pieces (default 60), nothing runs.
  • Every undo goes through the same save-first stash and the same lock as Add test-check: ask JEV what testing a change needs #1.
  • Tests check the hunk arithmetic against git's own diffs, and cover the untested-piece report and the limit refusal. Both mutants (undo writes nothing; no restore) fail the tests.

Historical result: bar prove-pieces-bar.md, results prove-pieces-results.json in /root/fleet-tools/state/test-check-eval-20260925/.

Bar Result
Catch ≥70% of gap PRs 7/8, pass (per-test alone: 4/8)
Flags ≤50% of quiet PRs 15/20, fail
Piece-run fails_on_new ≤20% 0/28, pass
Checkouts restored 28/28, pass

It finds the untested code reviewers complained about. But most approved PRs also have some untested piece, so as a gate it would block most work. The skill lists it as an optional step, and the README says to treat it as advice.

@jerryfane
jerryfane changed the base branch from feat/test-check to main September 25, 2026 22:28
@jerryfane

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #1 merged (now 2 commits on top of it). Still draft: the combined check failed its noise bar (15/20 approved PRs flagged, limit 10). Its skill and README text make the piece check optional advice, not a pass/fail step. Owner decision needed before this merges.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18d8cada7e9bf07c-1

Decision: approved
Summary: Executed review at exact head 0668968 (diff aa04331..0668968). vet/build/test all clean at head. Piece-mode mechanics verified live in throwaway repos: hunk undo arithmetic produces exactly one reverted hunk (replace/insert/delete/no-final-newline cases covered by fileHunks tests using git's own -U0 output); comment-only hunks skipped; added files, deleted files, added symlinks, and CRLF files each become correct pieces; --max-pieces refuses before any run; {name} in --test is rejected. Crash-safety verified by killing the run: SIGKILL mid-piece left a reverted file + manifest; a second run refused on the leftover stash and --restore refused while the killed pid was still a zombie, then once reaped --restore byte-restored the file and removed the stash; SIGINT during an applied piece exited 1 with the tree restored and no stash/lock leftovers. --list-tests prints the changed tests. Exit codes: 4 for piece_not_tested/too_many_pieces, 0 for proven. SKILL.md/README correctly keep --pieces advisory. Prior-verdicts file lists only PR #1 verdicts; no prior findings exist for PR #2 to answer. Two residual silent-skip gaps (advisory-level; they fail toward omission, not corruption): a modified binary code file yields no piece, and a JS/TS line starting with '#' (private field) is classified comment-only so its hunk is skipped; in both cases the outcome can still say proven.

Findings

  • Modified binary code files are silently skipped in piece mode. Observed live: repo with main bin.go='x\x00old' -> feat bin.go='x\x00new' plus a guarded text change: prove --pieces reported only 'guarded a.go:1' with outcome proven, exit 0; the bin.go change was never exercised. Added binary files do get a '(new file)' piece; only modifications are skipped. Suggest detecting binary diffs and emitting an unverifiable/skip marker or treating the whole file as one piece. (P3)
    • file: internal/prove/pieces.go
    • locator: codePieces -> fileHunks: git diff -U0 mergeBase -- p emits 'Binary files ... differ' with no hunks, so a modified binary code file produces zero pieces and is silently untested; outcome can still be proven
  • commentOnly treats '#' as a comment prefix for every language, misclassifying real JS/TS syntax (class private fields '#count = 0;', and similarly '--x' decrement via the '--' prefix). Observed live: a diff adding '+ #count = 0;' produced no piece for that line; outcome was still proven. Narrow '#' to Python-ish extensions or require '^#\s'. (P3)
    • file: internal/prove/pieces.go
    • locator: commentOnly(): '#' prefix matches JS/TS private-field declarations such as '#count = 0;', so a hunk that only adds such code is dropped as comment-only

Tests Run

  • CGO_ENABLED=0 go vet ./... && go build ./... && go test ./... at head 0668968 -> all 5 packages ok (a first attempt without CGO_ENABLED=0 failed on sandbox /usr/include permission, not the code)
  • fixture repo (feat: n>3->n>=3 guarded + n>10->n>=10 untested): test-check prove --pieces --test 'go test -count=1 .' -> outcome piece_not_tested, 'guarded a.go:3' + 'not_tested a.go:6', exit 4, tree clean
  • same fixture: prove --pieces --test 'go test -run {name} .' -> 'piece mode runs the test command as given; drop {name}', exit 1
  • fixture with added file, deleted file, CRLF edit, changed symlink, added binary .go, added symlink: piece run -> guarded a.go/crlf.go/gone.go/newfile.go pieces, not_tested new binary files, changed symlink skipped, exit 4, git status clean and CRLF bytes intact
  • SIGKILL mid-piece (manifest.json + a.go reverted observed): second prove run refused 'a previous prove run did not restore its files'; --restore initially refused while the killed pid was still a zombie ('prove run (pid 3985414) is still using this checkout'), then succeeded once reaped: exit 0, a.go byte-restored, stash gone
  • SIGINT during an applied piece: 'context canceled', exit 1, a.go restored to n>=3, no stash or lock leftovers
  • leftover stash from SIGKILL: prove --restore -> 'restored', exit 0, manifest consumed
  • prove --pieces --max-pieces 1 -> outcome too_many_pieces, test command never executed, exit 4
  • prove --list-tests -> 'TestGuarded', exit 0
  • modified-binary fixture: prove --pieces -> proven, exit 0, bin.go change skipped entirely (finding 1); added-symlink piece guarded and restored as a symlink ('test -L' true)

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane jerryfane closed this Sep 26, 2026
@jerryfane jerryfane reopened this Sep 26, 2026
@jerryfane

Copy link
Copy Markdown
Contributor Author

Independent review-only job local-review-review-router-18d8cada7e9bf07c-1 (g7-review, devin/swe-2) approved exact head 0668968831002b3b7afbd4f6d034004c3d5fd02a. The reviewer ran live break attempts, including SIGKILL mid-piece then --restore, CRLF files, and symlinks. CI is green at this head. Two P3s (binary code files skipped, #/-- comment heuristic) are filed as a follow-up. Merging as advisory, at the owner's request.

@jerryfane
jerryfane merged commit 470d632 into main Sep 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant