You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
prove --pieces: list changed code no test checks (advisory) - #2
Adds test-check prove --pieces and --list-tests, stacked on #1. Draft: the combined check failed its frozen noise bar, so the piece check stays advisory.
--pieces undoes one changed piece of code at a time and runs the tests; a piece that leaves them passing is not_tested. A piece is one diff hunk, one added file, or one deleted file.
Tests check the hunk arithmetic against git's own diffs, and cover the untested-piece report and the limit refusal. Both mutants (undo writes nothing; no restore) fail the tests.
Historical result: bar prove-pieces-bar.md, results prove-pieces-results.json in /root/fleet-tools/state/test-check-eval-20260925/.
Bar
Result
Catch ≥70% of gap PRs
7/8, pass (per-test alone: 4/8)
Flags ≤50% of quiet PRs
15/20, fail
Piece-run fails_on_new ≤20%
0/28, pass
Checkouts restored
28/28, pass
It finds the untested code reviewers complained about. But most approved PRs also have some untested piece, so as a gate it would block most work. The skill lists it as an optional step, and the README says to treat it as advice.
Rebased onto main after #1 merged (now 2 commits on top of it). Still draft: the combined check failed its noise bar (15/20 approved PRs flagged, limit 10). Its skill and README text make the piece check optional advice, not a pass/fail step. Owner decision needed before this merges.
Decision:approved Summary: Executed review at exact head 0668968 (diff aa04331..0668968). vet/build/test all clean at head. Piece-mode mechanics verified live in throwaway repos: hunk undo arithmetic produces exactly one reverted hunk (replace/insert/delete/no-final-newline cases covered by fileHunks tests using git's own -U0 output); comment-only hunks skipped; added files, deleted files, added symlinks, and CRLF files each become correct pieces; --max-pieces refuses before any run; {name} in --test is rejected. Crash-safety verified by killing the run: SIGKILL mid-piece left a reverted file + manifest; a second run refused on the leftover stash and --restore refused while the killed pid was still a zombie, then once reaped --restore byte-restored the file and removed the stash; SIGINT during an applied piece exited 1 with the tree restored and no stash/lock leftovers. --list-tests prints the changed tests. Exit codes: 4 for piece_not_tested/too_many_pieces, 0 for proven. SKILL.md/README correctly keep --pieces advisory. Prior-verdicts file lists only PR #1 verdicts; no prior findings exist for PR #2 to answer. Two residual silent-skip gaps (advisory-level; they fail toward omission, not corruption): a modified binary code file yields no piece, and a JS/TS line starting with '#' (private field) is classified comment-only so its hunk is skipped; in both cases the outcome can still say proven.
Findings
Modified binary code files are silently skipped in piece mode. Observed live: repo with main bin.go='x\x00old' -> feat bin.go='x\x00new' plus a guarded text change: prove --pieces reported only 'guarded a.go:1' with outcome proven, exit 0; the bin.go change was never exercised. Added binary files do get a '(new file)' piece; only modifications are skipped. Suggest detecting binary diffs and emitting an unverifiable/skip marker or treating the whole file as one piece. (P3)
file: internal/prove/pieces.go
locator: codePieces -> fileHunks: git diff -U0 mergeBase -- p emits 'Binary files ... differ' with no hunks, so a modified binary code file produces zero pieces and is silently untested; outcome can still be proven
commentOnly treats '#' as a comment prefix for every language, misclassifying real JS/TS syntax (class private fields '#count = 0;', and similarly '--x' decrement via the '--' prefix). Observed live: a diff adding '+ #count = 0;' produced no piece for that line; outcome was still proven. Narrow '#' to Python-ish extensions or require '^#\s'. (P3)
file: internal/prove/pieces.go
locator: commentOnly(): '#' prefix matches JS/TS private-field declarations such as '#count = 0;', so a hunk that only adds such code is dropped as comment-only
Tests Run
CGO_ENABLED=0 go vet ./... && go build ./... && go test ./... at head 0668968 -> all 5 packages ok (a first attempt without CGO_ENABLED=0 failed on sandbox /usr/include permission, not the code)
same fixture: prove --pieces --test 'go test -run {name} .' -> 'piece mode runs the test command as given; drop {name}', exit 1
fixture with added file, deleted file, CRLF edit, changed symlink, added binary .go, added symlink: piece run -> guarded a.go/crlf.go/gone.go/newfile.go pieces, not_tested new binary files, changed symlink skipped, exit 4, git status clean and CRLF bytes intact
SIGKILL mid-piece (manifest.json + a.go reverted observed): second prove run refused 'a previous prove run did not restore its files'; --restore initially refused while the killed pid was still a zombie ('prove run (pid 3985414) is still using this checkout'), then succeeded once reaped: exit 0, a.go byte-restored, stash gone
SIGINT during an applied piece: 'context canceled', exit 1, a.go restored to n>=3, no stash or lock leftovers
Independent review-only job local-review-review-router-18d8cada7e9bf07c-1 (g7-review, devin/swe-2) approved exact head 0668968831002b3b7afbd4f6d034004c3d5fd02a. The reviewer ran live break attempts, including SIGKILL mid-piece then --restore, CRLF files, and symlinks. CI is green at this head. Two P3s (binary code files skipped, #/-- comment heuristic) are filed as a follow-up. Merging as advisory, at the owner's request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
test-check prove --piecesand--list-tests, stacked on #1. Draft: the combined check failed its frozen noise bar, so the piece check stays advisory.--piecesundoes one changed piece of code at a time and runs the tests; a piece that leaves them passing isnot_tested. A piece is one diff hunk, one added file, or one deleted file.--max-pieces(default 60), nothing runs.Historical result: bar
prove-pieces-bar.md, resultsprove-pieces-results.jsonin/root/fleet-tools/state/test-check-eval-20260925/.fails_on_new≤20%It finds the untested code reviewers complained about. But most approved PRs also have some untested piece, so as a gate it would block most work. The skill lists it as an optional step, and the README says to treat it as advice.