Asks JEV what testing a code change needs, so agents write the regression tests that matter and skip the ones that only add cost. Proposed in gitmoot/gitmoot#2265.
It sends JEV the diff and the nearest existing tests, and gets back one advice:
| advice | meaning |
|---|---|
new_regression_test |
Behavior changed and nothing nearby tests it: add a test that fails on the old code. |
extend_existing_test |
A nearby test already covers this area: add a case to it. |
one_off_check |
Real change, but a permanent test would be brittle or only restate code: run it once. |
no_permanent_test |
No behavior change: build and run the existing tests. |
need_context |
JEV could not decide. |
It also reports two probabilities:
- covered: whether the change's own tests would fail on a regression.
- low-value: whether an added test only restates code, duplicates another test, tests internals, or would pass on the old code.
It is advice, not a gate. The agent still proves the change works; see the skill.
JEV reading a diff cannot tell whether a test would pass on the old code. So
test-check prove runs it:
test-check prove --test "go test ./pkg -v -run '^{name}$'"- It runs each test the change adds or edits on the working tree. Each must pass.
- It puts every changed non-test file back to its base version and runs each test again. Each must fail.
- It restores the files.
{name} is filled in for each test. The tests are taken from the change's diff for Go and Python; pass --each A,B for other runners. The command must print each test's name (use a verbose runner); otherwise the test is reported as test_not_run. Every test must go red on its own. A change is not proven just because one good test sits next to a weak one.
Without {name}, the command runs once for the whole change, and one red test is enough.
Red/green only checks the tests you wrote. --pieces checks the code you changed:
test-check prove --list-tests # the tests this change adds or edits
test-check prove --pieces --test "go test ./pkg -run '^(TestA|TestB)$'"It undoes one changed piece of code at a time (one diff hunk, a whole added file, or a whole deleted file) and runs the command. The command must fail every time.
- A piece that leaves the command passing is reported
not_tested, and the outcome ispiece_not_tested(exit 4). - A compile/import-only failure is
inconclusive, notguarded, and makes the overall outcomeinconclusive(exit 4). - Comment- and whitespace-only hunks are skipped.
- Above
--max-pieces(default 60), nothing runs and the outcome istoo_many_pieces. - Each piece goes through the same save-first stash, so
--restorerecovers an interrupted run.
Treat the list as advice. On 20 past PRs that reviewers approved without test findings, 15 had at least one not_tested piece: often logging, wording or defaults that need no test.
Test files and fixtures keep their new versions for both runs. The reverted files are saved under the git dir before anything changes. An interrupted or crashed run is recovered with test-check prove --restore, and a new run refuses to start until then.
Only one prove run may use a checkout at a time: a lock under the git dir makes a second run, or --restore, refuse while the first is alive. A lock left by a dead process is taken over. A test that hangs on the old code until --timeout counts as red.
Outcomes:
proven: exit 0. Behavioral failure on old code, pass on new code.inconclusive: exit 4. The old run only showed a build/import failure or unavailable Python module API (reason: old_code_build_failed), or did not mention the selected test (reason: old_test_not_observed).not_red_on_old: exit 4. The test passes without the fix.fails_on_new: exit 4.no_test_in_change: exit 4.test_not_run: exit 4.no_code_in_change: exit 0. Only tests changed.- Caller cancellation or test-check's own command deadline: exit 1 after restoration, never proof. Interrupted per-test/piece rows remain inconclusive; a framework's own assertion or panic is still a test failure, not a tool deadline.
A build_error_suspected flag marks an old-code failure that looks like a compile or import error rather than a failed assertion. This is not proof: test through an interface the old code already had. Actual Go/Python assertion-failure summaries take precedence over diagnostic words in assertion messages. In per-test mode, any inconclusive result prevents an overall proven; new-code failures still report fails_on_new.
test-check # local checkout, incl. uncommitted files
test-check --repo OWNER/REPO --pr N # a pull request's current head
test-check --repo OWNER/REPO --compare BASE...HEAD
test-check --json # machine-readableExit codes:
0: advice printed.1: the change could not be read.2: usage error.3: JEV unavailable. No advice is printed; answer the skill's four questions yourself.
The GitHub modes use the gh CLI. The key comes from $OPENROUTER_API_KEY,
else the OPENROUTER_API_KEY line in ~/.config/gitmoot/keychain.env. The
default model is OpenRouter's rolling ~typesafe/jev-latest alias (including the
leading ~), not a pinned release. The System One endpoint is unchanged.
The response may name a concrete model; do not infer that release from the alias.
Advice and proof commands automatically save a concise receipt after the command finishes (and, for proof, after restoration). Help, list-tests, restore, and usage errors do not create receipts.
The default is $XDG_STATE_HOME/test-check/receipts, or $HOME/.local/state/test-check/receipts. Storage must be outside the checkout. Tool-owned directories are created with mode 0700, files with 0600; existing unsafe directories are refused, never chmodded. Each invocation has an independent directory containing receipt.json and its output_ref diagnostic link.
Receipts include mode, command, available repository/base/HEAD/tool revision, times, result, warnings, piece outcomes, and named-test execution evidence. Advice hashes its already-collected input to distinguish uncommitted snapshots without storing the diff or prompt. Proof streams the already-enumerated changed paths under its existing lock before running tests, hashing base/HEAD, framed path names, modes, deletions, link targets, and file bytes (64 MiB total limit). This is a changed-input identity, not a complete runtime-environment snapshot: ignored files and external inputs are not covered. Missing/unreadable/oversized input identity stays unknown with a warning, never changing the core proof result. observed means a verbose Go/Python runner marker was seen; unsupported runners or missing/truncated output are unknown, not proof a test was skipped. Whole-command named-test rows are aggregate_only, not individual proofs. Receipt metadata is evidence of a command, not an approval or quality judgment.
Diagnostics are bounded tails, not full raw logs: up to 2,000 bytes per output and 256 outputs, with truncation noted. No environment, credentials, configuration, prompts, or diffs are deliberately collected. Advice command options are retained, but --title values (including --title=...) are explicitly [redacted] because they become prompt content; that command is not fully replayable. Other command and runner output literals may still be sensitive; they are not automatically redacted or uploaded.
The receipt path is printed on stderr; JSON results add receipt and receipt_status. If saving fails, the original exit/outcome is preserved, stderr warns, and JSON contains receipt_status: failed and receipt_error, never a fake path. Controlled failures and interruptions can produce receipts after recovery; SIGKILL/power loss cannot guarantee one. Receipt writes never replace or remove prove's recovery stash.
- Old-code build/import-only failures now return
inconclusive(exit 4), including per-test and piece results. Diagnostic words in actual Go/Python assertion failures no longer imply build failure. - Advice and proof executions now save private, local receipts with command/result evidence and bounded diagnostic links. Receipt storage failures do not change proof/advice outcomes.
go build -o ~/.local/bin/test-check ./cmd/test-check
ln -s "$PWD/skills/test-check" ~/.claude/skills/test-check