Skip to content

About

Asks JEV whether a code change needs a regression test, a one-off check, or no test (gitmoot/gitmoot#2265)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

Repository files navigation

test-check

Asks JEV what testing a code change needs, so agents write the regression tests that matter and skip the ones that only add cost. Proposed in gitmoot/gitmoot#2265.

It sends JEV the diff and the nearest existing tests, and gets back one advice:

advice meaning
new_regression_test Behavior changed and nothing nearby tests it: add a test that fails on the old code.
extend_existing_test A nearby test already covers this area: add a case to it.
one_off_check Real change, but a permanent test would be brittle or only restate code: run it once.
no_permanent_test No behavior change: build and run the existing tests.
need_context JEV could not decide.

It also reports two probabilities:

  • covered: whether the change's own tests would fail on a regression.
  • low-value: whether an added test only restates code, duplicates another test, tests internals, or would pass on the old code.

It is advice, not a gate. The agent still proves the change works; see the skill.

Prove a test goes red

JEV reading a diff cannot tell whether a test would pass on the old code. So test-check prove runs it:

test-check prove --test "go test ./pkg -v -run '^{name}$'"
  1. It runs each test the change adds or edits on the working tree. Each must pass.
  2. It puts every changed non-test file back to its base version and runs each test again. Each must fail.
  3. It restores the files.

{name} is filled in for each test. The tests are taken from the change's diff for Go and Python; pass --each A,B for other runners. The command must print each test's name (use a verbose runner); otherwise the test is reported as test_not_run. Every test must go red on its own. A change is not proven just because one good test sits next to a weak one.

Without {name}, the command runs once for the whole change, and one red test is enough.

Which changed pieces no test checks (advisory)

Red/green only checks the tests you wrote. --pieces checks the code you changed:

test-check prove --list-tests                       # the tests this change adds or edits
test-check prove --pieces --test "go test ./pkg -run '^(TestA|TestB)$'"

It undoes one changed piece of code at a time (one diff hunk, a whole added file, or a whole deleted file) and runs the command. The command must fail every time.

  • A piece that leaves the command passing is reported not_tested, and the outcome is piece_not_tested (exit 4).
  • A compile/import-only failure is inconclusive, not guarded, and makes the overall outcome inconclusive (exit 4).
  • Comment- and whitespace-only hunks are skipped.
  • Above --max-pieces (default 60), nothing runs and the outcome is too_many_pieces.
  • Each piece goes through the same save-first stash, so --restore recovers an interrupted run.

Treat the list as advice. On 20 past PRs that reviewers approved without test findings, 15 had at least one not_tested piece: often logging, wording or defaults that need no test.

Test files and fixtures keep their new versions for both runs. The reverted files are saved under the git dir before anything changes. An interrupted or crashed run is recovered with test-check prove --restore, and a new run refuses to start until then.

Only one prove run may use a checkout at a time: a lock under the git dir makes a second run, or --restore, refuse while the first is alive. A lock left by a dead process is taken over. A test that hangs on the old code until --timeout counts as red.

Outcomes:

  • proven: exit 0. Behavioral failure on old code, pass on new code.
  • inconclusive: exit 4. The old run only showed a build/import failure or unavailable Python module API (reason: old_code_build_failed), or did not mention the selected test (reason: old_test_not_observed).
  • not_red_on_old: exit 4. The test passes without the fix.
  • fails_on_new: exit 4.
  • no_test_in_change: exit 4.
  • test_not_run: exit 4.
  • no_code_in_change: exit 0. Only tests changed.
  • Caller cancellation or test-check's own command deadline: exit 1 after restoration, never proof. Interrupted per-test/piece rows remain inconclusive; a framework's own assertion or panic is still a test failure, not a tool deadline.

A build_error_suspected flag marks an old-code failure that looks like a compile or import error rather than a failed assertion. This is not proof: test through an interface the old code already had. Actual Go/Python assertion-failure summaries take precedence over diagnostic words in assertion messages. In per-test mode, any inconclusive result prevents an overall proven; new-code failures still report fails_on_new.

Use

test-check                                   # local checkout, incl. uncommitted files
test-check --repo OWNER/REPO --pr N          # a pull request's current head
test-check --repo OWNER/REPO --compare BASE...HEAD
test-check --json                            # machine-readable

Exit codes:

  • 0: advice printed.
  • 1: the change could not be read.
  • 2: usage error.
  • 3: JEV unavailable. No advice is printed; answer the skill's four questions yourself.

The GitHub modes use the gh CLI. The key comes from $OPENROUTER_API_KEY, else the OPENROUTER_API_KEY line in ~/.config/gitmoot/keychain.env. The default model is OpenRouter's rolling ~typesafe/jev-latest alias (including the leading ~), not a pinned release. The System One endpoint is unchanged. The response may name a concrete model; do not infer that release from the alias.

Local receipts

Advice and proof commands automatically save a concise receipt after the command finishes (and, for proof, after restoration). Help, list-tests, restore, and usage errors do not create receipts.

The default is $XDG_STATE_HOME/test-check/receipts, or $HOME/.local/state/test-check/receipts. Storage must be outside the checkout. Tool-owned directories are created with mode 0700, files with 0600; existing unsafe directories are refused, never chmodded. Each invocation has an independent directory containing receipt.json and its output_ref diagnostic link.

Receipts include mode, command, available repository/base/HEAD/tool revision, times, result, warnings, piece outcomes, and named-test execution evidence. Advice hashes its already-collected input to distinguish uncommitted snapshots without storing the diff or prompt. Proof streams the already-enumerated changed paths under its existing lock before running tests, hashing base/HEAD, framed path names, modes, deletions, link targets, and file bytes (64 MiB total limit). This is a changed-input identity, not a complete runtime-environment snapshot: ignored files and external inputs are not covered. Missing/unreadable/oversized input identity stays unknown with a warning, never changing the core proof result. observed means a verbose Go/Python runner marker was seen; unsupported runners or missing/truncated output are unknown, not proof a test was skipped. Whole-command named-test rows are aggregate_only, not individual proofs. Receipt metadata is evidence of a command, not an approval or quality judgment.

Diagnostics are bounded tails, not full raw logs: up to 2,000 bytes per output and 256 outputs, with truncation noted. No environment, credentials, configuration, prompts, or diffs are deliberately collected. Advice command options are retained, but --title values (including --title=...) are explicitly [redacted] because they become prompt content; that command is not fully replayable. Other command and runner output literals may still be sensitive; they are not automatically redacted or uploaded.

The receipt path is printed on stderr; JSON results add receipt and receipt_status. If saving fails, the original exit/outcome is preserved, stderr warns, and JSON contains receipt_status: failed and receipt_error, never a fake path. Controlled failures and interruptions can produce receipts after recovery; SIGKILL/power loss cannot guarantee one. Receipt writes never replace or remove prove's recovery stash.

Changes

  • Old-code build/import-only failures now return inconclusive (exit 4), including per-test and piece results. Diagnostic words in actual Go/Python assertion failures no longer imply build failure.
  • Advice and proof executions now save private, local receipts with command/result evidence and bounded diagnostic links. Receipt storage failures do not change proof/advice outcomes.

Install

go build -o ~/.local/bin/test-check ./cmd/test-check
ln -s "$PWD/skills/test-check" ~/.claude/skills/test-check

About

Asks JEV whether a code change needs a regression test, a one-off check, or no test (gitmoot/gitmoot#2265)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages