Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ test-check prove --pieces --test "go test ./pkg -run '^(TestA|TestB)$'"

It undoes one changed piece of code at a time (one diff hunk, a whole added file, or a whole deleted file) and runs the command. The command must fail every time.
- A piece that leaves the command passing is reported `not_tested`, and the outcome is `piece_not_tested` (exit 4).
- A compile/import-only failure is `inconclusive`, not `guarded`, and makes the overall outcome `inconclusive` (exit 4).
- Comment- and whitespace-only hunks are skipped.
- Above `--max-pieces` (default 60), nothing runs and the outcome is `too_many_pieces`.
- Each piece goes through the same save-first stash, so `--restore` recovers an interrupted run.
Expand All @@ -61,14 +62,16 @@ Test files and fixtures keep their new versions for both runs. The reverted file
Only one prove run may use a checkout at a time: a lock under the git dir makes a second run, or `--restore`, refuse while the first is alive. A lock left by a dead process is taken over. A test that hangs on the old code until `--timeout` counts as red.

Outcomes:
- `proven`: exit 0.
- `proven`: exit 0. Behavioral failure on old code, pass on new code.
- `inconclusive`: exit 4. The old run only showed a build/import failure or unavailable Python module API (`reason: old_code_build_failed`), or did not mention the selected test (`reason: old_test_not_observed`).
- `not_red_on_old`: exit 4. The test passes without the fix.
- `fails_on_new`: exit 4.
- `no_test_in_change`: exit 4.
- `test_not_run`: exit 4.
- `no_code_in_change`: exit 0. Only tests changed.
- Caller cancellation or test-check's own command deadline: exit 1 after restoration, never proof. Interrupted per-test/piece rows remain inconclusive; a framework's own assertion or panic is still a test failure, not a tool deadline.

A `build_error_suspected` flag marks an old-code failure that looks like a compile or import error rather than a failed assertion.
A `build_error_suspected` flag marks an old-code failure that looks like a compile or import error rather than a failed assertion. This is not proof: test through an interface the old code already had. Actual Go/Python assertion-failure summaries take precedence over diagnostic words in assertion messages. In per-test mode, any inconclusive result prevents an overall `proven`; new-code failures still report `fails_on_new`.

## Use

Expand All @@ -89,6 +92,23 @@ The GitHub modes use the `gh` CLI. The key comes from `$OPENROUTER_API_KEY`,
else the `OPENROUTER_API_KEY` line in `~/.config/gitmoot/keychain.env`. The
model is pinned to `typesafe/jev-1.13`.

## Local receipts

Advice and proof commands automatically save a concise receipt after the command finishes (and, for proof, after restoration). Help, list-tests, restore, and usage errors do not create receipts.

The default is `$XDG_STATE_HOME/test-check/receipts`, or `$HOME/.local/state/test-check/receipts`. Storage must be outside the checkout. Tool-owned directories are created with mode `0700`, files with `0600`; existing unsafe directories are refused, never chmodded. Each invocation has an independent directory containing `receipt.json` and its `output_ref` diagnostic link.

Receipts include mode, command, available repository/base/HEAD/tool revision, times, result, warnings, piece outcomes, and named-test execution evidence. Advice hashes its already-collected input to distinguish uncommitted snapshots without storing the diff or prompt. Proof streams the already-enumerated changed paths under its existing lock before running tests, hashing base/HEAD, framed path names, modes, deletions, link targets, and file bytes (64 MiB total limit). This is a **changed-input identity**, not a complete runtime-environment snapshot: ignored files and external inputs are not covered. Missing/unreadable/oversized input identity stays unknown with a warning, never changing the core proof result. `observed` means a verbose Go/Python runner marker was seen; unsupported runners or missing/truncated output are `unknown`, not proof a test was skipped. Whole-command named-test rows are `aggregate_only`, not individual proofs. Receipt metadata is evidence of a command, not an approval or quality judgment.

Diagnostics are **bounded tails**, not full raw logs: up to 2,000 bytes per output and 256 outputs, with truncation noted. No environment, credentials, configuration, prompts, or diffs are deliberately collected. Advice command options are retained, but `--title` values (including `--title=...`) are explicitly `[redacted]` because they become prompt content; that command is not fully replayable. Other command and runner output literals may still be sensitive; they are not automatically redacted or uploaded.

The receipt path is printed on stderr; JSON results add `receipt` and `receipt_status`. If saving fails, the original exit/outcome is preserved, stderr warns, and JSON contains `receipt_status: failed` and `receipt_error`, never a fake path. Controlled failures and interruptions can produce receipts after recovery; SIGKILL/power loss cannot guarantee one. Receipt writes never replace or remove prove's recovery stash.

## Changes

- Old-code build/import-only failures now return `inconclusive` (exit 4), including per-test and piece results. Diagnostic words in actual Go/Python assertion failures no longer imply build failure.
- Advice and proof executions now save private, local receipts with command/result evidence and bounded diagnostic links. Receipt storage failures do not change proof/advice outcomes.

## Install

```sh
Expand Down
29 changes: 27 additions & 2 deletions cmd/test-check/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ func run(args []string, stdout, stderr io.Writer) int {
put back files a crashed prove run left reverted

Exit: 0 advice printed / proven, 1 error, 2 usage, 3 JEV unavailable,
4 not proven (passes on the old code, fails on the new, or no test changed).
4 not proven (including inconclusive old-code build/import failures).
Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain.env.`)
}
dir := fs.String("dir", ".", "local git checkout")
Expand All @@ -91,6 +91,10 @@ Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain.
}

ctx := context.Background()
rec := startReceipt("advice", *dir, *base, adviceCommand(args), *pr == 0 && *compare == "")
if *repo != "" {
rec.record.Repo = *repo
}
var in check.Input
var err error
switch {
Expand All @@ -108,25 +112,46 @@ Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain.
in.Repo = *repo
}
if err != nil {
fields := rec.save("error", err, nil, stderr)
if *asJSON {
writeReceiptError(stdout, err, fields)
}
fmt.Fprintf(stderr, "test-check: %v\n", err)
return exitSource
}
rec.input(in)

key := apiKey()
if key == "" {
err := fmt.Errorf("JEV unavailable: no %s", apiKeyName)
fields := rec.save("unavailable", err, nil, stderr)
if *asJSON {
writeReceiptError(stdout, err, fields)
}
fmt.Fprintf(stderr, "test-check: JEV unavailable: no %s. Answer the four questions in the test-check skill yourself.\n", apiKeyName)
return exitUnavailable
}
result, err := check.Evaluate(ctx, newJudge(key), *model, in)
if err != nil {
fields := rec.save("unavailable", err, nil, stderr)
if *asJSON {
writeReceiptError(stdout, err, fields)
}
fmt.Fprintf(stderr, "test-check: JEV unavailable: %v. Answer the four questions in the test-check skill yourself.\n", err)
return exitUnavailable
}
if !result.DiffComplete {
rec.record.Warnings = append(rec.record.Warnings, "diff incomplete")
}
fields := rec.save(result.Advice, nil, nil, stderr)

if *asJSON {
encoder := json.NewEncoder(stdout)
encoder.SetIndent("", " ")
_ = encoder.Encode(result)
_ = encoder.Encode(struct {
check.Result
receiptFields
}{result, fields})
return 0
}
fmt.Fprintf(stdout, "advice: %s\n", result.Advice)
Expand Down
14 changes: 14 additions & 0 deletions cmd/test-check/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,20 @@ func withJudge(t *testing.T, judge check.Judge, key string) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
}

// CLI tests must never publish synthetic evidence in the user's receipt store.
func TestMain(m *testing.M) {
state, err := os.MkdirTemp("", "test-check-state-")
if err != nil {
panic(err)
}
if err := os.Setenv("XDG_STATE_HOME", state); err != nil {
panic(err)
}
code := m.Run()
_ = os.RemoveAll(state)
os.Exit(code)
}

func TestRunPrintsAdviceForALocalChange(t *testing.T) {
withJudge(t, stubJudge{advice: check.AdviceNewTest}, "k")
var stdout, stderr bytes.Buffer
Expand Down
110 changes: 110 additions & 0 deletions cmd/test-check/proof_classification_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
package main

import (
"bytes"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)

// Exercise the public CLI contract through interfaces available before this fix:
// a build-only old run must never return a successful proof, and diagnostic
// words in an actual assertion must not be reported as a build failure.
func TestProofDistinguishesBuildFailureFromBehavior(t *testing.T) {
t.Setenv("XDG_STATE_HOME", t.TempDir())
cases := []struct {
name, file, old, new, testFile, test, command string
build bool
}{
{"go_build", "value.go", "package value\nfunc Value() int { return 1 }\n", "package value\nfunc Value() int { return 1 }; func Added() int { return 2 }\n", "value_test.go", "package value\nimport \"testing\"\nfunc TestValue(t *testing.T) { if Added() != 2 { t.Fatal(\"wrong value\") } }\n", "go test -count=1 -v -run '^TestValue$' .", true},
{"go_assertion", "value.go", "package value\nfunc Value() int { return 1 }\n", "package value\nfunc Value() int { return 2 }\n", "value_test.go", "package value\nimport \"testing\"\nfunc TestValue(t *testing.T) { if Value() != 2 { t.Fatal(\"ImportError: expected diagnostic differs\") } }\n", "go test -count=1 -v -run '^TestValue$' .", false},
{"go_panic", "value.go", "package value\nfunc Value() int { return 1 }\n", "package value\nfunc Value() int { return 2 }\n", "value_test.go", "package value\nimport \"testing\"\nfunc TestValue(t *testing.T) { if Value() != 2 { panic(\"behavioral failure\") } }\n", "go test -count=1 -v -run '^TestValue$' .", false},
{"python_import", "value.py", "def value(): return 1\n", "def value(): return 1\ndef added(): return 2\n", "test_value.py", "import unittest\nfrom value import added\nclass ValueTest(unittest.TestCase):\n def test_value(self): self.assertEqual(added(), 2)\n", "python3 -B -m unittest -v test_value", true},
{"python_attribute", "value.py", "def value(): return 1\n", "def value(): return 1\ndef added(): return 2\n", "test_value.py", "import unittest\nimport value\nclass ValueTest(unittest.TestCase):\n def test_value(self): self.assertEqual(value.added(), 2)\n", "python3 -B -m unittest -v test_value", true},
{"python_assertion", "value.py", "def value(): return 1\n", "def value(): return 2\n", "test_value.py", "import unittest\nfrom value import value\nclass ValueTest(unittest.TestCase):\n def test_value(self): self.assertEqual(value(), 2, 'ImportError: expected diagnostic differs')\n", "python3 -B -m unittest -v test_value", false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
proofGit(t, dir, "init", "-q", "-b", "main")
proofWrite(t, dir, "go.mod", "module value\n\ngo 1.23\n")
proofWrite(t, dir, tc.file, tc.old)
proofGit(t, dir, "add", ".")
proofGit(t, dir, "-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "base")
proofWrite(t, dir, tc.file, tc.new)
proofWrite(t, dir, tc.testFile, tc.test)
for _, mode := range []string{"whole", "each", "pieces"} {
args := []string{"prove", "--dir", dir, "--base", "main", "--json", "--test", tc.command}
if mode == "each" {
name := "TestValue"
if strings.HasPrefix(tc.name, "python") {
name = "test_value"
}
args[len(args)-1] = strings.ReplaceAll(tc.command, name, "{name}")
args = append(args, "--each", name)
}
if mode == "pieces" {
args = append(args, "--pieces")
}
var stdout, stderr bytes.Buffer
code := run(args, &stdout, &stderr)
var report struct {
Outcome string `json:"outcome"`
Build bool `json:"build_error_suspected"`
Tests []struct {
Outcome string `json:"outcome"`
Build bool `json:"build_error_suspected"`
} `json:"tests"`
Pieces []struct {
Outcome string `json:"outcome"`
Build bool `json:"build_error_suspected"`
} `json:"pieces"`
}
if err := json.Unmarshal(stdout.Bytes(), &report); err != nil {
t.Fatalf("%s: %v: %s %s", mode, err, stdout.String(), stderr.String())
}
if tc.build {
if code != exitNotProven || report.Outcome == "proven" || !report.Build {
t.Fatalf("%s: build-only failure claimed proof: exit=%d report=%s stderr=%s", mode, code, stdout.String(), stderr.String())
}
for _, result := range report.Tests {
if result.Outcome == "proven" || !result.Build {
t.Fatalf("%s: test claimed proof: %+v", mode, result)
}
}
for _, result := range report.Pieces {
if result.Outcome == "guarded" || !result.Build {
t.Fatalf("%s: piece claimed behavioral coverage: %+v", mode, result)
}
}
} else if code != 0 || report.Outcome != "proven" || report.Build {
t.Fatalf("%s: assertion rejected: exit=%d report=%s stderr=%s", mode, code, stdout.String(), stderr.String())
}
got, err := os.ReadFile(filepath.Join(dir, tc.file))
if err != nil || string(got) != tc.new {
t.Fatalf("%s: working file not restored: %q %v", mode, got, err)
}
}
})
}
}

func proofGit(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", append([]string{"-C", dir}, args...)...)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
return string(out)
}

func proofWrite(t *testing.T, dir, name, content string) {
t.Helper()
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
142 changes: 142 additions & 0 deletions cmd/test-check/proof_interruption_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
package main

import (
"bytes"
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"time"

"github.com/gitmoot/test-check/internal/prove"
)

func TestRunnerDeadlinesNeverProveRegression(t *testing.T) {
t.Setenv("XDG_STATE_HOME", t.TempDir())
for _, stage := range []string{"old", "new"} {
for _, mode := range []string{"whole", "each", "pieces"} {
t.Run(stage+"/"+mode, func(t *testing.T) {
dir := runnerFixture(t)
proofWrite(t, dir, "value_test.sh", "echo TestValue\nif grep -q '^"+stage+"$' value.go; then sleep 30; fi\ngrep -q '^new$' value.go\n")
args := []string{"prove", "--dir", dir, "--base", "main", "--json", "--timeout", "1s", "--test", "sh value_test.sh"}
if mode == "each" {
args[len(args)-1] = "sh value_test.sh # {name}"
args = append(args, "--each", "TestValue")
}
if mode == "pieces" {
args = append(args, "--pieces")
}
var stdout, stderr bytes.Buffer
code := run(args, &stdout, &stderr)
var report struct {
Outcome, Receipt string
Tests []struct{ Outcome string }
Pieces []struct{ Outcome string }
}
if err := json.Unmarshal(stdout.Bytes(), &report); err != nil {
t.Fatalf("%v: %s %s", err, stdout.String(), stderr.String())
}
if code != exitSource || report.Outcome == "proven" {
t.Fatalf("owned timeout claimed proof: exit=%d %s", code, stdout.String())
}
for _, child := range report.Tests {
if child.Outcome == "proven" {
t.Fatalf("interrupted test claimed proof: %s", stdout.String())
}
}
for _, child := range report.Pieces {
if child.Outcome == "guarded" {
t.Fatalf("interrupted piece claimed coverage: %s", stdout.String())
}
}
raw, err := os.ReadFile(report.Receipt)
if err != nil {
t.Fatal(err)
}
var receipt struct{ Outcome string }
if err := json.Unmarshal(raw, &receipt); err != nil || receipt.Outcome != "error" {
t.Fatalf("timeout receipt: %s %v", raw, err)
}
content, err := os.ReadFile(filepath.Join(dir, "value.go"))
if err != nil || string(content) != "new\n" {
t.Fatalf("restoration: %q %v", content, err)
}
})
}
}
}

func TestCanceledOldRunNeverClaimsChildProof(t *testing.T) {
for _, mode := range []string{"whole", "each", "pieces"} {
t.Run(mode, func(t *testing.T) {
dir := runnerFixture(t)
ready := filepath.Join(t.TempDir(), "ready")
proofWrite(t, dir, "value_test.sh", "echo TestValue\nif grep -q '^new$' value.go; then exit 0; fi\ntouch '"+ready+"'\nsleep 30\n")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
stopped := make(chan struct{})
go func() {
defer close(stopped)
tick := time.NewTicker(time.Millisecond)
defer tick.Stop()
for {
if _, err := os.Stat(ready); err == nil {
cancel()
return
}
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}()
opts := prove.Options{Dir: dir, Base: "main", Test: "sh value_test.sh"}
if mode == "each" {
opts.Test += " # {name}"
opts.Each = []string{"TestValue"}
}
if mode == "pieces" {
opts.Pieces = true
}
report, err := prove.Run(ctx, opts)
<-stopped
if !errors.Is(err, context.Canceled) {
t.Fatalf("want caller cancellation, got %v", err)
}
if mode == "each" && len(report.Tests) != 1 {
t.Fatalf("lost interrupted result: %+v", report)
}
if mode == "pieces" && len(report.Pieces) != 1 {
t.Fatalf("lost interrupted piece: %+v", report)
}
for _, child := range report.Tests {
if child.Outcome == "proven" {
t.Fatalf("canceled child claimed proof: %+v", child)
}
}
for _, child := range report.Pieces {
if child.Outcome == "guarded" {
t.Fatalf("canceled piece claimed coverage: %+v", child)
}
}
content, readErr := os.ReadFile(filepath.Join(dir, "value.go"))
if readErr != nil || string(content) != "new\n" {
t.Fatalf("restoration: %q %v", content, readErr)
}
})
}
}

func runnerFixture(t *testing.T) string {
t.Helper()
dir := t.TempDir()
proofGit(t, dir, "init", "-q", "-b", "main")
proofWrite(t, dir, "value.go", "old\n")
proofGit(t, dir, "add", ".")
proofGit(t, dir, "-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "base")
proofWrite(t, dir, "value.go", "new\n")
return dir
}
Loading
Loading