diff --git a/README.md b/README.md index 59968be..562e6f7 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ test-check prove --pieces --test "go test ./pkg -run '^(TestA|TestB)$'" It undoes one changed piece of code at a time (one diff hunk, a whole added file, or a whole deleted file) and runs the command. The command must fail every time. - A piece that leaves the command passing is reported `not_tested`, and the outcome is `piece_not_tested` (exit 4). +- A compile/import-only failure is `inconclusive`, not `guarded`, and makes the overall outcome `inconclusive` (exit 4). - Comment- and whitespace-only hunks are skipped. - Above `--max-pieces` (default 60), nothing runs and the outcome is `too_many_pieces`. - Each piece goes through the same save-first stash, so `--restore` recovers an interrupted run. @@ -61,14 +62,16 @@ Test files and fixtures keep their new versions for both runs. The reverted file Only one prove run may use a checkout at a time: a lock under the git dir makes a second run, or `--restore`, refuse while the first is alive. A lock left by a dead process is taken over. A test that hangs on the old code until `--timeout` counts as red. Outcomes: -- `proven`: exit 0. +- `proven`: exit 0. Behavioral failure on old code, pass on new code. +- `inconclusive`: exit 4. The old run only showed a build/import failure or unavailable Python module API (`reason: old_code_build_failed`), or did not mention the selected test (`reason: old_test_not_observed`). - `not_red_on_old`: exit 4. The test passes without the fix. - `fails_on_new`: exit 4. - `no_test_in_change`: exit 4. - `test_not_run`: exit 4. - `no_code_in_change`: exit 0. Only tests changed. +- Caller cancellation or test-check's own command deadline: exit 1 after restoration, never proof. Interrupted per-test/piece rows remain inconclusive; a framework's own assertion or panic is still a test failure, not a tool deadline. -A `build_error_suspected` flag marks an old-code failure that looks like a compile or import error rather than a failed assertion. +A `build_error_suspected` flag marks an old-code failure that looks like a compile or import error rather than a failed assertion. This is not proof: test through an interface the old code already had. Actual Go/Python assertion-failure summaries take precedence over diagnostic words in assertion messages. In per-test mode, any inconclusive result prevents an overall `proven`; new-code failures still report `fails_on_new`. ## Use @@ -89,6 +92,23 @@ The GitHub modes use the `gh` CLI. The key comes from `$OPENROUTER_API_KEY`, else the `OPENROUTER_API_KEY` line in `~/.config/gitmoot/keychain.env`. The model is pinned to `typesafe/jev-1.13`. +## Local receipts + +Advice and proof commands automatically save a concise receipt after the command finishes (and, for proof, after restoration). Help, list-tests, restore, and usage errors do not create receipts. + +The default is `$XDG_STATE_HOME/test-check/receipts`, or `$HOME/.local/state/test-check/receipts`. Storage must be outside the checkout. Tool-owned directories are created with mode `0700`, files with `0600`; existing unsafe directories are refused, never chmodded. Each invocation has an independent directory containing `receipt.json` and its `output_ref` diagnostic link. + +Receipts include mode, command, available repository/base/HEAD/tool revision, times, result, warnings, piece outcomes, and named-test execution evidence. Advice hashes its already-collected input to distinguish uncommitted snapshots without storing the diff or prompt. Proof streams the already-enumerated changed paths under its existing lock before running tests, hashing base/HEAD, framed path names, modes, deletions, link targets, and file bytes (64 MiB total limit). This is a **changed-input identity**, not a complete runtime-environment snapshot: ignored files and external inputs are not covered. Missing/unreadable/oversized input identity stays unknown with a warning, never changing the core proof result. `observed` means a verbose Go/Python runner marker was seen; unsupported runners or missing/truncated output are `unknown`, not proof a test was skipped. Whole-command named-test rows are `aggregate_only`, not individual proofs. Receipt metadata is evidence of a command, not an approval or quality judgment. + +Diagnostics are **bounded tails**, not full raw logs: up to 2,000 bytes per output and 256 outputs, with truncation noted. No environment, credentials, configuration, prompts, or diffs are deliberately collected. Advice command options are retained, but `--title` values (including `--title=...`) are explicitly `[redacted]` because they become prompt content; that command is not fully replayable. Other command and runner output literals may still be sensitive; they are not automatically redacted or uploaded. + +The receipt path is printed on stderr; JSON results add `receipt` and `receipt_status`. If saving fails, the original exit/outcome is preserved, stderr warns, and JSON contains `receipt_status: failed` and `receipt_error`, never a fake path. Controlled failures and interruptions can produce receipts after recovery; SIGKILL/power loss cannot guarantee one. Receipt writes never replace or remove prove's recovery stash. + +## Changes + +- Old-code build/import-only failures now return `inconclusive` (exit 4), including per-test and piece results. Diagnostic words in actual Go/Python assertion failures no longer imply build failure. +- Advice and proof executions now save private, local receipts with command/result evidence and bounded diagnostic links. Receipt storage failures do not change proof/advice outcomes. + ## Install ```sh diff --git a/cmd/test-check/main.go b/cmd/test-check/main.go index cc3566e..9f4ef66 100644 --- a/cmd/test-check/main.go +++ b/cmd/test-check/main.go @@ -68,7 +68,7 @@ func run(args []string, stdout, stderr io.Writer) int { put back files a crashed prove run left reverted Exit: 0 advice printed / proven, 1 error, 2 usage, 3 JEV unavailable, -4 not proven (passes on the old code, fails on the new, or no test changed). +4 not proven (including inconclusive old-code build/import failures). Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain.env.`) } dir := fs.String("dir", ".", "local git checkout") @@ -91,6 +91,10 @@ Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain. } ctx := context.Background() + rec := startReceipt("advice", *dir, *base, adviceCommand(args), *pr == 0 && *compare == "") + if *repo != "" { + rec.record.Repo = *repo + } var in check.Input var err error switch { @@ -108,25 +112,46 @@ Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain. in.Repo = *repo } if err != nil { + fields := rec.save("error", err, nil, stderr) + if *asJSON { + writeReceiptError(stdout, err, fields) + } fmt.Fprintf(stderr, "test-check: %v\n", err) return exitSource } + rec.input(in) key := apiKey() if key == "" { + err := fmt.Errorf("JEV unavailable: no %s", apiKeyName) + fields := rec.save("unavailable", err, nil, stderr) + if *asJSON { + writeReceiptError(stdout, err, fields) + } fmt.Fprintf(stderr, "test-check: JEV unavailable: no %s. Answer the four questions in the test-check skill yourself.\n", apiKeyName) return exitUnavailable } result, err := check.Evaluate(ctx, newJudge(key), *model, in) if err != nil { + fields := rec.save("unavailable", err, nil, stderr) + if *asJSON { + writeReceiptError(stdout, err, fields) + } fmt.Fprintf(stderr, "test-check: JEV unavailable: %v. Answer the four questions in the test-check skill yourself.\n", err) return exitUnavailable } + if !result.DiffComplete { + rec.record.Warnings = append(rec.record.Warnings, "diff incomplete") + } + fields := rec.save(result.Advice, nil, nil, stderr) if *asJSON { encoder := json.NewEncoder(stdout) encoder.SetIndent("", " ") - _ = encoder.Encode(result) + _ = encoder.Encode(struct { + check.Result + receiptFields + }{result, fields}) return 0 } fmt.Fprintf(stdout, "advice: %s\n", result.Advice) diff --git a/cmd/test-check/main_test.go b/cmd/test-check/main_test.go index 2ebe8a7..3145951 100644 --- a/cmd/test-check/main_test.go +++ b/cmd/test-check/main_test.go @@ -62,6 +62,20 @@ func withJudge(t *testing.T, judge check.Judge, key string) { t.Setenv("XDG_CONFIG_HOME", t.TempDir()) } +// CLI tests must never publish synthetic evidence in the user's receipt store. +func TestMain(m *testing.M) { + state, err := os.MkdirTemp("", "test-check-state-") + if err != nil { + panic(err) + } + if err := os.Setenv("XDG_STATE_HOME", state); err != nil { + panic(err) + } + code := m.Run() + _ = os.RemoveAll(state) + os.Exit(code) +} + func TestRunPrintsAdviceForALocalChange(t *testing.T) { withJudge(t, stubJudge{advice: check.AdviceNewTest}, "k") var stdout, stderr bytes.Buffer diff --git a/cmd/test-check/proof_classification_test.go b/cmd/test-check/proof_classification_test.go new file mode 100644 index 0000000..0d1d2bb --- /dev/null +++ b/cmd/test-check/proof_classification_test.go @@ -0,0 +1,110 @@ +package main + +import ( + "bytes" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// Exercise the public CLI contract through interfaces available before this fix: +// a build-only old run must never return a successful proof, and diagnostic +// words in an actual assertion must not be reported as a build failure. +func TestProofDistinguishesBuildFailureFromBehavior(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + cases := []struct { + name, file, old, new, testFile, test, command string + build bool + }{ + {"go_build", "value.go", "package value\nfunc Value() int { return 1 }\n", "package value\nfunc Value() int { return 1 }; func Added() int { return 2 }\n", "value_test.go", "package value\nimport \"testing\"\nfunc TestValue(t *testing.T) { if Added() != 2 { t.Fatal(\"wrong value\") } }\n", "go test -count=1 -v -run '^TestValue$' .", true}, + {"go_assertion", "value.go", "package value\nfunc Value() int { return 1 }\n", "package value\nfunc Value() int { return 2 }\n", "value_test.go", "package value\nimport \"testing\"\nfunc TestValue(t *testing.T) { if Value() != 2 { t.Fatal(\"ImportError: expected diagnostic differs\") } }\n", "go test -count=1 -v -run '^TestValue$' .", false}, + {"go_panic", "value.go", "package value\nfunc Value() int { return 1 }\n", "package value\nfunc Value() int { return 2 }\n", "value_test.go", "package value\nimport \"testing\"\nfunc TestValue(t *testing.T) { if Value() != 2 { panic(\"behavioral failure\") } }\n", "go test -count=1 -v -run '^TestValue$' .", false}, + {"python_import", "value.py", "def value(): return 1\n", "def value(): return 1\ndef added(): return 2\n", "test_value.py", "import unittest\nfrom value import added\nclass ValueTest(unittest.TestCase):\n def test_value(self): self.assertEqual(added(), 2)\n", "python3 -B -m unittest -v test_value", true}, + {"python_attribute", "value.py", "def value(): return 1\n", "def value(): return 1\ndef added(): return 2\n", "test_value.py", "import unittest\nimport value\nclass ValueTest(unittest.TestCase):\n def test_value(self): self.assertEqual(value.added(), 2)\n", "python3 -B -m unittest -v test_value", true}, + {"python_assertion", "value.py", "def value(): return 1\n", "def value(): return 2\n", "test_value.py", "import unittest\nfrom value import value\nclass ValueTest(unittest.TestCase):\n def test_value(self): self.assertEqual(value(), 2, 'ImportError: expected diagnostic differs')\n", "python3 -B -m unittest -v test_value", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + proofGit(t, dir, "init", "-q", "-b", "main") + proofWrite(t, dir, "go.mod", "module value\n\ngo 1.23\n") + proofWrite(t, dir, tc.file, tc.old) + proofGit(t, dir, "add", ".") + proofGit(t, dir, "-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "base") + proofWrite(t, dir, tc.file, tc.new) + proofWrite(t, dir, tc.testFile, tc.test) + for _, mode := range []string{"whole", "each", "pieces"} { + args := []string{"prove", "--dir", dir, "--base", "main", "--json", "--test", tc.command} + if mode == "each" { + name := "TestValue" + if strings.HasPrefix(tc.name, "python") { + name = "test_value" + } + args[len(args)-1] = strings.ReplaceAll(tc.command, name, "{name}") + args = append(args, "--each", name) + } + if mode == "pieces" { + args = append(args, "--pieces") + } + var stdout, stderr bytes.Buffer + code := run(args, &stdout, &stderr) + var report struct { + Outcome string `json:"outcome"` + Build bool `json:"build_error_suspected"` + Tests []struct { + Outcome string `json:"outcome"` + Build bool `json:"build_error_suspected"` + } `json:"tests"` + Pieces []struct { + Outcome string `json:"outcome"` + Build bool `json:"build_error_suspected"` + } `json:"pieces"` + } + if err := json.Unmarshal(stdout.Bytes(), &report); err != nil { + t.Fatalf("%s: %v: %s %s", mode, err, stdout.String(), stderr.String()) + } + if tc.build { + if code != exitNotProven || report.Outcome == "proven" || !report.Build { + t.Fatalf("%s: build-only failure claimed proof: exit=%d report=%s stderr=%s", mode, code, stdout.String(), stderr.String()) + } + for _, result := range report.Tests { + if result.Outcome == "proven" || !result.Build { + t.Fatalf("%s: test claimed proof: %+v", mode, result) + } + } + for _, result := range report.Pieces { + if result.Outcome == "guarded" || !result.Build { + t.Fatalf("%s: piece claimed behavioral coverage: %+v", mode, result) + } + } + } else if code != 0 || report.Outcome != "proven" || report.Build { + t.Fatalf("%s: assertion rejected: exit=%d report=%s stderr=%s", mode, code, stdout.String(), stderr.String()) + } + got, err := os.ReadFile(filepath.Join(dir, tc.file)) + if err != nil || string(got) != tc.new { + t.Fatalf("%s: working file not restored: %q %v", mode, got, err) + } + } + }) + } +} + +func proofGit(t *testing.T, dir string, args ...string) string { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("git %v: %v: %s", args, err, out) + } + return string(out) +} + +func proofWrite(t *testing.T, dir, name, content string) { + t.Helper() + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644); err != nil { + t.Fatal(err) + } +} diff --git a/cmd/test-check/proof_interruption_test.go b/cmd/test-check/proof_interruption_test.go new file mode 100644 index 0000000..6604af3 --- /dev/null +++ b/cmd/test-check/proof_interruption_test.go @@ -0,0 +1,142 @@ +package main + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "os" + "path/filepath" + "testing" + "time" + + "github.com/gitmoot/test-check/internal/prove" +) + +func TestRunnerDeadlinesNeverProveRegression(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + for _, stage := range []string{"old", "new"} { + for _, mode := range []string{"whole", "each", "pieces"} { + t.Run(stage+"/"+mode, func(t *testing.T) { + dir := runnerFixture(t) + proofWrite(t, dir, "value_test.sh", "echo TestValue\nif grep -q '^"+stage+"$' value.go; then sleep 30; fi\ngrep -q '^new$' value.go\n") + args := []string{"prove", "--dir", dir, "--base", "main", "--json", "--timeout", "1s", "--test", "sh value_test.sh"} + if mode == "each" { + args[len(args)-1] = "sh value_test.sh # {name}" + args = append(args, "--each", "TestValue") + } + if mode == "pieces" { + args = append(args, "--pieces") + } + var stdout, stderr bytes.Buffer + code := run(args, &stdout, &stderr) + var report struct { + Outcome, Receipt string + Tests []struct{ Outcome string } + Pieces []struct{ Outcome string } + } + if err := json.Unmarshal(stdout.Bytes(), &report); err != nil { + t.Fatalf("%v: %s %s", err, stdout.String(), stderr.String()) + } + if code != exitSource || report.Outcome == "proven" { + t.Fatalf("owned timeout claimed proof: exit=%d %s", code, stdout.String()) + } + for _, child := range report.Tests { + if child.Outcome == "proven" { + t.Fatalf("interrupted test claimed proof: %s", stdout.String()) + } + } + for _, child := range report.Pieces { + if child.Outcome == "guarded" { + t.Fatalf("interrupted piece claimed coverage: %s", stdout.String()) + } + } + raw, err := os.ReadFile(report.Receipt) + if err != nil { + t.Fatal(err) + } + var receipt struct{ Outcome string } + if err := json.Unmarshal(raw, &receipt); err != nil || receipt.Outcome != "error" { + t.Fatalf("timeout receipt: %s %v", raw, err) + } + content, err := os.ReadFile(filepath.Join(dir, "value.go")) + if err != nil || string(content) != "new\n" { + t.Fatalf("restoration: %q %v", content, err) + } + }) + } + } +} + +func TestCanceledOldRunNeverClaimsChildProof(t *testing.T) { + for _, mode := range []string{"whole", "each", "pieces"} { + t.Run(mode, func(t *testing.T) { + dir := runnerFixture(t) + ready := filepath.Join(t.TempDir(), "ready") + proofWrite(t, dir, "value_test.sh", "echo TestValue\nif grep -q '^new$' value.go; then exit 0; fi\ntouch '"+ready+"'\nsleep 30\n") + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + stopped := make(chan struct{}) + go func() { + defer close(stopped) + tick := time.NewTicker(time.Millisecond) + defer tick.Stop() + for { + if _, err := os.Stat(ready); err == nil { + cancel() + return + } + select { + case <-ctx.Done(): + return + case <-tick.C: + } + } + }() + opts := prove.Options{Dir: dir, Base: "main", Test: "sh value_test.sh"} + if mode == "each" { + opts.Test += " # {name}" + opts.Each = []string{"TestValue"} + } + if mode == "pieces" { + opts.Pieces = true + } + report, err := prove.Run(ctx, opts) + <-stopped + if !errors.Is(err, context.Canceled) { + t.Fatalf("want caller cancellation, got %v", err) + } + if mode == "each" && len(report.Tests) != 1 { + t.Fatalf("lost interrupted result: %+v", report) + } + if mode == "pieces" && len(report.Pieces) != 1 { + t.Fatalf("lost interrupted piece: %+v", report) + } + for _, child := range report.Tests { + if child.Outcome == "proven" { + t.Fatalf("canceled child claimed proof: %+v", child) + } + } + for _, child := range report.Pieces { + if child.Outcome == "guarded" { + t.Fatalf("canceled piece claimed coverage: %+v", child) + } + } + content, readErr := os.ReadFile(filepath.Join(dir, "value.go")) + if readErr != nil || string(content) != "new\n" { + t.Fatalf("restoration: %q %v", content, readErr) + } + }) + } +} + +func runnerFixture(t *testing.T) string { + t.Helper() + dir := t.TempDir() + proofGit(t, dir, "init", "-q", "-b", "main") + proofWrite(t, dir, "value.go", "old\n") + proofGit(t, dir, "add", ".") + proofGit(t, dir, "-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "base") + proofWrite(t, dir, "value.go", "new\n") + return dir +} diff --git a/cmd/test-check/prove.go b/cmd/test-check/prove.go index 868f2ce..8161f4c 100644 --- a/cmd/test-check/prove.go +++ b/cmd/test-check/prove.go @@ -20,6 +20,7 @@ const exitNotProven = 4 var proveMessages = map[string]string{ prove.OutcomeProven: "The tests fail on the old code and pass on the new: they would catch a regression of this change.", + prove.OutcomeInconclusive: "The old-code run did not establish a behavioral test failure. A build/import failure or unavailable old API is not regression proof; use an interface the old code also has.", prove.OutcomeNotRedOnOld: "The tests PASS on the old code too, so they would not catch a regression. Make them fail without the fix.", prove.OutcomeFailsOnNew: "The tests FAIL on the new code. Fix that before proving.", prove.OutcomeNoTestChanged: "The change has no test files. If a test is needed, add one; otherwise record the one-off check you ran.", @@ -67,7 +68,8 @@ func runProve(args []string, stdout, stderr io.Writer) int { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() if *restore { - if err := prove.Restore(ctx, *dir); err != nil { + err := prove.Restore(ctx, *dir) + if err != nil { fmt.Fprintf(stderr, "test-check prove: %v\n", err) return exitSource } @@ -84,9 +86,25 @@ func runProve(args []string, stdout, stderr io.Writer) int { fmt.Fprintln(stderr, "test-check prove: --each needs {name} in --test") return exitUsage } + commandArgs := make([]string, len(args)) + for i, arg := range args { + commandArgs[i] = strconv.Quote(arg) + } + rec := startReceipt("prove", *dir, *base, "test-check prove "+strings.Join(commandArgs, " "), true) report, err := prove.Run(ctx, prove.Options{Dir: *dir, Base: *base, Test: *test, Each: names, Timeout: *timeout, Pieces: *pieces, MaxPieces: *maxPieces}) + if err != nil { + report.Outcome = "error" + } + fields := rec.proof(report, *test, err, stderr) if err != nil { fmt.Fprintf(stderr, "test-check prove: %v\n", err) + if *asJSON { + _ = json.NewEncoder(stdout).Encode(struct { + prove.Report + Error string `json:"error"` + receiptFields + }{report, err.Error(), fields}) + } return exitSource } code := exitNotProven @@ -96,12 +114,18 @@ func runProve(args []string, stdout, stderr io.Writer) int { if *asJSON { encoder := json.NewEncoder(stdout) encoder.SetIndent("", " ") - _ = encoder.Encode(report) + _ = encoder.Encode(struct { + prove.Report + receiptFields + }{report, fields}) return code } fmt.Fprintf(stdout, "outcome: %s\n%s\n", report.Outcome, proveMessages[report.Outcome]) + if report.Reason != "" { + fmt.Fprintf(stdout, "reason: %s\n", report.Reason) + } if report.BuildErrorSuspected { - fmt.Fprintln(stdout, "warning: the old-code run looks like a build or import error, not a failed assertion. That only shows the test uses new code; for a bug fix, test the behavior through an interface the old code also has.") + fmt.Fprintln(stdout, "warning: the old-code run looks like a build/import error or unavailable old API, not a failed assertion. For a bug fix, test the behavior through an interface the old code also has.") } for _, res := range report.Tests { note := "" @@ -133,7 +157,7 @@ func runProve(args []string, stdout, stderr io.Writer) int { switch report.Outcome { case prove.OutcomeFailsOnNew: fmt.Fprintf(stdout, "--- new-code run (tail) ---\n%s\n", report.NewOutput) - case prove.OutcomeProven, prove.OutcomeNotRedOnOld: + case prove.OutcomeProven, prove.OutcomeNotRedOnOld, prove.OutcomeInconclusive: fmt.Fprintf(stdout, "--- old-code run (tail) ---\n%s\n", report.OldOutput) } return code diff --git a/cmd/test-check/receipt.go b/cmd/test-check/receipt.go new file mode 100644 index 0000000..bdc6deb --- /dev/null +++ b/cmd/test-check/receipt.go @@ -0,0 +1,223 @@ +package main + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "path/filepath" + "regexp" + "runtime/debug" + "strconv" + "strings" + "time" + + "github.com/gitmoot/test-check/internal/check" + "github.com/gitmoot/test-check/internal/prove" + "github.com/gitmoot/test-check/internal/receipt" + "github.com/gitmoot/test-check/internal/source" +) + +// receiptFields are additive JSON metadata; stdout remains one JSON value. +type receiptFields struct { + Receipt string `json:"receipt,omitempty"` + ReceiptStatus string `json:"receipt_status"` + ReceiptError string `json:"receipt_error,omitempty"` +} + +type commandReceipt struct{ record receipt.Record } + +func startReceipt(mode, dir, base, command string, local bool) *commandReceipt { + r := &commandReceipt{record: receipt.Record{Mode: mode, Base: base, Command: command, StartedAt: time.Now().UTC()}} + if info, ok := debug.ReadBuildInfo(); ok { + for _, setting := range info.Settings { + if setting.Key == "vcs.revision" { + r.record.ToolRevision = setting.Value + } + } + } + if !local { + return r + } + // Metadata is best effort and bounded; it must never prevent proving or + // recovery. It is collected before any proof swaps files. + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + defer cancel() + git := source.Git(ctx, dir) + root, err := git("rev-parse", "--show-toplevel") + if err != nil { + r.record.Checkout, _ = filepath.Abs(dir) + r.record.Repo = r.record.Checkout + r.record.Warnings = append(r.record.Warnings, "repository identity unavailable") + return r + } + r.record.Repo = strings.TrimSpace(root) + r.record.Checkout = r.record.Repo + head, err := git("rev-parse", "HEAD") + if err == nil { + r.record.Head = strings.TrimSpace(head) + } + if base == "" { + base, _ = source.DefaultBase(git) + } + if base != "" { + mergeBase, err := git("merge-base", base, "HEAD") + if err == nil { + r.record.Base = strings.TrimSpace(mergeBase) + } + } + return r +} + +func (r *commandReceipt) input(in check.Input) { + if in.Repo != "" { + r.record.Repo = in.Repo + } + if in.Head != "" { + r.record.Head = in.Head + } + // Hash the already-collected input without copying or retaining its diff. + h := sha256.New() + _ = json.NewEncoder(h).Encode(in) + r.record.WorktreeDigest = hex.EncodeToString(h.Sum(nil)) + if in.Incomplete { + r.record.Warnings = append(r.record.Warnings, "input snapshot incomplete") + } +} + +func (r *commandReceipt) save(outcome string, runErr error, outputs []receipt.Output, stderr io.Writer) receiptFields { + r.record.Outcome = outcome + r.record.FinishedAt = time.Now().UTC() + if runErr != nil { + r.record.Error = runErr.Error() + } + // Save is local and independent of a canceled run context. The caller has + // already restored files before arriving here. + paths, err := receipt.Save("", r.record, outputs) + if err != nil { + fmt.Fprintf(stderr, "test-check: warning: receipt could not be saved: %v\n", err) + return receiptFields{ReceiptStatus: "failed", ReceiptError: err.Error()} + } + fmt.Fprintf(stderr, "receipt: %s\n", paths.Receipt) + return receiptFields{Receipt: paths.Receipt, ReceiptStatus: "saved"} +} + +func (r *commandReceipt) proof(report prove.Report, command string, runErr error, stderr io.Writer) receiptFields { + if report.Base != "" { + r.record.Base = report.Base + } + if report.Head != "" { + r.record.Head = report.Head + } + r.record.WorktreeDigest = report.WorktreeDigest + if report.WorktreeDigest == "" { + warning := report.SnapshotWarning + if warning == "" { + warning = "changed-input snapshot unknown (proof did not reach input capture)" + } + r.record.Warnings = append(r.record.Warnings, warning) + } + var outputs []receipt.Output + addOutput := func(label, text string) { + if text != "" { + outputs = append(outputs, receipt.Output{Label: label, Text: text}) + } + } + addOutput("new", report.NewOutput) + addOutput("old", report.OldOutput) + for _, test := range report.Tests { + item := receipt.Test{Name: test.Name, Command: strings.ReplaceAll(command, prove.NamePlaceholder, test.Name), Outcome: test.Outcome, + NewExecution: executionObserved(test.NewOutput, test.Name), OldExecution: executionObserved(test.OldOutput, test.Name)} + if test.BuildErrorSuspected { + item.Warnings = append(item.Warnings, "old_code_build_failed") + } + if test.Outcome == prove.OutcomeNotRun { + item.NewExecution = "not_observed" + } + r.record.Tests = append(r.record.Tests, item) + addOutput(test.Name+"/new", test.NewOutput) + addOutput(test.Name+"/old", test.OldOutput) + } + for _, piece := range report.Pieces { + r.record.Pieces = append(r.record.Pieces, receipt.Piece{File: piece.File, Line: piece.Line, Outcome: piece.Outcome, Reason: piece.Reason, BuildErrorSuspected: piece.BuildErrorSuspected}) + addOutput(fmt.Sprintf("%s:%d/old", piece.File, piece.Line), piece.OldOutput) + } + if len(report.Tests) == 0 { + seen := make(map[string]bool) + for _, text := range []string{report.NewOutput, report.OldOutput} { + for _, match := range runnerNames.FindAllStringSubmatch(text, -1) { + name := "" + for _, captured := range match[1:] { + if captured != "" { + name = captured + break + } + } + if seen[name] { + continue + } + seen[name] = true + r.record.Tests = append(r.record.Tests, receipt.Test{Name: name, Command: command, Outcome: "aggregate_only", + NewExecution: executionObserved(report.NewOutput, name), OldExecution: executionObserved(report.OldOutput, name)}) + } + } + } + if report.BuildErrorSuspected { + r.record.Warnings = append(r.record.Warnings, "old_code_build_failed") + } + if report.Reason != "" && report.Reason != "old_code_build_failed" { + r.record.Warnings = append(r.record.Warnings, report.Reason) + } + outcome := report.Outcome + if runErr != nil { + outcome = "error" + } + return r.save(outcome, runErr, outputs, stderr) +} + +var runnerNames = regexp.MustCompile(`(?m)^(?:=== RUN\s+(\S+)|[^\r\n]*::(test\S+)\s+(?:PASSED|FAILED)|((?:test)\S*) \([^\r\n]+\) \.\.\.)`) + +// Unsupported runners and truncated output are unknown, not evidence that a +// test was skipped. Only concrete verbose Go/Python markers show execution. +func executionObserved(output, name string) string { + quoted := regexp.QuoteMeta(name) + marker := regexp.MustCompile(`(?m)^(=== RUN\s+` + quoted + `(?:\s|$)|[^\r\n]*::` + quoted + `(?:\s|\[)|` + quoted + ` \([^\r\n]+\) \.\.\.)`) + if marker.MatchString(output) { + return "observed" + } + return "unknown" +} + +func writeReceiptError(stdout io.Writer, err error, fields receiptFields) { + _ = json.NewEncoder(stdout).Encode(struct { + Error string `json:"error"` + receiptFields + }{err.Error(), fields}) +} + +// Keep the invocation options, but never persist the user-supplied advice +// title: it becomes part of the model prompt. Its presence remains explicit. +func adviceCommand(args []string) string { + var command strings.Builder + command.WriteString("test-check") + redactNext := false + for _, arg := range args { + if redactNext { + arg, redactNext = "[redacted]", false + } else { + flag, _, hasValue := strings.Cut(arg, "=") + if flag == "--title" || flag == "-title" { + if hasValue { + arg = flag + "=[redacted]" + } else { + redactNext = true + } + } + } + command.WriteByte(' ') + command.WriteString(strconv.Quote(arg)) + } + return command.String() +} diff --git a/cmd/test-check/receipt_integration_test.go b/cmd/test-check/receipt_integration_test.go new file mode 100644 index 0000000..ae0f66d --- /dev/null +++ b/cmd/test-check/receipt_integration_test.go @@ -0,0 +1,187 @@ +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestProofAutomaticallyPersistsPrivateReceipt(t *testing.T) { + state := t.TempDir() + t.Setenv("XDG_STATE_HOME", state) + dir := receiptFixture(t) + var stdout, stderr bytes.Buffer + code := run([]string{"prove", "--dir", dir, "--base", "main", "--json", "--each", "TestValue", "--test", "echo '=== RUN {name}'; sh value_test.sh"}, &stdout, &stderr) + if code != 0 { + t.Fatalf("exit=%d: %s %s", code, stdout.String(), stderr.String()) + } + var response struct { + Outcome, Receipt string + ReceiptStatus string `json:"receipt_status"` + } + if err := json.Unmarshal(stdout.Bytes(), &response); err != nil { + t.Fatal(err) + } + if response.Outcome != "proven" || response.Receipt == "" || response.ReceiptStatus != "saved" { + t.Fatalf("missing durable evidence: %s", stdout.String()) + } + if !strings.HasPrefix(response.Receipt, state+string(os.PathSeparator)) || strings.HasPrefix(response.Receipt, dir+string(os.PathSeparator)) { + t.Fatalf("unexpected storage: %s", response.Receipt) + } + raw, err := os.ReadFile(response.Receipt) + if err != nil { + t.Fatal(err) + } + var record struct { + Command, Head, Outcome, OutputRef string + } + // Decode the documented snake-case keys, not private package fields. + var data map[string]json.RawMessage + if err := json.Unmarshal(raw, &data); err != nil { + t.Fatal(err) + } + _ = json.Unmarshal(data["command"], &record.Command) + _ = json.Unmarshal(data["head"], &record.Head) + _ = json.Unmarshal(data["outcome"], &record.Outcome) + _ = json.Unmarshal(data["output_ref"], &record.OutputRef) + var tests []struct { + Name string `json:"name"` + New string `json:"new_execution"` + Old string `json:"old_execution"` + } + if err := json.Unmarshal(data["tests"], &tests); err != nil { + t.Fatal(err) + } + if record.Command == "" || len(record.Head) != 40 || record.Outcome != "proven" || len(tests) != 1 || tests[0].Name != "TestValue" || tests[0].New != "observed" || tests[0].Old != "observed" { + t.Fatalf("incomplete receipt: %s", raw) + } + if record.OutputRef == "" { + t.Fatalf("missing diagnostic link: %s", raw) + } + detail := record.OutputRef + if !filepath.IsAbs(detail) { + detail = filepath.Join(filepath.Dir(response.Receipt), detail) + } + details, err := os.ReadFile(detail) + if err != nil || !bytes.Contains(details, []byte("=== RUN")) { + t.Fatalf("detail link: %v: %s", err, details) + } + for _, path := range []string{response.Receipt, detail} { + info, err := os.Stat(path) + if err != nil || info.Mode().Perm() != 0o600 { + t.Fatalf("private file %s: %v %v", path, info, err) + } + } +} + +func TestReceiptFailurePreservesProofAndReportsNoPath(t *testing.T) { + state := filepath.Join(t.TempDir(), "occupied") + if err := os.WriteFile(state, []byte("unchanged"), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("XDG_STATE_HOME", state) + dir := receiptFixture(t) + var stdout, stderr bytes.Buffer + code := run([]string{"prove", "--dir", dir, "--base", "main", "--json", "--test", "sh value_test.sh"}, &stdout, &stderr) + var response struct { + Outcome, Receipt string + Error string `json:"receipt_error"` + Status string `json:"receipt_status"` + } + if err := json.Unmarshal(stdout.Bytes(), &response); err != nil { + t.Fatal(err) + } + if code != 0 || response.Outcome != "proven" || response.Receipt != "" || response.Error == "" || response.Status != "failed" || !strings.Contains(stderr.String(), "receipt could not be saved") { + t.Fatalf("exit=%d response=%s stderr=%s", code, stdout.String(), stderr.String()) + } + content, err := os.ReadFile(filepath.Join(dir, "value.txt")) + if err != nil || string(content) != "new\n" { + t.Fatalf("restoration: %q %v", content, err) + } +} + +func TestReceiptsDistinguishChangedInputAtSameCommit(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + dir := receiptFixture(t) + var previousDigest, previousHead string + for _, text := range []string{"new\n", "new\nanother changed input\n"} { + proofWrite(t, dir, "value.txt", text) + var stdout, stderr bytes.Buffer + code := run([]string{"prove", "--dir", dir, "--base", "main", "--json", "--test", "sh value_test.sh"}, &stdout, &stderr) + if code != 0 { + t.Fatalf("exit=%d: %s %s", code, stdout.String(), stderr.String()) + } + var response struct{ Receipt string } + if err := json.Unmarshal(stdout.Bytes(), &response); err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(response.Receipt) + if err != nil { + t.Fatal(err) + } + var record struct { + Head string + Digest string `json:"worktree_digest"` + } + if err := json.Unmarshal(data, &record); err != nil { + t.Fatal(err) + } + if len(record.Digest) != 64 || len(record.Head) != 40 { + t.Fatalf("unknown identity: %s", data) + } + if previousDigest != "" && (record.Digest == previousDigest || record.Head != previousHead) { + t.Fatalf("different dirty inputs not distinguished: %s", data) + } + previousDigest, previousHead = record.Digest, record.Head + } +} + +func TestAdviceReceiptPreservesOptionsWithoutPromptTitle(t *testing.T) { + t.Setenv("XDG_STATE_HOME", t.TempDir()) + t.Setenv("HOME", t.TempDir()) + t.Setenv(apiKeyName, "") + dir := receiptFixture(t) + for _, titleArgs := range [][]string{{"--title", "private prompt title"}, {"--title=private prompt title"}} { + args := []string{"--dir", dir, "--base", "main", "--model", "fixture-model", "--json"} + args = append(args, titleArgs...) + var stdout, stderr bytes.Buffer + if code := run(args, &stdout, &stderr); code != exitUnavailable { + t.Fatalf("exit=%d: %s %s", code, stdout.String(), stderr.String()) + } + var response struct{ Receipt string } + if err := json.Unmarshal(stdout.Bytes(), &response); err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(response.Receipt) + if err != nil { + t.Fatal(err) + } + var record struct{ Command string } + if err := json.Unmarshal(raw, &record); err != nil { + t.Fatal(err) + } + for _, option := range []string{"--dir", dir, "--base", "main", "--model", "fixture-model", "--json", "--title", "[redacted]"} { + if !strings.Contains(record.Command, option) { + t.Fatalf("missing option %q in command %q", option, record.Command) + } + } + if bytes.Contains(raw, []byte("private prompt title")) { + t.Fatal("receipt persisted raw prompt title") + } + } +} + +func receiptFixture(t *testing.T) string { + t.Helper() + dir := t.TempDir() + proofGit(t, dir, "init", "-q", "-b", "main") + proofWrite(t, dir, "value.txt", "old\n") + proofGit(t, dir, "add", ".") + proofGit(t, dir, "-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "base") + proofWrite(t, dir, "value.txt", "new\n") + proofWrite(t, dir, "value_test.sh", "grep -q '^new$' value.txt\n") + return dir +} diff --git a/internal/prove/pieces.go b/internal/prove/pieces.go index 6895432..3264cbc 100644 --- a/internal/prove/pieces.go +++ b/internal/prove/pieces.go @@ -35,6 +35,8 @@ type PieceResult struct { Line int `json:"line"` Lines int `json:"lines"` Outcome string `json:"outcome"` + Reason string `json:"reason,omitempty"` + OldOutput string `json:"old_output,omitempty"` BuildErrorSuspected bool `json:"build_error_suspected"` Snippet string `json:"snippet"` } @@ -215,7 +217,7 @@ func applyPiece(r repo, pc piece) error { func runPieces(ctx context.Context, r repo, mergeBase string, opts Options, report Report) (Report, error) { pieces, err := codePieces(r, mergeBase, report.RevertedFiles) if err != nil { - return Report{}, err + return report, err } limit := opts.MaxPieces if limit <= 0 { @@ -234,8 +236,9 @@ func runPieces(ctx context.Context, r repo, mergeBase string, opts Options, repo } newOut, newErr := runTest(ctx, r.root, opts.Test, opts.Timeout) report.NewOutput = tail(newOut) - if ctx.Err() != nil { - return Report{}, ctx.Err() + if interruptedRun(newErr) { + report.Reason = "new_run_interrupted" + return report, newErr } if newErr != nil { report.Outcome = OutcomeFailsOnNew @@ -246,26 +249,39 @@ func runPieces(ctx context.Context, r repo, mergeBase string, opts Options, repo res := PieceResult{File: pc.file, Line: pc.line, Lines: pc.lines, Snippet: pc.snippet} if err := applyPiece(r, pc); err != nil { if errors.Is(err, errStashTaken) { - return Report{}, err + return report, err } if restoreErr := Restore(context.WithoutCancel(ctx), r.root); restoreErr != nil { - return Report{}, fmt.Errorf("undo piece %s:%d failed: %v; restore also failed: %w", pc.file, pc.line, err, restoreErr) + return report, fmt.Errorf("undo piece %s:%d failed: %v; restore also failed: %w", pc.file, pc.line, err, restoreErr) } - return Report{}, fmt.Errorf("undo piece %s:%d failed (files restored): %w", pc.file, pc.line, err) + return report, fmt.Errorf("undo piece %s:%d failed (files restored): %w", pc.file, pc.line, err) } out, testErr := runTest(ctx, r.root, opts.Test, opts.Timeout) + res.OldOutput = tail(out) + if interruptedRun(testErr) { + res.Outcome, res.Reason = OutcomeInconclusive, "old_run_interrupted" + } if err := Restore(context.WithoutCancel(ctx), r.root); err != nil { - return Report{}, fmt.Errorf("restore failed; recover with `test-check prove --restore`: %w", err) + report.Pieces = append(report.Pieces, res) + return report, fmt.Errorf("restore failed; recover with `test-check prove --restore`: %w", err) } - if ctx.Err() != nil { - return Report{}, ctx.Err() + if interruptedRun(testErr) { + report.Pieces = append(report.Pieces, res) + return report, testErr } if testErr == nil { res.Outcome = PieceUnguarded - report.Outcome = OutcomePieceNotTested + if report.Outcome != OutcomeInconclusive { + report.Outcome = OutcomePieceNotTested + } } else { - res.Outcome = PieceGuarded - res.BuildErrorSuspected = buildError.MatchString(out) + res.Outcome, res.Reason, res.BuildErrorSuspected = oldFailure(out, "") + if res.Outcome == OutcomeProven { + res.Outcome = PieceGuarded + } else { + report.Outcome, report.Reason = res.Outcome, res.Reason + } + report.BuildErrorSuspected = report.BuildErrorSuspected || res.BuildErrorSuspected } report.Pieces = append(report.Pieces, res) } diff --git a/internal/prove/prove.go b/internal/prove/prove.go index 74bc51b..4d7d4cc 100644 --- a/internal/prove/prove.go +++ b/internal/prove/prove.go @@ -25,6 +25,7 @@ import ( const ( OutcomeProven = "proven" + OutcomeInconclusive = "inconclusive" OutcomeNotRedOnOld = "not_red_on_old" OutcomeFailsOnNew = "fails_on_new" OutcomeNoTestChanged = "no_test_in_change" @@ -45,7 +46,12 @@ const ( // Report is the outcome of one prove run. type Report struct { - Outcome string `json:"outcome"` + Outcome string `json:"outcome"` + Reason string `json:"reason,omitempty"` + Base string `json:"base,omitempty"` + Head string `json:"head,omitempty"` + WorktreeDigest string `json:"worktree_digest,omitempty"` + SnapshotWarning string `json:"snapshot_warning,omitempty"` // BuildErrorSuspected: the old-code run failed with what looks like a // compile or import error, not a failing assertion. A test of a brand-new // API always does this; it proves nothing about a regression. @@ -65,6 +71,7 @@ type Report struct { type TestResult struct { Name string `json:"name"` Outcome string `json:"outcome"` + Reason string `json:"reason,omitempty"` BuildErrorSuspected bool `json:"build_error_suspected"` NewOutput string `json:"new_output,omitempty"` OldOutput string `json:"old_output,omitempty"` @@ -147,7 +154,11 @@ func Run(ctx context.Context, opts Options) (Report, error) { if err != nil { return Report{}, err } - var report Report + report := Report{Base: mergeBase} + report.Head, report.WorktreeDigest, err = snapshotDigest(ctx, r, mergeBase, changed) + if err != nil { + report.SnapshotWarning = "working-tree snapshot unavailable: " + err.Error() + } for _, p := range changed { if IsTestSide(p) { report.TestFiles = append(report.TestFiles, p) @@ -166,7 +177,7 @@ func Run(ctx context.Context, opts Options) (Report, error) { if opts.Pieces { if strings.Contains(opts.Test, NamePlaceholder) { - return Report{}, errors.New("piece mode runs the test command as given; drop {name}") + return report, errors.New("piece mode runs the test command as given; drop {name}") } return runPieces(ctx, r, mergeBase, opts, report) } @@ -176,10 +187,10 @@ func Run(ctx context.Context, opts Options) (Report, error) { names := opts.Each if len(names) == 0 { if names, err = ChangedTestNames(r.git, mergeBase, r.root, report.TestFiles); err != nil { - return Report{}, err + return report, err } if len(names) == 0 { - return Report{}, errors.New("{name} given but no added or edited test functions were found in the change; pass --each NAME,...") + return report, errors.New("{name} given but no added or edited test functions were found in the change; pass --each NAME,...") } } return runEach(ctx, r, mergeBase, opts, report, names) @@ -189,8 +200,9 @@ func runWhole(ctx context.Context, r repo, mergeBase string, opts Options, repor newOut, newErr := runTest(ctx, r.root, opts.Test, opts.Timeout) report.NewOutput = tail(newOut) if newErr != nil { - if ctx.Err() != nil { - return Report{}, ctx.Err() + if interruptedRun(newErr) { + report.Reason = "new_run_interrupted" + return report, newErr } report.Outcome = OutcomeFailsOnNew return report, nil @@ -199,16 +211,19 @@ func runWhole(ctx context.Context, r repo, mergeBase string, opts Options, repor var oldErr error if err := onBase(ctx, r, mergeBase, report.RevertedFiles, func() { oldOut, oldErr = runTest(ctx, r.root, opts.Test, opts.Timeout) + report.OldOutput = tail(oldOut) }); err != nil { - return Report{}, err + return report, err + } + if interruptedRun(oldErr) { + report.Reason = "old_run_interrupted" + return report, oldErr } - report.OldOutput = tail(oldOut) if oldErr == nil { report.Outcome = OutcomeNotRedOnOld return report, nil } - report.Outcome = OutcomeProven - report.BuildErrorSuspected = buildError.MatchString(oldOut) + report.Outcome, report.Reason, report.BuildErrorSuspected = oldFailure(oldOut, "") return report, nil } @@ -216,14 +231,16 @@ func runWhole(ctx context.Context, r repo, mergeBase string, opts Options, repor // the ones that passed on the old code, so the tree is swapped a single time. func runEach(ctx context.Context, r repo, mergeBase string, opts Options, report Report, names []string) (Report, error) { results := make([]TestResult, len(names)) + report.Tests = results var candidates []int for i, name := range names { results[i].Name = name out, err := runTest(ctx, r.root, strings.ReplaceAll(opts.Test, NamePlaceholder, name), opts.Timeout) - if ctx.Err() != nil { - return Report{}, ctx.Err() - } results[i].NewOutput = tail(out) + if interruptedRun(err) { + results[i].Outcome, results[i].Reason = OutcomeInconclusive, "new_run_interrupted" + return report, err + } switch { case err != nil: results[i].Outcome = OutcomeFailsOnNew @@ -234,6 +251,7 @@ func runEach(ctx context.Context, r repo, mergeBase string, opts Options, report } } if len(candidates) > 0 { + var interrupted error if err := onBase(ctx, r, mergeBase, report.RevertedFiles, func() { for _, i := range candidates { if ctx.Err() != nil { @@ -241,23 +259,30 @@ func runEach(ctx context.Context, r repo, mergeBase string, opts Options, report } out, err := runTest(ctx, r.root, strings.ReplaceAll(opts.Test, NamePlaceholder, names[i]), opts.Timeout) results[i].OldOutput = tail(out) + if interruptedRun(err) { + results[i].Outcome, results[i].Reason = OutcomeInconclusive, "old_run_interrupted" + interrupted = err + return + } if err == nil { results[i].Outcome = OutcomeNotRedOnOld continue } - results[i].Outcome = OutcomeProven - results[i].BuildErrorSuspected = buildError.MatchString(out) + results[i].Outcome, results[i].Reason, results[i].BuildErrorSuspected = oldFailure(out, names[i]) } }); err != nil { - return Report{}, err + return report, err + } + if interrupted != nil { + return report, interrupted } } - report.Tests = results report.Outcome = OutcomeProven - for _, worst := range []string{OutcomeNotRedOnOld, OutcomeNotRun, OutcomeFailsOnNew} { + for _, worst := range []string{OutcomeNotRedOnOld, OutcomeInconclusive, OutcomeNotRun, OutcomeFailsOnNew} { for _, res := range results { if res.Outcome == worst { report.Outcome = worst + report.Reason = res.Reason } } } @@ -600,12 +625,35 @@ func runTest(ctx context.Context, root, command string, timeout time.Duration) ( cmd.WaitDelay = 5 * time.Second out, err := cmd.CombinedOutput() if runCtx.Err() == context.DeadlineExceeded && ctx.Err() == nil { - return string(out), fmt.Errorf("test command timed out after %s", timeout) + return string(out), fmt.Errorf("test command timed out after %s: %w", timeout, context.DeadlineExceeded) + } + if ctx.Err() != nil { + return string(out), ctx.Err() } return string(out), err } -var buildError = regexp.MustCompile(`(?m)(undefined: |cannot find package|no required module|build failed|compilation failed|error TS\d+|Cannot find module|ModuleNotFoundError|ImportError: |NameError: name|error: cannot find '|error\[E0(425|433)\]|SyntaxError: )`) +func interruptedRun(err error) bool { + return errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) +} + +var buildError = regexp.MustCompile(`(?m)(undefined: |cannot find package|no required module|build failed|compilation failed|error TS\d+|Cannot find module|ModuleNotFoundError|ImportError: |NameError: name|error: cannot find '|error\[E0(425|433)\]|SyntaxError: |^(?:E\s+)?AttributeError: module '[^']+' has no attribute )`) + +// Runner failure summaries take precedence over diagnostic-looking assertion +// messages. In particular, an assertion containing "ImportError:" is not an +// import failure. These are the existing Go/Python runner contracts, not a +// parser for arbitrary test frameworks. +var assertionFailure = regexp.MustCompile(`(?m)^(--- FAIL: \S+|FAILED [^\r\n]*::\S+|FAIL: test\S*[ (])`) + +func oldFailure(out, name string) (outcome, reason string, buildFailed bool) { + if !assertionFailure.MatchString(out) && buildError.MatchString(out) { + return OutcomeInconclusive, "old_code_build_failed", true + } + if name != "" && !strings.Contains(out, name) { + return OutcomeInconclusive, "old_test_not_observed", false + } + return OutcomeProven, "", false +} func tail(out string) string { if len(out) <= tailBytes { diff --git a/internal/prove/prove_test.go b/internal/prove/prove_test.go index 68da87c..f4ee05a 100644 --- a/internal/prove/prove_test.go +++ b/internal/prove/prove_test.go @@ -104,7 +104,7 @@ func TestRunOutcomesAndTheWorkingTreeIsRestored(t *testing.T) { {"test that catches the bug", "grep -q 'return errStop' retry.go\n", OutcomeProven, false}, {"test that passes on the old code", "grep -q 'func Retry' retry.go && true\n", OutcomeNotRedOnOld, false}, {"test that fails on the new code", "grep -q 'return nil' retry.go\n", OutcomeFailsOnNew, false}, - {"red only because a new name is undefined", "grep -q errStop stop.go || { echo 'undefined: errStop'; exit 1; }\n", OutcomeProven, true}, + {"red only because a new name is undefined", "grep -q errStop stop.go || { echo 'undefined: errStop'; exit 1; }\n", OutcomeInconclusive, true}, {"no test changed", "", OutcomeNoTestChanged, false}, } for _, tc := range cases { diff --git a/internal/prove/snapshot.go b/internal/prove/snapshot.go new file mode 100644 index 0000000..4d78b83 --- /dev/null +++ b/internal/prove/snapshot.go @@ -0,0 +1,94 @@ +package prove + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "os" + "path/filepath" + "strings" +) + +// snapshotDigest identifies the already-enumerated change before any runner or +// temporary revert. Stream contents, never retain another full diff or copy. +// The base and HEAD identify unchanged tracked inputs; changed paths include +// test files and untracked files. Symlinks are hashed as links, not followed. +func snapshotDigest(ctx context.Context, r repo, base string, paths []string) (head, digest string, err error) { + head, err = r.git("rev-parse", "HEAD") + if err != nil { + return "", "", err + } + head = strings.TrimSpace(head) + h := sha256.New() + fmt.Fprintf(h, "base:%s\nhead:%s\n", base, head) + const maxSnapshotBytes = 64 << 20 + var total int64 + var buffer [32 << 10]byte + for _, name := range paths { + fmt.Fprintf(h, "path:%d:%s\n", len(name), name) + if err := ctx.Err(); err != nil { + return head, "", err + } + path := filepath.Join(r.root, name) + info, statErr := os.Lstat(path) + if os.IsNotExist(statErr) { + io.WriteString(h, "deleted\n") + continue + } + if statErr != nil { + return head, "", statErr + } + fmt.Fprintf(h, "mode:%s\n", info.Mode()) + if info.Mode()&os.ModeSymlink != 0 { + target, readErr := os.Readlink(path) + if readErr != nil { + return head, "", readErr + } + fmt.Fprintf(h, "link:%d:%s\n", len(target), target) + continue + } + if !info.Mode().IsRegular() { + return head, "", fmt.Errorf("cannot snapshot non-regular changed path %q", name) + } + if info.Size() > maxSnapshotBytes-total { + return head, "", fmt.Errorf("changed input exceeds %d-byte snapshot limit", maxSnapshotBytes) + } + fmt.Fprintf(h, "size:%d\n", info.Size()) + file, openErr := os.Open(path) + if openErr != nil { + return head, "", openErr + } + var readErr error + for { + if readErr = ctx.Err(); readErr != nil { + break + } + var n int + n, readErr = file.Read(buffer[:]) + total += int64(n) + if total > maxSnapshotBytes { + readErr = fmt.Errorf("changed input exceeds %d-byte snapshot limit", maxSnapshotBytes) + break + } + h.Write(buffer[:n]) + if readErr == io.EOF { + readErr = nil + break + } + if readErr != nil { + break + } + } + closeErr := file.Close() + if readErr != nil { + return head, "", readErr + } + if closeErr != nil { + return head, "", closeErr + } + io.WriteString(h, "\n") + } + return head, hex.EncodeToString(h.Sum(nil)), nil +} diff --git a/internal/prove/snapshot_test.go b/internal/prove/snapshot_test.go new file mode 100644 index 0000000..2d210bd --- /dev/null +++ b/internal/prove/snapshot_test.go @@ -0,0 +1,64 @@ +package prove + +import ( + "context" + "os" + "path/filepath" + "testing" +) + +func TestChangedInputSnapshotDistinguishesLinksAndBoundsReads(t *testing.T) { + dir := fixture(t) + r, err := open(context.Background(), dir) + if err != nil { + t.Fatal(err) + } + base, err := r.git("merge-base", "main", "HEAD") + if err != nil { + t.Fatal(err) + } + target := filepath.Join(t.TempDir(), "outside") + if err := os.WriteFile(target, []byte("first"), 0o600); err != nil { + t.Fatal(err) + } + link := filepath.Join(dir, "link") + if err := os.Symlink(target, link); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + _, first, err := snapshotDigest(context.Background(), r, base, []string{"link"}) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(target, []byte("changed external content"), 0o600); err != nil { + t.Fatal(err) + } + _, second, err := snapshotDigest(context.Background(), r, base, []string{"link"}) + if err != nil || first != second { + t.Fatalf("link target content was followed: %s %s %v", first, second, err) + } + if err := os.Mkdir(filepath.Join(dir, "directory"), 0o700); err != nil { + t.Fatal(err) + } + file, err := os.Create(filepath.Join(dir, "oversized")) + if err != nil { + t.Fatal(err) + } + if err := file.Truncate((64 << 20) + 1); err != nil { + t.Fatal(err) + } + if err := file.Close(); err != nil { + t.Fatal(err) + } + for _, name := range []string{"directory", "oversized"} { + _, digest, err := snapshotDigest(context.Background(), r, base, []string{name}) + if err == nil || digest != "" { + t.Fatalf("%s: unsupported identity claimed: %q %v", name, digest, err) + } + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, digest, err := snapshotDigest(ctx, r, base, []string{"retry.go"}) + if err == nil || digest != "" { + t.Fatalf("canceled snapshot claimed identity: %q %v", digest, err) + } +} diff --git a/internal/receipt/receipt.go b/internal/receipt/receipt.go new file mode 100644 index 0000000..bdf9ab0 --- /dev/null +++ b/internal/receipt/receipt.go @@ -0,0 +1,259 @@ +// Package receipt stores local evidence independently of prove's recovery state. +// A receipt describes an invocation, not approval or a merge gate. Command and +// runner output may contain secrets supplied by the user; files stay private and +// local, with no claim that their contents have been redacted. +package receipt + +import ( + "crypto/rand" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "time" +) + +const ( + Version = 1 + // Match the existing prove runner's diagnostic tail rather than retaining + // unbounded command output. Limit count and labels as well as output text. + outputBytes = 2000 + outputLimit = 256 + labelBytes = 256 +) + +// Record deliberately excludes input patches, prompts, environment and config. +// Missing identity or execution evidence must remain unknown, not inferred. +type Record struct { + Version int `json:"version"` + Mode string `json:"mode"` + Repo string `json:"repo,omitempty"` + Base string `json:"base,omitempty"` + Head string `json:"head,omitempty"` + WorktreeDigest string `json:"worktree_digest,omitempty"` + Command string `json:"command,omitempty"` + Outcome string `json:"outcome"` + Error string `json:"error,omitempty"` + ToolRevision string `json:"tool_revision"` + StartedAt time.Time `json:"started_at"` + FinishedAt time.Time `json:"finished_at"` + Tests []Test `json:"tests,omitempty"` + Pieces []Piece `json:"pieces,omitempty"` + Warnings []string `json:"warnings,omitempty"` + OutputRef string `json:"output_ref"` + // Checkout is only a storage safety boundary, not persisted metadata. When + // absent an absolute Repo path supplies the same boundary. + Checkout string `json:"-"` +} + +// Execution values are observed, not_observed, or unknown. They describe named +// test evidence from the runner, not whether a shell process was launched. +type Test struct { + Name string `json:"name"` + Command string `json:"command,omitempty"` + Outcome string `json:"outcome"` + NewExecution string `json:"new_execution"` + OldExecution string `json:"old_execution"` + Warnings []string `json:"warnings,omitempty"` +} + +type Piece struct { + File string `json:"file"` + Line int `json:"line"` + Outcome string `json:"outcome"` + Reason string `json:"reason,omitempty"` + BuildErrorSuspected bool `json:"build_error_suspected"` +} + +type Output struct { + Label string `json:"label"` + Text string `json:"text"` +} + +type Paths struct { + Receipt string + Details string +} + +type details struct { + Version int `json:"version"` + Outputs []Output `json:"outputs"` + Truncated bool `json:"truncated"` + OmittedOutputs int `json:"omitted_outputs,omitempty"` +} + +// Save writes details first, then atomically publishes receipt.json as the +// completion marker. Each invocation has its own unpredictable directory. +// Call only after restoration; errors must not replace the original run result. +// No model context is accepted: cancellation of advice must not cancel saving +// its outcome. This does not guarantee capture after SIGKILL or power loss. +// Empty root uses XDG_STATE_HOME/test-check/receipts, falling back to +// ~/.local/state/test-check/receipts. Existing directories are never chmodded. +func Save(root string, record Record, outputs []Output) (paths Paths, err error) { + var ownedParent string + if root == "" { + state := os.Getenv("XDG_STATE_HOME") + if state == "" || !filepath.IsAbs(state) { + home, homeErr := os.UserHomeDir() + if homeErr != nil { + return Paths{}, fmt.Errorf("receipt home: %w", homeErr) + } + state = filepath.Join(home, ".local", "state") + } + ownedParent = filepath.Join(state, "test-check") + root = filepath.Join(ownedParent, "receipts") + } + root, err = filepath.Abs(root) + if err != nil { + return Paths{}, err + } + checkout := record.Checkout + if checkout == "" && filepath.IsAbs(record.Repo) { + checkout = record.Repo + } + if checkout != "" { + resolvedRoot, resolveErr := resolveFuturePath(root) + if resolveErr != nil { + return Paths{}, resolveErr + } + resolvedCheckout, resolveErr := resolveFuturePath(checkout) + if resolveErr != nil { + return Paths{}, resolveErr + } + rel, relErr := filepath.Rel(resolvedCheckout, resolvedRoot) + if relErr != nil { + return Paths{}, relErr + } + if rel != ".." && !strings.HasPrefix(rel, ".."+string(os.PathSeparator)) { + return Paths{}, errors.New("receipt storage must be outside the checkout") + } + } + if ownedParent != "" { + if err := privateDir(ownedParent); err != nil { + return Paths{}, err + } + } + if err := privateDir(root); err != nil { + return Paths{}, err + } + // Root anchors operations even if an ancestor is renamed. Fixed artifact + // names cannot follow a symlink outside this directory. + storage, err := os.OpenRoot(root) + if err != nil { + return Paths{}, err + } + defer storage.Close() + var random [16]byte + if _, err := rand.Read(random[:]); err != nil { + return Paths{}, err + } + name := time.Now().UTC().Format("20060102T150405Z") + "-" + hex.EncodeToString(random[:]) + if err := storage.Mkdir(name, 0o700); err != nil { + return Paths{}, err + } + defer func() { + if err != nil { + // Only this invocation's unpublished artifacts are removed. + _ = storage.RemoveAll(name) + } + }() + dir, err := storage.OpenRoot(name) + if err != nil { + return Paths{}, err + } + defer dir.Close() + paths = Paths{ + Receipt: filepath.Join(root, name, "receipt.json"), + Details: filepath.Join(root, name, "detail.json"), + } + record.Version = Version + record.OutputRef = paths.Details + if err = publish(dir, "detail.json", boundedDetails(outputs)); err != nil { + return Paths{}, err + } + if err = publish(dir, "receipt.json", record); err != nil { + return Paths{}, err + } + return paths, nil +} + +func privateDir(path string) error { + if err := os.MkdirAll(path, 0o700); err != nil { + return err + } + info, err := os.Lstat(path) + if err != nil { + return err + } + if !info.IsDir() || info.Mode()&os.ModeSymlink != 0 || info.Mode().Perm()&0o077 != 0 { + return fmt.Errorf("receipt directory %q must be a private directory, not a symlink", path) + } + return nil +} + +// Resolve existing ancestors too, so a symlink cannot hide a checkout-contained +// destination that has not been created yet. +func resolveFuturePath(path string) (string, error) { + abs, err := filepath.Abs(path) + if err != nil { + return "", err + } + resolved, err := filepath.EvalSymlinks(abs) + if err == nil { + return resolved, nil + } + if !errors.Is(err, os.ErrNotExist) { + return "", err + } + parent := filepath.Dir(abs) + if parent == abs { + return "", err + } + resolved, err = resolveFuturePath(parent) + if err != nil { + return "", err + } + return filepath.Join(resolved, filepath.Base(abs)), nil +} + +func publish(dir *os.Root, name string, value any) error { + file, err := dir.OpenFile("."+name+".tmp", os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600) + if err != nil { + return err + } + encoder := json.NewEncoder(file) + encoder.SetIndent("", " ") + err = encoder.Encode(value) + if err == nil { + err = file.Sync() + } + closeErr := file.Close() + if err != nil { + return err + } + if closeErr != nil { + return closeErr + } + return dir.Rename("."+name+".tmp", name) +} + +func boundedDetails(outputs []Output) details { + count := min(len(outputs), outputLimit) + result := details{Version: Version, Outputs: make([]Output, count), OmittedOutputs: len(outputs) - count} + result.Truncated = result.OmittedOutputs > 0 + for i, output := range outputs[:count] { + if len(output.Label) > labelBytes { + output.Label = strings.ToValidUTF8(output.Label[:labelBytes], "") + "..." + result.Truncated = true + } + if len(output.Text) > outputBytes { + output.Text = "..." + strings.ToValidUTF8(output.Text[len(output.Text)-outputBytes:], "") + result.Truncated = true + } + result.Outputs[i] = output + } + return result +} diff --git a/internal/receipt/receipt_test.go b/internal/receipt/receipt_test.go new file mode 100644 index 0000000..525c6a3 --- /dev/null +++ b/internal/receipt/receipt_test.go @@ -0,0 +1,269 @@ +package receipt + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +// Concurrent invocations must not overwrite one another or expose a receipt +// whose detail link is missing. The warning distinguishes each invocation. +func TestConcurrentReceiptsArePrivateAndComplete(t *testing.T) { + root := filepath.Join(t.TempDir(), "receipts") + const runs = 24 + var workers sync.WaitGroup + var paths sync.Map + stop := make(chan struct{}) + observerDone := make(chan struct{}) + go func() { + defer close(observerDone) + for { + select { + case <-stop: + return + default: + } + files, _ := filepath.Glob(filepath.Join(root, "*", "receipt.json")) + for _, file := range files { + var record Record + data, err := os.ReadFile(file) + if err != nil { + t.Errorf("read published receipt: %v", err) + return + } + if err := json.Unmarshal(data, &record); err != nil { + t.Errorf("partially published receipt: %v", err) + return + } + var detail details + data, err = os.ReadFile(record.OutputRef) + if err != nil { + t.Errorf("receipt published before detail: %v", err) + return + } + if err := json.Unmarshal(data, &detail); err != nil { + t.Errorf("partially published detail: %v", err) + return + } + } + } + }() + for i := range runs { + workers.Add(1) + go func() { + defer workers.Done() + warning := fmt.Sprintf("run %d: old runner failed to build", i) + path, err := Save(root, Record{Outcome: "inconclusive", Warnings: []string{warning}}, []Output{{Label: "old", Text: warning}}) + if err != nil { + t.Error(err) + return + } + if _, duplicate := paths.LoadOrStore(path.Receipt, true); duplicate { + t.Error("concurrent invocation reused a receipt path") + } + var record Record + readJSON(t, path.Receipt, &record) + if len(record.Warnings) != 1 || record.Warnings[0] != warning { + t.Error("warning lost or overwritten by another invocation") + } + if record.OutputRef != path.Details { + t.Error("receipt does not link its own detail") + } + var detail details + readJSON(t, record.OutputRef, &detail) + if len(detail.Outputs) != 1 || detail.Outputs[0].Text != warning { + t.Error("output lost or overwritten by another invocation") + } + assertMode(t, path.Receipt, 0o600) + assertMode(t, path.Details, 0o600) + assertMode(t, filepath.Dir(path.Receipt), 0o700) + }() + } + workers.Wait() + close(stop) + <-observerDone + assertMode(t, root, 0o700) + files, err := filepath.Glob(filepath.Join(root, "*", "receipt.json")) + if err != nil || len(files) != runs { + t.Fatalf("retained receipts: %d, error: %v", len(files), err) + } +} + +func TestUnsafeStorageDoesNotTouchCheckoutOrChangePermissions(t *testing.T) { + base := t.TempDir() + checkout := filepath.Join(base, "checkout") + if err := os.Mkdir(checkout, 0o700); err != nil { + t.Fatal(err) + } + // Stand in for recovery state: saving a failed run must not touch it. + recovery := filepath.Join(checkout, "recovery") + if err := os.WriteFile(recovery, []byte("restore this original"), 0o600); err != nil { + t.Fatal(err) + } + unsafe := filepath.Join(base, "public") + if err := os.Mkdir(unsafe, 0o700); err != nil { + t.Fatal(err) + } + if err := os.Chmod(unsafe, 0o755); err != nil { + t.Fatal(err) + } + link := filepath.Join(base, "linked") + if err := os.Symlink(checkout, link); err != nil { + t.Fatal(err) + } + private := filepath.Join(base, "private") + if err := os.Mkdir(private, 0o700); err != nil { + t.Fatal(err) + } + privateLink := filepath.Join(base, "private-link") + if err := os.Symlink(private, privateLink); err != nil { + t.Fatal(err) + } + cases := []string{ + unsafe, + privateLink, + filepath.Join(checkout, "receipts"), + filepath.Join(link, "receipts"), + recovery, // An occupied path must fail even when tests run as root. + } + for _, root := range cases { + path, err := Save(root, Record{Checkout: checkout, Outcome: "failed"}, nil) + if err == nil || path != (Paths{}) { + t.Errorf("unsafe storage %q returned paths %#v, error %v", root, path, err) + } + } + assertMode(t, unsafe, 0o755) + original, err := os.ReadFile(recovery) + if err != nil || string(original) != "restore this original" { + t.Fatalf("receipt save changed recovery state: %q, %v", original, err) + } + entries, err := os.ReadDir(checkout) + if err != nil || len(entries) != 1 { + t.Fatalf("failed receipt created artifacts in checkout: %v, %v", entries, err) + } +} + +func TestDefaultStorageIsPrivateAndIndependentOfCheckout(t *testing.T) { + state := t.TempDir() + t.Setenv("XDG_STATE_HOME", state) + checkout := t.TempDir() + path, err := Save("", Record{Repo: checkout}, nil) + if err != nil { + t.Fatal(err) + } + root := filepath.Join(state, "test-check", "receipts") + if !strings.HasPrefix(path.Receipt, root+string(os.PathSeparator)) { + t.Fatalf("receipt not in durable state directory: %s", path.Receipt) + } + assertMode(t, filepath.Dir(root), 0o700) + assertMode(t, root, 0o700) + entries, err := os.ReadDir(checkout) + if err != nil || len(entries) != 0 { + t.Fatalf("receipt storage touched checkout: %v, %v", entries, err) + } + // Tool-owned parent may not silently relax or repair existing permissions. + if err := os.Chmod(filepath.Dir(root), 0o755); err != nil { + t.Fatal(err) + } + if _, err := Save("", Record{Repo: checkout}, nil); err == nil { + t.Fatal("accepted unsafe preexisting tool state directory") + } + assertMode(t, filepath.Dir(root), 0o755) +} + +func TestBoundedDetailDoesNotAlterOriginalDiagnostic(t *testing.T) { + text := strings.Repeat("private diagnostic ", 400) + "final assertion failure" + outputs := make([]Output, outputLimit+1) + for i := range outputs { + outputs[i] = Output{Label: strings.Repeat("x", labelBytes+1), Text: text} + } + path, err := Save(filepath.Join(t.TempDir(), "receipts"), Record{Warnings: []string{"original warning"}}, outputs) + if err != nil { + t.Fatal(err) + } + var detail details + readJSON(t, path.Details, &detail) + if !detail.Truncated || detail.OmittedOutputs != 1 || len(detail.Outputs) != outputLimit { + t.Fatal("unbounded detail or silent output omission") + } + for _, output := range detail.Outputs { + if len(output.Text) > outputBytes+3 || !strings.HasSuffix(output.Text, "final assertion failure") { + t.Fatal("detail did not retain bounded diagnostic tail") + } + if len(output.Label) > labelBytes+3 { + t.Fatal("unbounded output label") + } + } + for _, output := range outputs { + if output.Text != text || len(output.Label) != labelBytes+1 { + t.Fatal("saving a receipt mutated the caller's original diagnostic") + } + } +} + +// A failed publication must leave earlier evidence intact and must not return +// a path that callers could mistake for a successfully saved receipt. +func TestFailedSaveDoesNotReplaceEarlierEvidence(t *testing.T) { + root := filepath.Join(t.TempDir(), "receipts") + first, err := Save(root, Record{Outcome: "original"}, []Output{{Text: "keep"}}) + if err != nil { + t.Fatal(err) + } + original, err := os.ReadFile(first.Receipt) + if err != nil { + t.Fatal(err) + } + // time.Time's JSON encoder rejects this year after details have been + // written, exercising cleanup of a controlled late write failure. + path, err := Save(root, Record{FinishedAt: time.Date(10000, 1, 1, 0, 0, 0, 0, time.UTC)}, nil) + if err == nil || path != (Paths{}) { + t.Fatalf("failed publication returned %#v, error %v", path, err) + } + entries, err := os.ReadDir(root) + if err != nil || len(entries) != 1 { + t.Fatalf("failed publication left artifacts: %v, %v", entries, err) + } + after, err := os.ReadFile(first.Receipt) + if err != nil || string(after) != string(original) { + t.Fatalf("failed save changed earlier evidence: %v", err) + } + occupied := filepath.Join(t.TempDir(), "occupied") + if err := os.WriteFile(occupied, []byte("do not overwrite"), 0o600); err != nil { + t.Fatal(err) + } + path, err = Save(occupied, Record{Outcome: "failed"}, nil) + if err == nil || path != (Paths{}) { + t.Fatalf("occupied storage returned %#v, error %v", path, err) + } + after, err = os.ReadFile(occupied) + if err != nil || string(after) != "do not overwrite" { + t.Fatalf("failed storage creation changed existing file: %v", err) + } +} + +func readJSON(t *testing.T, path string, target any) { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(data, target); err != nil { + t.Fatal(err) + } +} + +func assertMode(t *testing.T, path string, want os.FileMode) { + t.Helper() + info, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != want { + t.Errorf("%s permissions %04o, want %04o", path, info.Mode().Perm(), want) + } +} diff --git a/skills/test-check/SKILL.md b/skills/test-check/SKILL.md index 0ef16e7..f36888b 100644 --- a/skills/test-check/SKILL.md +++ b/skills/test-check/SKILL.md @@ -24,15 +24,19 @@ A permanent test earns its place only if it protects behavior someone relies on | outcome | exit | meaning | |---|---|---| | `proven` | 0 | Red on the old code, green on the new. Done. | + | `inconclusive` | 4 | The old run only failed to build/import or lacked a Python module API, or did not show the selected test. This is not behavioral regression proof; use an old-compatible interface and a verbose runner. | | `not_red_on_old` | 4 | The test passes without the fix. Make it fail without the fix — unless it is a guard test ("X is *not* matched", "Y stays unchanged") or only its fixture changed: those pass on old code by design; say so in the PR. | | `fails_on_new` | 4 | The test fails on your code. Fix that first. | | `no_test_in_change` | 4 | No test file changed. Add one, or treat it as a one-off check. | | `test_not_run` | 4 | The command never mentioned the test, so it probably selected nothing. Fix the command or the name. | - A `build_error_suspected` warning means the old run failed only because the test uses new names; for a bug fix, test through an interface the old code already had. If a run is interrupted, `test-check prove --restore` puts your files back. For a one-off check: run it and keep the output. + `build_error_suspected: true` with `reason: old_code_build_failed` is **inconclusive**, never proof. An assertion's message containing a diagnostic word is not itself a build failure. If a run is interrupted, `test-check prove --restore` puts your files back. For a one-off check: run it and keep the output. + Caller cancellation and test-check's own command deadline return exit 1 after restoration, never proof; interrupted child rows remain inconclusive. A test framework's own assertion/panic is not a tool deadline. Optionally, look for changed code no test checks: `test-check prove --pieces --test ""` (`test-check prove --list-tests` prints them). It undoes each changed piece of code on its own and lists every `not_tested` piece. Many are fine (logging, wording, defaults); use the list to decide what deserves a test, not as a pass/fail. Done when the PR description records the `prove` outcome (or the one-off command and what you observed) — not before. + Commands automatically save private local receipts outside the checkout under `$XDG_STATE_HOME/test-check/receipts` (fallback `$HOME/.local/state/test-check/receipts`): command, available commit identity, named-test runner evidence, result/warnings, and links to bounded diagnostic tails, not full logs. The path is printed on stderr and included as `receipt` in JSON. Unknown execution is not evidence a test ran. Receipt-save failures warn without changing the original outcome/exit; receipts do not prove quality, upload data, or replace recovery state. Commands/output may contain sensitive literals; do not publish receipts without checking them. + `test-check` is advice. Your own evidence decides; never skip verification because it said no test. ## The four questions