diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..c883dd4 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,50 @@ +name: ci + +on: + push: + branches: [main] + pull_request: + +permissions: + contents: read + +concurrency: + group: ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + name: format, vet, build, test + runs-on: ubuntu-latest + env: + CGO_ENABLED: "0" + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + cache: true + + - name: gofmt + run: | + unformatted="$(gofmt -l .)" + if [ -n "$unformatted" ]; then + echo "these files are not gofmt-clean:" + echo "$unformatted" + exit 1 + fi + + - name: go vet + run: go vet ./... + + - name: go build + run: go build ./... + + - name: go test + run: go test ./... + + - name: go.mod is tidy + run: | + go mod tidy + git diff --exit-code go.mod diff --git a/README.md b/README.md index 04e1a1d..5e6a06b 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,80 @@ # test-check -Asks JEV whether a code change needs a regression test, an extended existing -test, a one-off check, or no permanent test. Proposed in gitmoot/gitmoot#2265. +Asks [JEV](https://openrouter.ai/typesafe) what testing a code change needs, so +agents write the regression tests that matter and skip the ones that only add +cost. Proposed in gitmoot/gitmoot#2265. + +It sends JEV the diff and the nearest existing tests, and gets back one +**advice**: + +| advice | meaning | +|---|---| +| `new_regression_test` | Behavior changed and nothing nearby tests it: add a test that fails on the old code. | +| `extend_existing_test` | A nearby test already covers this area: add a case to it. | +| `one_off_check` | Real change, but a permanent test would be brittle or only restate code: run it once. | +| `no_permanent_test` | No behavior change: build and run the existing tests. | +| `need_context` | JEV could not decide. | + +It also reports two probabilities: +- **covered**: whether the change's own tests would fail on a regression. +- **low-value**: whether an added test only restates code, duplicates another test, tests internals, or would pass on the old code. + +It is advice, not a gate. The agent still proves the change works; see the +[skill](skills/test-check/SKILL.md). + +## Prove a test goes red + +JEV reading a diff cannot tell whether a test would pass on the old code. So +`test-check prove` runs it: + +```sh +test-check prove --test "go test ./pkg -v -run '^{name}$'" +``` + +1. It runs each test the change adds or edits on the working tree. Each must pass. +2. It puts every changed non-test file back to its base version and runs each test again. Each must fail. +3. It restores the files. + +`{name}` is filled in for each test. The tests are taken from the change's diff for Go and Python; pass `--each A,B` for other runners. The command must print each test's name (use a verbose runner); otherwise the test is reported as `test_not_run`. Every test must go red on its own. A change is not proven just because one good test sits next to a weak one. + +Without `{name}`, the command runs once for the whole change, and one red test is enough. + +Test files and fixtures keep their new versions for both runs. The reverted files are saved under the git dir before anything changes. An interrupted or crashed run is recovered with `test-check prove --restore`, and a new run refuses to start until then. + +Only one prove run may use a checkout at a time: a lock under the git dir makes a second run, or `--restore`, refuse while the first is alive. A lock left by a dead process is taken over. A test that hangs on the old code until `--timeout` counts as red. + +Outcomes: +- `proven`: exit 0. +- `not_red_on_old`: exit 4. The test passes without the fix. +- `fails_on_new`: exit 4. +- `no_test_in_change`: exit 4. +- `test_not_run`: exit 4. +- `no_code_in_change`: exit 0. Only tests changed. + +A `build_error_suspected` flag marks an old-code failure that looks like a compile or import error rather than a failed assertion. + +## Use + +```sh +test-check # local checkout, incl. uncommitted files +test-check --repo OWNER/REPO --pr N # a pull request's current head +test-check --repo OWNER/REPO --compare BASE...HEAD +test-check --json # machine-readable +``` + +Exit codes: +- `0`: advice printed. +- `1`: the change could not be read. +- `2`: usage error. +- `3`: JEV unavailable. No advice is printed; answer the skill's four questions yourself. + +The GitHub modes use the `gh` CLI. The key comes from `$OPENROUTER_API_KEY`, +else the `OPENROUTER_API_KEY` line in `~/.config/gitmoot/keychain.env`. The +model is pinned to `typesafe/jev-1.13`. + +## Install + +```sh +go build -o ~/.local/bin/test-check ./cmd/test-check +ln -s "$PWD/skills/test-check" ~/.claude/skills/test-check +``` diff --git a/cmd/test-check/main.go b/cmd/test-check/main.go new file mode 100644 index 0000000..7c1ccea --- /dev/null +++ b/cmd/test-check/main.go @@ -0,0 +1,165 @@ +// Command test-check asks JEV what testing a code change needs, and proves +// that a change's tests go red on the old code and green on the new. +package main + +import ( + "bufio" + "context" + "encoding/json" + "flag" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" + + "github.com/gitmoot/test-check/internal/check" + "github.com/gitmoot/test-check/internal/jev" + "github.com/gitmoot/test-check/internal/source" +) + +const ( + exitUsage = 2 + exitSource = 1 + exitUnavailable = 3 + apiKeyName = "OPENROUTER_API_KEY" +) + +// newJudge is a variable so tests can substitute a fake. +var newJudge = func(key string) check.Judge { + return &jev.Client{ + Endpoint: jev.DefaultEndpoint, + APIKey: key, + HTTP: http.DefaultClient, + Timeout: jev.DefaultTimeout, + MaxRetries: jev.DefaultMaxRetries, + } +} + +func main() { + os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) +} + +func run(args []string, stdout, stderr io.Writer) int { + if len(args) > 0 && args[0] == "prove" { + return runProve(args[1:], stdout, stderr) + } + fs := flag.NewFlagSet("test-check", flag.ContinueOnError) + fs.SetOutput(stderr) + fs.Usage = func() { + fmt.Fprintln(stderr, `Usage: + test-check [--dir DIR] [--base REF] [--title TEXT] [--json] + the change in a local checkout since its merge base with REF + (default: the remote default branch), including uncommitted files + test-check --repo OWNER/REPO --pr N [--json] + a pull request's current head + test-check --repo OWNER/REPO --compare BASE...HEAD [--title TEXT] [--json] + the change between two commits + test-check prove --test CMD [--each A,B] [--dir DIR] [--base REF] [--timeout DUR] [--json] + run CMD on the new code (must pass), then with every non-test change + reverted (must fail); files are restored afterwards. With {name} in + CMD, every added or edited test must go red on its own + test-check prove --restore [--dir DIR] + put back files a crashed prove run left reverted + +Exit: 0 advice printed / proven, 1 error, 2 usage, 3 JEV unavailable, +4 not proven (passes on the old code, fails on the new, or no test changed). +Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain.env.`) + } + dir := fs.String("dir", ".", "local git checkout") + base := fs.String("base", "", "base ref for a local change") + title := fs.String("title", "", "short description of the change") + repo := fs.String("repo", "", "OWNER/REPO on GitHub") + pr := fs.Int("pr", 0, "pull request number") + compare := fs.String("compare", "", "BASE...HEAD commits on GitHub") + model := fs.String("model", jev.DefaultModel, "JEV model") + asJSON := fs.Bool("json", false, "print the result as JSON") + if err := fs.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return exitUsage + } + if fs.NArg() != 0 || (*pr > 0 && *compare != "") || ((*pr > 0 || *compare != "") && *repo == "") { + fs.Usage() + return exitUsage + } + + ctx := context.Background() + var in check.Input + var err error + switch { + case *pr > 0: + in, err = source.PullRequest(ctx, *repo, *pr) + case *compare != "": + b, h, ok := strings.Cut(*compare, "...") + if !ok || b == "" || h == "" { + fmt.Fprintln(stderr, "test-check: --compare must be BASE...HEAD") + return exitUsage + } + in, err = source.Compare(ctx, *repo, b, h, *title) + default: + in, err = source.Local(ctx, *dir, *base, *title) + in.Repo = *repo + } + if err != nil { + fmt.Fprintf(stderr, "test-check: %v\n", err) + return exitSource + } + + key := apiKey() + if key == "" { + fmt.Fprintf(stderr, "test-check: JEV unavailable: no %s. Answer the four questions in the test-check skill yourself.\n", apiKeyName) + return exitUnavailable + } + result, err := check.Evaluate(ctx, newJudge(key), *model, in) + if err != nil { + fmt.Fprintf(stderr, "test-check: JEV unavailable: %v. Answer the four questions in the test-check skill yourself.\n", err) + return exitUnavailable + } + + if *asJSON { + encoder := json.NewEncoder(stdout) + encoder.SetIndent("", " ") + _ = encoder.Encode(result) + return 0 + } + fmt.Fprintf(stdout, "advice: %s\n", result.Advice) + fmt.Fprintf(stdout, "next: %s\n", result.Next) + fmt.Fprintf(stdout, "covered by this change's own tests: p=%.2f\n", result.Covered) + fmt.Fprintf(stdout, "low-value tests added: p=%.2f\n", result.LowValue) + if len(result.NearbyTests) > 0 { + fmt.Fprintf(stdout, "nearby tests: %s\n", strings.Join(result.NearbyTests, ", ")) + } + if !result.DiffComplete { + fmt.Fprintln(stdout, "note: the diff was truncated or missing parts; judge the omitted files yourself") + } + return 0 +} + +// apiKey prefers the environment, then the Gitmoot keychain file the fleet +// already keeps the key in. Any read failure means no key. +func apiKey() string { + if key := strings.TrimSpace(os.Getenv(apiKeyName)); key != "" { + return key + } + configDir, err := os.UserConfigDir() + if err != nil { + return "" + } + file, err := os.Open(filepath.Join(configDir, "gitmoot", "keychain.env")) + if err != nil { + return "" + } + defer file.Close() + scanner := bufio.NewScanner(file) + for scanner.Scan() { + name, value, ok := strings.Cut(strings.TrimSpace(scanner.Text()), "=") + if ok && strings.TrimSpace(strings.TrimPrefix(name, "export ")) == apiKeyName { + value = strings.TrimSpace(value) + return strings.Trim(strings.TrimSpace(value), `"'`) + } + } + return "" +} diff --git a/cmd/test-check/main_test.go b/cmd/test-check/main_test.go new file mode 100644 index 0000000..2ebe8a7 --- /dev/null +++ b/cmd/test-check/main_test.go @@ -0,0 +1,145 @@ +package main + +import ( + "bytes" + "context" + "errors" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/gitmoot/test-check/internal/check" + "github.com/gitmoot/test-check/internal/jev" +) + +type stubJudge struct { + advice string + err error +} + +func (s stubJudge) Evaluate(context.Context, jev.Request) (jev.Exchange, error) { + if s.err != nil { + return jev.Exchange{}, s.err + } + zero := 0.0 + return jev.Exchange{Response: jev.Response{Answers: map[string]jev.Answer{ + "advice": {Choice: s.advice}, + "covered_by_diff_tests": {Noul: &zero}, + "low_value_tests_added": {Noul: &zero}, + }}}, nil +} + +func repoWithChange(t *testing.T) string { + t.Helper() + dir := t.TempDir() + git := func(args ...string) { + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } + } + git("init", "-q", "-b", "main") + if err := os.WriteFile(filepath.Join(dir, "a.go"), []byte("package a\n"), 0o644); err != nil { + t.Fatal(err) + } + git("add", ".") + git("commit", "-q", "-m", "base") + if err := os.WriteFile(filepath.Join(dir, "a.go"), []byte("package a\n\nvar X = 1\n"), 0o644); err != nil { + t.Fatal(err) + } + return dir +} + +func withJudge(t *testing.T, judge check.Judge, key string) { + t.Helper() + prev := newJudge + newJudge = func(string) check.Judge { return judge } + t.Cleanup(func() { newJudge = prev }) + t.Setenv("OPENROUTER_API_KEY", key) + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) +} + +func TestRunPrintsAdviceForALocalChange(t *testing.T) { + withJudge(t, stubJudge{advice: check.AdviceNewTest}, "k") + var stdout, stderr bytes.Buffer + if code := run([]string{"--dir", repoWithChange(t), "--base", "main"}, &stdout, &stderr); code != 0 { + t.Fatalf("exit %d: %s", code, stderr.String()) + } + if !strings.Contains(stdout.String(), "advice: new_regression_test") || !strings.Contains(stdout.String(), "goes red on the old code") { + t.Fatalf("stdout = %q", stdout.String()) + } +} + +// Without JEV the tool must not print advice it did not get. +func TestRunWithoutJEVExitsUnavailableWithNoAdvice(t *testing.T) { + dir := repoWithChange(t) + for name, key := range map[string]string{"no key": "", "JEV error": "k"} { + t.Run(name, func(t *testing.T) { + withJudge(t, stubJudge{err: errors.New("HTTP 503")}, key) + var stdout, stderr bytes.Buffer + if code := run([]string{"--dir", dir, "--base", "main"}, &stdout, &stderr); code != exitUnavailable { + t.Fatalf("exit %d, want %d", code, exitUnavailable) + } + if stdout.Len() != 0 || !strings.Contains(stderr.String(), "four questions") { + t.Fatalf("stdout = %q, stderr = %q", stdout.String(), stderr.String()) + } + }) + } +} + +func TestAPIKeyFallsBackToGitmootKeychain(t *testing.T) { + config := t.TempDir() + t.Setenv("OPENROUTER_API_KEY", "") + t.Setenv("XDG_CONFIG_HOME", config) + if err := os.MkdirAll(filepath.Join(config, "gitmoot"), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(config, "gitmoot", "keychain.env"), []byte("OTHER=x\nOPENROUTER_API_KEY=from-keychain\n"), 0o600); err != nil { + t.Fatal(err) + } + if got := apiKey(); got != "from-keychain" { + t.Fatalf("apiKey = %q", got) + } +} + +func TestRunRejectsConflictingModes(t *testing.T) { + var stdout, stderr bytes.Buffer + for _, args := range [][]string{ + {"--pr", "3"}, + {"--repo", "o/r", "--pr", "3", "--compare", "a...b"}, + {"--repo", "o/r", "--compare", "a..b"}, + } { + if code := run(args, &stdout, &stderr); code != exitUsage { + t.Fatalf("run(%v) = %d, want %d", args, code, exitUsage) + } + } +} + +func TestProveExitCodes(t *testing.T) { + dir := repoWithChange(t) + if err := os.WriteFile(filepath.Join(dir, "a_test.sh"), []byte("grep -q 'X = 1' a.go\n"), 0o644); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + if code := run([]string{"prove", "--dir", dir, "--base", "main", "--test", "sh a_test.sh"}, &stdout, &stderr); code != 0 { + t.Fatalf("proven change exit %d: %s%s", code, stdout.String(), stderr.String()) + } + if !strings.Contains(stdout.String(), "outcome: proven") { + t.Fatalf("stdout = %q", stdout.String()) + } + if err := os.WriteFile(filepath.Join(dir, "a_test.sh"), []byte("grep -q 'package a' a.go\n"), 0o644); err != nil { + t.Fatal(err) + } + stdout.Reset() + if code := run([]string{"prove", "--dir", dir, "--base", "main", "--test", "sh a_test.sh"}, &stdout, &stderr); code != exitNotProven { + t.Fatalf("weak test exit %d, want %d: %s", code, exitNotProven, stdout.String()) + } + for _, args := range [][]string{{"prove"}, {"prove", "--restore", "--test", "true"}} { + if code := run(args, &stdout, &stderr); code != exitUsage { + t.Fatalf("run(%v) = %d, want %d", args, code, exitUsage) + } + } +} diff --git a/cmd/test-check/prove.go b/cmd/test-check/prove.go new file mode 100644 index 0000000..0ea400c --- /dev/null +++ b/cmd/test-check/prove.go @@ -0,0 +1,112 @@ +package main + +import ( + "context" + "encoding/json" + "flag" + "fmt" + "io" + "os" + "os/signal" + "strings" + "syscall" + "time" + + "github.com/gitmoot/test-check/internal/prove" +) + +const exitNotProven = 4 + +var proveMessages = map[string]string{ + prove.OutcomeProven: "The tests fail on the old code and pass on the new: they would catch a regression of this change.", + prove.OutcomeNotRedOnOld: "The tests PASS on the old code too, so they would not catch a regression. Make them fail without the fix.", + prove.OutcomeFailsOnNew: "The tests FAIL on the new code. Fix that before proving.", + prove.OutcomeNoTestChanged: "The change has no test files. If a test is needed, add one; otherwise record the one-off check you ran.", + prove.OutcomeNoCodeChanged: "Only tests changed; there is no old code to prove them against.", + prove.OutcomeNotRun: "A test never appeared in the command's output on the new code, so the command probably did not run it. Use a verbose runner (go test -v, pytest -v) and check the name.", +} + +func runProve(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("test-check prove", flag.ContinueOnError) + fs.SetOutput(stderr) + dir := fs.String("dir", ".", "local git checkout") + base := fs.String("base", "", "base ref (default: the remote default branch)") + test := fs.String("test", "", "test command, run with sh -c at the repository root; with {name}, each test is proven on its own") + each := fs.String("each", "", "comma-separated test names for {name} (default: the Go/Python tests the change adds or edits)") + timeout := fs.Duration("timeout", 10*time.Minute, "limit for each test run") + restore := fs.Bool("restore", false, "restore files a crashed prove run left reverted") + asJSON := fs.Bool("json", false, "print the report as JSON") + if err := fs.Parse(args); err != nil { + if err == flag.ErrHelp { + return 0 + } + return exitUsage + } + if fs.NArg() != 0 || (*restore == (strings.TrimSpace(*test) != "")) { + fmt.Fprintln(stderr, "test-check prove: pass exactly one of --test CMD or --restore") + return exitUsage + } + // Ctrl-C and SIGTERM cancel the run; Run then restores before returning. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + if *restore { + if err := prove.Restore(ctx, *dir); err != nil { + fmt.Fprintf(stderr, "test-check prove: %v\n", err) + return exitSource + } + fmt.Fprintln(stdout, "restored") + return 0 + } + var names []string + for _, name := range strings.Split(*each, ",") { + if name = strings.TrimSpace(name); name != "" { + names = append(names, name) + } + } + if len(names) > 0 && !strings.Contains(*test, prove.NamePlaceholder) { + fmt.Fprintln(stderr, "test-check prove: --each needs {name} in --test") + return exitUsage + } + report, err := prove.Run(ctx, prove.Options{Dir: *dir, Base: *base, Test: *test, Each: names, Timeout: *timeout}) + if err != nil { + fmt.Fprintf(stderr, "test-check prove: %v\n", err) + return exitSource + } + code := exitNotProven + if report.Outcome == prove.OutcomeProven || report.Outcome == prove.OutcomeNoCodeChanged { + code = 0 + } + if *asJSON { + encoder := json.NewEncoder(stdout) + encoder.SetIndent("", " ") + _ = encoder.Encode(report) + return code + } + fmt.Fprintf(stdout, "outcome: %s\n%s\n", report.Outcome, proveMessages[report.Outcome]) + if report.BuildErrorSuspected { + fmt.Fprintln(stdout, "warning: the old-code run looks like a build or import error, not a failed assertion. That only shows the test uses new code; for a bug fix, test the behavior through an interface the old code also has.") + } + for _, res := range report.Tests { + note := "" + if res.BuildErrorSuspected { + note = " (old run looks like a build error)" + } + fmt.Fprintf(stdout, " %-16s %s%s\n", res.Outcome, res.Name, note) + } + if len(report.TestFiles) > 0 { + fmt.Fprintf(stdout, "test files: %s\n", strings.Join(report.TestFiles, ", ")) + } + if len(report.RevertedFiles) > 0 && report.Outcome != prove.OutcomeNoTestChanged { + fmt.Fprintf(stdout, "reverted for the old-code run: %s\n", strings.Join(report.RevertedFiles, ", ")) + } + if len(report.Tests) > 0 { + return code + } + switch report.Outcome { + case prove.OutcomeFailsOnNew: + fmt.Fprintf(stdout, "--- new-code run (tail) ---\n%s\n", report.NewOutput) + case prove.OutcomeProven, prove.OutcomeNotRedOnOld: + fmt.Fprintf(stdout, "--- old-code run (tail) ---\n%s\n", report.OldOutput) + } + return code +} diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..dcc37f7 --- /dev/null +++ b/go.mod @@ -0,0 +1,3 @@ +module github.com/gitmoot/test-check + +go 1.26 diff --git a/internal/check/check.go b/internal/check/check.go new file mode 100644 index 0000000..f75e289 --- /dev/null +++ b/internal/check/check.go @@ -0,0 +1,281 @@ +// Package check asks JEV what testing a code change needs. It is advice, not a +// gate: the agent still decides, and every answer still requires the change to +// be verified. +package check + +import ( + "context" + "errors" + "fmt" + "sort" + "strings" + + "github.com/gitmoot/test-check/internal/jev" +) + +const ( + AdviceNewTest = "new_regression_test" + AdviceExtendTest = "extend_existing_test" + AdviceOneOff = "one_off_check" + AdviceNoTest = "no_permanent_test" + AdviceNeedContext = "need_context" + + // A probability ABOVE these thresholds sets the flag. + CoveredThreshold = 0.5 + LowValueThreshold = 0.5 + + DiffBudgetBytes = 40000 + TestsBudgetBytes = 16000 + + questionAdvice = "advice" + questionCovered = "covered_by_diff_tests" + questionLowValue = "low_value_tests_added" + truncatedMarker = "\n[... truncated ...]\n" +) + +// File is one changed file. Patch is empty when the source could not provide +// it (binary, or too large for GitHub), which makes the diff incomplete. +type File struct { + Path string + Patch string +} + +// NearbyTest is an existing test file that is not part of the change. +type NearbyTest struct { + Path string + Content string +} + +// Input is one change and the tests around it. +type Input struct { + // Incomplete: the source knows files are missing from Files. + Incomplete bool + Repo string + Title string + Head string + Files []File + Nearby []NearbyTest +} + +// Result is JEV's advice and the one next step it implies. +type Result struct { + Advice string `json:"advice"` + // NeedsTestWork: the advice asks for a permanent test the change does not + // yet have, or an added test looks low-value. + NeedsTestWork bool `json:"needs_test_work"` + Next string `json:"next"` + Covered float64 `json:"covered_by_diff_tests"` + LowValue float64 `json:"low_value_tests_added"` + Probabilities map[string]float64 `json:"probabilities"` + DiffComplete bool `json:"diff_complete"` + NearbyTests []string `json:"nearby_tests"` +} + +// Judge evaluates one JEV request; *jev.Client satisfies it. +type Judge interface { + Evaluate(ctx context.Context, request jev.Request) (jev.Exchange, error) +} + +// Questions are the three questions sent with every change. +func Questions() map[string]jev.Question { + return map[string]jev.Question{ + questionAdvice: { + Type: "choice", + Instructions: "Decide what testing this code change needs. Judge the diff itself and the nearby existing tests. A permanent test earns its place only if it protects behavior a user or caller relies on AND would fail on a plausible regression of this change. Pick what the change needs even if the diff already includes it. Choose need_context only when essential evidence is missing, never because the change is large or complex.", + Criteria: map[string]string{ + AdviceNewTest: "Fixes a bug or adds behavior that no nearby test exercises; a regression would go unnoticed. Needs a new test that fails on the old code.", + AdviceExtendTest: "Changes behavior an existing nearby test already covers; add a case or assertion to that test rather than a new test.", + AdviceOneOff: "Behavior changes, but a permanent test would be brittle, slow, or only restate the code (UI layout, wiring, configuration, external services, scripts). Verify once by running it.", + AdviceNoTest: "No behavior changes: docs, comments, formatting, renames, version bumps, or a refactor existing tests already cover.", + AdviceNeedContext: "Essential evidence is missing, so no option can be chosen.", + }, + }, + questionCovered: { + Type: "noul", + Instructions: "Do tests added or changed in this diff exercise the changed behavior, so that a plausible regression of this change would make them fail? Answer no if the diff adds or changes no tests, or if its tests only check unrelated code, restate the source, or would still pass on the old code.", + Criteria: map[string]string{ + "true": "The diff's own tests would fail on a plausible regression of the change.", + "false": "The diff has no such tests.", + }, + }, + questionLowValue: { + Type: "noul", + Instructions: "Does the diff add or change tests that are low-value: they only restate source text or constants, copy fixtures or lists, assert private implementation details or call order, duplicate a stronger existing test, or would still pass on the old broken code? Answer no if the diff adds or changes no tests.", + Criteria: map[string]string{ + "true": "At least one added or changed test is low-value in one of those ways.", + "false": "No added or changed test is low-value, or the diff has no tests.", + }, + }, + } +} + +// Evaluate asks JEV about one change. Any failure to get a complete answer is +// an error: the caller must not present a partial answer as advice. +func Evaluate(ctx context.Context, judge Judge, model string, in Input) (Result, error) { + if judge == nil { + return Result{}, errors.New("no JEV client") + } + state, complete, nearby := BuildState(in) + exchange, err := judge.Evaluate(ctx, jev.Request{Model: model, State: state, Questions: Questions()}) + if err != nil { + return Result{}, err + } + answers := exchange.Response.Answers + advice, ok := answers[questionAdvice] + if !ok { + return Result{}, fmt.Errorf("%s answer missing", questionAdvice) + } + switch advice.Choice { + case AdviceNewTest, AdviceExtendTest, AdviceOneOff, AdviceNoTest, AdviceNeedContext: + default: + return Result{}, fmt.Errorf("%s answer %q is not an offered option", questionAdvice, advice.Choice) + } + covered, err := probability(answers, questionCovered) + if err != nil { + return Result{}, err + } + lowValue, err := probability(answers, questionLowValue) + if err != nil { + return Result{}, err + } + result := Result{ + Advice: advice.Choice, + Covered: covered, + LowValue: lowValue, + Probabilities: advice.Probabilities, + DiffComplete: complete, + NearbyTests: nearby, + } + result.NeedsTestWork = needsTestWork(result) + result.Next = nextStep(result) + return result, nil +} + +func probability(answers map[string]jev.Answer, key string) (float64, error) { + answer, ok := answers[key] + if !ok { + return 0, fmt.Errorf("%s answer missing", key) + } + if answer.Noul == nil || !(*answer.Noul >= 0 && *answer.Noul <= 1) { + return 0, fmt.Errorf("%s answer has no probability in [0,1]", key) + } + return *answer.Noul, nil +} + +func needsTestWork(r Result) bool { + missing := (r.Advice == AdviceNewTest || r.Advice == AdviceExtendTest) && r.Covered <= CoveredThreshold + return missing || r.LowValue > LowValueThreshold +} + +func nextStep(r Result) string { + var next string + switch r.Advice { + case AdviceNewTest, AdviceExtendTest: + switch { + case r.Covered > CoveredThreshold: + next = "The change's own tests look like they cover it. Prove it: they must go red on the old code and green on the new." + case r.Advice == AdviceExtendTest && len(r.NearbyTests) > 0: + next = "Add a case to an existing test (nearest: " + r.NearbyTests[0] + ") that goes red on the old code and green on the new." + case r.Advice == AdviceExtendTest: + next = "Add a case to the existing test that covers this behavior; it must go red on the old code and green on the new." + default: + next = "Add a regression test that goes red on the old code and green on the new." + } + case AdviceOneOff: + next = "Skip a permanent test. Run the changed path once and record the command and what you observed." + case AdviceNoTest: + next = "No permanent test needed. Still build and run the existing tests." + default: + next = "JEV lacked context. Answer the four questions in the test-check skill yourself." + } + if r.LowValue > LowValueThreshold { + next += " Warning: an added test looks low-value (restates code, duplicates another test, tests internals, or would pass on the old code); rework or drop it." + } + return next +} + +// BuildState renders the model input. Every changed path and nearby test path +// is always listed; diff and test text are cut to fair shares of their budgets +// so one huge file cannot hide the others. +func BuildState(in Input) (state map[string]any, complete bool, nearby []string) { + complete = !in.Incomplete + names := make([]string, len(in.Files)) + texts := make([]string, len(in.Files)) + for i, file := range in.Files { + if file.Patch == "" { + complete = false + } + added, deleted := 0, 0 + for _, line := range strings.Split(file.Patch, "\n") { + switch { + case strings.HasPrefix(line, "+++"), strings.HasPrefix(line, "---"): + case strings.HasPrefix(line, "+"): + added++ + case strings.HasPrefix(line, "-"): + deleted++ + } + } + kind := "code" + switch { + case IsTestFile(file.Path): + kind = "test" + case !IsCode(file.Path): + kind = "other" + } + names[i] = fmt.Sprintf("%s [%s] (+%d/-%d)", file.Path, kind, added, deleted) + texts[i] = "diff --git a/" + file.Path + " b/" + file.Path + "\n" + file.Patch + } + if !fitBudget(texts, DiffBudgetBytes) { + complete = false + } + tests := make([]map[string]any, len(in.Nearby)) + contents := make([]string, len(in.Nearby)) + nearby = make([]string, len(in.Nearby)) + for i, t := range in.Nearby { + nearby[i] = t.Path + tests[i] = map[string]any{"path": t.Path, "tests": TestNames(t.Content)} + contents[i] = "== " + t.Path + "\n" + t.Content + } + fitBudget(contents, TestsBudgetBytes) + return map[string]any{ + "repo": in.Repo, + "title": in.Title, + "head": in.Head, + "files": names, + "diffComplete": complete, + "diff": strings.Join(texts, ""), + "nearbyTests": tests, + "nearbyTestContent": strings.Join(contents, "\n"), + }, complete, nearby +} + +// fitBudget truncates texts in place so their total stays within budget, +// giving each a fair share, smallest first. A text whose share cannot hold the +// marker is dropped. It reports whether nothing was cut. +func fitBudget(texts []string, budget int) bool { + total := 0 + for _, t := range texts { + total += len(t) + } + if total <= budget { + return true + } + order := make([]int, len(texts)) + for i := range order { + order[i] = i + } + sort.SliceStable(order, func(a, b int) bool { return len(texts[order[a]]) < len(texts[order[b]]) }) + remaining := budget + for n, index := range order { + share := remaining / (len(order) - n) + if len(texts[index]) > share { + if share < len(truncatedMarker) { + texts[index] = "" + } else { + texts[index] = strings.ToValidUTF8(texts[index][:share-len(truncatedMarker)], "") + truncatedMarker + } + } + remaining -= len(texts[index]) + } + return false +} diff --git a/internal/check/check_test.go b/internal/check/check_test.go new file mode 100644 index 0000000..6ab9370 --- /dev/null +++ b/internal/check/check_test.go @@ -0,0 +1,148 @@ +package check + +import ( + "context" + "errors" + "math" + "reflect" + "strings" + "testing" + + "github.com/gitmoot/test-check/internal/jev" +) + +type fakeJudge struct { + advice string + covered float64 + lowValue float64 + drop string + err error + request jev.Request +} + +func (f *fakeJudge) Evaluate(_ context.Context, request jev.Request) (jev.Exchange, error) { + f.request = request + if f.err != nil { + return jev.Exchange{}, f.err + } + covered, lowValue := f.covered, f.lowValue + answers := map[string]jev.Answer{ + questionAdvice: {Type: "choice", Choice: f.advice, Probabilities: map[string]float64{f.advice: 0.9}}, + questionCovered: {Type: "noul", Noul: &covered}, + questionLowValue: {Type: "noul", Noul: &lowValue}, + } + delete(answers, f.drop) + return jev.Exchange{Response: jev.Response{Answers: answers}}, nil +} + +func sampleInput() Input { + return Input{ + Repo: "o/r", + Files: []File{{Path: "pkg/retry.go", Patch: "@@ -1 +1 @@\n-return nil\n+return err"}}, + Nearby: []NearbyTest{{Path: "pkg/retry_test.go", Content: "func TestRetryStops(t *testing.T) {}"}}, + } +} + +func TestEvaluateNextStepFollowsAdviceAndFlags(t *testing.T) { + cases := []struct { + name string + judge fakeJudge + wantNext string + needsWork bool + }{ + {"missing regression test", fakeJudge{advice: AdviceNewTest, covered: 0.1}, "Add a regression test that goes red", true}, + {"diff already carries the test", fakeJudge{advice: AdviceNewTest, covered: 0.8}, "must go red on the old code", false}, + {"extend names the nearest test", fakeJudge{advice: AdviceExtendTest, covered: 0.2}, "nearest: pkg/retry_test.go", true}, + {"one-off check", fakeJudge{advice: AdviceOneOff}, "Run the changed path once", false}, + {"no test still runs existing tests", fakeJudge{advice: AdviceNoTest}, "Still build and run the existing tests", false}, + {"low-value test flagged even when no test is needed", fakeJudge{advice: AdviceNoTest, lowValue: 0.7}, "looks low-value", true}, + {"need context points at the questions", fakeJudge{advice: AdviceNeedContext}, "four questions", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + judge := tc.judge + got, err := Evaluate(context.Background(), &judge, jev.DefaultModel, sampleInput()) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(got.Next, tc.wantNext) { + t.Fatalf("Next = %q, want it to contain %q", got.Next, tc.wantNext) + } + if got.NeedsTestWork != tc.needsWork { + t.Fatalf("NeedsTestWork = %v, want %v", got.NeedsTestWork, tc.needsWork) + } + }) + } +} + +func TestEvaluateRefusesIncompleteAnswers(t *testing.T) { + for name, judge := range map[string]*fakeJudge{ + "transport error": {err: errors.New("HTTP 503")}, + "missing advice": {advice: AdviceNoTest, drop: questionAdvice}, + "missing covered": {advice: AdviceNoTest, drop: questionCovered}, + "unoffered advice": {advice: "write_more_tests"}, + "NaN low-value answer": {advice: AdviceNoTest, lowValue: math.NaN()}, + } { + t.Run(name, func(t *testing.T) { + if got, err := Evaluate(context.Background(), judge, jev.DefaultModel, sampleInput()); err == nil { + t.Fatalf("Evaluate = %+v, want an error rather than partial advice", got) + } + }) + } + if _, err := Evaluate(context.Background(), nil, jev.DefaultModel, sampleInput()); err == nil { + t.Fatal("nil judge must be an error") + } +} + +func TestBuildStateSendsNearbyTestNamesAndKeepsEveryPathWithinBudget(t *testing.T) { + in := sampleInput() + in.Files = append(in.Files, File{Path: "gen/huge.go", Patch: "@@ -0,0 +1 @@\n+" + strings.Repeat("x", 3*DiffBudgetBytes)}) + state, complete, nearby := BuildState(in) + if complete { + t.Fatal("complete = true for a truncated diff") + } + diff := state["diff"].(string) + if len(diff) > DiffBudgetBytes || !strings.Contains(diff, "+return err") { + t.Fatalf("diff len %d; small file kept: %v", len(diff), strings.Contains(diff, "+return err")) + } + files := state["files"].([]string) + if len(files) != 2 || files[0] != "pkg/retry.go [code] (+1/-1)" { + t.Fatalf("files = %v", files) + } + tests := state["nearbyTests"].([]map[string]any) + if !reflect.DeepEqual(nearby, []string{"pkg/retry_test.go"}) || !reflect.DeepEqual(tests[0]["tests"], []string{"TestRetryStops"}) { + t.Fatalf("nearby = %v, tests = %v", nearby, tests) + } +} + +func TestNearbyTestPathsPrefersTestsNamingTheChangedFile(t *testing.T) { + all := []string{ + "pkg/retry.go", "pkg/retry_test.go", "pkg/retry_backoff_test.go", "pkg/other_test.go", "pkg/new_test.go", + "web/src/Player.tsx", "web/src/Player.test.tsx", "web/src/__tests__/player.spec.ts", + "app/billing.py", "tests/test_billing.py", + "Sources/App/Paywall.swift", "Tests/AppTests/PaywallTests.swift", + "docs/README.md", + } + changed := []string{"pkg/retry.go", "pkg/new_test.go", "web/src/Player.tsx", "app/billing.py", "Sources/App/Paywall.swift", "docs/README.md"} + got := NearbyTestPaths(changed, all) + want := []string{"pkg/retry_test.go", "web/src/Player.test.tsx", "Tests/AppTests/PaywallTests.swift", "tests/test_billing.py", "web/src/__tests__/player.spec.ts", "pkg/retry_backoff_test.go", "pkg/other_test.go"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("NearbyTestPaths =\n%v\nwant\n%v", got, want) + } + if got := NearbyTestPaths([]string{"docs/README.md"}, all); len(got) != 0 { + t.Fatalf("docs-only change got nearby tests %v", got) + } +} + +func TestTestNamesAcrossLanguages(t *testing.T) { + content := "func TestGo(t *testing.T) {}\n" + + "func testSwiftXC() {}\n" + + "@Test(\"x\")\nfunc swiftTesting() {}\n" + + "def test_python():\n" + + "#[test]\nfn rust_case() {}\n" + + "it('renders the cut', () => {})\n" + want := []string{"TestGo", "testSwiftXC", "swiftTesting", "test_python", "rust_case", "renders the cut"} + if got := TestNames(content); !reflect.DeepEqual(got, want) { + t.Fatalf("TestNames = %v, want %v", got, want) + } +} diff --git a/internal/check/tests.go b/internal/check/tests.go new file mode 100644 index 0000000..abe5cff --- /dev/null +++ b/internal/check/tests.go @@ -0,0 +1,144 @@ +package check + +import ( + "path" + "regexp" + "sort" + "strings" +) + +// MaxNearbyTests bounds how many existing test files are shown to the model. +const MaxNearbyTests = 8 + +var testFilePattern = regexp.MustCompile(`(_test\.go$|\.(test|spec)\.[cm]?[jt]sx?$|(^|/)test_[^/]*\.py$|_test\.py$|Tests?\.swift$|_test\.rs$|_spec\.rb$|_test\.(sh|bash)$|\.bats$|(^|/)(tests?|__tests__|spec|Tests)/)`) + +// IsTestFile reports whether a repository path looks like test code. +func IsTestFile(p string) bool { + return testFilePattern.MatchString(p) +} + +var codeExtensions = map[string]bool{ + ".go": true, ".ts": true, ".tsx": true, ".js": true, ".jsx": true, ".mjs": true, ".cjs": true, + ".py": true, ".swift": true, ".rs": true, ".rb": true, ".kt": true, ".java": true, ".vue": true, +} + +// IsCode reports whether a non-test path is source code whose behavior a test +// could protect. Docs, config, and assets are not. +func IsCode(p string) bool { + return codeExtensions[strings.ToLower(path.Ext(p))] && !IsTestFile(p) +} + +var testNamePatterns = []*regexp.Regexp{ + regexp.MustCompile(`(?m)^\s*func\s+(Test\w+)\s*\(`), // Go + regexp.MustCompile(`(?m)^\s*(?:@\w+\s+)*func\s+(test\w+)\s*\(`), // Swift XCTest + regexp.MustCompile(`(?m)@Test[^\n]*\n\s*func\s+(\w+)`), // Swift Testing + regexp.MustCompile(`(?m)^\s*(?:async\s+)?def\s+(test_\w+)\s*\(`), // Python + regexp.MustCompile(`(?m)#\[(?:tokio::)?test\]\s*(?:async\s+)?fn\s+(\w+)`), // Rust + regexp.MustCompile("(?m)\\b(?:it|test)\\s*\\(\\s*['\"`]([^'\"`\\n]{1,100})"), // JS/TS +} + +// TestNames lists the test cases declared in one test file, in file order, +// bounded so a huge file cannot crowd out the others. +func TestNames(content string) []string { + type hit struct { + at int + name string + } + var hits []hit + for _, pattern := range testNamePatterns { + for _, m := range pattern.FindAllStringSubmatchIndex(content, -1) { + hits = append(hits, hit{at: m[2], name: content[m[2]:m[3]]}) + } + } + sort.Slice(hits, func(a, b int) bool { return hits[a].at < hits[b].at }) + names := make([]string, 0, min(len(hits), 40)) + for _, h := range hits { + if len(names) == 40 { + break + } + names = append(names, h.name) + } + return names +} + +// NearbyTestPaths picks existing test files most likely to cover the changed +// code: test files that name a changed file (same directory first), then other +// tests in a changed file's directory. Files the change itself touches are +// excluded; their content is already in the diff. +func NearbyTestPaths(changed, all []string) []string { + touched := map[string]bool{} + for _, p := range changed { + touched[p] = true + } + type candidate struct { + path string + rank int + } + best := map[string]int{} + for _, c := range changed { + if !IsCode(c) { + continue + } + dir := path.Dir(c) + stem := strings.ToLower(strings.TrimSuffix(path.Base(c), path.Ext(c))) + for _, p := range all { + if touched[p] || !IsTestFile(p) { + continue + } + base := strings.ToLower(path.Base(p)) + sameDir := path.Dir(p) == dir + exact := testSubject(base) == stem + named := len(stem) >= 3 && strings.Contains(base, stem) + rank := 0 + switch { + case exact && sameDir: + rank = 5 + case exact: + rank = 4 + case named && sameDir: + rank = 3 + case named: + rank = 2 + case sameDir: + rank = 1 + } + if rank > best[p] { + best[p] = rank + } + } + } + candidates := make([]candidate, 0, len(best)) + for p, rank := range best { + if rank > 0 { + candidates = append(candidates, candidate{p, rank}) + } + } + sort.Slice(candidates, func(a, b int) bool { + if candidates[a].rank != candidates[b].rank { + return candidates[a].rank > candidates[b].rank + } + return candidates[a].path < candidates[b].path + }) + out := make([]string, 0, min(len(candidates), MaxNearbyTests)) + for _, c := range candidates { + if len(out) == MaxNearbyTests { + break + } + out = append(out, c.path) + } + return out +} + +// testSubject strips test markers from a lower-cased test file name, so +// "retry_test.go", "retry.test.ts", "test_retry.py" and "RetryTests.swift" +// all name "retry". +func testSubject(base string) string { + name, _, _ := strings.Cut(base, ".") + name = strings.TrimPrefix(name, "test_") + for _, suffix := range []string{"_test", "_spec", "tests", "test"} { + if trimmed, ok := strings.CutSuffix(name, suffix); ok && trimmed != "" { + return trimmed + } + } + return name +} diff --git a/internal/jev/client.go b/internal/jev/client.go new file mode 100644 index 0000000..e072055 --- /dev/null +++ b/internal/jev/client.go @@ -0,0 +1,271 @@ +package jev + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strconv" + "strings" + "sync" + "time" +) + +// maxResponseBytes bounds how much of a response is read. An answer is small; +// anything far larger is not a response this client can trust. +const maxResponseBytes = 4 << 20 + +// APIError is a failed evaluation, with whether retrying could help. +type APIError struct { + Status int + Message string + Retryable bool +} + +func (e *APIError) Error() string { + if e.Status == 0 { + return "jev: " + e.Message + } + return fmt.Sprintf("jev: HTTP %d: %s", e.Status, e.Message) +} + +// Client calls the evaluation endpoint with bounded time, retries, and +// request rate. +type Client struct { + Endpoint string + APIKey string + HTTP *http.Client + Timeout time.Duration + MaxRetries int + // MinInterval spaces requests from one client. + MinInterval time.Duration + // BackoffBase and MaxBackoff shape exponential backoff. Retry-After is + // honoured but never beyond MaxBackoff. + BackoffBase time.Duration + MaxBackoff time.Duration + // Sleep and Now are injectable so tests exercise backoff without waiting. + Sleep func(ctx context.Context, d time.Duration) error + Now func() time.Time + + mu sync.Mutex + lastRequest time.Time +} + +// Exchange is one completed call: what was sent, what came back, and how many +// attempts it took. +type Exchange struct { + Body []byte + Response Response + Attempts int +} + +// Evaluate sends one request and returns the parsed response. +func (c *Client) Evaluate(ctx context.Context, request Request) (Exchange, error) { + body, err := Encode(request) + if err != nil { + return Exchange{}, &APIError{Message: "encode request: " + err.Error()} + } + exchange := Exchange{Body: body} + + // At least one attempt: a negative MaxRetries must not return success + // without ever contacting the server. + attempts := max(c.MaxRetries, 0) + 1 + var last error + for attempt := range attempts { + if err := c.pace(ctx); err != nil { + return exchange, &APIError{Message: "cancelled: " + err.Error()} + } + exchange.Attempts++ + response, wait, err := c.once(ctx, body) + if err == nil { + exchange.Response = response + return exchange, nil + } + last = err + + var apiErr *APIError + if !errors.As(err, &apiErr) || !apiErr.Retryable || attempt == attempts-1 { + break + } + if wait <= 0 { + wait = c.backoff(attempt) + } + if wait > c.maxBackoff() { + wait = c.maxBackoff() + } + if err := c.sleep(ctx, wait); err != nil { + return exchange, &APIError{Message: "cancelled during backoff: " + err.Error()} + } + } + return exchange, last +} + +// once performs a single attempt. The returned duration is a server-requested +// wait, when one was given. +func (c *Client) once(ctx context.Context, body []byte) (Response, time.Duration, error) { + attemptCtx, cancel := ctx, context.CancelFunc(func() {}) + if c.Timeout > 0 { + attemptCtx, cancel = context.WithTimeout(ctx, c.Timeout) + } + defer cancel() + + req, err := http.NewRequestWithContext(attemptCtx, http.MethodPost, c.Endpoint, bytes.NewReader(body)) + if err != nil { + return Response{}, 0, &APIError{Message: "build request: " + err.Error()} + } + req.Header.Set("Authorization", "Bearer "+c.APIKey) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + + client := c.httpClient() + resp, err := client.Do(req) + if err != nil { + // Timeouts and connection failures are transient by nature. + return Response{}, 0, &APIError{Message: redactKey(err.Error(), c.APIKey), Retryable: ctx.Err() == nil} + } + defer resp.Body.Close() + + raw, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1)) + if err != nil { + return Response{}, 0, &APIError{Status: resp.StatusCode, Message: "read response: " + err.Error(), Retryable: true} + } + if len(raw) > maxResponseBytes { + return Response{}, 0, &APIError{Status: resp.StatusCode, Message: "response exceeds the size bound"} + } + + if resp.StatusCode != http.StatusOK { + return Response{}, retryAfter(resp.Header.Get("Retry-After")), &APIError{ + Status: resp.StatusCode, + Message: summarize(raw, c.APIKey), + Retryable: retryableStatus(resp.StatusCode), + } + } + + var response Response + if err := json.Unmarshal(raw, &response); err != nil { + return Response{}, 0, &APIError{Status: resp.StatusCode, Message: "decode response: " + err.Error()} + } + if response.Answers == nil { + return Response{}, 0, &APIError{Status: resp.StatusCode, Message: "response carried no answers"} + } + return response, 0, nil +} + +// retryableStatus: 429 and 529 ask the client to back off and retry, and a 5xx +// is a server fault worth another try. A 401 or 422 fails the same way every +// time. +func retryableStatus(status int) bool { + switch { + case status == http.StatusTooManyRequests, status == 529: + return true + case status >= 500 && status <= 599: + return true + default: + return false + } +} + +// retryAfter parses a Retry-After header given in seconds. The HTTP-date form +// is ignored; exponential backoff covers it. +func retryAfter(header string) time.Duration { + seconds, err := strconv.Atoi(strings.TrimSpace(header)) + if err != nil || seconds < 0 { + return 0 + } + return time.Duration(seconds) * time.Second +} + +// pace enforces the minimum interval between requests. +func (c *Client) pace(ctx context.Context) error { + c.mu.Lock() + now := c.now() + wait := time.Duration(0) + if !c.lastRequest.IsZero() { + if next := c.lastRequest.Add(c.MinInterval); next.After(now) { + wait = next.Sub(now) + } + } + c.lastRequest = now.Add(wait) + c.mu.Unlock() + if wait <= 0 { + return nil + } + return c.sleep(ctx, wait) +} + +func (c *Client) backoff(attempt int) time.Duration { + base := c.BackoffBase + if base <= 0 { + base = 500 * time.Millisecond + } + wait := base << attempt + if wait <= 0 || wait > c.maxBackoff() { + return c.maxBackoff() + } + return wait +} + +func (c *Client) maxBackoff() time.Duration { + if c.MaxBackoff <= 0 { + return 10 * time.Second + } + return c.MaxBackoff +} + +// httpClient preserves the caller's transport and timeout but refuses +// redirects: the key may not be forwarded to a different endpoint. +func (c *Client) httpClient() http.Client { + base := c.HTTP + if base == nil { + base = http.DefaultClient + } + client := *base + client.CheckRedirect = func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + } + return client +} + +func (c *Client) now() time.Time { + if c.Now != nil { + return c.Now() + } + return time.Now() +} + +func (c *Client) sleep(ctx context.Context, d time.Duration) error { + if c.Sleep != nil { + return c.Sleep(ctx, d) + } + timer := time.NewTimer(d) + defer timer.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + return nil + } +} + +// summarize shortens an error body for reporting, and strips the key in case a +// server echoed it back. +func summarize(raw []byte, key string) string { + text := strings.TrimSpace(string(raw)) + if len(text) > 300 { + text = text[:300] + "..." + } + if text == "" { + text = "empty response body" + } + return redactKey(text, key) +} + +func redactKey(text, key string) string { + if key == "" { + return text + } + return strings.ReplaceAll(text, key, "[redacted]") +} diff --git a/internal/jev/client_test.go b/internal/jev/client_test.go new file mode 100644 index 0000000..f40d29c --- /dev/null +++ b/internal/jev/client_test.go @@ -0,0 +1,141 @@ +package jev + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" +) + +type fakeServer struct { + mu sync.Mutex + calls int + headers []http.Header + paths []string + bodies []string + server *httptest.Server +} + +func newFakeServer(t *testing.T, reply func(n int) (int, http.Header, string)) *fakeServer { + t.Helper() + f := &fakeServer{} + f.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var raw json.RawMessage + _ = json.NewDecoder(r.Body).Decode(&raw) + f.mu.Lock() + f.calls++ + n := f.calls + f.headers = append(f.headers, r.Header.Clone()) + f.paths = append(f.paths, r.URL.Path) + f.bodies = append(f.bodies, string(raw)) + f.mu.Unlock() + status, header, body := reply(n) + for key, values := range header { + for _, value := range values { + w.Header().Add(key, value) + } + } + w.WriteHeader(status) + _, _ = w.Write([]byte(body)) + })) + t.Cleanup(f.server.Close) + return f +} + +func testClient(endpoint string) *Client { + return &Client{ + Endpoint: endpoint, + APIKey: "test-key", + Timeout: 5 * time.Second, + MaxRetries: 2, + Sleep: func(context.Context, time.Duration) error { return nil }, + } +} + +const okAnswer = `{"model":"typesafe/jev-1.13","answers":{"risk":{"type":"noul","noul":0.12}}}` + +func TestClientSendsBearerToSystemOneAndDecodesAnswers(t *testing.T) { + server := newFakeServer(t, func(int) (int, http.Header, string) { return http.StatusOK, nil, okAnswer }) + client := testClient(server.server.URL + "/api/v1/systemone") + exchange, err := client.Evaluate(context.Background(), Request{ + Model: DefaultModel, + State: map[string]any{"diff": "a < b"}, + Questions: map[string]Question{"risk": {Type: "noul", Instructions: "x"}}, + }) + if err != nil { + t.Fatalf("Evaluate: %v", err) + } + if got := server.headers[0].Get("Authorization"); got != "Bearer test-key" { + t.Fatalf("Authorization = %q", got) + } + if server.paths[0] != "/api/v1/systemone" { + t.Fatalf("path = %q", server.paths[0]) + } + if !strings.Contains(server.bodies[0], `"a < b"`) { + t.Fatalf("diff text was HTML-escaped: %s", server.bodies[0]) + } + answer := exchange.Response.Answers["risk"] + if answer.Noul == nil || *answer.Noul != 0.12 { + t.Fatalf("noul = %v", answer.Noul) + } +} + +func TestClientRetriesServerErrorsThenSucceeds(t *testing.T) { + server := newFakeServer(t, func(n int) (int, http.Header, string) { + if n == 1 { + return http.StatusBadGateway, nil, "upstream down" + } + return http.StatusOK, nil, okAnswer + }) + exchange, err := testClient(server.server.URL).Evaluate(context.Background(), Request{Model: DefaultModel}) + if err != nil { + t.Fatalf("Evaluate: %v", err) + } + if exchange.Attempts != 2 { + t.Fatalf("attempts = %d, want 2", exchange.Attempts) + } +} + +func TestClientDoesNotRetryAuthFailureAndRedactsKey(t *testing.T) { + server := newFakeServer(t, func(int) (int, http.Header, string) { + return http.StatusUnauthorized, nil, "bad key test-key" + }) + exchange, err := testClient(server.server.URL).Evaluate(context.Background(), Request{Model: DefaultModel}) + var apiErr *APIError + if !errors.As(err, &apiErr) || apiErr.Status != http.StatusUnauthorized || apiErr.Retryable { + t.Fatalf("err = %v", err) + } + if exchange.Attempts != 1 { + t.Fatalf("attempts = %d, want 1", exchange.Attempts) + } + if strings.Contains(err.Error(), "test-key") { + t.Fatalf("key leaked into error: %v", err) + } +} + +func TestClientCapsRetryAfterAtMaxBackoff(t *testing.T) { + server := newFakeServer(t, func(n int) (int, http.Header, string) { + if n == 1 { + return http.StatusTooManyRequests, http.Header{"Retry-After": {"3600"}}, "slow down" + } + return http.StatusOK, nil, okAnswer + }) + client := testClient(server.server.URL) + client.MaxBackoff = 2 * time.Second + var waited []time.Duration + client.Sleep = func(_ context.Context, d time.Duration) error { + waited = append(waited, d) + return nil + } + if _, err := client.Evaluate(context.Background(), Request{Model: DefaultModel}); err != nil { + t.Fatalf("Evaluate: %v", err) + } + if len(waited) != 1 || waited[0] != 2*time.Second { + t.Fatalf("waited = %v, want [2s]", waited) + } +} diff --git a/internal/jev/request.go b/internal/jev/request.go new file mode 100644 index 0000000..13537b0 --- /dev/null +++ b/internal/jev/request.go @@ -0,0 +1,69 @@ +// Package jev calls TypeSafe's JEV judge model through OpenRouter's System One +// endpoint. Copied from github.com/gitmoot/gitmoot/internal/jev (itself a +// trimmed port of workspace-janitor's client); keep the two in step. +package jev + +import ( + "bytes" + "encoding/json" + "time" +) + +const ( + // DefaultEndpoint is OpenRouter's System One route. + DefaultEndpoint = "https://openrouter.ai/api/v1/systemone" + // DefaultModel is pinned so a model release never silently changes a + // decision surface that was measured against a specific version. + DefaultModel = "typesafe/jev-1.13" + DefaultTimeout = 20 * time.Second + DefaultMaxRetries = 2 +) + +// Request is the body of POST /api/v1/systemone: a state, a pinned model, +// and a map of typed questions. +type Request struct { + State any `json:"state"` + Model string `json:"model"` + Questions map[string]Question `json:"questions"` +} + +// Question is a typed question: noul (yes/no probability), choice, or score. +type Question struct { + Type string `json:"type"` + Instructions any `json:"instructions"` + Criteria any `json:"criteria,omitempty"` +} + +// Encode renders a request exactly as it is sent. HTML escaping is off so +// diff text such as "<" reaches the model as written. +func Encode(request Request) ([]byte, error) { + var buf bytes.Buffer + encoder := json.NewEncoder(&buf) + encoder.SetEscapeHTML(false) + if err := encoder.Encode(request); err != nil { + return nil, err + } + return bytes.TrimSuffix(buf.Bytes(), []byte("\n")), nil +} + +// Response is the body of a successful evaluation. +type Response struct { + Model string `json:"model"` + Answers map[string]Answer `json:"answers"` + Usage Usage `json:"usage"` +} + +// Answer is one typed answer. Pointers distinguish a missing value from zero. +type Answer struct { + Type string `json:"type"` + Noul *float64 `json:"noul,omitempty"` + Choice string `json:"choice,omitempty"` + Probabilities map[string]float64 `json:"probabilities,omitempty"` + Confidence *float64 `json:"confidence,omitempty"` +} + +// Usage is the token accounting the API reports. +type Usage struct { + InputTokens int64 `json:"input_tokens"` + OutputTokens int64 `json:"output_tokens"` +} diff --git a/internal/prove/each_test.go b/internal/prove/each_test.go new file mode 100644 index 0000000..97abdf0 --- /dev/null +++ b/internal/prove/each_test.go @@ -0,0 +1,74 @@ +package prove + +import ( + "context" + "reflect" + "testing" +) + +// goFixture: the fix changes Stop's boundary. TestStopAtThree catches it; +// TestStopAtTen passes on both versions, the weak test reviewers kept finding. +func goFixture(t *testing.T) string { + t.Helper() + t.Setenv("GOTOOLCHAIN", "local") + t.Setenv("GOFLAGS", "-buildvcs=false") + dir := t.TempDir() + git(t, dir, "init", "-q", "-b", "main") + put(t, dir, "go.mod", "module example.com/stop\n\ngo 1.22\n") + put(t, dir, "stop.go", "package stop\n\nfunc Stop(n int) bool { return n > 3 }\n") + put(t, dir, "stop_test.go", "package stop\n\nimport \"testing\"\n\nfunc TestStopExisting(t *testing.T) {\n\tif !Stop(9) {\n\t\tt.Fatal(\"9\")\n\t}\n}\n") + git(t, dir, "add", ".") + git(t, dir, "commit", "-q", "-m", "base") + git(t, dir, "checkout", "-q", "-b", "fix") + put(t, dir, "stop.go", "package stop\n\nfunc Stop(n int) bool { return n >= 3 }\n") + put(t, dir, "stop_test.go", "package stop\n\nimport \"testing\"\n\nfunc TestStopExisting(t *testing.T) {\n\tif !Stop(9) {\n\t\tt.Fatal(\"9\")\n\t}\n}\n\nfunc TestStopAtThree(t *testing.T) {\n\tif !Stop(3) {\n\t\tt.Fatal(\"3\")\n\t}\n}\n\nfunc TestStopAtTen(t *testing.T) {\n\tif !Stop(10) {\n\t\tt.Fatal(\"10\")\n\t}\n}\n") + return dir +} + +func TestEachFindsTheWeakTestTheWholeRunHides(t *testing.T) { + dir := goFixture(t) + before := snapshot(t, dir) + ctx := context.Background() + + whole, err := Run(ctx, Options{Dir: dir, Base: "main", Test: "go test -count=1 ."}) + if err != nil { + t.Fatal(err) + } + if whole.Outcome != OutcomeProven { + t.Fatalf("whole-change outcome = %s, want proven (one good test hides the weak one)", whole.Outcome) + } + + each, err := Run(ctx, Options{Dir: dir, Base: "main", Test: "go test -count=1 -v -run '^{name}$' ."}) + if err != nil { + t.Fatal(err) + } + got := map[string]string{} + var order []string + for _, res := range each.Tests { + got[res.Name] = res.Outcome + order = append(order, res.Name) + } + if !reflect.DeepEqual(order, []string{"TestStopAtThree", "TestStopAtTen"}) { + t.Fatalf("auto-selected tests = %v, want only the added ones", order) + } + if got["TestStopAtThree"] != OutcomeProven || got["TestStopAtTen"] != OutcomeNotRedOnOld { + t.Fatalf("per-test outcomes = %v", got) + } + if each.Outcome != OutcomeNotRedOnOld { + t.Fatalf("overall = %s, want not_red_on_old", each.Outcome) + } + if after := snapshot(t, dir); !reflect.DeepEqual(before, after) { + t.Fatalf("files changed by prove:\nbefore %v\nafter %v", before, after) + } +} + +func TestEachReportsATestTheCommandNeverRan(t *testing.T) { + dir := goFixture(t) + report, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "go test -count=1 -v -run '^{name}$' .", Each: []string{"TestStopAtThree", "TestMissing"}}) + if err != nil { + t.Fatal(err) + } + if report.Outcome != OutcomeNotRun || report.Tests[1].Outcome != OutcomeNotRun || report.Tests[0].Outcome != OutcomeProven { + t.Fatalf("report = %+v", report) + } +} diff --git a/internal/prove/procgroup_other.go b/internal/prove/procgroup_other.go new file mode 100644 index 0000000..94b3487 --- /dev/null +++ b/internal/prove/procgroup_other.go @@ -0,0 +1,10 @@ +//go:build !unix + +package prove + +import "os/exec" + +func setProcessGroup(*exec.Cmd) {} + +// processAlive cannot probe other processes here; assume a lock is live. +func processAlive(int) bool { return true } diff --git a/internal/prove/procgroup_unix.go b/internal/prove/procgroup_unix.go new file mode 100644 index 0000000..6f5f0b1 --- /dev/null +++ b/internal/prove/procgroup_unix.go @@ -0,0 +1,24 @@ +//go:build unix + +package prove + +import ( + "errors" + "os/exec" + "syscall" +) + +// processAlive reports whether pid exists (EPERM still means it does). +func processAlive(pid int) bool { + err := syscall.Kill(pid, 0) + return err == nil || errors.Is(err, syscall.EPERM) +} + +// setProcessGroup makes cancellation kill the whole test process tree, not +// just the shell, so a child cannot keep running against reverted files. +func setProcessGroup(cmd *exec.Cmd) { + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + cmd.Cancel = func() error { + return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) + } +} diff --git a/internal/prove/prove.go b/internal/prove/prove.go new file mode 100644 index 0000000..21e79f8 --- /dev/null +++ b/internal/prove/prove.go @@ -0,0 +1,605 @@ +// Package prove checks that a change's tests go red on the old code and green +// on the new: it runs the test command on the working tree, then temporarily +// puts the non-test files back to their base versions, runs it again, and +// restores them. Test files and everything else stay as they are, so build +// caches, dependencies and the environment are identical for both runs. +package prove + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + "time" + + "github.com/gitmoot/test-check/internal/check" + "github.com/gitmoot/test-check/internal/source" +) + +const ( + OutcomeProven = "proven" + OutcomeNotRedOnOld = "not_red_on_old" + OutcomeFailsOnNew = "fails_on_new" + OutcomeNoTestChanged = "no_test_in_change" + OutcomeNoCodeChanged = "no_code_in_change" + // OutcomeNotRun: in per-test mode, the command's output on the new code + // never mentioned the test, so it probably selected nothing. + OutcomeNotRun = "test_not_run" + + // NamePlaceholder in Options.Test switches to per-test mode. + NamePlaceholder = "{name}" + + // stashDir is under the per-worktree git dir, so it never shows up as a + // change and cannot be committed. + stashDir = "test-check-prove" + manifestName = "manifest.json" + tailBytes = 2000 +) + +// Report is the outcome of one prove run. +type Report struct { + Outcome string `json:"outcome"` + // BuildErrorSuspected: the old-code run failed with what looks like a + // compile or import error, not a failing assertion. A test of a brand-new + // API always does this; it proves nothing about a regression. + BuildErrorSuspected bool `json:"build_error_suspected"` + TestFiles []string `json:"test_files"` + RevertedFiles []string `json:"reverted_files"` + NewOutput string `json:"new_output,omitempty"` + OldOutput string `json:"old_output,omitempty"` + // Tests holds one result per test in per-test mode. The overall Outcome is + // proven only when every test is. + Tests []TestResult `json:"tests,omitempty"` +} + +// TestResult is one test's red/green result in per-test mode. +type TestResult struct { + Name string `json:"name"` + Outcome string `json:"outcome"` + BuildErrorSuspected bool `json:"build_error_suspected"` + NewOutput string `json:"new_output,omitempty"` + OldOutput string `json:"old_output,omitempty"` +} + +// Options configure one run. Test is run with `sh -c` at the repository root. +// If Test contains {name}, each test is proven on its own: the command is run +// once per name with {name} replaced, and every test must go red by itself. +// Names are Each, or when Each is empty the tests the change adds or edits. +type Options struct { + Dir string + Base string + Test string + Each []string + Timeout time.Duration +} + +var testSupport = regexp.MustCompile(`(^|/)(testdata|fixtures?|__snapshots__|__fixtures__)/`) + +// IsTestSide reports whether a changed path belongs to the tests: test files +// and their fixtures are kept at their new versions for the old-code run. +func IsTestSide(p string) bool { + return check.IsTestFile(p) || testSupport.MatchString(p) +} + +type repo struct { + git func(...string) (string, error) + root string + gitDir string +} + +func open(ctx context.Context, dir string) (repo, error) { + git := source.Git(ctx, dir) + root, err := git("rev-parse", "--show-toplevel") + if err != nil { + return repo{}, err + } + gitDir, err := git("rev-parse", "--absolute-git-dir") + if err != nil { + return repo{}, err + } + return repo{git: git, root: strings.TrimSpace(root), gitDir: strings.TrimSpace(gitDir)}, nil +} + +// Run proves one change. It returns an error only when it could not run the +// check; a test that does not go red is a Report, not an error. +func Run(ctx context.Context, opts Options) (Report, error) { + if strings.TrimSpace(opts.Test) == "" { + return Report{}, errors.New("no test command") + } + r, err := open(ctx, opts.Dir) + if err != nil { + return Report{}, err + } + release, err := acquire(r) + if err != nil { + return Report{}, err + } + defer release() + stash := filepath.Join(r.gitDir, stashDir) + if _, err := os.Stat(stash); err == nil { + return Report{}, fmt.Errorf("a previous prove run did not restore its files; run `test-check prove --restore` first (%s)", stash) + } + base := opts.Base + if base == "" { + if base, err = source.DefaultBase(r.git); err != nil { + return Report{}, err + } + } + mergeBase, err := r.git("merge-base", base, "HEAD") + if err != nil { + return Report{}, err + } + mergeBase = strings.TrimSpace(mergeBase) + changed, err := changedPaths(r, mergeBase) + if err != nil { + return Report{}, err + } + var report Report + for _, p := range changed { + if IsTestSide(p) { + report.TestFiles = append(report.TestFiles, p) + } else { + report.RevertedFiles = append(report.RevertedFiles, p) + } + } + switch { + case len(report.TestFiles) == 0: + report.Outcome = OutcomeNoTestChanged + return report, nil + case len(report.RevertedFiles) == 0: + report.Outcome = OutcomeNoCodeChanged + return report, nil + } + + if !strings.Contains(opts.Test, NamePlaceholder) { + return runWhole(ctx, r, mergeBase, opts, report) + } + names := opts.Each + if len(names) == 0 { + if names, err = ChangedTestNames(r.git, mergeBase, r.root, report.TestFiles); err != nil { + return Report{}, err + } + if len(names) == 0 { + return Report{}, errors.New("{name} given but no added or edited test functions were found in the change; pass --each NAME,...") + } + } + return runEach(ctx, r, mergeBase, opts, report, names) +} + +func runWhole(ctx context.Context, r repo, mergeBase string, opts Options, report Report) (Report, error) { + newOut, newErr := runTest(ctx, r.root, opts.Test, opts.Timeout) + report.NewOutput = tail(newOut) + if newErr != nil { + if ctx.Err() != nil { + return Report{}, ctx.Err() + } + report.Outcome = OutcomeFailsOnNew + return report, nil + } + var oldOut string + var oldErr error + if err := onBase(ctx, r, mergeBase, report.RevertedFiles, func() { + oldOut, oldErr = runTest(ctx, r.root, opts.Test, opts.Timeout) + }); err != nil { + return Report{}, err + } + report.OldOutput = tail(oldOut) + if oldErr == nil { + report.Outcome = OutcomeNotRedOnOld + return report, nil + } + report.Outcome = OutcomeProven + report.BuildErrorSuspected = buildError.MatchString(oldOut) + return report, nil +} + +// runEach runs every test on the new code first, then reverts once and runs +// the ones that passed on the old code, so the tree is swapped a single time. +func runEach(ctx context.Context, r repo, mergeBase string, opts Options, report Report, names []string) (Report, error) { + results := make([]TestResult, len(names)) + var candidates []int + for i, name := range names { + results[i].Name = name + out, err := runTest(ctx, r.root, strings.ReplaceAll(opts.Test, NamePlaceholder, name), opts.Timeout) + if ctx.Err() != nil { + return Report{}, ctx.Err() + } + results[i].NewOutput = tail(out) + switch { + case err != nil: + results[i].Outcome = OutcomeFailsOnNew + case !strings.Contains(out, name): + results[i].Outcome = OutcomeNotRun + default: + candidates = append(candidates, i) + } + } + if len(candidates) > 0 { + if err := onBase(ctx, r, mergeBase, report.RevertedFiles, func() { + for _, i := range candidates { + if ctx.Err() != nil { + return + } + out, err := runTest(ctx, r.root, strings.ReplaceAll(opts.Test, NamePlaceholder, names[i]), opts.Timeout) + results[i].OldOutput = tail(out) + if err == nil { + results[i].Outcome = OutcomeNotRedOnOld + continue + } + results[i].Outcome = OutcomeProven + results[i].BuildErrorSuspected = buildError.MatchString(out) + } + }); err != nil { + return Report{}, err + } + } + report.Tests = results + report.Outcome = OutcomeProven + for _, worst := range []string{OutcomeNotRedOnOld, OutcomeNotRun, OutcomeFailsOnNew} { + for _, res := range results { + if res.Outcome == worst { + report.Outcome = worst + } + } + } + for _, res := range results { + report.BuildErrorSuspected = report.BuildErrorSuspected || res.BuildErrorSuspected + } + return report, nil +} + +// onBase reverts the non-test files to the merge base, calls fn, and restores +// them even if the caller cancelled: the working tree must come back. +func onBase(ctx context.Context, r repo, mergeBase string, paths []string, fn func()) error { + if err := swapToBase(r, mergeBase, paths); err != nil { + if errors.Is(err, errStashTaken) { + return err + } + if restoreErr := Restore(context.WithoutCancel(ctx), r.root); restoreErr != nil { + return fmt.Errorf("revert to base failed: %v; restore also failed: %w", err, restoreErr) + } + return fmt.Errorf("revert to base failed (files restored): %w", err) + } + fn() + if err := Restore(context.WithoutCancel(ctx), r.root); err != nil { + return fmt.Errorf("restore failed; recover with `test-check prove --restore`: %w", err) + } + return ctx.Err() +} + +var ( + goTestDecl = regexp.MustCompile(`^func (Test\w+)\(`) + pyTestDecl = regexp.MustCompile(`^\s*(?:async\s+)?def (test_\w+)\(`) +) + +// ChangedTestNames lists the Go and Python test functions the change adds or +// edits. Within each hunk, a changed line belongs to the nearest test +// declaration above it — the hunk header's function, until the hunk declares +// another — so an unchanged test merely preceding new ones is not selected. +// Untracked test files contribute all of their tests. +func ChangedTestNames(git func(...string) (string, error), mergeBase, root string, testFiles []string) ([]string, error) { + seen := map[string]bool{} + var names []string + add := func(name string) { + if name != "" && !seen[name] { + seen[name] = true + names = append(names, name) + } + } + decl := func(line string) string { + for _, re := range []*regexp.Regexp{goTestDecl, pyTestDecl} { + if m := re.FindStringSubmatch(line); m != nil { + return m[1] + } + } + return "" + } + if len(testFiles) == 0 { + return nil, nil + } + diff, err := git(append([]string{"diff", "--no-color", "--no-ext-diff", mergeBase, "--"}, testFiles...)...) + if err != nil { + return nil, err + } + current := "" + // File headers ("+++ b/x", "--- a/x") appear only between "diff --git" and + // the first hunk; inside a hunk, a line starting "++" or "--" is content. + inHunk := false + for _, line := range strings.Split(diff, "\n") { + switch { + case strings.HasPrefix(line, "diff --git"): + current, inHunk = "", false + case !inHunk && (strings.HasPrefix(line, "+++") || strings.HasPrefix(line, "---")): + case !inHunk && !strings.HasPrefix(line, "@@"): + // index, mode and rename lines of the file header + case strings.HasPrefix(line, "@@"): + current, inHunk = "", true + if parts := strings.SplitN(line, "@@", 3); len(parts) == 3 { + current = decl(strings.TrimSpace(parts[2])) + } + case line == "": + default: + body := line[1:] + if name := decl(body); name != "" { + current = name + if line[0] == '+' { + add(name) + } + continue + } + if (line[0] == '+' || line[0] == '-') && strings.TrimSpace(body) != "" { + add(current) + } + } + } + untracked, err := git(append([]string{"ls-files", "--others", "--exclude-standard", "-z", "--"}, testFiles...)...) + if err != nil { + return nil, err + } + for _, p := range source.SplitZ(untracked) { + content, err := os.ReadFile(filepath.Join(root, p)) + if err != nil { + return nil, err + } + for _, line := range strings.Split(string(content), "\n") { + add(decl(line)) + } + } + return names, nil +} + +// changedPaths lists every path that differs from the merge base, including +// uncommitted, deleted and untracked files. +func changedPaths(r repo, mergeBase string) ([]string, error) { + tracked, err := r.git("diff", "--name-only", "--no-renames", "-z", mergeBase) + if err != nil { + return nil, err + } + untracked, err := r.git("ls-files", "--others", "--exclude-standard", "-z") + if err != nil { + return nil, err + } + return append(source.SplitZ(tracked), source.SplitZ(untracked)...), nil +} + +type entry struct { + Path string `json:"path"` + // Existed is false for a file the change added; restore recreates it. + Existed bool `json:"existed"` + Mode fs.FileMode `json:"mode"` + Link string `json:"link,omitempty"` + Saved string `json:"saved,omitempty"` +} + +// swapToBase saves the current version of each path under the git dir, writes +// the manifest BEFORE touching the working tree, then puts each path back to +// its base version (deleting it if the base did not have it). +func swapToBase(r repo, mergeBase string, paths []string) error { + stash := filepath.Join(r.gitDir, stashDir) + if err := os.Mkdir(stash, 0o700); err != nil { + if errors.Is(err, fs.ErrExist) { + return errStashTaken + } + return err + } + entries := make([]entry, len(paths)) + for i, p := range paths { + full := filepath.Join(r.root, p) + info, err := os.Lstat(full) + switch { + case errors.Is(err, fs.ErrNotExist): + entries[i] = entry{Path: p} + continue + case err != nil: + return err + } + entries[i] = entry{Path: p, Existed: true, Mode: info.Mode()} + if info.Mode()&fs.ModeSymlink != 0 { + if entries[i].Link, err = os.Readlink(full); err != nil { + return err + } + continue + } + content, err := os.ReadFile(full) + if err != nil { + return err + } + entries[i].Saved = strconv.Itoa(i) + if err := os.WriteFile(filepath.Join(stash, entries[i].Saved), content, 0o600); err != nil { + return err + } + } + manifest, err := json.Marshal(entries) + if err != nil { + return err + } + if err := writeSynced(filepath.Join(stash, manifestName), manifest); err != nil { + return err + } + for _, p := range paths { + if err := writeBase(r, mergeBase, p); err != nil { + return err + } + } + return nil +} + +// writeBase puts one path back to its merge-base version: a symlink as a +// symlink, and deleted only when the base does not have the path. +func writeBase(r repo, mergeBase, p string) error { + full := filepath.Join(r.root, p) + listing, err := r.git("ls-tree", mergeBase, "--", p) + if err != nil { + return err + } + if strings.TrimSpace(listing) == "" { + // Not in the base: the change added it. + if err := os.RemoveAll(full); err != nil { + return err + } + return nil + } + old, err := r.git("show", mergeBase+":"+p) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + return err + } + if err := os.RemoveAll(full); err != nil { + return err + } + switch { + case strings.HasPrefix(listing, "120000"): + return os.Symlink(old, full) + case strings.HasPrefix(listing, "100755"): + return os.WriteFile(full, []byte(old), 0o755) + default: + return os.WriteFile(full, []byte(old), 0o644) + } +} + +// Restore puts back every file a prove run reverted, from the manifest under +// the git dir, then removes the stash. It is safe to run again if it fails +// half way, and a no-op when nothing is stashed. +func Restore(ctx context.Context, dir string) error { + r, err := open(ctx, dir) + if err != nil { + return err + } + if pid := lockHolder(filepath.Join(r.gitDir, lockName)); pid > 0 && pid != os.Getpid() && processAlive(pid) { + return fmt.Errorf("prove run (pid %d) is still using this checkout; it restores its own files", pid) + } + stash := filepath.Join(r.gitDir, stashDir) + raw, err := os.ReadFile(filepath.Join(stash, manifestName)) + if errors.Is(err, fs.ErrNotExist) { + // A crash before the manifest was written left no working-tree change. + return os.RemoveAll(stash) + } + if err != nil { + return err + } + var entries []entry + if err := json.Unmarshal(raw, &entries); err != nil { + return fmt.Errorf("unreadable manifest %s: %w", stash, err) + } + for _, e := range entries { + full := filepath.Join(r.root, e.Path) + if !e.Existed { + if err := os.RemoveAll(full); err != nil { + return err + } + continue + } + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + return err + } + // A test may have created a directory where a reverted file was. + if err := os.RemoveAll(full); err != nil { + return err + } + if e.Link != "" { + if err := os.Symlink(e.Link, full); err != nil { + return err + } + continue + } + content, err := os.ReadFile(filepath.Join(stash, e.Saved)) + if err != nil { + return err + } + if err := os.WriteFile(full, content, e.Mode.Perm()); err != nil { + return err + } + } + return os.RemoveAll(stash) +} + +func writeSynced(path string, content []byte) error { + file, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600) + if err != nil { + return err + } + if _, err := file.Write(content); err != nil { + file.Close() + return err + } + if err := file.Sync(); err != nil { + file.Close() + return err + } + return file.Close() +} + +// runTest runs the command in its own process group, so a timeout or cancel +// stops everything it started, and returns combined output. +func runTest(ctx context.Context, root, command string, timeout time.Duration) (string, error) { + if timeout <= 0 { + timeout = 10 * time.Minute + } + runCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + cmd := exec.CommandContext(runCtx, "sh", "-c", command) + cmd.Dir = root + setProcessGroup(cmd) + // A grandchild holding the output pipe must not block the return. + cmd.WaitDelay = 5 * time.Second + out, err := cmd.CombinedOutput() + if runCtx.Err() == context.DeadlineExceeded && ctx.Err() == nil { + return string(out), fmt.Errorf("test command timed out after %s", timeout) + } + return string(out), err +} + +var buildError = regexp.MustCompile(`(?m)(undefined: |cannot find package|no required module|build failed|compilation failed|error TS\d+|Cannot find module|ModuleNotFoundError|ImportError: |NameError: name|error: cannot find '|error\[E0(425|433)\]|SyntaxError: )`) + +func tail(out string) string { + if len(out) <= tailBytes { + return out + } + return "..." + strings.ToValidUTF8(out[len(out)-tailBytes:], "") +} + +// errStashTaken: another run's stash exists; it must not be restored or +// removed by this run. +var errStashTaken = errors.New("another prove run's stash exists under the git dir; wait for it, or run `test-check prove --restore` if it crashed") + +const lockName = "test-check-prove.lock" + +// acquire takes the per-worktree prove lock with O_EXCL, so two runs never +// share the stash. A lock whose process is gone is taken over. +func acquire(r repo) (func(), error) { + path := filepath.Join(r.gitDir, lockName) + for attempt := 0; attempt < 2; attempt++ { + file, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600) + if err == nil { + _, _ = file.WriteString(strconv.Itoa(os.Getpid())) + _ = file.Close() + return func() { _ = os.Remove(path) }, nil + } + if !errors.Is(err, fs.ErrExist) { + return nil, err + } + if pid := lockHolder(path); pid > 0 && processAlive(pid) { + return nil, fmt.Errorf("another prove run (pid %d) is using this checkout", pid) + } + _ = os.Remove(path) + } + return nil, errors.New("could not take the prove lock") +} + +func lockHolder(path string) int { + raw, err := os.ReadFile(path) + if err != nil { + return 0 + } + pid, _ := strconv.Atoi(strings.TrimSpace(string(raw))) + return pid +} diff --git a/internal/prove/prove_test.go b/internal/prove/prove_test.go new file mode 100644 index 0000000..68da87c --- /dev/null +++ b/internal/prove/prove_test.go @@ -0,0 +1,202 @@ +package prove + +import ( + "context" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +func git(t *testing.T, dir string, args ...string) string { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } + return string(out) +} + +func put(t *testing.T, dir, rel, content string) { + t.Helper() + full := filepath.Join(dir, rel) + if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(full, []byte(content), 0o644); err != nil { + t.Fatal(err) + } +} + +// fixture: main has a buggy retry.go and a weak existing test. The branch fixes +// retry.go (committed), adds stop.go (untracked) and edits notes.md +// (uncommitted); tests are shell scripts that grep the code. +func fixture(t *testing.T) string { + t.Helper() + dir := t.TempDir() + git(t, dir, "init", "-q", "-b", "main") + put(t, dir, "retry.go", "func Retry() error { return nil }\n") + put(t, dir, "retry_test.sh", "grep -q 'func Retry' retry.go\n") + put(t, dir, "notes.md", "old\n") + git(t, dir, "add", ".") + git(t, dir, "commit", "-q", "-m", "base") + git(t, dir, "checkout", "-q", "-b", "fix") + put(t, dir, "retry.go", "func Retry() error { return errStop }\n") + git(t, dir, "commit", "-q", "-am", "fix retry") + put(t, dir, "stop.go", "var errStop error\n") + put(t, dir, "notes.md", "new\n") + return dir +} + +func snapshot(t *testing.T, dir string) map[string]string { + t.Helper() + files := map[string]string{} + err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() && d.Name() == ".git" { + return filepath.SkipDir + } + if !d.IsDir() { + content, err := os.ReadFile(path) + if err != nil { + return err + } + rel, _ := filepath.Rel(dir, path) + files[rel] = string(content) + } + return nil + }) + if err != nil { + t.Fatal(err) + } + return files +} + +func assertRestored(t *testing.T, dir string, before map[string]string) { + t.Helper() + after := snapshot(t, dir) + if len(after) != len(before) { + t.Fatalf("files after = %v, before = %v", after, before) + } + for path, content := range before { + if after[path] != content { + t.Fatalf("%s = %q after prove, want %q", path, after[path], content) + } + } + if _, err := os.Stat(filepath.Join(dir, ".git", stashDir)); !os.IsNotExist(err) { + t.Fatalf("stash left behind: %v", err) + } +} + +func TestRunOutcomesAndTheWorkingTreeIsRestored(t *testing.T) { + cases := []struct { + name string + test string // new content of retry_test.sh; "" leaves the test unchanged + want string + buildErr bool + }{ + {"test that catches the bug", "grep -q 'return errStop' retry.go\n", OutcomeProven, false}, + {"test that passes on the old code", "grep -q 'func Retry' retry.go && true\n", OutcomeNotRedOnOld, false}, + {"test that fails on the new code", "grep -q 'return nil' retry.go\n", OutcomeFailsOnNew, false}, + {"red only because a new name is undefined", "grep -q errStop stop.go || { echo 'undefined: errStop'; exit 1; }\n", OutcomeProven, true}, + {"no test changed", "", OutcomeNoTestChanged, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := fixture(t) + if tc.test != "" { + put(t, dir, "retry_test.sh", tc.test) + } + before := snapshot(t, dir) + report, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "sh retry_test.sh"}) + if err != nil { + t.Fatal(err) + } + if report.Outcome != tc.want || report.BuildErrorSuspected != tc.buildErr { + t.Fatalf("report = %+v, want %s (build error %v)", report, tc.want, tc.buildErr) + } + assertRestored(t, dir, before) + }) + } +} + +// The old-code run must see: committed code at base, an added file gone, and +// uncommitted non-test edits reverted too; the test file stays new. +func TestOldRunSeesEveryNonTestChangeReverted(t *testing.T) { + dir := fixture(t) + put(t, dir, "retry_test.sh", `grep -q 'return errStop' retry.go || { test ! -e stop.go && grep -qx old notes.md && echo OLD-STATE-OK; exit 1; }`+"\n") + before := snapshot(t, dir) + report, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "sh retry_test.sh"}) + if err != nil { + t.Fatal(err) + } + if report.Outcome != OutcomeProven || !strings.Contains(report.OldOutput, "OLD-STATE-OK") { + t.Fatalf("report = %+v", report) + } + for _, p := range []string{"retry.go", "stop.go", "notes.md"} { + if !strings.Contains(strings.Join(report.RevertedFiles, " "), p) { + t.Fatalf("reverted = %v, missing %s", report.RevertedFiles, p) + } + } + assertRestored(t, dir, before) +} + +// Ctrl-C during the old-code run must still put the new code back. +func TestCancelDuringOldRunStillRestores(t *testing.T) { + dir := fixture(t) + put(t, dir, "retry_test.sh", "grep -q 'return errStop' retry.go || sleep 30\n") + before := snapshot(t, dir) + ctx, cancel := context.WithCancel(context.Background()) + go func() { + for { + if _, err := os.Stat(filepath.Join(dir, ".git", stashDir, manifestName)); err == nil { + time.Sleep(200 * time.Millisecond) + cancel() + return + } + time.Sleep(20 * time.Millisecond) + } + }() + start := time.Now() + if _, err := Run(ctx, Options{Dir: dir, Base: "main", Test: "sh retry_test.sh"}); err == nil { + t.Fatal("cancelled run returned no error") + } + if time.Since(start) > 15*time.Second { + t.Fatal("cancel did not stop the test process") + } + assertRestored(t, dir, before) +} + +// A crash between reverting and restoring leaves the stash; the next run must +// refuse, and --restore must bring the new code back. +func TestLeftoverStashBlocksRunsUntilRestored(t *testing.T) { + dir := fixture(t) + before := snapshot(t, dir) + r, err := open(context.Background(), dir) + if err != nil { + t.Fatal(err) + } + mergeBase := strings.TrimSpace(git(t, dir, "merge-base", "main", "HEAD")) + if err := swapToBase(r, mergeBase, []string{"retry.go", "stop.go", "notes.md"}); err != nil { + t.Fatal(err) + } + if got := snapshot(t, dir)["retry.go"]; !strings.Contains(got, "return nil") { + t.Fatalf("swap did not revert retry.go: %q", got) + } + if _, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "true"}); err == nil || !strings.Contains(err.Error(), "--restore") { + t.Fatalf("Run with leftover stash err = %v", err) + } + if err := Restore(context.Background(), dir); err != nil { + t.Fatal(err) + } + assertRestored(t, dir, before) + if err := Restore(context.Background(), dir); err != nil { + t.Fatalf("second Restore = %v, want no-op", err) + } +} diff --git a/internal/prove/review_fixes_test.go b/internal/prove/review_fixes_test.go new file mode 100644 index 0000000..7cb2000 --- /dev/null +++ b/internal/prove/review_fixes_test.go @@ -0,0 +1,100 @@ +package prove + +import ( + "context" + "os" + "path/filepath" + "reflect" + "strconv" + "strings" + "testing" + + "github.com/gitmoot/test-check/internal/source" +) + +// Review P2: a changed content line starting "++" or "--" was read as a file +// header, so the test it belonged to was silently dropped. +func TestChangedTestNamesKeepsTestsWithPlusPlusAndMinusMinusLines(t *testing.T) { + dir := t.TempDir() + git(t, dir, "init", "-q", "-b", "main") + put(t, dir, "u_test.py", "def test_v():\n x = 1\n assert x\n\n\ndef test_w():\n s = 1\n--z\n assert s\n") + git(t, dir, "add", ".") + git(t, dir, "commit", "-q", "-m", "base") + base := strings.TrimSpace(git(t, dir, "rev-parse", "HEAD")) + put(t, dir, "u_test.py", "def test_v():\n x = 1\n++y\n assert x\n\n\ndef test_w():\n s = 1\n assert s\n") + names, err := ChangedTestNames(source.Git(context.Background(), dir), base, dir, []string{"u_test.py"}) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(names, []string{"test_v", "test_w"}) { + t.Fatalf("names = %v, want [test_v test_w]", names) + } +} + +// Review P2: a second run must not touch the first run's stash. While a live +// process holds the lock, Run and Restore refuse and leave the stash intact. +func TestLiveLockBlocksRunAndRestore(t *testing.T) { + dir := fixture(t) + gitDir := strings.TrimSpace(git(t, dir, "rev-parse", "--absolute-git-dir")) + stash := filepath.Join(gitDir, stashDir) + if err := os.Mkdir(stash, 0o700); err != nil { + t.Fatal(err) + } + put(t, stash, manifestName, `[{"path":"retry.go","existed":false}]`) + // A live process that is not this test: the parent (go test). + if err := os.WriteFile(filepath.Join(gitDir, lockName), []byte(strconv.Itoa(os.Getppid())), 0o600); err != nil { + t.Fatal(err) + } + if _, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "true"}); err == nil || !strings.Contains(err.Error(), "another prove run") { + t.Fatalf("Run err = %v, want refusal", err) + } + if err := Restore(context.Background(), dir); err == nil { + t.Fatal("Restore consumed a live run's stash") + } + if _, err := os.Stat(filepath.Join(stash, manifestName)); err != nil { + t.Fatalf("live run's manifest gone: %v", err) + } + if _, err := os.Stat(filepath.Join(dir, "retry.go")); err != nil { + t.Fatalf("working file touched: %v", err) + } +} + +// A lock left by a dead process is taken over. +func TestStaleLockIsTakenOver(t *testing.T) { + dir := fixture(t) + gitDir := strings.TrimSpace(git(t, dir, "rev-parse", "--absolute-git-dir")) + if err := os.WriteFile(filepath.Join(gitDir, lockName), []byte("999999999"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "sh retry_test.sh"}); err != nil { + t.Fatalf("Run with stale lock: %v", err) + } + if _, err := os.Stat(filepath.Join(gitDir, lockName)); !os.IsNotExist(err) { + t.Fatalf("lock not released: %v", err) + } +} + +// Review P3: a base-side symlink stays a symlink for the old-code run. +func TestOldRunSeesBaseSymlinkAsSymlink(t *testing.T) { + dir := fixture(t) + git(t, dir, "checkout", "-q", "main") + if err := os.Symlink("retry.go", filepath.Join(dir, "link.go")); err != nil { + t.Fatal(err) + } + git(t, dir, "add", ".") + git(t, dir, "commit", "-q", "-m", "link") + git(t, dir, "checkout", "-q", "-b", "fix2") + if err := os.Remove(filepath.Join(dir, "link.go")); err != nil { + t.Fatal(err) + } + put(t, dir, "link.go", "not a link\n") + put(t, dir, "retry.go", "func Retry() error { return errStop }\n") + put(t, dir, "link_test.sh", "test -L link.go && exit 1; exit 0\n") + report, err := Run(context.Background(), Options{Dir: dir, Base: "main", Test: "sh link_test.sh"}) + if err != nil { + t.Fatal(err) + } + if report.Outcome != OutcomeProven { + t.Fatalf("outcome = %s, want proven (old run must see a symlink)", report.Outcome) + } +} diff --git a/internal/source/diff.go b/internal/source/diff.go new file mode 100644 index 0000000..4f6ab25 --- /dev/null +++ b/internal/source/diff.go @@ -0,0 +1,69 @@ +// Package source collects a change and its nearby tests, either from a local +// git checkout or from GitHub. +package source + +import ( + "strconv" + "strings" + + "github.com/gitmoot/test-check/internal/check" +) + +// splitUnifiedDiff turns `git diff` output into one File per path, keeping each +// file's hunks (from the first "@@") as its patch, which matches the shape the +// GitHub API returns. A file with no hunks (binary, pure rename, mode change) +// gets an empty patch. +func splitUnifiedDiff(diff string) []check.File { + var files []check.File + for _, chunk := range strings.Split("\n"+diff, "\ndiff --git ")[1:] { + header, body, _ := strings.Cut(chunk, "\n") + path := diffPath(header, body) + if path == "" { + continue + } + patch := "" + if at := strings.Index(body, "\n@@"); at >= 0 { + patch = body[at+1:] + } else if strings.HasPrefix(body, "@@") { + patch = body + } + files = append(files, check.File{Path: path, Patch: strings.TrimRight(patch, "\n")}) + } + return files +} + +// diffPath prefers the "+++ b/" line, which is unambiguous even when a path +// contains spaces; deletions fall back to "--- a/". +func diffPath(header, body string) string { + for _, line := range strings.Split(body, "\n") { + if strings.HasPrefix(line, "@@") { + break + } + if rest, ok := strings.CutPrefix(line, "+++ b/"); ok { + return rest + } + } + for _, line := range strings.Split(body, "\n") { + if strings.HasPrefix(line, "@@") { + break + } + if rest, ok := strings.CutPrefix(line, "--- a/"); ok { + return rest + } + } + if _, b, ok := strings.Cut(header, " b/"); ok { + return b + } + return "" +} + +// newFilePatch renders an untracked file as an all-added patch. +func newFilePatch(content string) string { + lines := strings.Split(strings.TrimRight(content, "\n"), "\n") + var b strings.Builder + b.WriteString("@@ -0,0 +1," + strconv.Itoa(len(lines)) + " @@") + for _, line := range lines { + b.WriteString("\n+" + line) + } + return b.String() +} diff --git a/internal/source/github.go b/internal/source/github.go new file mode 100644 index 0000000..95f10ce --- /dev/null +++ b/internal/source/github.go @@ -0,0 +1,121 @@ +package source + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/url" + "os/exec" + "strings" + + "github.com/gitmoot/test-check/internal/check" +) + +// Gh runs the gh CLI and returns stdout. It is a variable so tests can fake it. +var Gh = func(ctx context.Context, args ...string) ([]byte, error) { + out, err := exec.CommandContext(ctx, "gh", args...).Output() + if err != nil { + var exit *exec.ExitError + if errors.As(err, &exit) { + return nil, fmt.Errorf("gh %s: %s", strings.Join(args, " "), strings.TrimSpace(string(exit.Stderr))) + } + return nil, err + } + return out, nil +} + +type ghFile struct { + Filename string `json:"filename"` + Patch string `json:"patch"` +} + +// PullRequest reads an open or merged pull request's current head. +func PullRequest(ctx context.Context, repo string, number int) (check.Input, error) { + var pr struct { + Title string `json:"title"` + Head struct { + SHA string `json:"sha"` + } `json:"head"` + } + if err := ghJSON(ctx, &pr, "api", fmt.Sprintf("repos/%s/pulls/%d", repo, number)); err != nil { + return check.Input{}, err + } + // GitHub serves at most 3000 files over 30 pages of 100. + var files []ghFile + for page := 1; page <= 30; page++ { + var batch []ghFile + if err := ghJSON(ctx, &batch, "api", fmt.Sprintf("repos/%s/pulls/%d/files?per_page=100&page=%d", repo, number, page)); err != nil { + return check.Input{}, err + } + files = append(files, batch...) + if len(batch) < 100 { + break + } + } + return fromGitHub(ctx, repo, pr.Title, pr.Head.SHA, files) +} + +// Compare reads the change between two commits, for example a pull request's +// first reviewed head against the commit its branch started from. +func Compare(ctx context.Context, repo, base, head, title string) (check.Input, error) { + var cmp struct { + Files []ghFile `json:"files"` + } + if err := ghJSON(ctx, &cmp, "api", fmt.Sprintf("repos/%s/compare/%s...%s", repo, base, head)); err != nil { + return check.Input{}, err + } + in, err := fromGitHub(ctx, repo, title, head, cmp.Files) + // The compare API lists at most 300 files. + in.Incomplete = len(cmp.Files) >= 300 + return in, err +} + +func fromGitHub(ctx context.Context, repo, title, head string, ghFiles []ghFile) (check.Input, error) { + if len(ghFiles) == 0 { + return check.Input{}, errors.New("the change has no files") + } + in := check.Input{Repo: repo, Title: title, Head: head} + for _, f := range ghFiles { + in.Files = append(in.Files, check.File{Path: f.Filename, Patch: f.Patch}) + } + var tree struct { + Tree []struct { + Path string `json:"path"` + Type string `json:"type"` + } `json:"tree"` + } + if err := ghJSON(ctx, &tree, "api", fmt.Sprintf("repos/%s/git/trees/%s?recursive=1", repo, head)); err != nil { + return check.Input{}, err + } + all := make([]string, 0, len(tree.Tree)) + for _, entry := range tree.Tree { + if entry.Type == "blob" { + all = append(all, entry.Path) + } + } + return withNearby(in, all, func(p string) (string, error) { + out, err := Gh(ctx, "api", "-H", "Accept: application/vnd.github.raw", + fmt.Sprintf("repos/%s/contents/%s?ref=%s", repo, escapePath(p), head)) + return string(out), err + }), nil +} + +func ghJSON(ctx context.Context, into any, args ...string) error { + raw, err := Gh(ctx, args...) + if err != nil { + return err + } + if err := json.Unmarshal(raw, into); err != nil { + return fmt.Errorf("decode gh %s: %w", strings.Join(args, " "), err) + } + return nil +} + +func escapePath(p string) string { + parts := strings.Split(p, "/") + for i, part := range parts { + parts[i] = url.PathEscape(part) + } + return strings.Join(parts, "/") +} diff --git a/internal/source/local.go b/internal/source/local.go new file mode 100644 index 0000000..631e608 --- /dev/null +++ b/internal/source/local.go @@ -0,0 +1,126 @@ +package source + +import ( + "context" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + + "github.com/gitmoot/test-check/internal/check" +) + +// Local reads the change in a git checkout: everything since the merge base +// with base, including uncommitted and untracked files, so it can run before a +// commit exists. An empty base means the remote's default branch. +func Local(ctx context.Context, dir, base, title string) (check.Input, error) { + git := Git(ctx, dir) + root, err := git("rev-parse", "--show-toplevel") + if err != nil { + return check.Input{}, err + } + root = strings.TrimSpace(root) + if base == "" { + base, err = DefaultBase(git) + if err != nil { + return check.Input{}, err + } + } + diff, err := git("diff", "--merge-base", base, "--no-color", "--no-ext-diff", "--no-renames") + if err != nil { + return check.Input{}, err + } + files := splitUnifiedDiff(diff) + untracked, err := git("ls-files", "--others", "--exclude-standard", "-z") + if err != nil { + return check.Input{}, err + } + for _, p := range SplitZ(untracked) { + content, err := os.ReadFile(filepath.Join(root, p)) + if err != nil { + return check.Input{}, err + } + patch := "" + if !strings.ContainsRune(string(content), 0) { + patch = newFilePatch(string(content)) + } + files = append(files, check.File{Path: p, Patch: patch}) + } + if len(files) == 0 { + return check.Input{}, fmt.Errorf("no changes since %s", base) + } + tracked, err := git("ls-files", "-z") + if err != nil { + return check.Input{}, err + } + all := append(SplitZ(tracked), SplitZ(untracked)...) + head, _ := git("rev-parse", "HEAD") + if title == "" { + if subject, err := git("log", "-1", "--format=%s", base+"..HEAD"); err == nil { + title = strings.TrimSpace(subject) + } + } + return withNearby(check.Input{Title: title, Head: strings.TrimSpace(head), Files: files}, all, func(p string) (string, error) { + content, err := os.ReadFile(filepath.Join(root, p)) + return string(content), err + }), nil +} + +// Git returns a runner for git commands in dir; errors carry git's stderr. +func Git(ctx context.Context, dir string) func(args ...string) (string, error) { + return func(args ...string) (string, error) { + cmd := exec.CommandContext(ctx, "git", append([]string{"-C", dir}, args...)...) + out, err := cmd.Output() + if err != nil { + var exit *exec.ExitError + if errors.As(err, &exit) { + return "", fmt.Errorf("git %s: %s", strings.Join(args, " "), strings.TrimSpace(string(exit.Stderr))) + } + return "", err + } + return string(out), nil + } +} + +// DefaultBase is the remote's default branch, else a local main or master. +func DefaultBase(git func(...string) (string, error)) (string, error) { + if ref, err := git("symbolic-ref", "--quiet", "--short", "refs/remotes/origin/HEAD"); err == nil { + return strings.TrimSpace(ref), nil + } + for _, candidate := range []string{"origin/main", "origin/master", "main", "master"} { + if _, err := git("rev-parse", "--verify", "--quiet", candidate); err == nil { + return candidate, nil + } + } + return "", errors.New("cannot find a base branch; pass --base") +} + +// SplitZ splits NUL-separated git output. +func SplitZ(out string) []string { + var paths []string + for _, p := range strings.Split(out, "\x00") { + if p != "" { + paths = append(paths, p) + } + } + return paths +} + +// withNearby attaches the nearest existing test files. A test file that cannot +// be read is skipped rather than failing the whole check. +func withNearby(in check.Input, all []string, read func(string) (string, error)) check.Input { + changed := make([]string, len(in.Files)) + for i, f := range in.Files { + changed[i] = f.Path + } + for _, p := range check.NearbyTestPaths(changed, all) { + content, err := read(p) + if err != nil { + continue + } + in.Nearby = append(in.Nearby, check.NearbyTest{Path: p, Content: content}) + } + return in +} diff --git a/internal/source/source_test.go b/internal/source/source_test.go new file mode 100644 index 0000000..c2cafb9 --- /dev/null +++ b/internal/source/source_test.go @@ -0,0 +1,134 @@ +package source + +import ( + "context" + "errors" + "os" + "os/exec" + "path/filepath" + "reflect" + "strconv" + "strings" + "testing" + + "github.com/gitmoot/test-check/internal/check" +) + +func gitIn(t *testing.T, dir string, args ...string) { + t.Helper() + cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) + cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } +} + +func write(t *testing.T, dir, rel, content string) { + t.Helper() + path := filepath.Join(dir, rel) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } +} + +// An agent runs test-check before committing: committed, staged, unstaged, +// and untracked changes since the base must all be seen, and nothing before it. +func TestLocalSeesEveryUncommittedKindOfChangeSinceTheBase(t *testing.T) { + dir := t.TempDir() + gitIn(t, dir, "init", "-q", "-b", "main") + write(t, dir, "pkg/retry.go", "package pkg\n\nfunc Retry() error { return nil }\n") + write(t, dir, "pkg/retry_test.go", "package pkg\n\nfunc TestRetryStops(t *testing.T) {}\n") + write(t, dir, "pkg/old.go", "package pkg\n") + gitIn(t, dir, "add", ".") + gitIn(t, dir, "commit", "-q", "-m", "base") + gitIn(t, dir, "checkout", "-q", "-b", "fix") + write(t, dir, "pkg/old.go", "package pkg\n\nvar committed = 1\n") + gitIn(t, dir, "commit", "-q", "-am", "Fix retry loop") + write(t, dir, "pkg/retry.go", "package pkg\n\nfunc Retry() error { return errStop }\n") + write(t, dir, "pkg/stop.go", "package pkg\n\nvar errStop error\n") + + in, err := Local(context.Background(), dir, "main", "") + if err != nil { + t.Fatal(err) + } + got := map[string]string{} + for _, f := range in.Files { + got[f.Path] = f.Patch + } + if len(got) != 3 || !strings.Contains(got["pkg/old.go"], "+var committed = 1") || + !strings.Contains(got["pkg/retry.go"], "+func Retry() error { return errStop }") || + !strings.Contains(got["pkg/stop.go"], "+var errStop error") { + t.Fatalf("files = %#v", got) + } + if in.Title != "Fix retry loop" { + t.Fatalf("title = %q", in.Title) + } + if len(in.Nearby) != 1 || in.Nearby[0].Path != "pkg/retry_test.go" || !strings.Contains(in.Nearby[0].Content, "TestRetryStops") { + t.Fatalf("nearby = %+v", in.Nearby) + } +} + +func TestLocalWithNoChangesIsAnError(t *testing.T) { + dir := t.TempDir() + gitIn(t, dir, "init", "-q", "-b", "main") + write(t, dir, "a.go", "package a\n") + gitIn(t, dir, "add", ".") + gitIn(t, dir, "commit", "-q", "-m", "base") + if _, err := Local(context.Background(), dir, "main", ""); err == nil || !strings.Contains(err.Error(), "no changes") { + t.Fatalf("err = %v", err) + } +} + +func TestSplitUnifiedDiffKeepsDeletionsAndBinaryPaths(t *testing.T) { + diff := "diff --git a/gone.go b/gone.go\ndeleted file mode 100644\n--- a/gone.go\n+++ /dev/null\n@@ -1 +0,0 @@\n-package gone\n" + + "diff --git a/logo.png b/logo.png\nBinary files a/logo.png and b/logo.png differ\n" + got := splitUnifiedDiff(diff) + want := []check.File{{Path: "gone.go", Patch: "@@ -1 +0,0 @@\n-package gone"}, {Path: "logo.png", Patch: ""}} + if !reflect.DeepEqual(got, want) { + t.Fatalf("splitUnifiedDiff = %#v", got) + } +} + +func TestPullRequestReadsFilesAcrossPagesAndNearbyTestsAtHead(t *testing.T) { + var calls []string + prev := Gh + Gh = func(_ context.Context, args ...string) ([]byte, error) { + call := strings.Join(args, " ") + calls = append(calls, call) + switch { + case call == "api repos/o/r/pulls/7": + return []byte(`{"title":"Stop retry loop","head":{"sha":"abc"}}`), nil + case call == "api repos/o/r/pulls/7/files?per_page=100&page=1": + page := make([]string, 100) + for i := range page { + page[i] = `{"filename":"src/gen` + strconv.Itoa(i) + `.go","patch":"+x"}` + } + page[0] = `{"filename":"pkg/retry.go","patch":"@@ -1 +1 @@\n-a\n+b"}` + return []byte("[" + strings.Join(page, ",") + "]"), nil + case call == "api repos/o/r/pulls/7/files?per_page=100&page=2": + return []byte(`[{"filename":"README.md","patch":"+x"}]`), nil + case call == "api repos/o/r/git/trees/abc?recursive=1": + return []byte(`{"tree":[{"path":"pkg","type":"tree"},{"path":"pkg/retry.go","type":"blob"},{"path":"pkg/retry_test.go","type":"blob"},{"path":"pkg/broken_test.go","type":"blob"}]}`), nil + case call == "api -H Accept: application/vnd.github.raw repos/o/r/contents/pkg/retry_test.go?ref=abc": + return []byte("func TestRetryStops(t *testing.T) {}"), nil + case strings.Contains(call, "contents/pkg/broken_test.go"): + return nil, errors.New("HTTP 404") + } + return nil, errors.New("unexpected gh call: " + call) + } + t.Cleanup(func() { Gh = prev }) + + in, err := PullRequest(context.Background(), "o/r", 7) + if err != nil { + t.Fatal(err) + } + if in.Title != "Stop retry loop" || in.Head != "abc" || len(in.Files) != 101 || in.Files[100].Path != "README.md" { + t.Fatalf("input = %+v", in) + } + if len(in.Nearby) != 1 || in.Nearby[0].Path != "pkg/retry_test.go" { + t.Fatalf("nearby = %+v (an unreadable test file must be skipped, not fatal)", in.Nearby) + } +} diff --git a/skills/test-check/SKILL.md b/skills/test-check/SKILL.md new file mode 100644 index 0000000..a56d08f --- /dev/null +++ b/skills/test-check/SKILL.md @@ -0,0 +1,43 @@ +--- +name: test-check +description: Decides what testing a code change needs — a regression test that goes red on the old code, a case in an existing test, a one-off check, or none. Use before writing or finalizing tests for a change, and when judging tests a change adds. +--- + +A permanent test earns its place only if it protects behavior someone relies on and goes **red** on a plausible regression: it fails on the old code and passes on the new. A test that cannot go red is cost without protection. + +## Steps + +1. **Ask.** In the checkout, run `test-check` (it reads everything since the merge base, including uncommitted files). For a pushed PR: `test-check --repo OWNER/REPO --pr N`. Done when it prints `advice:`, or exits 3 (JEV unavailable) — then answer the four questions below yourself and act on your answers. +2. **Act on `advice`:** + + | advice | do | + |---|---| + | `new_regression_test` | Write one test for the changed behavior. | + | `extend_existing_test` | Add a case to the named nearby test instead of a new test. | + | `one_off_check` | No permanent test. Run the changed path once. | + | `no_permanent_test` | No new test. Build and run the existing tests. | + | `need_context` | Answer the four questions yourself. | + + If it warns that an added test looks **low-value**, rework it until it can go red, or delete it. +3. **Prove it.** For a new or extended test, run `test-check prove --test ""` — for example `go test ./pkg -v -run '^{name}$'` or `python3 -m pytest -v -k '{name}'`. It finds the tests your change adds or edits (Go and Python; otherwise pass `--each A,B`), runs each on your new code, then again with every non-test change reverted, and restores your files. **Every** test must go red on its own. Outcomes: + + | outcome | exit | meaning | + |---|---|---| + | `proven` | 0 | Red on the old code, green on the new. Done. | + | `not_red_on_old` | 4 | The test passes without the fix: it protects nothing. Make it fail without the fix. | + | `fails_on_new` | 4 | The test fails on your code. Fix that first. | + | `no_test_in_change` | 4 | No test file changed. Add one, or treat it as a one-off check. | + | `test_not_run` | 4 | The command never mentioned the test, so it probably selected nothing. Fix the command or the name. | + + A `build_error_suspected` warning means the old run failed only because the test uses new names; for a bug fix, test through an interface the old code already had. If a run is interrupted, `test-check prove --restore` puts your files back. For a one-off check: run it and keep the output. Done when the PR description records the `prove` outcome (or the one-off command and what you observed) — not before. + +`test-check` is advice. Your own evidence decides; never skip verification because it said no test. + +## The four questions + +1. What behavior does the test protect, for whom? +2. What plausible bug would make it go red? +3. Why do the existing tests not catch that bug already? +4. Does it force test-only code, hooks, or exports into the product? If yes, test through the real interface instead. + +A test that restates source text or constants, copies fixtures, asserts private internals or call order, or duplicates a stronger test fails question 2 or 3: do not add it.