Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: ci

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
name: format, vet, build, test
runs-on: ubuntu-latest
env:
CGO_ENABLED: "0"
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

- name: gofmt
run: |
unformatted="$(gofmt -l .)"
if [ -n "$unformatted" ]; then
echo "these files are not gofmt-clean:"
echo "$unformatted"
exit 1
fi

- name: go vet
run: go vet ./...

- name: go build
run: go build ./...

- name: go test
run: go test ./...

- name: go.mod is tidy
run: |
go mod tidy
git diff --exit-code go.mod
80 changes: 78 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,80 @@
# test-check

Asks JEV whether a code change needs a regression test, an extended existing
test, a one-off check, or no permanent test. Proposed in gitmoot/gitmoot#2265.
Asks [JEV](https://openrouter.ai/typesafe) what testing a code change needs, so
agents write the regression tests that matter and skip the ones that only add
cost. Proposed in gitmoot/gitmoot#2265.

It sends JEV the diff and the nearest existing tests, and gets back one
**advice**:

| advice | meaning |
|---|---|
| `new_regression_test` | Behavior changed and nothing nearby tests it: add a test that fails on the old code. |
| `extend_existing_test` | A nearby test already covers this area: add a case to it. |
| `one_off_check` | Real change, but a permanent test would be brittle or only restate code: run it once. |
| `no_permanent_test` | No behavior change: build and run the existing tests. |
| `need_context` | JEV could not decide. |

It also reports two probabilities:
- **covered**: whether the change's own tests would fail on a regression.
- **low-value**: whether an added test only restates code, duplicates another test, tests internals, or would pass on the old code.

It is advice, not a gate. The agent still proves the change works; see the
[skill](skills/test-check/SKILL.md).

## Prove a test goes red

JEV reading a diff cannot tell whether a test would pass on the old code. So
`test-check prove` runs it:

```sh
test-check prove --test "go test ./pkg -v -run '^{name}$'"
```

1. It runs each test the change adds or edits on the working tree. Each must pass.
2. It puts every changed non-test file back to its base version and runs each test again. Each must fail.
3. It restores the files.

`{name}` is filled in for each test. The tests are taken from the change's diff for Go and Python; pass `--each A,B` for other runners. The command must print each test's name (use a verbose runner); otherwise the test is reported as `test_not_run`. Every test must go red on its own. A change is not proven just because one good test sits next to a weak one.

Without `{name}`, the command runs once for the whole change, and one red test is enough.

Test files and fixtures keep their new versions for both runs. The reverted files are saved under the git dir before anything changes. An interrupted or crashed run is recovered with `test-check prove --restore`, and a new run refuses to start until then.

Only one prove run may use a checkout at a time: a lock under the git dir makes a second run, or `--restore`, refuse while the first is alive. A lock left by a dead process is taken over. A test that hangs on the old code until `--timeout` counts as red.

Outcomes:
- `proven`: exit 0.
- `not_red_on_old`: exit 4. The test passes without the fix.
- `fails_on_new`: exit 4.
- `no_test_in_change`: exit 4.
- `test_not_run`: exit 4.
- `no_code_in_change`: exit 0. Only tests changed.

A `build_error_suspected` flag marks an old-code failure that looks like a compile or import error rather than a failed assertion.

## Use

```sh
test-check # local checkout, incl. uncommitted files
test-check --repo OWNER/REPO --pr N # a pull request's current head
test-check --repo OWNER/REPO --compare BASE...HEAD
test-check --json # machine-readable
```

Exit codes:
- `0`: advice printed.
- `1`: the change could not be read.
- `2`: usage error.
- `3`: JEV unavailable. No advice is printed; answer the skill's four questions yourself.

The GitHub modes use the `gh` CLI. The key comes from `$OPENROUTER_API_KEY`,
else the `OPENROUTER_API_KEY` line in `~/.config/gitmoot/keychain.env`. The
model is pinned to `typesafe/jev-1.13`.

## Install

```sh
go build -o ~/.local/bin/test-check ./cmd/test-check
ln -s "$PWD/skills/test-check" ~/.claude/skills/test-check
```
165 changes: 165 additions & 0 deletions cmd/test-check/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
// Command test-check asks JEV what testing a code change needs, and proves
// that a change's tests go red on the old code and green on the new.
package main

import (
"bufio"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"

"github.com/gitmoot/test-check/internal/check"
"github.com/gitmoot/test-check/internal/jev"
"github.com/gitmoot/test-check/internal/source"
)

const (
exitUsage = 2
exitSource = 1
exitUnavailable = 3
apiKeyName = "OPENROUTER_API_KEY"
)

// newJudge is a variable so tests can substitute a fake.
var newJudge = func(key string) check.Judge {
return &jev.Client{
Endpoint: jev.DefaultEndpoint,
APIKey: key,
HTTP: http.DefaultClient,
Timeout: jev.DefaultTimeout,
MaxRetries: jev.DefaultMaxRetries,
}
}

func main() {
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
}

func run(args []string, stdout, stderr io.Writer) int {
if len(args) > 0 && args[0] == "prove" {
return runProve(args[1:], stdout, stderr)
}
fs := flag.NewFlagSet("test-check", flag.ContinueOnError)
fs.SetOutput(stderr)
fs.Usage = func() {
fmt.Fprintln(stderr, `Usage:
test-check [--dir DIR] [--base REF] [--title TEXT] [--json]
the change in a local checkout since its merge base with REF
(default: the remote default branch), including uncommitted files
test-check --repo OWNER/REPO --pr N [--json]
a pull request's current head
test-check --repo OWNER/REPO --compare BASE...HEAD [--title TEXT] [--json]
the change between two commits
test-check prove --test CMD [--each A,B] [--dir DIR] [--base REF] [--timeout DUR] [--json]
run CMD on the new code (must pass), then with every non-test change
reverted (must fail); files are restored afterwards. With {name} in
CMD, every added or edited test must go red on its own
test-check prove --restore [--dir DIR]
put back files a crashed prove run left reverted

Exit: 0 advice printed / proven, 1 error, 2 usage, 3 JEV unavailable,
4 not proven (passes on the old code, fails on the new, or no test changed).
Key: $OPENROUTER_API_KEY, else OPENROUTER_API_KEY in ~/.config/gitmoot/keychain.env.`)
}
dir := fs.String("dir", ".", "local git checkout")
base := fs.String("base", "", "base ref for a local change")
title := fs.String("title", "", "short description of the change")
repo := fs.String("repo", "", "OWNER/REPO on GitHub")
pr := fs.Int("pr", 0, "pull request number")
compare := fs.String("compare", "", "BASE...HEAD commits on GitHub")
model := fs.String("model", jev.DefaultModel, "JEV model")
asJSON := fs.Bool("json", false, "print the result as JSON")
if err := fs.Parse(args); err != nil {
if err == flag.ErrHelp {
return 0
}
return exitUsage
}
if fs.NArg() != 0 || (*pr > 0 && *compare != "") || ((*pr > 0 || *compare != "") && *repo == "") {
fs.Usage()
return exitUsage
}

ctx := context.Background()
var in check.Input
var err error
switch {
case *pr > 0:
in, err = source.PullRequest(ctx, *repo, *pr)
case *compare != "":
b, h, ok := strings.Cut(*compare, "...")
if !ok || b == "" || h == "" {
fmt.Fprintln(stderr, "test-check: --compare must be BASE...HEAD")
return exitUsage
}
in, err = source.Compare(ctx, *repo, b, h, *title)
default:
in, err = source.Local(ctx, *dir, *base, *title)
in.Repo = *repo
}
if err != nil {
fmt.Fprintf(stderr, "test-check: %v\n", err)
return exitSource
}

key := apiKey()
if key == "" {
fmt.Fprintf(stderr, "test-check: JEV unavailable: no %s. Answer the four questions in the test-check skill yourself.\n", apiKeyName)
return exitUnavailable
}
result, err := check.Evaluate(ctx, newJudge(key), *model, in)
if err != nil {
fmt.Fprintf(stderr, "test-check: JEV unavailable: %v. Answer the four questions in the test-check skill yourself.\n", err)
return exitUnavailable
}

if *asJSON {
encoder := json.NewEncoder(stdout)
encoder.SetIndent("", " ")
_ = encoder.Encode(result)
return 0
}
fmt.Fprintf(stdout, "advice: %s\n", result.Advice)
fmt.Fprintf(stdout, "next: %s\n", result.Next)
fmt.Fprintf(stdout, "covered by this change's own tests: p=%.2f\n", result.Covered)
fmt.Fprintf(stdout, "low-value tests added: p=%.2f\n", result.LowValue)
if len(result.NearbyTests) > 0 {
fmt.Fprintf(stdout, "nearby tests: %s\n", strings.Join(result.NearbyTests, ", "))
}
if !result.DiffComplete {
fmt.Fprintln(stdout, "note: the diff was truncated or missing parts; judge the omitted files yourself")
}
return 0
}

// apiKey prefers the environment, then the Gitmoot keychain file the fleet
// already keeps the key in. Any read failure means no key.
func apiKey() string {
if key := strings.TrimSpace(os.Getenv(apiKeyName)); key != "" {
return key
}
configDir, err := os.UserConfigDir()
if err != nil {
return ""
}
file, err := os.Open(filepath.Join(configDir, "gitmoot", "keychain.env"))
if err != nil {
return ""
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
name, value, ok := strings.Cut(strings.TrimSpace(scanner.Text()), "=")
if ok && strings.TrimSpace(strings.TrimPrefix(name, "export ")) == apiKeyName {
value = strings.TrimSpace(value)
return strings.Trim(strings.TrimSpace(value), `"'`)
}
}
return ""
}
Loading
Loading