Repository navigation
Add test-check: ask JEV what testing a change needs - #1
Conversation
|
Added The historical bar was frozen before any run: Main bar failed.
Passed:
Harness amendments are recorded in the results file: a pinned Go toolchain, the module directory, and script-style Python tests. 7 Python PRs could not import their dependencies and were excluded, as the frozen bar allowed. Still draft; nothing installed and no agent rule added. |
|
Per-test mode added at The bar was frozen before the run:
What improved. It now catches both gap PRs whose tests pass without the fix, including the exact test the reviewer named on gitmoot#2254 ( What red/green cannot catch:
Still draft; nothing installed. |
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
…ks and restore edges
|
Round 1 response at
|
Decision: Findings
Tests Run
Raw runtime output was retained in local Gitmoot state and is not posted here. |
|
Independent review-only job |
First version of
test-check, from gitmoot/gitmoot#2265, built at the owner's request. Draft: it failed its frozen historical bar, so nothing is installed and no agent rule was added.What it does
new_regression_test/extend_existing_test/one_off_check/no_permanent_test/need_context), plus two probabilities: whether the change's own tests would catch a regression, and whether an added test looks low-value.skills/test-check/tells agents to prove every new test goes red on the old code and green on the new.Historical check
The bar and dataset were frozen before any run. Details are in
/root/fleet-tools/state/test-check-eval-20260925/.need_contextat most 10%; errors at most 5%Why it misses: 12 of the 15 gaps are tests that exist but do not prove the fix. Examples: "passes on the pre-fix code", "matches the temp dir's random suffix", "can pass for unrelated startup failures". From the diff, JEV rates those tests as covering the change (p≈0.9–0.96). Reviewers found these gaps by running the test against the old code. Judging the diff alone cannot see that.
Not merged or installed until the owner decides the direction.