Repository navigation
Description
Follow-up to #775 (finder moved to opencode-go / DeepSeek in #777). Bench and calibration numbers were explicitly out of scope there; the repository's recorded numbers still come from mistral/gemini/anthropic runs.
Why
A model swap changes the finder's recall and the judge's drift. #246 measured the two-stage design on a mistral finder, and #752 is about detecting judge drift between runs. The new default effort (GUARDIAN_OPENCODE_EFFORT=low) was chosen by analogy with the Anthropic default, not measured. Without this, a prompt or provider change has no baseline to regress against.
What to measure
- Finder recall / precision / noise on the curated corpus:
Per-PR and aggregate numbers;
GUARDIAN_PROVIDER=opencode-go GUARDIAN_MODEL=deepseek-v4.1-flash \ GUARDIAN_SKEPTIC=off \ uv run python scripts/guardian_bench.py --runs 3
--runs 3to see variance. Results append tobenchmarks/guardian/results.jsonl— commit the new lines. - Effort sweep: repeat with
GUARDIAN_OPENCODE_EFFORT=low,high,max(the live catalog offerslow|high|maxfor this model). Pick the default with the numbers behind it, and record cost per run from the reported token counts at $0.15/$0.60 per 1M in/out (peak hours double: 01:00–04:00 and 06:00–10:00 UTC, Mon–Fri). - Judge calibration / drift:
If guardian(bench): detect LLM-judge drift between runs with a frozen anchor set #752's frozen anchor set exists by then, re-judge it first and report flip count and direction; otherwise state explicitly that judge drift is unmeasured for this model. Note that the judge and the finder would then share a vendor — record that, since guardian: cross-model skeptic to cut recall-lean noise without killing recall #246's cross-vendor assumption was about the skeptic, not the judge.
uv run python scripts/guardian_calibrate.py run --provider opencode-go --model deepseek-v4.1-flash uv run python scripts/guardian_calibrate.py report
- Baseline comparison: diff the new numbers against the last recorded mistral/gemini rows in
benchmarks/guardian/results.jsonland against the published experiment notes (guardian: cross-model skeptic to cut recall-lean noise without killing recall #246). Name regressions, not just totals.
Credentials
Bench runs need OPENCODE_API_KEY in the environment; GitHub OIDC is only available inside Actions. The current Custom OIDC credential allows event_name=issue_comment, so a workflow_dispatch bench job cannot exchange the token as-is. Options, pick one and say why:
- add a second condition/credential in the opencode console for
event_name=workflow_dispatchonzaebee/codegraph-brain, then a small dispatchable workflow mints the token, runs the bench and uploadsresults.jsonl/calibration.jsonlas artifacts (no static key); - use an opencode service-account key locally (console → Settings → Service accounts), passed through the environment, never committed.
Done when
- a table per PR and aggregate (recall, precision, noise) for the default setting, with the exact command and environment recorded;
- the effort default is chosen and justified by the sweep;
- judge calibration/drift is recorded (or explicitly declared unmeasured, with the reason);
- cost per review is stated from token counts;
benchmarks/guardian/results.jsonl(andcalibration.jsonlif re-judged) is updated in a PR, and the numbers are pasted in this issue.
Not in scope
Prompt changes, the skeptic-vendor decision (#246), and bench-harness redesign unless the new model exposes a real gap. scripts/guardian_calibrate.py --provider help text still lists gemini | mistral | ollama; updating it is in scope only if the command works but the help is wrong.
Activity