Skip to content

guardian(bench): re-measure recall, precision and judge drift on opencode-go / deepseek-v4.1-flash #778

Open

Description

@zaebee

Follow-up to #775 (finder moved to opencode-go / DeepSeek in #777). Bench and calibration numbers were explicitly out of scope there; the repository's recorded numbers still come from mistral/gemini/anthropic runs.

Why

A model swap changes the finder's recall and the judge's drift. #246 measured the two-stage design on a mistral finder, and #752 is about detecting judge drift between runs. The new default effort (GUARDIAN_OPENCODE_EFFORT=low) was chosen by analogy with the Anthropic default, not measured. Without this, a prompt or provider change has no baseline to regress against.

What to measure

  1. Finder recall / precision / noise on the curated corpus:
    GUARDIAN_PROVIDER=opencode-go GUARDIAN_MODEL=deepseek-v4.1-flash \
    GUARDIAN_SKEPTIC=off \
    uv run python scripts/guardian_bench.py --runs 3
    Per-PR and aggregate numbers; --runs 3 to see variance. Results append to benchmarks/guardian/results.jsonl — commit the new lines.
  2. Effort sweep: repeat with GUARDIAN_OPENCODE_EFFORT=low, high, max (the live catalog offers low|high|max for this model). Pick the default with the numbers behind it, and record cost per run from the reported token counts at $0.15/$0.60 per 1M in/out (peak hours double: 01:00–04:00 and 06:00–10:00 UTC, Mon–Fri).
  3. Judge calibration / drift:
    uv run python scripts/guardian_calibrate.py run --provider opencode-go --model deepseek-v4.1-flash
    uv run python scripts/guardian_calibrate.py report
    If guardian(bench): detect LLM-judge drift between runs with a frozen anchor set #752's frozen anchor set exists by then, re-judge it first and report flip count and direction; otherwise state explicitly that judge drift is unmeasured for this model. Note that the judge and the finder would then share a vendor — record that, since guardian: cross-model skeptic to cut recall-lean noise without killing recall #246's cross-vendor assumption was about the skeptic, not the judge.
  4. Baseline comparison: diff the new numbers against the last recorded mistral/gemini rows in benchmarks/guardian/results.jsonl and against the published experiment notes (guardian: cross-model skeptic to cut recall-lean noise without killing recall #246). Name regressions, not just totals.

Credentials

Bench runs need OPENCODE_API_KEY in the environment; GitHub OIDC is only available inside Actions. The current Custom OIDC credential allows event_name=issue_comment, so a workflow_dispatch bench job cannot exchange the token as-is. Options, pick one and say why:

  • add a second condition/credential in the opencode console for event_name=workflow_dispatch on zaebee/codegraph-brain, then a small dispatchable workflow mints the token, runs the bench and uploads results.jsonl/calibration.jsonl as artifacts (no static key);
  • use an opencode service-account key locally (console → Settings → Service accounts), passed through the environment, never committed.

Done when

  • a table per PR and aggregate (recall, precision, noise) for the default setting, with the exact command and environment recorded;
  • the effort default is chosen and justified by the sweep;
  • judge calibration/drift is recorded (or explicitly declared unmeasured, with the reason);
  • cost per review is stated from token counts;
  • benchmarks/guardian/results.jsonl (and calibration.jsonl if re-judged) is updated in a PR, and the numbers are pasted in this issue.

Not in scope

Prompt changes, the skeptic-vendor decision (#246), and bench-harness redesign unless the new model exposes a real gap. scripts/guardian_calibrate.py --provider help text still lists gemini | mistral | ollama; updating it is in scope only if the command works but the help is wrong.

Refs #775, #246, #752.

Activity

  1. added
    tailLoose end: something left to check or finish later
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tailLoose end: something left to check or finish later

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions