[Feat] Re-score saved public231 decisions

[linear3735]

Required human contributor answers

1. What outcome are you trying to achieve?

   Answer: Saved public231 decision responses get reproducible accuracy scores: a fixed workload, a fixed collector version and retained response evidence determine every published number, failed and unprocessed questions stay in the 231-question denominator, and response validity, accuracy and calibration are reported separately.

2. What is the highest-leverage variable for that outcome?

   Answer: Replayability of the scoring evidence — given the same saved input and the same consumer/scorer versions, anyone recomputes the identical published numbers, and the planned 231-question denominator never silently shrinks. The absolute leaderboard values are secondary; without auditable replay they carry no information.

3. How will this work move that variable rather than create busywork?

   Answer: It ships a versioned offline consumer (quality_public231.py) that validates the saved workload, collector version and response evidence before calling the pinned JevBench scorer while preserving original probabilities; a required benchmark CI job that rebuilds the input fixture from the fixed public reference and original collector and fails on any skip; and an accuracy leaderboard with fixed versions, counts and update rules that blocks quiet data or methodology swaps. Scope stays split: request latency remains in Omni #131 and the historical ticket auditor stays outside this PR, so the evidence that exists stays trustworthy instead of being diluted.

4. Who is the customer or user, and what do they actually need?

   Answer: Maintainers and reviewers judging claimed accuracy scores need a redistributable, CI-enforced replay path rather than screenshots of past runs. People comparing decision-model quality (including us) need a stable 231-question accuracy and calibration reference, not latency or completion rates. They need one command that reconstructs the numbers — not more charts.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale: Yes. Only versioned scoring code in the repository plus a required fail-on-skip CI job make replay enforceable; external scripts or documentation cannot.

For a bug fix: link actual observed failure evidence.

Evidence: N/A — this is a feature on saved-response scoring, not a bug fix; no live runtime path changes (s1a, the agent loop, inference backends and serving code are untouched).

Why

Closes #68. Saved decision responses need reproducible accuracy scores that keep
failed and unprocessed questions in the denominator.

This PR depends on #52.
Merge #52 first, then update this branch and reconcile the shared CI and
CHANGELOG changes. The current diff against main also includes the #52 collector.

How

Check the saved workload, collector version and response evidence before calling
the pinned JevBench scorer. Preserve original probabilities and all 231 planned
questions. Report response validity, accuracy and calibration separately.

The required benchmark CI job fetches the fixed public reference and collector,
builds the input fixture and fails if any benchmark test skips.

What

• Add the offline public231 consumer and root-level tests.
• Add an accuracy leaderboard with fixed versions, counts and update rules.
• Keep request latency in Omni #131.

Review quality_public231.py, its tests and the CI job in that order. This change
does not alter s1a, the agent loop, inference backends or serving code.
The historical ticket auditor is outside this PR; its original source and evidence
are retained separately. This tool does not validate ticket episode completion.

Verification

GitHub CI for 66636af
passed on clean Linux runners: core Python 3.11/3.13, full and benchmarks.
The benchmark job passed with zero skips; its XML skip guard passed.
Ruff format/check, ty, build and smoke checks passed locally on this head
(pytest -q: 980 passed, 125 fixture-dependent skips, 198 subtests; bash scripts/smoke.sh: ok).
Windows and browser jobs were skipped by the PR workflow conditions.
Independent Kimi and Codex source reviews found no remaining blocking issues.
Birfy's scoped reviews covered the consumer slice and its #52 integration
(through 3400a601); subsequent head commits are upstream/main merges
(including the docs-gate PRs #70/#71) and the evidence-scope note 9df1322.

uv run ruff format --check .
uv run ruff check .
uv run ty check
JEVBENCH_ROOT=<pinned-checkout> OMNI_BENCH_SOURCE=<original-collector> uv run pytest tests/benchmarks -q
uv run pytest -q
bash scripts/smoke.sh

The saved text campaign gives 179/231 correct for Open-Jev-9B and 198/231 for
Open-Jev-27B-v1.1 in each round. Three repeats remain 231 unique questions.
These historical observations are not new model runs or kernel speedup claims.
Image/VL, screenshots, browser tasks and higher concurrency are outside this scope.

Demo / evidence

• Five-model report

• Pinned JevBench reference

• Public five-model summary bundle from #52: campaign 415076 aggregate summaries, run metadata and collector replay receipt. Per-question predictions, probabilities and raw responses are omitted; this is not a complete archive for independent re-scoring.

• That bundle covers the [Feat] Add a reproducible public231 decision benchmark #52 collector, not this PR's quality_public231.py consumer or the separate text campaign 416135. This PR's numeric replay attachment and a qualifying demo remain unpublished; [Feat] Add a reproducible public231 decision benchmark #52's review does not close those evidence gaps. The historical ticket replay does not validate this consumer.

• uv run ruff format --check . && uv run ruff check . && uv run ty check

• uv run pytest -q and scripts/smoke.sh

• CHANGELOG.md and the docs say what the code does now

[Birfy]

Scoped independent review of 7421849, including the new consumer slice and its #52 integration. No confirmed routing/scoring correctness defect from this pass; ticket verification still has the reproducibility gap below, so this is not overall approval.

Validation on Linux/Python 3.12:

• Exact source tree matched GitHub tree cae572d6396e7836e2d9d09f5699018293d7d701.
• Default suite: 757 passed, 163 skipped, 144 subtests passed.
• With all eight hash-pinned JevBench files, the exact companion #131 collector, and a locally reconstructed public231 input matching the published SHA-256: benchmark suite 91 passed, 40 skipped. All 9 new public231 consumer tests passed. These are synthetic saved-response checks, not revalidation of the reported model-quality numbers.
• Repository-wide Ruff check/format and configured ty check passed; smoke passed without live model calls.

All remaining 40 benchmark skips are ticket tests. The original ticket artifacts remain unavailable pending the stated redistribution permission; this review does not ask for restricted data to be published. Please make a redistributable synthetic fixture/contract available so the committed negative regressions execute in a clean contributor/CI checkout. The existing #52 dependency, pending replay evidence and exact-head CI gaps remain; keep Draft.