[Feat] Add a reproducible public231 decision benchmark

[linear3735]

Required human contributor answers

1. What outcome are you trying to achieve?

   Answer: Make accuracy and HTTP latency reproducible on the fixed 231-question JevBench workload.

2. What is the highest-leverage variable for that outcome?

   Answer: Completeness and replayability of the saved benchmark evidence.

3. How will this work move that variable rather than create busywork?

   Answer: Save HTTP evidence, pin the scorer, recompute summaries offline, and reject skipped benchmark tests in CI.

4. Who is the customer or user, and what do they actually need?

   Answer: Benchmark contributors and reviewers need recomputable results with clear failure counts and timing boundaries.

After answering the four questions: is a PR needed at all to achieve this outcome?

Yes. A shared, versioned tool and CI checks let future contributions reuse this workflow.

For a bug fix: link actual observed failure evidence.

Evidence: N/A — this is new benchmark infrastructure, not a bug fix; no existing runtime behavior changes (adds benchmarks/, its tests, docs and one CI job only).

Why

Compare decision services on the same 231 public JevBench questions, reporting accuracy and HTTP request latency together.

How

predict sends serial requests to an existing decision service and retains the raw HTTP evidence. aggregate validates that archive and uses the pinned JevBench scorer to recompute the results; neither command starts a service or allocates GPUs.

What

• Add benchmarks/ with predict to collect requests/responses and aggregate to validate and re-score them using the pinned JevBench reference. Failed and missing requests remain in the 231-question accuracy denominator.
• Report accuracy, schema validity, Brier, ECE and per-round/pooled p50/p95; document five measured H800 configurations.
• Add a CI benchmark job that fails on skipped tests.

Collection, replay and test commands

Verification

• Current-head CPU protocol/scorer check on 2026-10-13: 82 passed, zero skipped with the pinned JevBench reference. No model inference was performed.

• Historical CPU replay at 5c59dc4: the five retained archives reproduced counts exactly and numerical metrics within 1e-12; this is separate from the current-head test run.

• CI for 5c59dc42: benchmarks, core Python 3.11/3.13 and full passed; Windows/browser jobs were skipped by the PR workflow conditions.

• Head caea1147: merges upstream/main (docs-only gates, PRs [Docs] Review skills: merge-readiness gates and a readability focus #70/[Docs] Gate contributions on explicit human intent #71) and reapplies the benchmark-test utf-8 encoding fix dropped in the branch rebuild. The benchmark command code, README and results document are byte-identical to the recorded replay head 5c59dc4; the only benchmark diff at this head is the test-file encoding fix. Previously recorded local software checks on this head: ruff format --check, ruff check, ty check passed; pytest -q 979 passed, 117 skipped (fixture-dependent), 198 subtests passed; scripts/smoke.sh: ok. CI for caea1147: benchmarks, core Python 3.11/3.13 and full passed; Windows/browser jobs skipped by the PR workflow conditions.

• uv run ruff format --check . && uv run ruff check . && uv run ty check

• uv run pytest -q and scripts/smoke.sh

• CHANGELOG.md and the docs say what the code does now

Demo / evidence

Job 415076: one H800, serial requests, five excluded warmups and three rounds per model. Accuracy uses 231 unique questions per round; latency pools 693 measured HTTP requests. All five configurations completed 693/693 requests with 100% schema validity. Brier/ECE use all 693 valid distributions; the rounds are repeats of 231 unique tasks. Latency covers HTTP POST through full body receipt, excludes warmups, and uses interpolated percentiles.

Model	Service	Correct / 231	Accuracy	Validity	Brier	ECE	HTTP p50 / p95 ms
Intern-Decision-0.8B	official HF / T=1	164/231	70.9957%	100%	0.369906	0.111077	80.493 / 175.237
Intern-Decision-2B	official HF / T=1	181/231	78.3550%	100%	0.306049	0.115420	79.054 / 175.276
Intern-Decision-4B	official HF / T=1	200/231	86.5801%	100%	0.189601	0.067214	104.552 / 235.291
Open-Jev-9B	Omni native Rust/CUDA	179/231	77.4892%	100%	0.320752	0.086786	42.688 / 327.057
Open-Jev-27B-v1.1	Omni native Rust/CUDA	198/231	85.7143%	100%	0.240493	0.130333	68.877 / 1068.817

Intern uses its official HF service; Open-Jev uses Omni native Rust/CUDA. Backends and temperatures differ, so these are deployed-configuration comparisons, not isolated kernel speedups. These are decision-quality and request-latency measurements on a fixed static question set — not application-task success rates; application completion evidence is tracked separately in the roadmap (#54). Full results, per-round latency, checkpoints and settings

Recorded replay + downloadable outputs

public231-terminal-replay.mp4

The 39-second recorded terminal replay shows the actual aggregate CLI re-scoring all five saved H800 archives on CPU on 2026-10-11 at 5c59dc4, including the command lines, service/model identities and resulting aggregate summaries; reading pauses are added. It performs no new model inference; latency values come from the original HTTP requests. The video content matches the hash recorded in the download package and was inspected in the scoped review below. After this description update on 2026-10-13, one-byte GET requests returned 206 for both the video and ZIP, with and without GitHub authentication. The original MP4 is preserved (SHA-256 70287ba942385596b0aea95b5b4fd3e23110af1c7cdcfc27caf9aa5932b5f9ec).

Download benchmark outputs and provenance (ZIP). The focused package includes the five summaries, redacted environment/run metadata, terminal output, replay receipt and checksums; its 20 SHA256SUMS entries were re-verified against the packaged bytes on 2026-10-13; source provenance was checked separately against head caea1147, and the model JSONs/receipt/terminal files are byte-identical to the original package inspected by the review below. Questions, raw responses and per-question predictions are omitted; this is not a complete archive for independent re-scoring. Obtain JevBench inputs from upstream subject to its terms.

Birfy's scoped follow-up at 5c59dc4 inspected frames showing the five commands/results and the original, larger output package and verified all 25 of its checksums, the five summary hashes and the receipt/source-file hashes against the head tree; it addressed the original missing-recording/output finding as evidence of an author-recorded CPU replay. That review did not independently rerun aggregation and did not grant overall approval; questions/raw responses remain omitted by design.

Maintainer decisions requested

1. For this collection/scoring-only change, accept the benchmark CLI evidence under the video policy or specify the application demo required. No exception has been accepted yet.
2. Confirm whether the scoped recording and summary package are sufficient for review, or identify the additional replay material needed. The package omits the raw archive; any further disclosure must respect its rights and publication limits.

[Birfy]

Review of current head 40bf742e2b84ad6157bf2990dd5e4bab73571702 against 9790b52ef4d78a2215e8dd5a526d3a8daea9875a.

I did not find a blocking code defect in the collector or evidence replay during this pass. The collector preserves per-request raw bodies, replays scoring from the pinned JevBench source, and keeps failed/missing requests in the fixed 231-task denominator. CI is green (core 3.11, 3.13, Windows 3.11 and full; browser job skipped).

P2 — Missing required demo recording/evidence attachment. This adds a new benchmark workflow and the PR body explicitly says “Video and downloadable output evidence are not attached yet.” The report table and a description of a three-ticket replay do not let reviewers inspect the actual benchmark invocation/result. Please attach a short screen/terminal recording of the benchmark run or replay, plus the corresponding downloadable evidence/output referenced by the PR. Keep the full 231-question measurements distinct from the three-ticket illustrative replay, and show the command, service/model identity and resulting aggregate summary. I consider this an evidence blocker under the project’s requirement that new features include a recording.

Disposition: code review found no blocker; please add the recording/evidence before merge.

[Birfy]

Scoped follow-up to my October 8 missing-recording/output finding, at head 5c59dc4229c47d2deea31bd3fdb70ce844eaa3d5.

I downloaded the public231 MP4 and public231-output.zip, inspected frames covering all five commands/results and the separate historical ticket replay, and checked the package rather than only its links. All 25 entries in SHA256SUMS match; the public231 video matches its recorded hash; all five summary hashes match the receipt; the receipt's ten source-file hashes and candidate tree match this head. The summaries retain three rounds of 231 questions, 693 attempted/valid requests per model, and the displayed per-round correct counts 164/181/200/179/198. The benchmark implementation/tests are unchanged from my previous code-review head; the current changes add pinned-reference CI and clarify the README. Exact-head CI succeeds for benchmarks, Linux 3.11/3.13 and full; browser and Windows are skipped.

This addresses the specific absence of a benchmark recording and downloadable output in my previous review. The new materials provide inspectable command/model/result evidence for an author-recorded CPU replay of saved H800 responses. I did not independently re-run that aggregation: raw requests/responses and per-question predictions are omitted, and checksums establish internal consistency, not independent authenticity or the reported comparison against omitted original summaries. No restricted dataset redistribution is requested.

The separately labelled three-ticket recording/output remains historical Agents 9790b52 + Omni 4a79980, not validation of current-main application integration. Neither clip is new GPU inference or a fresh latency measurement. This scoped update does not grant overall approval, satisfy a separate PR's evidence requirements, or replace maintainer approval.