[Feat] Add a reproducible public231 decision benchmark

.github/workflows/test.yml

@@ -108,3 +108,33 @@ jobs:
       # The Chromium revision that playwright-core inside the runtime's pinned @playwright/mcp expects.
       - run: npx -y -p "$(uv run --no-sync python -c 'import s1a.entry; from s1a.config import DEFAULT_PLAYWRIGHT_MCP_ARGS as a; print(a.split()[-1])')" playwright-core install --with-deps chromium
       - run: S1A_BROWSER_TESTS=1 uv run pytest -q tests/system
+
+  benchmarks:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    env:
+      JEVBENCH_ROOT: ${{ github.workspace }}/_reference/jevbench
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/checkout@v7
+        with:
+          repository: fstandhartinger/jevbench
+          ref: 7ce310c7262ed49cc85853339a8a42459298e3f3
+          path: _reference/jevbench
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@v10.2.0
+        with:
+          python-version: "3.11"
+          enable-cache: true
+      - run: uv sync --extra dev
+      - name: Required benchmark tests
+        run: |
+          uv run pytest tests/benchmarks/test_public231.py -q --junitxml="${RUNNER_TEMP}/benchmark-tests.xml"
+          uv run python - <<'PY'
+          import os
+          import xml.etree.ElementTree as ET
+          from pathlib import Path
+
+          report = Path(os.environ["RUNNER_TEMP"]) / "benchmark-tests.xml"
+          raise SystemExit(bool(ET.parse(report).findall(".//skipped")))
+          PY

CHANGELOG.md

@@ -6,6 +6,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
 
 ### Added
 
+- Public231 decision-model benchmark with raw HTTP evidence, pinned upstream scoring,
+  complete-denominator quality metrics and per-round service latency.
+
 - Optional local Cua-S1 4B inference through the `cua-four-b` extra and `CUA_S1_VARIANT=4b`, supporting text and
   multimodal input. Desktop `--pixel-target` offers named screenshot points; clicks remain bound to the observed
   window and capture.

benchmarks/README.md

@@ -0,0 +1,83 @@
+# Decision model benchmarks
+
+Compare an existing decision service on JevBench's 231 public questions: easy48,
+original72 and hard111. The collector saves every request and response; a separate
+command re-scores them with the pinned upstream scorer.
+
+[H800 results](results/public231-h800.md) compare five open checkpoints. This is the
+public231 set, not the article's 248-question or 23-model leaderboard.
+
+## Run
+
+Use the repository's development environment and a running service. Clone
+[JevBench](https://github.com/fstandhartinger/jevbench) at
+`7ce310c7262ed49cc85853339a8a42459298e3f3`; the tool verifies the content hashes of
+all three datasets and every scorer module it imports. No dataset is vendored.
+
+```bash
+export JEVBENCH_ROOT=/absolute/path/to/jevbench
+python -m benchmarks.legs.leg1_public231 predict \
+  --endpoint http://127.0.0.1:8000 --path /v1/systemone \
+  --model open-jev-9b --thinking off --warmup 5 --rounds 3 \
+  --run-meta /absolute/path/to/environment.json \
+  --out /absolute/path/to/results/open-jev-9b
+python -m benchmarks.legs.leg1_public231 aggregate \
+  --run /absolute/path/to/results/open-jev-9b \
+  --out /absolute/path/to/results/open-jev-9b
+```
+
+Omni native Open-Jev uses `/v1/systemone`; Intern's official service uses
+`/v1/decisions`. `--model` is optional. The collector preserves state, questions and
+option order, sends `thinking.enabled=false`, and never sends targets or provenance.
+It makes one request at a time, without retries. It does not start services or choose GPUs.
+
+`environment.json` describes the **actual service**, without credentials:
+
+```json
+{
+  "checkpoint": {"repo": "owner/model", "revision": "full-checkpoint-commit"},
+  "backend": {"name": "serving-backend", "version": "full-source-commit"},
+  "sampling": {"temperature": 1.0},
+  "hardware": {"gpu": "GPU model"}
+}
+```
+
+Also record dtype, tokenizer/base/adapter/head revisions, loaded libraries, context
+limits, cache settings and job identity. The file is copied byte-for-byte and hashed;
+all original fields must match the run metadata. Omit `stop_reason` and conflicting
+collector fields. Temperature is recorded, not configured by the client. Authenticate
+outside this tool if needed; it does not accept or save API keys.
+
+## Reading the results
+
+- Each round's accuracy is **correct / 231**. Failures and unattempted questions stay
+  in the denominator. Three repeated rounds provide timing samples, not 693 unique questions.
+- Upstream schema validity, Brier and ECE are reported separately. Missing probabilities
+  are not replaced with one-hot predictions. Native `noul` scalars become `{no: 1-p, yes: p}`.
+- Per-round and pooled p50/p95 cover HTTP POST through full response receipt. All attempts,
+  successes and failures have separate counts; empty latency sets are null. Warmups are excluded.
+- `sufficient=true` requires five warmups and three complete 231-question rounds.
+  `--limit`, fewer rounds or missing attempts produce partial observations.
+- HTTP 401/403/429, interruption or three consecutive infrastructure failures stop collection.
+  HTTP 422 counts as a failed question and does not trigger that stop counter.
+
+Evidence goes in a fresh directory **outside the repository**. Each round contains
+`results.jsonl` and `raw/<sha256(task_id)>.json`; warmups are separate. Raw records retain
+the request, response text, decoded response, HTTP status, error, timestamp and latency.
+`run.meta.json` and `completion.json` record the plan and any stop reason.
+
+Aggregation checks source hashes, the environment, input order, raw hashes and record
+consistency before re-scoring. Orphan or missing raw files are errors. It never overwrites
+an existing summary; use a new output directory for another recomputation. These checks
+detect inconsistent evidence, not coordinated rewriting of an unsigned archive.
+
+## Tests
+
+```bash
+JEVBENCH_ROOT=/absolute/path/to/jevbench python -m pytest tests/benchmarks -q --confcutdir=tests/benchmarks
+ruff check benchmarks tests/benchmarks
+ruff format --check benchmarks tests/benchmarks
+```
+
+Reference-dependent tests skip if `JEVBENCH_ROOT` is absent. Mock tests
+check collection and scoring behavior; model quality and latency come from the linked real runs.

benchmarks/__init__.py

@@ -0,0 +1 @@
+"""Offline decision benchmark tools, separate from agent runtime."""

benchmarks/client.py

@@ -0,0 +1,68 @@
+"""One synchronous POST per decision request; no retries or model inference."""
+
+import json
+import math
+import time
+from urllib.parse import urlsplit
+
+import httpx
+
+
+def decode(text):
+    def invalid_constant(value):
+        raise ValueError(f"Non-JSON numeric constant: {value}")
+
+    def finite_float(value):
+        result = float(value)
+        if not math.isfinite(result):
+            raise ValueError(f"Non-finite JSON number: {value}")
+        return result
+
+    return json.loads(text, parse_constant=invalid_constant, parse_float=finite_float)
+
+
+class DecisionClient:
+    def __init__(self, endpoint, *, path, model=None, timeout_s=120, transport=None):
+        if not math.isfinite(timeout_s) or timeout_s <= 0:
+            raise ValueError("timeout_s must be finite and positive")
+        url = urlsplit(endpoint)
+        if (
+            url.scheme not in ("http", "https")
+            or not url.netloc
+            or url.username
+            or url.password
+            or url.query
+            or url.fragment
+        ):
+            raise ValueError("Use an HTTP endpoint without credentials or query parameters")
+        if not path.startswith("/") or "?" in path or "#" in path:
+            raise ValueError("path must be an absolute API path without query parameters")
+        self.url, self.model = endpoint.rstrip("/") + path, model
+        self.http = httpx.Client(timeout=timeout_s, transport=transport, follow_redirects=False, trust_env=False)
+
+    def request(self, body):
+        request = dict(body)
+        if self.model is not None:
+            request.setdefault("model", self.model)
+        request = decode(json.dumps(request, ensure_ascii=False, allow_nan=False))
+        raw = dict(request=request, response_body=None, response=None, http_status=None, client_error=None)
+        raw["ts"] = time.time()
+        started = time.perf_counter()
+        try:
+            response = self.http.post(self.url, json=request)
+            raw.update(http_status=response.status_code, response_body=response.text)
+        except httpx.HTTPError as exc:
+            raw["client_error"] = f"{type(exc).__name__}: {exc}"
+        except KeyboardInterrupt:
+            raw["client_error"] = "KeyboardInterrupt"
+        finally:
+            raw["latency_s"] = time.perf_counter() - started
+        if raw["response_body"] is not None:
+            try:
+                raw["response"] = decode(raw["response_body"])
+            except ValueError:
+                pass  # The original body remains available for offline diagnosis.
+        return raw
+
+    def close(self):
+        self.http.close()

benchmarks/evidence.py

@@ -0,0 +1,163 @@
+"""Pinned upstream scoring and reproducible public231 evidence."""
+
+import hashlib
+import importlib
+import json
+import os
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+
+from benchmarks.client import decode
+
+PIN = "7ce310c7262ed49cc85853339a8a42459298e3f3"
+HASHES = {
+    "jevbench/__init__.py": "64c28dc1534502c429a3b67eee79ab7909fab995b7d116c9b089f0f3bb8652f7",
+    "jevbench/tasks.py": "b9c2f8ba9a7301519656d79d47c31b8ab0542a5545498ce3b339d5e93b976812",
+    "jevbench/scoring.py": "6aa17d212ea20a3eeb6876149367842e8462c11dcb54d73df062fb406368331e",
+    "jevbench/summarize.py": "aa5f24da407d004cd80bad1cd957b6987aba152591084c096d1de51dd9775cbe",
+    "jevbench/metrics.py": "4038c8423ad1cf28601afb6babe777b71f08e379b9ca125b09b52c0ac7d60515",
+    "datasets/public/easy.jsonl": "231df3c2c8e88a1a8c137ebe85de96ba70fabd330849098ac7b3c52c70b7172b",
+    "datasets/public/original.jsonl": "5c2414edb3006b8bfcb70fda433f0f9ca015759433849f8d3104328a1f7c4180",
+    "datasets/public/hard.jsonl": "89e9e6becb33ed88c1de7d42dcc87531b2fb64cfaef4e1986faf7c37b3f80ebb",
+}
+
+
+def digest(data):
+    return hashlib.sha256(data).hexdigest()
+
+
+def dump(value):
+    return json.dumps(value, ensure_ascii=False, allow_nan=False)
+
+
+def write_json(path, value):
+    with Path(path).open("x", encoding="utf-8") as stream:
+        stream.write(dump(value) + "\n")
+
+
+def outside_repo(path):
+    path = Path(path).resolve()
+    if path.is_relative_to(Path(__file__).resolve().parents[1]):
+        raise ValueError("Evidence must be outside the source repository")
+    return path
+
+
+def load_reference():
+    root = Path(os.environ["JEVBENCH_ROOT"]).resolve()
+    for name, expected in HASHES.items():
+        if digest((root / name).read_bytes()) != expected:
+            raise ValueError(f"Pinned reference hash mismatch: {name}")
+    for name, module in tuple(sys.modules.items()):
+        if name == "jevbench" or name.startswith("jevbench."):
+            if not Path(module.__file__).resolve().is_relative_to(root):
+                raise ValueError("Another JevBench root is already imported")
+    sys.path.insert(0, str(root))
+    upstream = SimpleNamespace(
+        **{name: importlib.import_module(f"jevbench.{name}") for name in ("tasks", "scoring", "summarize", "metrics")}
+    )
+    items = []
+    for tier in ("easy", "original", "hard"):
+        for line, text in enumerate(
+            (root / f"datasets/public/{tier}.jsonl").read_text(encoding="utf-8").split("\n")[:-1], 1
+        ):
+            task = upstream.tasks.Task.from_dict(decode(text))
+            identity = dict(
+                suite=f"jevbench-{tier}",
+                task_id=task.id,
+                source_file=f"{tier}.jsonl",
+                source_line=line,
+                family=task.family,
+                tier=tier,
+                split=task.split,
+            )
+            items.append((identity, task))
+    dataset = dict(
+        jevbench_pin=PIN,
+        root=str(root),
+        hashes=HASHES,
+        dataset_hash=upstream.tasks.dataset_hash([task for _, task in items]),
+    )
+    return upstream, items, dataset
+
+
+def body_for(task, model):
+    body = {"state": task.state, "questions": {"decision": task.question}, "thinking": {"enabled": False}}
+    if model is not None:
+        body["model"] = model
+    return body
+
+
+def score_record(upstream, identity, task, raw, model, round_number, kind, raw_hash):
+    status, error, probs, source = "ok", None, None, "unavailable"
+    response = raw["response"]
+    code = raw["http_status"]
+    if raw["client_error"] is not None:
+        status, error = "client_error", raw["client_error"]
+    elif code is None or not 200 <= code < 300:
+        status, error = ("refused" if code == 422 else "http_error"), raw["response_body"]
+    else:
+        answers = response.get("answers") if isinstance(response, dict) else None
+        answer = answers.get("decision") if isinstance(answers, dict) else None
+        if not isinstance(answer, dict) or answer.get("type") != task.question["type"]:
+            status, error = "parse_error", "Missing or mismatched typed decision answer"
+        else:
+            probs = answer.get("probabilities")
+            source = "response_probabilities" if probs is not None else "unavailable"
+            if probs is None and task.question["type"] == "noul":
+                value = answer.get("noul")
+                if isinstance(value, (float, int)) and not isinstance(value, bool):
+                    probs, source = {"no": 1 - value, "yes": value}, "native_noul_scalar"
+    try:
+        scored = upstream.scoring.score_task(probs, task)
+    except OverflowError:
+        scored = upstream.scoring.score_task(None, task)
+        scored["error"] = "Probability exceeds the scorer's numeric range"
+    return dict(
+        identity,
+        round=round_number,
+        kind=kind,
+        ts=raw["ts"],
+        attempted=True,
+        ok=status == "ok",
+        status=status,
+        status_code=code,
+        error=error or scored.get("error"),
+        latency_s=raw["latency_s"],
+        latency_basis="client_single_request_wall_seconds",
+        **{key: scored.get(key) for key in ("valid", "strict_valid", "renormalized", "correct", "predicted", "probs")},
+        expected=task.expected,
+        candidate_count=len(task.labels),
+        probs_as_returned=probs,
+        probs_source=source,
+        score_source="jevbench_7ce310c7",
+        model_configured=model,
+        model=response.get("model") if isinstance(response, dict) else None,
+        usage=response.get("usage") if isinstance(response, dict) else None,
+        raw_sha256=raw_hash,
+    )
+
+
+def summarize(upstream, tasks, records):
+    # Upstream accuracy is attempted-only; keep it explicitly diagnostic.
+    projections = [dict(row, model=str(row["model"]) if row["model"] is not None else "<missing>") for row in records]
+    original = upstream.summarize.summarize(tasks, projections)
+    correct = sum(bool(row["correct"]) for row in records)
+    latency = {
+        name: upstream.metrics.latency_summary([row["latency_s"] for row in rows])
+        for name, rows in (
+            ("all_attempts", records),
+            ("success", [r for r in records if r["ok"]]),
+            ("failure", [r for r in records if not r["ok"]]),
+        )
+    }
+    return dict(
+        planned=len(tasks),
+        attempted=len(records),
+        valid=sum(r["valid"] for r in records),
+        correct=correct,
+        accuracy=correct / len(tasks),
+        coverage=len(records) / len(tasks),
+        latency=latency,
+        upstream_attempted_metrics=original,
+    )

benchmarks/legs/__init__.py

@@ -0,0 +1 @@
+"""Benchmark collection and aggregation command lines."""