From 76ea033919c07536867879539354dbd4c8be24b4 Mon Sep 17 00:00:00 2001 From: linear3735 Date: Thu, 8 Oct 2026 10:55:52 +0800 Subject: [PATCH 1/5] [Feat] Add reproducible public231 decision benchmarks --- CHANGELOG.md | 3 + benchmarks/README.md | 83 +++++ benchmarks/__init__.py | 1 + benchmarks/client.py | 68 ++++ benchmarks/evidence.py | 161 +++++++++ benchmarks/legs/__init__.py | 1 + benchmarks/legs/leg1_public231.py | 248 +++++++++++++ benchmarks/results/public231-h800.md | 92 +++++ tests/benchmarks/test_public231.py | 520 +++++++++++++++++++++++++++ 9 files changed, 1177 insertions(+) create mode 100644 benchmarks/README.md create mode 100644 benchmarks/__init__.py create mode 100644 benchmarks/client.py create mode 100644 benchmarks/evidence.py create mode 100644 benchmarks/legs/__init__.py create mode 100644 benchmarks/legs/leg1_public231.py create mode 100644 benchmarks/results/public231-h800.md create mode 100644 tests/benchmarks/test_public231.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 4297334..ef3966b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ### Added +- Public231 decision-model benchmark with raw HTTP evidence, pinned upstream scoring, + complete-denominator quality metrics and per-round service latency. + - Snake recipe and game client: `evals/snake` vendors the laya-mlx snake CLI (Apache-2.0) with single-game paced recording and a 16-game multigrid mode against any `/v1/systemone` backend; `recipes/snake` documents setup, diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..19f462e --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,83 @@ +# Decision model benchmarks + +Compare an existing decision service on JevBench's 231 public questions: easy48, +original72 and hard111. The collector saves every request and response; a separate +command re-scores them with the pinned upstream scorer. + +[H800 results](results/public231-h800.md) compare five open checkpoints. This is the +public231 set, not the article's 248-question or 23-model leaderboard. + +## Run + +Use the repository's development environment and a running service. Clone +[JevBench](https://github.com/fstandhartinger/jevbench) at +`7ce310c7262ed49cc85853339a8a42459298e3f3`; the tool verifies the content hashes of +all three datasets and every scorer module it imports. No dataset is vendored. + +```bash +export JEVBENCH_ROOT=/absolute/path/to/jevbench +python -m benchmarks.legs.leg1_public231 predict \ + --endpoint http://127.0.0.1:8000 --path /v1/systemone \ + --model open-jev-9b --thinking off --warmup 5 --rounds 3 \ + --run-meta /absolute/path/to/environment.json \ + --out /absolute/path/to/results/open-jev-9b +python -m benchmarks.legs.leg1_public231 aggregate \ + --run /absolute/path/to/results/open-jev-9b \ + --out /absolute/path/to/results/open-jev-9b +``` + +Omni native Open-Jev uses `/v1/systemone`; Intern's official service uses +`/v1/decisions`. `--model` is optional. The collector preserves state, questions and +option order, sends `thinking.enabled=false`, and never sends targets or provenance. +It makes one request at a time, without retries. It does not start services or choose GPUs. + +`environment.json` describes the **actual service**, without credentials: + +```json +{ + "checkpoint": {"repo": "owner/model", "revision": "full-checkpoint-commit"}, + "backend": {"name": "serving-backend", "version": "full-source-commit"}, + "sampling": {"temperature": 1.0}, + "hardware": {"gpu": "GPU model"} +} +``` + +Also record dtype, tokenizer/base/adapter/head revisions, loaded libraries, context +limits, cache settings and job identity. The file is copied byte-for-byte and hashed; +all original fields must match the run metadata. Omit `stop_reason` and conflicting +collector fields. Temperature is recorded, not configured by the client. Authenticate +outside this tool if needed; it does not accept or save API keys. + +## Reading the results + +- Each round's accuracy is **correct / 231**. Failures and unattempted questions stay + in the denominator. Three repeated rounds provide timing samples, not 693 unique questions. +- Upstream schema validity, Brier and ECE are reported separately. Missing probabilities + are not replaced with one-hot predictions. Native `noul` scalars become `{no: 1-p, yes: p}`. +- Per-round and pooled p50/p95 cover HTTP POST through full response receipt. All attempts, + successes and failures have separate counts; empty latency sets are null. Warmups are excluded. +- `sufficient=true` requires five warmups and three complete 231-question rounds. + `--limit`, fewer rounds or missing attempts produce partial observations. +- HTTP 401/403/429, interruption or three consecutive infrastructure failures stop collection. + HTTP 422 counts as a failed question and does not trigger that stop counter. + +Evidence goes in a fresh directory **outside the repository**. Each round contains +`results.jsonl` and `raw/.json`; warmups are separate. Raw records retain +the request, response text, decoded response, HTTP status, error, timestamp and latency. +`run.meta.json` and `completion.json` record the plan and any stop reason. + +Aggregation checks source hashes, the environment, input order, raw hashes and record +consistency before re-scoring. Orphan or missing raw files are errors. It never overwrites +an existing summary; use a new output directory for another recomputation. These checks +detect inconsistent evidence, not coordinated rewriting of an unsigned archive. + +## Tests + +```bash +JEVBENCH_ROOT=/absolute/path/to/jevbench python -m pytest tests/benchmarks -q --confcutdir=tests/benchmarks +ruff check benchmarks tests/benchmarks +ruff format --check benchmarks tests/benchmarks +``` + +Reference-dependent tests skip if `JEVBENCH_ROOT` is absent. Mock and loopback tests +check collection and scoring behavior; model quality and latency come from the linked real runs. diff --git a/benchmarks/__init__.py b/benchmarks/__init__.py new file mode 100644 index 0000000..391ce6c --- /dev/null +++ b/benchmarks/__init__.py @@ -0,0 +1 @@ +"""Offline decision benchmark tools, separate from agent runtime.""" diff --git a/benchmarks/client.py b/benchmarks/client.py new file mode 100644 index 0000000..38d6f00 --- /dev/null +++ b/benchmarks/client.py @@ -0,0 +1,68 @@ +"""One synchronous POST per decision request; no retries or model inference.""" + +import json +import math +import time +from urllib.parse import urlsplit + +import httpx + + +def decode(text): + def invalid_constant(value): + raise ValueError(f"Non-JSON numeric constant: {value}") + + def finite_float(value): + result = float(value) + if not math.isfinite(result): + raise ValueError(f"Non-finite JSON number: {value}") + return result + + return json.loads(text, parse_constant=invalid_constant, parse_float=finite_float) + + +class DecisionClient: + def __init__(self, endpoint, *, path, model=None, timeout_s=120, transport=None): + if not math.isfinite(timeout_s) or timeout_s <= 0: + raise ValueError("timeout_s must be finite and positive") + url = urlsplit(endpoint) + if ( + url.scheme not in ("http", "https") + or not url.netloc + or url.username + or url.password + or url.query + or url.fragment + ): + raise ValueError("Use an HTTP endpoint without credentials or query parameters") + if not path.startswith("/") or "?" in path or "#" in path: + raise ValueError("path must be an absolute API path without query parameters") + self.url, self.model = endpoint.rstrip("/") + path, model + self.http = httpx.Client(timeout=timeout_s, transport=transport, follow_redirects=False, trust_env=False) + + def request(self, body): + request = dict(body) + if self.model is not None: + request.setdefault("model", self.model) + request = decode(json.dumps(request, ensure_ascii=False, allow_nan=False)) + raw = dict(request=request, response_body=None, response=None, http_status=None, client_error=None) + raw["ts"] = time.time() + started = time.perf_counter() + try: + response = self.http.post(self.url, json=request) + raw.update(http_status=response.status_code, response_body=response.text) + except httpx.HTTPError as exc: + raw["client_error"] = f"{type(exc).__name__}: {exc}" + except KeyboardInterrupt: + raw["client_error"] = "KeyboardInterrupt" + finally: + raw["latency_s"] = time.perf_counter() - started + if raw["response_body"] is not None: + try: + raw["response"] = decode(raw["response_body"]) + except ValueError: + pass # The original body remains available for offline diagnosis. + return raw + + def close(self): + self.http.close() diff --git a/benchmarks/evidence.py b/benchmarks/evidence.py new file mode 100644 index 0000000..3940380 --- /dev/null +++ b/benchmarks/evidence.py @@ -0,0 +1,161 @@ +"""Pinned upstream scoring and reproducible public231 evidence.""" + +import hashlib +import importlib +import json +import os +import sys +from pathlib import Path +from types import SimpleNamespace + +from benchmarks.client import decode + +PIN = "7ce310c7262ed49cc85853339a8a42459298e3f3" +HASHES = { + "jevbench/__init__.py": "64c28dc1534502c429a3b67eee79ab7909fab995b7d116c9b089f0f3bb8652f7", + "jevbench/tasks.py": "b9c2f8ba9a7301519656d79d47c31b8ab0542a5545498ce3b339d5e93b976812", + "jevbench/scoring.py": "6aa17d212ea20a3eeb6876149367842e8462c11dcb54d73df062fb406368331e", + "jevbench/summarize.py": "aa5f24da407d004cd80bad1cd957b6987aba152591084c096d1de51dd9775cbe", + "jevbench/metrics.py": "4038c8423ad1cf28601afb6babe777b71f08e379b9ca125b09b52c0ac7d60515", + "datasets/public/easy.jsonl": "231df3c2c8e88a1a8c137ebe85de96ba70fabd330849098ac7b3c52c70b7172b", + "datasets/public/original.jsonl": "5c2414edb3006b8bfcb70fda433f0f9ca015759433849f8d3104328a1f7c4180", + "datasets/public/hard.jsonl": "89e9e6becb33ed88c1de7d42dcc87531b2fb64cfaef4e1986faf7c37b3f80ebb", +} + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def dump(value): + return json.dumps(value, ensure_ascii=False, allow_nan=False) + + +def write_json(path, value): + with Path(path).open("x", encoding="utf-8") as stream: + stream.write(dump(value) + "\n") + + +def outside_repo(path): + path = Path(path).resolve() + if path.is_relative_to(Path(__file__).resolve().parents[1]): + raise ValueError("Evidence must be outside the source repository") + return path + + +def load_reference(): + root = Path(os.environ["JEVBENCH_ROOT"]).resolve() + for name, expected in HASHES.items(): + if digest((root / name).read_bytes()) != expected: + raise ValueError(f"Pinned reference hash mismatch: {name}") + for name, module in tuple(sys.modules.items()): + if name == "jevbench" or name.startswith("jevbench."): + if not Path(module.__file__).resolve().is_relative_to(root): + raise ValueError("Another JevBench root is already imported") + sys.path.insert(0, str(root)) + upstream = SimpleNamespace( + **{name: importlib.import_module(f"jevbench.{name}") for name in ("tasks", "scoring", "summarize", "metrics")} + ) + items = [] + for tier in ("easy", "original", "hard"): + for line, text in enumerate((root / f"datasets/public/{tier}.jsonl").read_text().split("\n")[:-1], 1): + task = upstream.tasks.Task.from_dict(decode(text)) + identity = dict( + suite=f"jevbench-{tier}", + task_id=task.id, + source_file=f"{tier}.jsonl", + source_line=line, + family=task.family, + tier=tier, + split=task.split, + ) + items.append((identity, task)) + dataset = dict( + jevbench_pin=PIN, + root=str(root), + hashes=HASHES, + dataset_hash=upstream.tasks.dataset_hash([task for _, task in items]), + ) + return upstream, items, dataset + + +def body_for(task, model): + body = {"state": task.state, "questions": {"decision": task.question}, "thinking": {"enabled": False}} + if model is not None: + body["model"] = model + return body + + +def score_record(upstream, identity, task, raw, model, round_number, kind, raw_hash): + status, error, probs, source = "ok", None, None, "unavailable" + response = raw["response"] + code = raw["http_status"] + if raw["client_error"] is not None: + status, error = "client_error", raw["client_error"] + elif code is None or not 200 <= code < 300: + status, error = ("refused" if code == 422 else "http_error"), raw["response_body"] + else: + answers = response.get("answers") if isinstance(response, dict) else None + answer = answers.get("decision") if isinstance(answers, dict) else None + if not isinstance(answer, dict) or answer.get("type") != task.question["type"]: + status, error = "parse_error", "Missing or mismatched typed decision answer" + else: + probs = answer.get("probabilities") + source = "response_probabilities" if probs is not None else "unavailable" + if probs is None and task.question["type"] == "noul": + value = answer.get("noul") + if isinstance(value, (float, int)) and not isinstance(value, bool): + probs, source = {"no": 1 - value, "yes": value}, "native_noul_scalar" + try: + scored = upstream.scoring.score_task(probs, task) + except OverflowError: + scored = upstream.scoring.score_task(None, task) + scored["error"] = "Probability exceeds the scorer's numeric range" + return dict( + identity, + round=round_number, + kind=kind, + ts=raw["ts"], + attempted=True, + ok=status == "ok", + status=status, + status_code=code, + error=error or scored.get("error"), + latency_s=raw["latency_s"], + latency_basis="client_single_request_wall_seconds", + **{key: scored.get(key) for key in ("valid", "strict_valid", "renormalized", "correct", "predicted", "probs")}, + expected=task.expected, + candidate_count=len(task.labels), + probs_as_returned=probs, + probs_source=source, + score_source="jevbench_7ce310c7", + model_configured=model, + model=response.get("model") if isinstance(response, dict) else None, + usage=response.get("usage") if isinstance(response, dict) else None, + raw_sha256=raw_hash, + ) + + +def summarize(upstream, tasks, records): + # Upstream accuracy is attempted-only; keep it explicitly diagnostic. + projections = [dict(row, model=str(row["model"]) if row["model"] is not None else "") for row in records] + original = upstream.summarize.summarize(tasks, projections) + correct = sum(bool(row["correct"]) for row in records) + latency = { + name: upstream.metrics.latency_summary([row["latency_s"] for row in rows]) + for name, rows in ( + ("all_attempts", records), + ("success", [r for r in records if r["ok"]]), + ("failure", [r for r in records if not r["ok"]]), + ) + } + return dict( + planned=len(tasks), + attempted=len(records), + valid=sum(r["valid"] for r in records), + correct=correct, + accuracy=correct / len(tasks), + coverage=len(records) / len(tasks), + latency=latency, + upstream_attempted_metrics=original, + ) diff --git a/benchmarks/legs/__init__.py b/benchmarks/legs/__init__.py new file mode 100644 index 0000000..56d7447 --- /dev/null +++ b/benchmarks/legs/__init__.py @@ -0,0 +1 @@ +"""Benchmark collection and aggregation command lines.""" diff --git a/benchmarks/legs/leg1_public231.py b/benchmarks/legs/leg1_public231.py new file mode 100644 index 0000000..7630fc1 --- /dev/null +++ b/benchmarks/legs/leg1_public231.py @@ -0,0 +1,248 @@ +"""Collect serial public231 rounds or validate and aggregate saved evidence.""" + +import argparse +import copy +from contextlib import closing +import math +import os +import sys +import time +from pathlib import Path + +from benchmarks.client import DecisionClient, decode +from benchmarks.evidence import ( + body_for, + digest, + dump, + load_reference, + outside_repo, + score_record, + summarize, + write_json, +) + + +def validate_environment(meta): + for group, fields in (("checkpoint", ("repo", "revision")), ("backend", ("name", "version"))): + if any(not meta.get(group, {}).get(field) for field in fields): + raise ValueError(f"run-meta requires actual {group}: {fields}") + value = meta.get("sampling", {}).get("temperature") + if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value <= 0: + raise ValueError("run-meta requires actual positive sampling.temperature") + if not meta.get("hardware"): + raise ValueError("run-meta requires actual hardware") + + +def predict(args): + upstream, items, dataset = load_reference() + environment = args.run_meta.read_bytes() + meta = decode(environment) + validate_environment(meta) + if "stop_reason" in meta: + raise ValueError("stop_reason is collector-owned state; omit it from the supplied environment") + if args.rounds < 1 or not 0 <= args.warmup <= 231 or not 1 <= args.limit <= 231: + raise ValueError("Invalid rounds, warmup or limit") + output = outside_repo(args.out) + plan = dict( + warmup=args.warmup, + rounds=args.rounds, + tasks=231, + limit=args.limit, + order="dataset sequential", + identities=[identity for identity, _ in items], + ) + collector = dict( + run_id=output.name, + argv=sys.argv, + endpoint=args.endpoint, + path=args.path, + model_configured=args.model, + response_parse="typed", + thinking=False, + dataset=dataset, + plan=plan, + env_file=str(args.run_meta.resolve()), + env_sha256=digest(environment), + stop_reason=None, + ts=time.time(), + ) + if any(key in meta and meta[key] != value for key, value in collector.items()): + raise ValueError("Supplied environment conflicts with collector metadata") + meta.update(collector) + with closing(DecisionClient(args.endpoint, path=args.path, model=args.model, timeout_s=args.timeout)) as client: + output.mkdir(parents=True, exist_ok=False) + with (output / "environment.json").open("xb") as stream: + stream.write(environment) + write_json(output / "run.meta.json", meta) + stop, errors = None, 0 + try: + for number in range(args.rounds + 1): + kind = "warmup" if number == 0 else "measured" + directory = output / ("warmup" if number == 0 else f"round-{number}") + (directory / "raw").mkdir(parents=True) + selected = items[: args.warmup if number == 0 else args.limit] + write_json( + directory / "predict.meta.json", dict(round=number, planned=args.warmup if number == 0 else 231) + ) + with (directory / ("records.jsonl" if number == 0 else "results.jsonl")).open("x") as stream: + for identity, task in selected: + raw = client.request(body_for(task, args.model)) + raw_path = directory / "raw" / (digest(task.id.encode()) + ".json") + write_json(raw_path, raw) + record = score_record( + upstream, identity, task, raw, args.model, number, kind, digest(raw_path.read_bytes()) + ) + stream.write(dump(record) + "\n") + stream.flush() + os.fsync(stream.fileno()) + if raw["client_error"] == "KeyboardInterrupt": + stop = "interrupted" + break + errors = errors + 1 if not record["ok"] and record["status_code"] != 422 else 0 + if record["status_code"] in (401, 403, 429) or errors >= 3: + stop = "access/rate limit or three consecutive infrastructure errors" + break + if stop: + break + except KeyboardInterrupt: + stop = "interrupted" + except Exception as exc: + stop = f"{type(exc).__name__}: {exc}" + raise + finally: + write_json(output / "completion.json", dict(stop_reason=stop, ts=time.time())) + meta["stop_reason"] = stop + write_json(output / "run.meta.final.json", meta) + (output / "run.meta.final.json").replace(output / "run.meta.json") + return output + + +def read_round(directory, expected, upstream, model, number): + path = directory / ("records.jsonl" if number == 0 else "results.jsonl") + records = [] + if path.exists(): + with path.open(encoding="utf-8") as stream: + records = [decode(line) for line in stream] + if len(records) > len(expected): + raise ValueError("Too many or duplicate records") + expected_raw = {digest(task.id.encode()) + ".json" for _, task in expected[: len(records)]} + actual_raw = {path.name for path in (directory / "raw").glob("*")} + if expected_raw != actual_raw: + raise ValueError( + f"Raw/record mismatch in {directory}: missing={sorted(expected_raw - actual_raw)}, " + f"orphan={sorted(actual_raw - expected_raw)}" + ) + for row, (identity, task) in zip(records, expected): + raw_path = directory / "raw" / (digest(task.id.encode()) + ".json") + data = raw_path.read_bytes() + raw = decode(data) + if dump(raw["request"]) != dump(body_for(task, model)): + raise ValueError("Raw request differs from frozen task") + parsed = None + if raw["response_body"] is not None: + try: + parsed = decode(raw["response_body"]) + except ValueError: + pass + if raw["response"] != parsed: + raise ValueError("Decoded response differs from original body") + if ( + not isinstance(raw["latency_s"], (int, float)) + or not math.isfinite(raw["latency_s"]) + or raw["latency_s"] < 0 + ): + raise ValueError("Invalid measured latency") + recomputed = score_record( + upstream, identity, task, raw, model, number, "warmup" if number == 0 else "measured", digest(data) + ) + if row != recomputed: + raise ValueError("Record differs from raw evidence or planned identity/order") + return records + + +def aggregate(args): + upstream, items, dataset = load_reference() + run = outside_repo(args.run) + meta = decode((run / "run.meta.json").read_text()) + validate_environment(meta) + environment = (run / "environment.json").read_bytes() + if digest(environment) != meta["env_sha256"]: + raise ValueError("Environment source hash mismatch") + env = decode(environment) + if any(key not in meta or meta[key] != value for key, value in env.items()): + raise ValueError("Run metadata differs from supplied environment") + plan = meta["plan"] + if ( + any(meta["dataset"][key] != dataset[key] for key in ("jevbench_pin", "hashes", "dataset_hash")) + or plan["identities"] != [identity for identity, _ in items] + or plan["tasks"] != 231 + ): + raise ValueError("Frozen plan/reference mismatch") + if plan["rounds"] < 1 or not 0 <= plan["warmup"] <= 231 or not 1 <= plan["limit"] <= 231: + raise ValueError("Invalid frozen run plan") + expected_dirs = {f"round-{i}" for i in range(1, plan["rounds"] + 1)} + if {p.name for p in run.glob("round-*")} - expected_dirs: + raise ValueError("Unknown round directory") + warmup = read_round(run / "warmup", items[: plan["warmup"]], upstream, meta["model_configured"], 0) + rounds, pooled_tasks, pooled_records = [], [], [] + for number in range(1, plan["rounds"] + 1): + records = read_round( + run / f"round-{number}", items[: plan["limit"]], upstream, meta["model_configured"], number + ) + tasks = [task for _, task in items] + rounds.append(dict(round=number, **summarize(upstream, tasks, records))) + # Give repeated tasks distinct IDs only for upstream duplicate checking. + for task in tasks: + clone = copy.copy(task) + clone.id = f"{number}:{task.id}" + clone.group = f"{number}:{task.group}" if task.group else None + pooled_tasks.append(clone) + pooled_records.extend(dict(row, task_id=f"{number}:{row['task_id']}") for row in records) + sufficient = ( + plan["rounds"] == 3 + and plan["warmup"] == 5 + and plan["limit"] == 231 + and len(warmup) == 5 + and all(r["attempted"] == 231 for r in rounds) + ) + result = dict( + sufficient=sufficient, + status="sufficient" if sufficient else "insufficient", + warmup_attempted=len(warmup), + rounds=rounds, + pooled=summarize(upstream, pooled_tasks, pooled_records), + distinct_tasks=231, + cost_usd=None, + cost_basis="unmetered is not free", + latency_success_basis="HTTP success with a typed decision structure; schema validity is separate", + ) + output = outside_repo(args.out) + output.mkdir(parents=True, exist_ok=True) + write_json(output / "summary.json", result) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest="command", required=True) + collector = commands.add_parser("predict") + collector.add_argument("--endpoint", required=True) + collector.add_argument("--path", required=True, choices=("/v1/systemone", "/v1/decisions")) + collector.add_argument("--model") + collector.add_argument("--thinking", choices=("off",), default="off") + collector.add_argument("--warmup", type=int, default=5) + collector.add_argument("--rounds", type=int, default=3) + collector.add_argument("--limit", type=int, default=231) + collector.add_argument("--timeout", type=float, default=120) + collector.add_argument("--run-meta", type=Path, required=True) + collector.add_argument("--out", type=Path, required=True) + merger = commands.add_parser("aggregate") + merger.add_argument("--run", type=Path, required=True) + merger.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + result = predict(args) if args.command == "predict" else aggregate(args) + print(result if isinstance(result, Path) else dump(result)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/results/public231-h800.md b/benchmarks/results/public231-h800.md new file mode 100644 index 0000000..c08a993 --- /dev/null +++ b/benchmarks/results/public231-h800.md @@ -0,0 +1,92 @@ +# Public231 on one H800 + +Five open checkpoints, one NVIDIA H800, one HTTP request at a time. Each service +received five warmups followed by the same 231 questions in the same order for three +rounds. Latency covers the complete HTTP request and response. No profiler was active. + +Open-Jev used System1-Omni's native Rust/CUDA service; Intern-Decision used its +upstream HF service. These results compare deployed configurations. Their ratios +include different architectures, templates and execution paths, not just model size +or kernel speed. + +## Results + +| Model | Service | Correct / 231 | Accuracy | Validity | Brier | ECE | p50 / p95 ms | +|---|---|---:|---:|---:|---:|---:|---:| +| Intern-Decision-0.8B | HF / T=1 | 164/231 | 70.9957% | 100% | 0.369906 | 0.111077 | 80.493 / 175.237 | +| Intern-Decision-2B | HF / T=1 | 181/231 | 78.3550% | 100% | 0.306049 | 0.115420 | 79.054 / 175.276 | +| Intern-Decision-4B | HF / T=1 | 200/231 | 86.5801% | 100% | 0.189601 | 0.067214 | 104.552 / 235.291 | +| Open-Jev-9B | Omni native | 179/231 | 77.4892% | 100% | 0.320752 | 0.086786 | 42.688 / 327.057 | +| Open-Jev-27B-v1.1 | Omni native | 198/231 | 85.7143% | 100% | 0.240493 | 0.130333 | 68.877 / 1068.817 | + +All five configurations completed 693/693 measured requests, with no failed or invalid +responses. Predictions and probabilities matched across the three rounds. Accuracy +therefore has 231 unique questions; the repeated rounds are timing repetitions. +Brier/ECE use all 693 valid distributions. Failure latency has n=0 and null quantiles. + +Open-Jev 9B had the lowest median latency. Intern 4B answered 200/231 correctly versus +Open-Jev 27B's 198/231, and had a lower p95; Open-Jev 27B had a lower median. This small +set does not establish a general quality ranking or a uniform 2–3× speed advantage. + +## Per-round latency + +| Model | Round 1 p50 / p95 ms | Round 2 p50 / p95 ms | Round 3 p50 / p95 ms | +|---|---:|---:|---:| +| Intern-Decision-0.8B | 84.534 / 180.725 | 73.301 / 173.430 | 73.485 / 172.817 | +| Intern-Decision-2B | 81.454 / 180.338 | 74.251 / 173.638 | 74.154 / 173.411 | +| Intern-Decision-4B | 107.711 / 240.246 | 98.628 / 232.829 | 98.065 / 232.709 | +| Open-Jev-9B | 43.166 / 317.867 | 42.624 / 317.644 | 42.525 / 317.579 | +| Open-Jev-27B-v1.1 | 68.925 / 1030.605 | 68.892 / 1034.555 | 68.821 / 1033.953 | + +Pooled quantiles above are computed from all raw timings, not from these three quantiles. + +## Configuration + +- BF16; client concurrency 1; no retries; thinking disabled. +- Intern: official source `3572c8a68b5df5dafe02d0e093989ba8ec0183bc`, + PyTorch 2.9.1+cu128, torchvision 0.24.1+cu128, Transformers 5.14.1, SDPA, + temperature 1, no calibration artifact, no KV cache. Optional FLA/causal-conv1d + fast paths were unavailable, so the service used the upstream Torch fallback. +- Open-Jev: Omni `4a79980d8a75fd063cb3f8247e06215e288b18ac`, CUDA 13, + merged LoRA and decision head, Graph and prefix cache disabled. 9B evaluates + candidates sequentially; 27B packs candidates. A single service request is not + necessarily one candidate sequence. +- Context limits: Intern 8192 tokens for the complete chat; Open-Jev 16384 per + candidate. Native usage counts tokens summed over candidates, so the two services' + token counts are not directly comparable. + +| Checkpoint | Revision | Temperature | Spawn-to-ready s | +|---|---|---:|---:| +| [Intern 0.8B](https://huggingface.co/internlm/Intern-Decision-0.8B) | `85a0cc5a99d67ea8d56dfe98115689212867171d` | 1 | 16.084 | +| [Intern 2B](https://huggingface.co/internlm/Intern-Decision-2B) | `8797836c65fc91a2435b1fb6850b5f0aabd75cc3` | 1 | 14.045 | +| [Intern 4B](https://huggingface.co/internlm/Intern-Decision-4B) | `0e5e6aa7d6d750e2b1504ba11a8136cb58aeb3cd` | 1 | 16.048 | +| [Open-Jev 9B](https://huggingface.co/ZefanCai/Open-Jev-9B) | `47e966881e489511c0c7f5633a9e1960a676a551` | 1.8969118766 | 8.015 | +| [Open-Jev 27B v1.1](https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1) | `28cf73067d5b337860bbef3c85b8b82ba8730956` | 2.5343690298 | 20.029 | + +Cold startup is excluded from request latency and includes native internal warmup. +Open 9B's base is `c202236235762e1c871ad0ccb60c8ee5ba337b9a`; Open 27B's base is +`1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`. + +## Inputs and verification + +[JevBench](https://github.com/fstandhartinger/jevbench/tree/7ce310c7262ed49cc85853339a8a42459298e3f3) +provides easy48, original72 and hard111 and the scorer. Source and data hashes are +pinned in `benchmarks/evidence.py`. The article's complete 248-question set was not +available; this is not a reproduction of its 23-model leaderboard. + +The run used Agents base `9790b52ef4d78a2215e8dd5a526d3a8daea9875a`, dated +2026-10-07 UTC, Slurm job 415076. All 3490 raw requests, including warmups, were +checked for source identity, order, response hash and absence of target leakage. +A non-author independently re-scored the saved responses with the pinned scorer +and recomputed latency from raw samples. The final collector's offline aggregation +matched counts exactly and floating-point metrics within 1e-12. + +Collection preceded fixes to metadata validation, URL validation before file writes, +and JSONL physical-line reading. These fixes preserve the HTTP and scoring paths. +Verification replays the saved responses; it is not a second model run. + +Actual loaded-library maps and the weight hashes were recorded. The native binary +SHA-256 was `a0694633308ea0ad0ca2a302bfb16cbb73fd3367cd36cd142596761a71d23afc`; +CUDA library SHA-256 was `a0db4905df4fe2fb5d006acaaa09b5b51dbd3933cc8e9b20ec8114f23def8b20`. +Use the [collection and replay commands](../README.md#run) with the fixed inputs and +record the actual serving environment. Large raw traces and weights stay outside Git. diff --git a/tests/benchmarks/test_public231.py b/tests/benchmarks/test_public231.py new file mode 100644 index 0000000..0f5720e --- /dev/null +++ b/tests/benchmarks/test_public231.py @@ -0,0 +1,520 @@ +"""CPU protocol fixtures only; no model service or network connection.""" + +import copy +import json +import os +from pathlib import Path +from types import SimpleNamespace + +import httpx +import pytest + +from benchmarks.client import DecisionClient +from benchmarks.evidence import body_for, digest, dump, load_reference, score_record, summarize +from benchmarks.legs import leg1_public231 as cli + + +@pytest.fixture +def reference(): + if not os.environ.get("JEVBENCH_ROOT"): + pytest.skip("Set JEVBENCH_ROOT to the pinned external reference") + return load_reference() + + +def test_multifield_single_post_preserves_input(): + body = { + "state": [{"context": "structured"}], + "questions": { + "second": {"type": "choice", "criteria": {"z": "last", "a": "first"}}, + "first": {"type": "noul", "instructions": "accept?"}, + }, + "thinking": {"enabled": False}, + } + before, calls = copy.deepcopy(body), [] + + def handler(request): + calls.append(json.loads(request.content)) + assert str(request.url) == "http://fixture.invalid/v1/decisions" + return httpx.Response(200, text=' {"answers": {"second": {}, "first": {}}, "model":"fixture"}\n') + + client = DecisionClient("http://fixture.invalid", path="/v1/decisions", transport=httpx.MockTransport(handler)) + raw = client.request(body) + client.close() + assert body == before and calls == [body] + assert list(calls[0]["questions"]) == ["second", "first"] + assert list(calls[0]["questions"]["second"]["criteria"]) == ["z", "a"] + assert raw["response_body"].startswith(" ") and raw["response_body"].endswith("\n") + assert set(raw["response"]["answers"]) == {"second", "first"} + assert raw["http_status"] == 200 and raw["latency_s"] > 0 + + +@pytest.mark.parametrize("status,text", [(422, "too long"), (500, "failure"), (200, "broken json")]) +def test_raw_bad_responses_survive(status, text): + client = DecisionClient( + "http://fixture.invalid", + path="/v1/systemone", + transport=httpx.MockTransport(lambda _: httpx.Response(status, text=text)), + ) + raw = client.request({"questions": {}}) + client.close() + assert (raw["http_status"], raw["response_body"], raw["response"]) == (status, text, None) + assert raw["latency_s"] > 0 + + +@pytest.mark.parametrize("error", [httpx.ReadTimeout("fixture timeout"), KeyboardInterrupt()]) +def test_network_error_has_real_elapsed(error): + def handler(_): + raise error + + client = DecisionClient("http://fixture.invalid", path="/v1/systemone", transport=httpx.MockTransport(handler)) + raw = client.request({"state": "fixture"}) + client.close() + assert raw["http_status"] is None and raw["response_body"] is None + assert raw["client_error"] and raw["latency_s"] > 0 + + +@pytest.mark.parametrize("url", ["http://user:secret@fixture.invalid", "http://fixture.invalid?key=x"]) +def test_credentials_cannot_enter_endpoint_evidence(url): + with pytest.raises(ValueError): + DecisionClient(url, path="/v1/systemone") + + +def raw_answer(answer, *, status=200, seconds=1): + response = {"answers": {"decision": answer}, "model": "fixture", "usage": {"decision_count": 1}} + return dict( + request={}, + response=response, + response_body=dump(response), + http_status=status, + client_error=None, + latency_s=seconds, + ts=123.0, + ) + + +@pytest.mark.parametrize( + "probs,valid,renormalized", + [ + ({"no": 0.4, "yes": 0.595}, True, True), + ({"no": 0.4, "yes": 0.5}, False, False), + ({"no": 0.4}, False, False), + ({"no": 0.4, "yes": 0.6, "extra": 0}, False, False), + ({"no": -0.1, "yes": 1.1}, False, False), + ], +) +def test_upstream_alone_decides_probability_band(reference, probs, valid, renormalized): + upstream, items, _ = reference + identity, task = next(item for item in items if item[1].question["type"] == "noul") + row = score_record( + upstream, + identity, + task, + raw_answer({"type": "noul", "probabilities": probs}), + "configured", + 1, + "measured", + "fixture-hash", + ) + assert row["ok"] and row["valid"] is valid and row["renormalized"] is renormalized + assert row["probs_as_returned"] == probs + if renormalized: + assert row["strict_valid"] is False and sum(row["probs"].values()) == pytest.approx(1) + if not valid: + assert row["correct"] is False and row["probs"] is None + + +def test_native_scalar_and_missing_probabilities(reference): + upstream, items, _ = reference + for kind in ("noul", "choice", "score"): + identity, task = next(item for item in items if item[1].question["type"] == kind) + answer = {"type": kind, "noul": 0.6} if kind == "noul" else {"type": kind, kind: task.expected} + row = score_record(upstream, identity, task, raw_answer(answer), None, 1, "measured", "h") + assert row["valid"] is (kind == "noul") + assert row["model_configured"] is None and row["model"] == "fixture" + if kind == "noul": + assert row["probs"] == {"no": 0.4, "yes": 0.6} + else: + assert row["probs"] is None and row["probs_as_returned"] is None + + +@pytest.mark.parametrize("answers", [None, [], "bad", {"decision": None}, {"decision": {"type": "wrong"}}]) +def test_malformed_typed_structure_is_recorded(reference, answers): + upstream, items, _ = reference + identity, task = items[0] + raw = raw_answer({}) + raw["response"]["answers"] = answers + row = score_record(upstream, identity, task, raw, None, 1, "measured", "h") + assert row["status"] == "parse_error" and not row["ok"] and not row["valid"] + + +def test_nonfinite_response_is_not_json(): + client = DecisionClient( + "http://fixture.invalid", + path="/v1/systemone", + transport=httpx.MockTransport( + lambda _: httpx.Response( + 200, text='{"answers":{"decision":{"type":"noul","probabilities":{"no":NaN,"yes":1}}}}' + ) + ), + ) + raw = client.request({}) + client.close() + assert raw["response"] is None and "NaN" in raw["response_body"] + + +@pytest.fixture +def collect(reference, monkeypatch, tmp_path): + upstream, items, _ = reference + env = tmp_path / "actual-environment.json" + env.write_text( + dump( + dict( + checkpoint={"repo": "CPU-fixture", "revision": "fixture-only"}, + backend={"name": "httpx.MockTransport", "version": httpx.__version__}, + sampling={"temperature": 1.0}, + hardware="CPU protocol fixture, no inference", + dtype="fixture-original", + tokenizer={"revision": "fixture-tokenizer"}, + base={"revision": "fixture-base"}, + adapter={"revision": "fixture-adapter"}, + head={"revision": "fixture-head"}, + context_limit=16384, + cache={"enabled": False}, + job_id="CPU-fixture", + ) + ) + ) + lookup = {dump(body_for(task, "configured")): task for _, task in items} + + def run(*, warmup=5, rounds=3, limit=231, statuses=None, response_text=None): + calls = [] + + def handler(request): + body = json.loads(request.content) + assert "targets" not in body and "provenance" not in body and "expected" not in body + assert body["thinking"] == {"enabled": False} + task = lookup[dump(body)] + calls.append(body) + status = statuses[min(len(calls) - 1, len(statuses) - 1)] if statuses else 200 + if response_text is not None: + return httpx.Response(status, text=response_text) + answer = { + "type": task.question["type"], + "probabilities": {label: float(label == str(task.expected)) for label in task.labels}, + } + return httpx.Response(status, json={"answers": {"decision": answer}, "model": "fixture-response"}) + + class FixtureClient(DecisionClient): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs, transport=httpx.MockTransport(handler)) + + def request(self, body): + raw = super().request(body) + raw["latency_s"] = 999 if len(calls) <= warmup else (1 if raw["http_status"] == 200 else 120) + return raw + + monkeypatch.setattr(cli, "DecisionClient", FixtureClient) + output = tmp_path / f"run-{len(list(tmp_path.glob('run-*')))}" + args = SimpleNamespace( + endpoint="http://fixture.invalid", + path="/v1/decisions", + model="configured", + timeout=10, + run_meta=env, + out=output, + warmup=warmup, + rounds=rounds, + limit=limit, + ) + cli.predict(args) + return output, calls, args + + return run + + +def aggregate(run, out=None): + return cli.aggregate(SimpleNamespace(run=run, out=out or run)) + + +@pytest.mark.parametrize( + "endpoint", ["http://user:DUMMY_SECRET@fixture.invalid", "http://fixture.invalid?key=DUMMY_SECRET"] +) +def test_predict_cli_rejects_credentials_before_output(collect, monkeypatch, endpoint): + run, calls, args = collect(warmup=0, rounds=1, limit=1) + args.out = run.parent / "rejected-endpoint" + monkeypatch.setattr( + "sys.argv", + [ + "public231", + "predict", + "--endpoint", + endpoint, + "--path", + args.path, + "--run-meta", + str(args.run_meta), + "--out", + str(args.out), + ], + ) + with pytest.raises(ValueError, match="without credentials"): + cli.main() + assert not args.out.exists() and len(calls) == 1 + assert all("DUMMY_SECRET" not in path.read_text() for path in run.parent.rglob("*") if path.is_file()) + + +@pytest.mark.parametrize("failure", ["mkdir", "environment.json", "run.meta.json"]) +def test_predict_closes_client_when_output_setup_fails(collect, monkeypatch, failure): + run, calls, args = collect(warmup=0, rounds=1, limit=1) + args.out = run.parent / "failed-output" + created, original_client, original_open = [], cli.DecisionClient, Path.open + + def client(*args, **kwargs): + result = original_client(*args, **kwargs) + created.append(result) + return result + + def open_file(path, *options, **kwargs): + if path == args.out / failure: + raise OSError("fixture output failure") + return original_open(path, *options, **kwargs) + + if failure == "mkdir": + args.out.mkdir() + monkeypatch.setattr(cli, "DecisionClient", client) + monkeypatch.setattr(Path, "open", open_file) + with pytest.raises(OSError): + cli.predict(args) + assert len(created) == 1 and created[0].http.is_closed + assert len(calls) == 1 + + +@pytest.mark.parametrize("separator", ["\u2028", "\u0085", "\u2029"]) +def test_collect_and_aggregate_preserve_unicode_in_error_body(collect, separator): + run, calls, _ = collect( + warmup=0, rounds=1, limit=1, statuses=[422], response_text=dump({"detail": f"bad{separator}input"}) + ) + with (run / "round-1/results.jsonl").open() as stream: + records = [json.loads(line) for line in stream] + assert len(calls) == len(records) == 1 and records[0]["status"] == "refused" + assert separator in records[0]["error"] + result = aggregate(run) + assert result["pooled"]["planned"] == 231 and result["pooled"]["attempted"] == 1 + assert result["pooled"]["correct"] == result["pooled"]["accuracy"] == 0 + assert not result["sufficient"] + + +@pytest.mark.parametrize("line", ["\n", "{broken}\n"]) +def test_aggregate_rejects_blank_or_malformed_physical_line(collect, line): + run, _, _ = collect(warmup=0, rounds=1, limit=1) + (run / "round-1/results.jsonl").write_text(line) + with pytest.raises(ValueError): + aggregate(run) + + +def test_complete_5_plus_3_rounds_and_no_warmup_in_latency(collect): + run, calls, _ = collect() + result = aggregate(run) + assert len(calls) == 698 and result["sufficient"] + assert result["pooled"]["planned"] == result["pooled"]["correct"] == 693 + assert result["pooled"]["latency"]["all_attempts"] == {"n": 693, "p50_s": 1, "p95_s": 1} + assert result["pooled"]["latency"]["failure"] == {"n": 0, "p50_s": None, "p95_s": None} + assert all(round_["attempted"] == 231 for round_ in result["rounds"]) + assert len(list((run / "warmup/raw").glob("*.json"))) == 5 + second = aggregate(run, run / "offline-recompute") + assert result == second + with pytest.raises(FileExistsError): + aggregate(run) + + +@pytest.mark.parametrize("options", [{"limit": 10}, {"warmup": 0}, {"rounds": 1}]) +def test_debug_run_is_insufficient_with_fixed_denominator(collect, options): + run, _, _ = collect(**options) + result = aggregate(run) + assert not result["sufficient"] + if "limit" in options: + assert result["pooled"]["accuracy"] == 30 / 693 + assert result["rounds"][0]["accuracy"] == 10 / 231 + + +@pytest.mark.parametrize( + "statuses,attempted", [([401], 1), ([403], 1), ([429], 1), ([500], 3), ([500, 500, 422, 500, 500, 500], 6)] +) +def test_stop_policy_leaves_unattempted_in_plan(collect, statuses, attempted): + run, calls, _ = collect(warmup=0, statuses=statuses) + result = aggregate(run) + assert len(calls) == attempted + assert result["pooled"]["accuracy"] == 0 and result["pooled"]["planned"] == 693 + assert result["pooled"]["attempted"] == attempted and not result["sufficient"] + assert json.loads((run / "run.meta.json").read_text())["stop_reason"] + + +def test_failures_and_partial_denominator_with_two_latencies(collect): + run, _, _ = collect(warmup=0, limit=2, rounds=1, statuses=[200, 500]) + result = aggregate(run)["rounds"][0] + assert result["accuracy"] == 1 / 231 + assert result["latency"]["success"] == {"n": 1, "p50_s": 1, "p95_s": 1} + assert result["latency"]["failure"] == {"n": 1, "p50_s": 120, "p95_s": 120} + assert result["latency"]["all_attempts"]["p50_s"] == 60.5 + assert result["latency"]["all_attempts"]["p95_s"] == pytest.approx(114.05) + + +@pytest.mark.parametrize("change", ["duplicate", "unknown", "score", "raw", "environment", "order"]) +def test_offline_tampering_rejected(collect, change): + run, _, _ = collect(warmup=0, rounds=1, limit=2) + path = run / "round-1/results.jsonl" + rows = [json.loads(line) for line in path.read_text().splitlines()] + if change == "duplicate": + rows[1] = rows[0] + elif change == "unknown": + rows[0]["task_id"] = "unknown" + elif change == "score": + rows[0]["correct"] = False + elif change == "order": + rows.reverse() + elif change == "raw": + raw = run / "round-1/raw" / (digest(rows[0]["task_id"].encode()) + ".json") + raw.write_text(raw.read_text() + " ") + else: + path_env = run / "environment.json" + path_env.write_text(path_env.read_text() + " ") + path.write_text("".join(dump(row) + "\n" for row in rows)) + with pytest.raises(ValueError): + aggregate(run) + + +def test_output_and_environment_boundaries(collect, tmp_path): + run, _, args = collect(warmup=0, limit=1, rounds=1) + with pytest.raises(FileExistsError): + cli.predict(args) + args.out = Path(__file__).resolve().parents[2] / "forbidden-evidence" + with pytest.raises(ValueError, match="outside"): + cli.predict(args) + env = json.loads(args.run_meta.read_text()) + env["sampling"]["temperature"] = 0 + args.run_meta.write_text(dump(env)) + args.out = tmp_path / "invalid-temperature" + with pytest.raises(ValueError, match="temperature"): + cli.predict(args) + assert not args.out.exists() + + +def test_reference_hash_tampering_rejected(reference, monkeypatch, tmp_path): + _, _, dataset = reference + from benchmarks.evidence import HASHES + + root = tmp_path / "reference" + for name in HASHES: + destination = root / name + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes((Path(dataset["root"]) / name).read_bytes()) + (root / "jevbench/scoring.py").write_text("raise RuntimeError('must not execute')\n") + monkeypatch.setenv("JEVBENCH_ROOT", str(root)) + with pytest.raises(ValueError, match="hash mismatch"): + load_reference() + + +def test_empty_summary_has_null_latency(reference): + upstream, items, _ = reference + result = summarize(upstream, [task for _, task in items], []) + assert result["accuracy"] == 0 and result["coverage"] == 0 + assert all(value == {"n": 0, "p50_s": None, "p95_s": None} for value in result["latency"].values()) + + +@pytest.mark.parametrize("literal", ["1e309", "NaN", "Infinity"]) +def test_nonfinite_wire_numbers_are_retained_as_parse_failures(literal): + response = '{"answers":{"decision":{"type":"noul","noul":' + literal + "}}}" + client = DecisionClient( + "http://fixture.invalid", + path="/v1/systemone", + transport=httpx.MockTransport(lambda _: httpx.Response(200, text=response)), + ) + raw = client.request({}) + client.close() + assert raw["response"] is None and raw["response_body"] == response + + +def test_scorer_numeric_overflow_counts_wrong(reference): + upstream, items, _ = reference + identity, task = next(item for item in items if item[1].question["type"] == "noul") + probs = {"no": 10**400, "yes": 0} + row = score_record( + upstream, identity, task, raw_answer({"type": "noul", "probabilities": probs}), "configured", 1, "measured", "h" + ) + assert row["ok"] and not row["valid"] and not row["correct"] + assert row["probs_as_returned"] == probs + + +@pytest.mark.parametrize("directory,filename", [("warmup", "records.jsonl"), ("round-1", "results.jsonl")]) +@pytest.mark.parametrize("change", ["missing_row", "missing_records_file", "missing_raw", "unknown_raw"]) +def test_raw_and_records_must_correspond(collect, directory, filename, change): + run, _, _ = collect(warmup=2, rounds=1, limit=2) + directory = run / directory + path = directory / filename + rows = path.read_text().splitlines() + row = json.loads(rows[-1]) + raw = directory / "raw" / (digest(row["task_id"].encode()) + ".json") + changed = raw + if change == "missing_row": + path.write_text(rows[0] + "\n") + elif change == "missing_records_file": + path.unlink() + elif change == "missing_raw": + raw.unlink() + else: + changed = directory / "raw/unknown.json" + changed.write_bytes(raw.read_bytes()) + with pytest.raises(ValueError, match="Raw/record mismatch") as error: + aggregate(run) + assert changed.name in str(error.value) + assert not (run / "summary.json").exists() + + +@pytest.mark.parametrize("field", ["dtype", "tokenizer", "base", "adapter", "head", "context_limit", "cache", "job_id"]) +@pytest.mark.parametrize("change", ["different", "missing"]) +def test_all_environment_fields_must_match(collect, field, change): + run, _, _ = collect(warmup=0, rounds=1, limit=1) + path = run / "run.meta.json" + meta = json.loads(path.read_text()) + if change == "different": + meta[field] = "fixture-contradiction" + else: + del meta[field] + path.write_text(dump(meta)) + with pytest.raises(ValueError, match="supplied environment"): + aggregate(run) + assert not (run / "summary.json").exists() + + +def test_conflicting_environment_controls_rejected_before_collection(collect): + run, calls, args = collect(warmup=0, rounds=1, limit=1) + env = json.loads(args.run_meta.read_text()) + env["endpoint"] = "http://conflicting.invalid" + args.run_meta.write_text(dump(env)) + args.out = run.parent / "conflicting-run" + with pytest.raises(ValueError, match="collector"): + cli.predict(args) + assert len(calls) == 1 and not args.out.exists() + + +def test_matching_environment_controls_are_compatible(collect): + run, _, args = collect(warmup=0, rounds=1, limit=1) + env = json.loads(args.run_meta.read_text()) + env.update(endpoint=args.endpoint, path=args.path, thinking=False) + args.run_meta.write_text(dump(env)) + args.out = run.parent / "matching-run" + cli.predict(args) + assert aggregate(args.out)["pooled"]["attempted"] == 1 + + +@pytest.mark.parametrize("stop_reason", [None, "stale stop reason"]) +def test_environment_stop_reason_rejected_before_request(collect, stop_reason): + run, calls, args = collect(warmup=0, rounds=1, limit=1, statuses=[401]) + assert aggregate(run)["pooled"]["attempted"] == 1 + env = json.loads(args.run_meta.read_text()) + env["stop_reason"] = stop_reason + args.run_meta.write_text(dump(env)) + args.out = run.parent / "invalid-stop-reason" + with pytest.raises(ValueError, match="stop_reason"): + cli.predict(args) + assert len(calls) == 1 and not args.out.exists() From 40bf742e2b84ad6157bf2990dd5e4bab73571702 Mon Sep 17 00:00:00 2001 From: linear3735 Date: Thu, 8 Oct 2026 11:25:49 +0800 Subject: [PATCH 2/5] Fix benchmark evidence encoding across locales --- benchmarks/evidence.py | 4 ++- benchmarks/legs/leg1_public231.py | 6 ++-- tests/benchmarks/test_public231.py | 49 ++++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+), 3 deletions(-) diff --git a/benchmarks/evidence.py b/benchmarks/evidence.py index 3940380..dff9de5 100644 --- a/benchmarks/evidence.py +++ b/benchmarks/evidence.py @@ -58,7 +58,9 @@ def load_reference(): ) items = [] for tier in ("easy", "original", "hard"): - for line, text in enumerate((root / f"datasets/public/{tier}.jsonl").read_text().split("\n")[:-1], 1): + for line, text in enumerate( + (root / f"datasets/public/{tier}.jsonl").read_text(encoding="utf-8").split("\n")[:-1], 1 + ): task = upstream.tasks.Task.from_dict(decode(text)) identity = dict( suite=f"jevbench-{tier}", diff --git a/benchmarks/legs/leg1_public231.py b/benchmarks/legs/leg1_public231.py index 7630fc1..97e88cc 100644 --- a/benchmarks/legs/leg1_public231.py +++ b/benchmarks/legs/leg1_public231.py @@ -84,7 +84,9 @@ def predict(args): write_json( directory / "predict.meta.json", dict(round=number, planned=args.warmup if number == 0 else 231) ) - with (directory / ("records.jsonl" if number == 0 else "results.jsonl")).open("x") as stream: + with (directory / ("records.jsonl" if number == 0 else "results.jsonl")).open( + "x", encoding="utf-8" + ) as stream: for identity, task in selected: raw = client.request(body_for(task, args.model)) raw_path = directory / "raw" / (digest(task.id.encode()) + ".json") @@ -163,7 +165,7 @@ def read_round(directory, expected, upstream, model, number): def aggregate(args): upstream, items, dataset = load_reference() run = outside_repo(args.run) - meta = decode((run / "run.meta.json").read_text()) + meta = decode((run / "run.meta.json").read_text(encoding="utf-8")) validate_environment(meta) environment = (run / "environment.json").read_bytes() if digest(environment) != meta["env_sha256"]: diff --git a/tests/benchmarks/test_public231.py b/tests/benchmarks/test_public231.py index 0f5720e..16da0e4 100644 --- a/tests/benchmarks/test_public231.py +++ b/tests/benchmarks/test_public231.py @@ -3,6 +3,8 @@ import copy import json import os +import subprocess +import sys from pathlib import Path from types import SimpleNamespace @@ -518,3 +520,50 @@ def test_environment_stop_reason_rejected_before_request(collect, stop_reason): with pytest.raises(ValueError, match="stop_reason"): cli.predict(args) assert len(calls) == 1 and not args.out.exists() + + +def test_cli_uses_utf8_independent_of_locale(reference, tmp_path): + script = r""" +import json +import sys +from pathlib import Path +import httpx +from benchmarks.client import DecisionClient +from benchmarks.evidence import write_json +from benchmarks.legs import leg1_public231 as cli + +root = Path(sys.argv[1]) +environment = root / "environment.json" +write_json(environment, dict( + checkpoint=dict(repo="fixture", revision="fixture"), + backend=dict(name="MockTransport", version=httpx.__version__), + sampling=dict(temperature=1), hardware="CPU \u6d4b\u8bd5", +)) +class Client(DecisionClient): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs, transport=httpx.MockTransport( + lambda _: httpx.Response(422, text="refused\u2028request") + )) +cli.DecisionClient = Client +run = root / "run" +sys.argv = ["public231", "predict", "--endpoint", "http://fixture.invalid", + "--path", "/v1/decisions", "--run-meta", str(environment), + "--out", str(run), "--warmup", "0", "--rounds", "1", "--limit", "2"] +cli.main() +sys.argv = ["public231", "aggregate", "--run", str(run), "--out", str(root / "summary")] +cli.main() +with (run / "round-1/results.jsonl").open(encoding="utf-8") as stream: + records = [json.loads(line) for line in stream] +assert len(records) == 2 and all(row["error"] == "refused\u2028request" for row in records) +summary = json.loads((root / "summary/summary.json").read_text(encoding="utf-8")) +assert summary["pooled"]["attempted"] == 2 and summary["pooled"]["planned"] == 231 +""" + result = subprocess.run( + [sys.executable, "-c", script, str(tmp_path)], + env={**os.environ, "PYTHONUTF8": "0", "PYTHONCOERCECLOCALE": "0", "LC_ALL": "C", "PYTHONIOENCODING": "utf-8"}, + capture_output=True, + text=True, + encoding="utf-8", + timeout=30, + ) + assert result.returncode == 0, result.stderr From 7421849c045061307d077047df097bf771d736cb Mon Sep 17 00:00:00 2001 From: linear3735 Date: Sat, 10 Oct 2026 19:15:15 +0800 Subject: [PATCH 3/5] bench: re-score saved text and ticket decisions Re-score pinned public231 probabilities and verify ticket decisions against accepted actions and committed environment state. Changed lines relative to stacked base: 2264 authored code/test lines and 98 documentation lines (2362 total additions + deletions). No third-party source copied in this commit. Depends on public231 PR #52. --- CHANGELOG.md | 3 + benchmarks/README.md | 95 +++ benchmarks/evidence.py | 15 +- benchmarks/quality_public231.py | 217 +++++ benchmarks/quality_ticket.py | 887 ++++++++++++++++++++ tests/benchmarks/test_quality_public231.py | 232 ++++++ tests/benchmarks/test_quality_ticket.py | 913 +++++++++++++++++++++ 7 files changed, 2360 insertions(+), 2 deletions(-) create mode 100644 benchmarks/quality_public231.py create mode 100644 benchmarks/quality_ticket.py create mode 100644 tests/benchmarks/test_quality_public231.py create mode 100644 tests/benchmarks/test_quality_ticket.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e289d1..786ebe0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ### Added +- Offline quality consumers for shared Omni public231 raw and original ticket-router + evidence, retaining planned denominators, original scoring and provisional completion gates. + - Public231 decision-model benchmark with raw HTTP evidence, pinned upstream scoring, complete-denominator quality metrics and per-round service latency. diff --git a/benchmarks/README.md b/benchmarks/README.md index 19f462e..93a2d44 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -81,3 +81,98 @@ ruff format --check benchmarks tests/benchmarks Reference-dependent tests skip if `JEVBENCH_ROOT` is absent. Mock and loopback tests check collection and scoring behavior; model quality and latency come from the linked real runs. + +## Offline quality from shared raw + +These consumers re-score saved Omni requests and original ticket episodes without +another model run. The existing `predict` and `aggregate` commands and their +default timing basis are unchanged. + +For public231, provide the frozen input manifest, the independently frozen Omni +`bench.py` interface (`5aa20a2c…`), and one `--run` path per planned round in order: + +```bash +export JEVBENCH_ROOT=/absolute/path/to/pinned/jevbench +python -m benchmarks.quality_public231 \ + --input-manifest /absolute/path/to/input-r1/manifest.json \ + --collector /absolute/path/to/omni-raw-interface-v2/bench.py \ + --model-config-id open-9b \ + --run /absolute/path/to/round-1 --run /absolute/path/to/round-2 \ + --run /absolute/path/to/round-3 \ + --out /absolute/path/to/new-quality-output +``` + +Each raw directory must contain the complete frozen `requests.jsonl`, `config.json`, +`completion.json` and saved `responses.jsonl` when measurement started. The Omni +interface checks file/response hashes, IDs, decoding and its saved wire verdict. +The quality consumer checks the pinned inputs and request order, then uses the +original JevBench scorer on the raw HTTP/typed probabilities. An Omni +`invalid_response` can still be a valid relaxed JevBench distribution. Wire validity, +distribution validity, strict validity, renormalization and label correctness remain +separate. Choice/score hard labels are never substituted for missing probabilities; +noul scalars use `{no: 1-p, yes: p}`. Accuracy uses the original argmax rule, including +lexical tie breaking; ordinal expected value is used for MAE. + +Missing round directories and started requests without a saved terminal remain +in the plan. Three rounds have 231 unique tasks and 693 planned observations. +The ordered CLI paths define the round plan; the producer saves one directory per +round. A saved `metadata.round`, when present, must match that ordered round; +the remaining model/configuration metadata must agree between available rounds. +The resulting `summary.json` and full-plan `records.jsonl` carry original raw +pointers and hashes. Quality output uses null timestamps/latencies; it does not +report speed. The raw clock is POST through body decoding, JSON and schema validation, +and the saved response is decoded HTTPX text, not compressed wire bytes. + +Ticket quality needs the existing System1-Agents environment, including openJiuwen. +Public231 does not need that framework. Use the frozen ticket producer r3 preparation +and the **whole** measurement root, including unsuccessful or missing cells: + +```bash +python -m benchmarks.quality_ticket \ + --preparation /absolute/path/to/ticket-gpu-prepare/r3 \ + --run /absolute/path/to/measurement-root \ + --out /absolute/path/to/new-ticket-quality-output +``` + +This version accepts the frozen `open-9b`, `open-27b` and `laya-english` cells, +seed0, one episode, batch30, maxsteps30, timeout300 and decision deadline120. +It reuses the original fixture/environment and choice validator, checks every +POST/retry against the current public ticket and queues, and follows original +validated slot → accepted act → committed environment step → returned job. +Decisions and actions must occur inside the original `Runner.run_agent` call; +cleanup/release, episode return, annotate/close, write_job and stop must follow +the original caller's order. Laya-served response provenance is checked with the +original model's response augmentation or captured health/routing fallback; +other wire fields remain exact comparisons. +Labels come only from the fixed fixture. Requests, probabilities/confidence, +before/after observations, original job ticks/views, loaded source and cell/asset +identity are checked; archived report scores are diagnostic comparisons. Non-decision +HTTP bodies are also checked for exact saved bytes, hash and viewing text. Captured +worker/frontend executables must match launch paths and recorded asset hashes, and +the ready/after process IDs and maps must remain consistent. +Remote artifact paths are relocated only inside that cell's recorded arm prefix. + +Each cell reports fixture-derived `route_correct / 30`, `accepted_processed / 30`, +`complete_episode`, and `strict_batch_success` (normal complete episode and all +30 routes correct). Timeout/error/missing evidence retain verified prefixes and +the full 30 denominator. `MODEL_NAME` must be empty, rethink off, and original +episode `rethink` must be boolean `false`, with `chat_calls` zero. Route counts +describe individually verified calls under the recorded cell identity. Damaged +cleanup, process/asset cross-checks or framework copies block complete/strict +results without removing those closed routes; a broken ticket association blocks +that ticket. `process_identity_state` reports that separate process/asset check. +`answer` is an explicitly **provisional** normal-type gate; +missing, empty or incomplete framework source captures cannot be complete/strict. +all results require independent review of the actual captured framework source. +Other/unknown result types cannot be complete or strict successes. Exact recorded +asset/source consistency does not establish historical binary build ancestry. + +Both tools require fresh output outside the repository. They detect inconsistent +unsigned archives, not coordinated rewriting of all evidence. No real model quality, +GPU execution or framework normal-type acceptance is established by their CPU mocks. + +The new tests additionally use `OMNI_BENCH_SOURCE`, `PUBLIC231_INPUT_MANIFEST`, and +`TICKET_PREPARATION_ROOT` pointing to those frozen external inputs. When the CPU +environment lacks openJiuwen, ticket tests stub only framework imports and execute +the repository's actual environment/validator; that is separate from real framework +integration. Run the benchmark tests with the same pytest command above. diff --git a/benchmarks/evidence.py b/benchmarks/evidence.py index dff9de5..79174a6 100644 --- a/benchmarks/evidence.py +++ b/benchmarks/evidence.py @@ -88,7 +88,18 @@ def body_for(task, model): return body -def score_record(upstream, identity, task, raw, model, round_number, kind, raw_hash): +def score_record( + upstream, + identity, + task, + raw, + model, + round_number, + kind, + raw_hash, + *, + latency_basis="client_single_request_wall_seconds", +): status, error, probs, source = "ok", None, None, "unavailable" response = raw["response"] code = raw["http_status"] @@ -124,7 +135,7 @@ def score_record(upstream, identity, task, raw, model, round_number, kind, raw_h status_code=code, error=error or scored.get("error"), latency_s=raw["latency_s"], - latency_basis="client_single_request_wall_seconds", + latency_basis=latency_basis, **{key: scored.get(key) for key in ("valid", "strict_valid", "renormalized", "correct", "predicted", "probs")}, expected=task.expected, candidate_count=len(task.labels), diff --git a/benchmarks/quality_public231.py b/benchmarks/quality_public231.py new file mode 100644 index 0000000..df18d15 --- /dev/null +++ b/benchmarks/quality_public231.py @@ -0,0 +1,217 @@ +"""Score the same frozen Omni raw offline with the pinned public231 scorer.""" + +import argparse +import copy +import importlib.util +from pathlib import Path + +from benchmarks.client import decode +from benchmarks.evidence import ( + body_for, + digest, + dump, + load_reference, + outside_repo, + score_record, + summarize, + write_json, +) + +COLLECTOR_SHA256 = "5aa20a2c612b7da4cd2f1c1442d37e77feb9d6b9ada4375b59a08f4cada63044" +RAW_CLOCK = "post_to_body_json_and_schema_validation" + + +def load_collector(path): + # Only this reviewed interface is executable; never import a file selected by raw metadata. + if digest(path.read_bytes()) != COLLECTOR_SHA256: + raise ValueError("Frozen Omni collector hash mismatch") + spec = importlib.util.spec_from_file_location("public231_omni_raw", path) + collector = importlib.util.module_from_spec(spec) + spec.loader.exec_module(collector) + collector.SOURCE_SHA256 = COLLECTOR_SHA256 + return collector + + +def adapt_record(upstream, identity, task, saved, model, number, raw_file): + response = None + if saved["response"] is not None: + try: + response = decode(saved["response"]) + except ValueError: + pass + # invalid_response is the Omni wire verdict, not the pinned scorer's probability verdict. + client_error = saved["error"] if saved["error_kind"] in ("timeout", "transport", "interrupted") else None + raw = dict( + response=response, + response_body=saved["response"], + http_status=saved["status"], + client_error=client_error, + ts=None, + latency_s=None, + ) + row = score_record( + upstream, identity, task, raw, model, number, "measured", saved["response_sha256"], latency_basis=None + ) + row.update( + wire_valid=saved["error_kind"] is None, + wire_error_kind=saved["error_kind"], + raw_record_state=saved["state"], + raw_file=raw_file, + raw_latency_basis=RAW_CLOCK, + ) + return row + + +def verify_inputs(manifest_path, items, dataset): + manifest = decode(manifest_path.read_text(encoding="utf-8")) + path = manifest_path.parent / manifest["output"]["path"] + if path.resolve() != manifest_path.with_name("public231.jsonl").resolve(): + raise ValueError("Frozen input path must be public231.jsonl beside its manifest") + data = path.read_bytes() + if digest(data) != manifest["output"]["sha256"]: + raise ValueError("Frozen input hash mismatch") + if ( + manifest["sources"]["jevbench_pin"] != dataset["jevbench_pin"] + or manifest["sources"]["reference_dataset_hash"] != dataset["dataset_hash"] + ): + raise ValueError("Frozen input reference mismatch") + expected = [] + for identity, task in items: + target = task.expected == "yes" if task.question["type"] == "noul" else task.expected + expected.append( + dict( + id=task.id, + request=body_for(task, None), + expected={"decision": target}, + source={**identity, "jevbench_pin": dataset["jevbench_pin"]}, + labels=task.labels, + provenance=task.provenance, + group=task.group, + reference_expected=task.expected, + ) + ) + if data != "".join(dump(row) + "\n" for row in expected).encode("utf-8"): + raise ValueError("Input order, requests or evaluator fields differ from the pinned dataset") + return data + + +def aggregate(runs, input_manifest, collector, model_config_id): + if not runs or len({path.resolve() for path in runs}) != len(runs) or not model_config_id.strip(): + raise ValueError("Provide unique planned round paths and a model_config_id") + upstream, items, dataset = load_reference() + frozen = verify_inputs(input_manifest, items, dataset) + tasks = [task for _, task in items] + rounds, all_rows, pooled_tasks, attempted_pooled = [], [], [], [] + binding = None + for number, run in enumerate(runs, 1): + records, source, config, started_ids = {}, None, None, set() + if run.exists(): + if (run / "requests.jsonl").read_bytes() != frozen: + raise ValueError("Omni requests differ from the complete frozen 231 plan") + config = decode((run / "config.json").read_text(encoding="utf-8")) + if config["runner_sha256"] != COLLECTOR_SHA256 or config["latency_basis"] != RAW_CLOCK: + raise ValueError("Omni runner or clock differs from the frozen interface") + identity = {key: value for key, value in config.items() if key not in ("python", "httpx")} + metadata = dict(config.get("metadata", {})) + recorded_round = metadata.pop("round", None) + if "round" in config.get("metadata", {}) and (type(recorded_round) is not int or recorded_round != number): + raise ValueError("Recorded round differs from the ordered planned round") + identity["metadata"] = metadata + if binding is not None and identity != binding: + raise ValueError("Model configuration differs between planned rounds") + binding = identity + collector.summarize_saved(run) # raw/config/response hashes, typed decode and complete plan associations + completion = decode((run / "completion.json").read_text(encoding="utf-8")) + started_ids = set(completion["attempted_ids"]) + source = dict( + run=str(run.resolve()), + config_sha256=completion["config_sha256"], + manifest_sha256=completion["manifest_sha256"], + responses_sha256=completion["responses_sha256"], + stop_reason=completion["stop_reason"], + phase=config["phase"], + recorded_round=recorded_round, + ) + if (run / "responses.jsonl").exists(): + records = { + row["id"]: row + for row in ( + decode(line) for line in (run / "responses.jsonl").read_text(encoding="utf-8").splitlines() + ) + } + rows, attempted = [], [] + for identity, task in items: + if task.id in records: + row = adapt_record( + upstream, identity, task, records[task.id], config["model"], number, str(run / "responses.jsonl") + ) + attempted.append(row) + else: + started = task.id in started_ids + row = dict( + identity, + round=number, + attempted=started, + ok=False, + valid=False, + strict_valid=False, + renormalized=False, + correct=False, + predicted=None, + probs=None, + status="started_without_saved_terminal" if started else "unexecuted", + expected=task.expected, + model=None, + model_configured=config["model"] if config else None, + probs_source="unavailable", + raw_file=None, + raw_sha256=None, + latency_s=None, + latency_basis=None, + ts=None, + ) + if started: + attempted.append(row) + rows.append(dict(row, model_config_id=model_config_id, planned_run=str(run.resolve()))) + clone = copy.copy(task) + clone.id = f"{number}:{task.id}" + clone.group = f"{number}:{task.group}" if task.group else None + pooled_tasks.append(clone) + all_rows.extend(rows) + attempted_pooled.extend(dict(row, task_id=f"{number}:{row['task_id']}") for row in attempted) + rounds.append(dict(round=number, source=source, **summarize(upstream, tasks, attempted))) + result = dict( + schema="public231-omni-offline-quality-v1", + model_config_id=model_config_id, + distinct_tasks=231, + rounds=rounds, + pooled=summarize(upstream, pooled_tasks, attempted_pooled), + dataset=dataset, + input_manifest_sha256=digest(input_manifest.read_bytes()), + model_binding=binding, + identity_basis="saved metadata; build ancestry not independently proven", + latency_reported=False, + raw_latency_basis=RAW_CLOCK, + evidence_complete=all(row["source"] and row["source"]["stop_reason"] == "completed" for row in rounds), + ) + return result, all_rows + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--run", type=Path, action="append", required=True, help="one path for each planned round") + parser.add_argument("--input-manifest", type=Path, required=True) + parser.add_argument("--collector", type=Path, required=True, help="frozen Omni bench.py; only imported offline") + parser.add_argument("--model-config-id", required=True) + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + result, records = aggregate(args.run, args.input_manifest, load_collector(args.collector), args.model_config_id) + output = outside_repo(args.out) + output.mkdir(parents=True, exist_ok=False) + write_json(output / "summary.json", result) + (output / "records.jsonl").write_text("".join(dump(row) + "\n" for row in records), encoding="utf-8") + print(dump(result)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/quality_ticket.py b/benchmarks/quality_ticket.py new file mode 100644 index 0000000..0e56d45 --- /dev/null +++ b/benchmarks/quality_ticket.py @@ -0,0 +1,887 @@ +"""Independently verify frozen ticket-router raw evidence offline, with all 3 x 30 cases.""" + +import argparse +import asyncio +import base64 +import importlib.util +import json +from pathlib import Path +from urllib.parse import urlsplit + +from benchmarks.client import decode +from benchmarks.evidence import digest, dump, outside_repo, write_json + +BINDING_SHA256 = "8bd28a3624842cdaf7fcd35dac187dc0378f029d559daa2f870ed3ca4d7a573a" +SCHEMA_SHA256 = "34b2014a5f62db9801a52eab7831de6ffa4ef6af0cd3ce3bde8cb5b75b7499af" +CELLS = { + "open-9b": ("jev", "open-jev-9b"), + "open-27b": ("jev", "open-jev-27b-v1.1"), + "laya-english": ("laya-served", "english"), +} +NORMAL_BASIS = "answer_only_provisional_pending_actual_framework_source_verification" + + +def read_json(path): + return decode(path.read_text(encoding="utf-8")) + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def load_preparation(path): + require(digest((path / "binding.json").read_bytes()) == BINDING_SHA256, "Frozen producer binding hash mismatch") + require(digest((path / "schema-r1.md").read_bytes()) == SCHEMA_SHA256, "Frozen producer schema hash mismatch") + binding = read_json(path / "binding.json") + source = path / "evidence.py" + require( + digest(source.read_bytes()) == binding["source_sha256"]["evidence.py"], "Frozen journal parser hash mismatch" + ) + spec = importlib.util.spec_from_file_location("ticket_raw_journal", source) + producer = importlib.util.module_from_spec(spec) + spec.loader.exec_module(producer) + # Imports below use the current checkout, whose relevant contracts must match this producer. + current = Path(__file__).resolve().parents[1] + for name in ( + "s1a/agents/ticket_router.py", + "s1a/decision_models/validation.py", + "s1a/decision_models/types.py", + "s1a/decision_models/jev.py", + "s1a/decision_models/laya.py", + "s1a/decision_models/served.py", + "s1a/tool/models.py", + "s1a/tool/loop.py", + "s1a/jobs.py", + ): + require( + digest((current / name).read_bytes()) == binding["source_sha256"]["runtime/" + name], + f"Current contract differs from frozen producer: {name}", + ) + fixture = path / "runtime/s1a/agents/_data/ticket_router_eval.jsonl" + require(digest(fixture.read_bytes()) == binding["fixture_sha256"], "Frozen ticket fixture hash mismatch") + require( + digest((path / "planned-cases.jsonl").read_bytes()) == binding["planned_cases_sha256"], + "Frozen ticket plan hash mismatch", + ) + return path, binding, producer + + +def body_bytes(record): + raw = base64.b64decode(record["base64"], validate=True) + require(len(raw) == record["bytes"] and digest(raw) == record["sha256"], "HTTP body length/hash mismatch") + require(raw.decode("utf-8", errors="replace") == record["utf8"], "HTTP body viewing text mismatch") + return raw + + +def bytes_json(record): + return decode(body_bytes(record)) + + +def relocated(path, remote_arm, arm): + suffix = Path(path).relative_to(Path(remote_arm)) + local = (arm / suffix).resolve() + require(local.is_relative_to(arm.resolve()), "Original artifact path escapes current cell") + return local + + +def verify_identity(root, arm, item, binding): + identity = read_json(arm / "identity.json") + require( + identity["benchmark_run_id"] == root.name and identity["planned_model"] == arm.name, + "Current run/cell identity mismatch", + ) + require( + identity["requested_model"] == CELLS[arm.name][1] + and identity["identity_state"] == "current_exact_assets_recorded", + "Actual model identity unavailable", + ) + assets_hash = digest(json.dumps(identity["actual_assets"], sort_keys=True, ensure_ascii=False).encode("utf-8")) + require(assets_hash == identity["model_identity_sha256"], "Actual asset identity hash mismatch") + require(identity["source_binding_sha256"] == BINDING_SHA256, "Cell source binding mismatch") + require(identity["job_id"] == read_json(root / "status.json")["job_id"], "Job identity mismatch") + python = read_json(arm / "python-identity.json") + loaded = read_json(arm / "python-loaded-after.json") + require( + python["actual_s1a_root"] == str(Path(python["expected_s1a_root"]) / "s1a"), + "Actual loaded Agents root differs from requested source", + ) + for modules in (python["modules"], loaded): + for name, row in modules.items(): + if name == "s1a" or name.startswith("s1a."): + suffix = Path(row["path"]).relative_to(python["expected_s1a_root"]) + require( + row["sha256"] == binding["source_sha256"].get("runtime/" + suffix.as_posix()), + f"Loaded source hash mismatch: {name}", + ) + for name in ("s1a.agents.ticket_router", "s1a.decision_models.validation", "s1a.tool.models", "s1a.tool.loop"): + require(name in modules, f"Missing actual loaded source: {name}") + env = python["environment"] + require( + env["MODEL_NAME"] == "" and env["S1A_HOME"] == str(Path(item["arm"]) / "s1a-home"), + "Chat model must be empty and S1A_HOME must belong to this cell", + ) + require( + env["S1A_DECISION_TIMEOUT_S"] == "120" and env["LAYA_SERVED_TIMEOUT_S"] == "120", + "Actual decision deadline differs from 120s", + ) + model_field = "LAYA_SERVED_MODEL" if arm.name == "laya-english" else "TYPESAFE_MODEL" + require(env[model_field] == identity["requested_model"], "Actual request model environment mismatch") + return identity, python + + +def verify_processes(arm, identity): + launch = read_json(arm / "launch.json") + assets = identity["actual_assets"] + binaries = assets["binaries"] if arm.name == "laya-english" else assets["binary_sha256"] + ready = read_json(arm / "process-ready.json") + after = read_json(arm / "process-after.json") + for phase, capture in (("ready", ready), ("after", after)): + for role in ("worker", "frontend"): + process = capture[role] + require( + process["exe"] == launch[role] + and process["argv"] == [launch[role]] + and process["exe_sha256"] == binaries[launch[role]], + f"Actual {phase} {role} executable differs from launch/assets", + ) + require( + type(process["pid"]) is int and process["pid"] > 0 and process["pid"] == ready[role]["pid"], + f"Actual {role} process changed during episode", + ) + maps = (arm / f"{role}-{phase}.maps").read_bytes() + require(digest(maps) == process["maps_sha256"], f"Actual {phase} {role} maps hash mismatch") + mapped = { + line.split(maxsplit=5)[5] for line in maps.decode().splitlines() if len(line.split(maxsplit=5)) == 6 + } + require( + process["exe"] in mapped and set(process["loaded_libraries"]).issubset(mapped), + f"Actual {phase} {role} executable/libraries absent from captured maps", + ) + + +def verify_decision(events, observation, queues, rules, identity, choice_faults, *, journal=None): + slots = [row for row in events if row["kind"] == "slot_start"] + require(len(slots) == 1, "Decision must have exactly one original slot start") + slot = slots[0] + context = slot["context"] + require( + all( + {key: value for key, value in row["context"].items() if key != "selected_key"} == context for row in events + ), + "Decision context/environment instance differs across original calls", + ) + require( + slot["observation"] == observation and slot["candidates"] == queues, "Slot current ticket/candidates mismatch" + ) + terminals = {row.get("start_event"): row for row in events if row["kind"].endswith("_terminal")} + posts = [row for row in events if row["kind"] == "http_start" and row["method"] == "POST"] + require( + all(type(row["post_attempt_index"]) is int for row in posts) + and [row["post_attempt_index"] for row in posts] == list(range(1, len(posts) + 1)), + "POST retry sequence mismatch", + ) + expected_request = dict( + model=identity["requested_model"], + state=observation, + questions={"pick": {"type": "choice", "criteria": queues, "instructions": {"rules": rules}}}, + ) + replies = [] + for post in posts: + url = urlsplit(post["url"]) + require( + post["category"] == "decision_post" + and url.hostname in ("127.0.0.1", "localhost") + and url.port == 18188 + and url.path == "/v1/systemone", + "Unexpected decision POST destination", + ) + require( + dump(bytes_json(post["request_body"])) == dump(expected_request), + "Actual POST differs from current public ticket/queues/rules/model", + ) + terminal = terminals[post["event_index"]] + if terminal["status"] == "response": + # HTTP errors are still decoded/hash checked; non-JSON error bodies remain diagnostic. + try: + payload = bytes_json(terminal["response_body"]) + except (UnicodeError, json.JSONDecodeError): + require(not 200 <= terminal["http_status"] < 300, "Successful response is not JSON") + continue + if 200 <= terminal["http_status"] < 300: + replies.append((post, terminal, payload)) + slot_result = terminals[slot["event_index"]] + if slot_result["status"] != "returned" or not slot_result.get("new_decisions"): + return None + require( + slot_result.get("state_error") is None and len(slot_result["new_decisions"]) == 1, + "Returned slot is blocked or has multiple decisions", + ) + tick = slot_result["new_decisions"][0] + validations = [row for row in events if row["kind"] == "decision_validation_start"] + require(len(validations) == 1, "Missing or duplicate original validation start") + validation = validations[0] + require(validation["observation"] == {"state": observation, "images": []}, "Validator observation mismatch") + require( + validation["questions"] == {"pick": {"options": queues, "rules": [rules], "goal": "", "operation": ""}}, + "Validator offered question mismatch", + ) + result = terminals[validation["event_index"]] + require(result["status"] == "returned" and replies, "No successful POST for original validated decision") + validated = result["result"] + answer = validated["answers"]["pick"] + faults = choice_faults( + {"choice": answer["key"], "probabilities": answer["probabilities"], "confidence": answer["confidence"]}, + list(queues), + ) + require(not faults, "Invalid original validated probabilities/confidence: " + "; ".join(faults)) + payload = replies[-1][2] + wire = payload["answers"]["pick"] + require( + wire.get("type") == "choice" and not choice_faults(wire, list(queues)), + "Invalid wire choice probabilities/confidence", + ) + require( + (wire["choice"], wire["probabilities"], wire["confidence"]) + == (answer["key"], answer["probabilities"], answer["confidence"]), + "Wire and original validation result differ", + ) + expected_raw = payload + if identity["requested_model"] == "english": + from s1a.decision_models.served import served_by_from_health + + sent = payload.get("served_by") + if isinstance(sent, dict): + served_by = {**sent, "source": "response"} + else: + history = events if journal is None else journal + starts = {row["event_index"]: row for row in history if row["kind"] == "http_start"} + health = {} + for row in history: + if row["event_index"] >= replies[-1][0]["event_index"]: + break + start = starts.get(row.get("start_event"), {}) + url = urlsplit(start.get("url", "")) + if ( + row["kind"] == "http_terminal" + and row.get("status") == "response" + and row.get("http_status") == 200 + and start.get("method") == "GET" + and url.hostname in ("127.0.0.1", "localhost") + and url.port == 18188 + and url.path == "/health" + ): + try: + reading = bytes_json(row["response_body"]) + except (UnicodeError, json.JSONDecodeError): + continue + if isinstance(reading, dict) and reading: + health = reading + provenance = validated["raw"]["served_by"] + read_at = provenance.get("read_at") + require(read_at is None or isinstance(read_at, str), "Invalid original health reading timestamp") + served_by = served_by_from_health(health, payload.get("routing"), read_at) + expected_raw = {**payload, "served_by": served_by} + require( + all(validated["raw"].get(key) == value for key, value in expected_raw.items()), + "Validated raw differs from actual HTTP reply", + ) + require( + tick["key"] == answer["key"] + and tick["probabilities"] == answer["probabilities"] + and tick["confidence"] == round(answer["confidence"], 3) + and tick["model"] == validated["model"], + "Slot tick differs from original validation result", + ) + require( + slot["event_index"] < validation["event_index"] < posts[0]["event_index"] + and replies[-1][1]["event_index"] < result["event_index"] < slot_result["event_index"], + "Decision/POST/validation/slot event order mismatch", + ) + return tick, context, slot_result + + +async def score_cell(root, arm, item, binding, producer, fixture, TicketRouterEnv, queues, rules, choice_faults): + identity, python = verify_identity(root, arm, item, binding) + events, integrity = producer.read_journal(arm / "journal.jsonl", identity) + errors = [f"journal: {row['error']}" for row in integrity] + # Pair/identity faults affect their events; a broken cleanup cannot erase closed ticket evidence. + damaged_events = {row[key] for row in integrity for key in ("event_index", "start_event") if key in row} + malformed_lines = [row["line"] for row in integrity if row["code"] == "invalid_json_event"] + if events and any(row["code"] == "missing_final_newline" for row in integrity): + damaged_events.add(events[-1]["event_index"]) + terminals = {row.get("start_event"): row for row in events if row["kind"].endswith("_terminal")} + env = TicketRouterEnv(0, tickets=fixture, batch_size=30) + await env.reset() + planned_ids = env.report()["ticket_ids"] + labels = {row["id"]: row["label"] for row in fixture} + observations = [] + for ticket_id in planned_ids: + observations.append(await env.observe()) + await env.step(labels[ticket_id]) + observations.append(await env.observe()) + planned = [ + dict( + suite="ticket_router", + episode_seed=0, + sequence=index + 1, + ticket_id=ticket_id, + public_ticket=observations[index]["ticket"], + public_ticket_sha256=digest(dump(observations[index]["ticket"]).encode("utf-8")), + expected_next_ticket_id=planned_ids[index + 1] if index < 29 else None, + planned_denominator=30, + ) + for index, ticket_id in enumerate(planned_ids) + ] + initial = read_json(arm / "planned-initial.json") + actual_plan = read_json(arm / "planned-cases.json") + require( + initial["planned_denominator"] == actual_plan["planned_denominator"] == 30 and initial["cases"] == planned, + "Prebuilt initial plan differs from fixture-derived seed0 order/public fields", + ) + require( + initial["identity"]["benchmark_run_id"] == root.name + and initial["identity"]["planned_model"] == arm.name + and initial["identity"]["identity_state"] == "planned_contract_only", + "Prebuilt plan identity differs from this run/cell", + ) + require( + actual_plan["identity"] + == {key: identity[key] for key in ("benchmark_run_id", "model_identity_sha256")} | {"record": "identity.json"}, + "Current actual plan identity mismatch", + ) + require( + actual_plan["cases"] + == [ + { + **case, + "logical_key": [root.name, identity["model_identity_sha256"], "ticket_router", 0, case["ticket_id"]], + } + for case in planned + ], + "Current actual plan/logical keys mismatch", + ) + cases = [ + dict( + ticket_id=ticket_id, + planned_sequence=index + 1, + expected=labels[ticket_id], + attempted=False, + post_attempts=0, + accepted=False, + correct=False, + predicted=None, + errors=[], + ) + for index, ticket_id in enumerate(planned_ids) + ] + decisions = {} + for row in events: + context = row.get("context") + if context is None: + if row["kind"] in ("http_start", "http_terminal"): + try: + for field in ("request_body", "response_body"): + if field in row: + body_bytes(row[field]) + except (KeyError, TypeError, ValueError) as error: + errors.append(f"Non-decision HTTP evidence: {error}") + if row["kind"] == "http_start" and row["method"] == "POST": + errors.append("POST outside original decision context; zero-chat provenance insufficient") + continue + ticket_id = context.get("case_id") + if ticket_id not in planned_ids: + errors.append("Unknown current ticket in journal context") + continue + index = planned_ids.index(ticket_id) + expected_key = [root.name, identity["model_identity_sha256"], "ticket_router", 0, ticket_id] + if ( + context.get("logical_key") != expected_key + or any( + type(context.get(key)) is not int + for key in ("decision_index", "planned_sequence", "env_index", "episode_seed") + ) + or context.get("decision_index", 0) <= 0 + or context.get("planned_sequence") != index + 1 + or context.get("env_index") != index + or context.get("suite") != "ticket_router" + or context.get("episode_seed") != 0 + ): + cases[index]["errors"].append("Current case/seed/logical identity mismatch") + decisions.setdefault(context.get("decision_index"), []).append(row) + verified_ticks, verified_acts = [], [] + for related in decisions.values(): + context = related[0]["context"] + ticket_id = context.get("case_id") + index = planned_ids.index(ticket_id) + case = cases[index] + case["attempted"] = True + case["post_attempts"] += sum(row["kind"] == "http_start" and row["method"] == "POST" for row in related) + try: + require( + not case["errors"] + and not any(row["event_index"] in damaged_events for row in related) + and not any(line <= max(row["event_index"] for row in related) for line in malformed_lines), + "Journal/current case integrity unavailable", + ) + verified = verify_decision( + related, observations[index], queues, rules, identity, choice_faults, journal=events + ) + acts = [row for row in related if row["kind"] == "act_start"] + steps = [row for row in related if row["kind"] == "env_step_start"] + if verified is None: + require(not acts and not steps, "No valid slot decision for executed act/step") + continue + tick, slot_context, slot_result = verified + verified_ticks.append(tick) + if not acts: + require(not steps, "Environment step without original act") + continue + require(len(acts) == len(steps) == 1 and not case["accepted"], "Duplicate act/step for planned ticket") + act, step = acts[0], steps[0] + act_result, step_result = terminals[act["event_index"]], terminals[step["event_index"]] + key = tick["key"] + act_context = {**slot_context, "selected_key": key} + require( + act["context"] == step["context"] == act_context + and act["inputs"] == {"key": key} + and act["association_matches"] is True + and step["key"] == key, + "Selected key/accepted act association mismatch", + ) + require( + act_result["status"] == step_result["status"] == "returned" + and act_result["act_calls"] == [{"key": key, "accepted": True}], + "Original act/step was not accepted", + ) + require( + slot_result["event_index"] + < act["event_index"] + < step["event_index"] + < step_result["event_index"] + < act_result["event_index"], + "Original act/step event order mismatch", + ) + require( + step["before"] == dict(index=index, done=False, observation=observations[index], candidates=queues) + and step_result["after"] + == dict( + index=index + 1, + done=index == 29, + observation=observations[index + 1], + candidates={} if index == 29 else queues, + ), + "Original environment step before/after differs from seeded fixture", + ) + for view, expected_index in ((act["before"], index), (act_result["after"], index + 1)): + require( + view["state"] == observations[expected_index] + and view["done"] == (expected_index == 30) + and view["candidates"] == ({} if expected_index == 30 else queues), + "Original act view differs from seeded fixture", + ) + require(decode(act_result["raw_tool_result"]) == act_result["after"], "Original act output/view mismatch") + case.update( + accepted=True, + correct=key == labels[ticket_id], + predicted=key, + accepted_event=act_result["event_index"], + step_event=step_result["event_index"], + ) + verified_acts.append((case, act, act_result)) + except (AttributeError, KeyError, TypeError, ValueError, OverflowError) as error: + case["errors"].append(str(error)) + for case in cases: + errors.extend(f"{case['ticket_id']}: {error}" for error in case["errors"]) + process_identity_state = "verified" + try: + verify_processes(arm, identity) + except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: + process_identity_state = "invalid" + errors.append(str(error)) + # Partial/timeout episodes retain independently verified accepted prefixes. + complete, result_type, job_source = False, None, None + try: + writes = [ + row + for row in events + if row["kind"] == "lifecycle_terminal" + and row.get("stage") == "write_job" + and row.get("status") == "returned" + ] + require(len(writes) == 1, "Missing unique actual write_job result") + job = relocated(writes[0]["result"], item["arm"], arm) + require( + job.is_relative_to((arm / "s1a-home/evals/results/ticket_router").resolve()), + "Original job outside current cell", + ) + paths = list(job.glob("*/agent/episode.json")) + require(len(paths) == 1, "Missing unique original episode in actual returned job") + path = paths[0] + episode, trial, config = ( + read_json(path), + read_json(path.parents[1] / "result.json"), + read_json(job / "config.json"), + ) + for name in ("result.json", "summary.json"): + read_json(job / name) # Existence/JSON only: score and DONE are not an oracle. + job_source = dict( + job=str(job), + original_path=writes[0]["result"], + episode_sha256=digest(path.read_bytes()), + trial_sha256=digest((path.parents[1] / "result.json").read_bytes()), + ) + require( + config["agents"] == [{"name": "s1a-evals/" + CELLS[arm.name][0], "model_name": CELLS[arm.name][0]}] + and config["datasets"] == [{"name": "ticket_router", "n_tasks": 1}], + "Original job policy/plan mismatch", + ) + require( + trial["task_name"] == "ticket_router/0" and trial["source"] == "ticket_router", + "Original trial seed/task mismatch", + ) + extra, metadata = episode["extra"], trial["agent_result"]["metadata"] + result_type = extra.get("result_type") + require( + extra["rethink"] is False + and not extra.get("rethinks") + and type(metadata["chat_calls"]) is int + and metadata["chat_calls"] == 0, + "Original episode must have rethink off and zero chat", + ) + require(episode["decisions"] == verified_ticks, "Original job decisions differ from verified slot ticks") + accepted = [case for case in cases if case["accepted"]] + report = extra["ticket_router"] + expected_routes = [ + { + "id": case["ticket_id"], + "expected": case["expected"], + "predicted": case["predicted"], + "correct": case["correct"], + } + for case in accepted + ] + require( + report["seed"] == 0 + and report["batch_sha256"] == binding["batch_sha256"] + and report["total"] == 30 + and report["processed"] == len(accepted) + and report["ticket_ids"] == planned_ids + and report["unprocessed_ids"] == planned_ids[len(accepted) :] + and report["routes"] == expected_routes + and report["correct"] == sum(case["correct"] for case in accepted), + "Original report differs from fixture-derived routes/labels/full denominator", + ) + require( + [case["planned_sequence"] for case in accepted] == list(range(1, len(accepted) + 1)), + "Accepted routes are not a prefix", + ) + views = episode["views"] + require( + len(views) == len(accepted) + 1 and episode["final_state"] == observations[len(accepted)], + "Original job views/final state mismatch", + ) + for index, view in enumerate(views): + require( + view["state"] == observations[index] + and view["candidates"] == ({} if index == 30 else queues) + and view["done"] == (index == 30) + and view["score"] == sum(case["correct"] for case in accepted[:index]), + "Original job view differs from independently replayed prefix", + ) + for index, (_, act, act_result) in enumerate(verified_acts): + require( + act["before"] == views[index] and act_result["after"] == views[index + 1], + "Original job/act views mismatch", + ) + require( + metadata["steps"] == len(accepted) + and metadata["decisions"] == len(verified_ticks) + and metadata["invalid_keys"] == 0, + "Original job action counts mismatch", + ) + lifecycle = [row for row in events if row["kind"] == "lifecycle_terminal"] + required = { + stage: [row for row in lifecycle if row.get("stage") == stage] for stage in producer.REQUIRED_LIFECYCLES + } + require( + all(len(rows) == 1 and rows[0]["status"] == "returned" for rows in required.values()), + "Original lifecycle incomplete or abnormal", + ) + require( + required["Runner.run_agent"][0].get("result_type") == result_type + and required["Runner.stop"][0]["event_index"] > writes[0]["event_index"], + "Original runner result/stop mismatch", + ) + runner = required["Runner.run_agent"][0] + require( + all( + runner["start_event"] < row["event_index"] < runner["event_index"] + for row in events + if row.get("context") is not None + ), + "Decision/act/environment evidence outside original Runner.run_agent", + ) + resets = [row for row in events if row["kind"] == "env_reset_terminal"] + require(len(resets) == 1, "Original episode must have exactly one environment reset") + cli = [row for row in events if row["kind"] == "cli_start"] + require(len(cli) == 1, "Missing unique original CLI start") + ordered = [cli[0]["event_index"]] + for stage in ("Runner.start", "build_model", "make_series", "DecisionModel.warm", "TicketRouterEnv.__init__"): + row = required[stage][0] + ordered.extend((row["start_event"], row["event_index"])) + ordered.extend((required["run_episode"][0]["start_event"], resets[0]["start_event"], resets[0]["event_index"])) + for stage in ( + "DeepAgent.ensure_initialized", + "Runner.run_agent", + "DeepAgent.cleanup_task_resources", + "ability_manager.teardown_tools", + "Runner.release", + ): + row = required[stage][0] + ordered.extend((row["start_event"], row["event_index"])) + ordered.append(required["run_episode"][0]["event_index"]) + for stage in ("annotate", "DecisionModel.close", "write_job", "Runner.stop"): + row = required[stage][0] + ordered.extend((row["start_event"], row["event_index"])) + require(all(left < right for left, right in zip(ordered, ordered[1:])), "Original lifecycle order mismatch") + processes = [row for row in events if row["kind"] == "process_terminal"] + expected_argv = [ + "s1a", + "run", + "ticket_router", + "--model", + CELLS[arm.name][0], + "--rethink", + "off", + "--episodes", + "1", + "--seed", + "0", + "--max-steps", + "30", + "--timeout", + "300", + "--dataset", + str(Path(python["expected_s1a_root"]) / "s1a/agents/_data/ticket_router_eval.jsonl"), + "--batch-size", + "30", + "--log", + ] + require(cli[0]["argv"] == expected_argv, "Actual CLI plan/budget differs from frozen contract") + require( + len(processes) == 1 + and events[-1] == processes[0] + and processes[0].get("state") == "returned" + and processes[0].get("exit_code") == 0 + and processes[0].get("stage") == "original_cli_entry" + and not processes[0].get("error") + and not processes[0].get("received_signal") + and item.get("collector_exit_code") == 0, + "Original process/collector did not complete normally", + ) + require( + not any( + row["kind"] == "process_signal" or (row["kind"] != "http_terminal" and row.get("status") == "exception") + for row in events + ), + "Abnormal non-HTTP termination", + ) + require( + trial.get("exception_info") is None and result_type == "answer", + "Non-normal or unknown original result_type", + ) + require(len(accepted) == 30 and not errors, "Incomplete accepted coverage or evidence errors") + complete = True + except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: + errors.append(str(error)) + try: + framework = framework_sources(arm, item["arm"]) + if framework["state"] != "verified": + errors.append("Actual framework source capture missing or incomplete") + complete = False + except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: + framework = {"state": "invalid", "record": str(arm / "framework-result-type-source.json")} + errors.append(str(error)) + complete = False + correct = sum(case["correct"] for case in cases if case["accepted"]) + processed = sum(case["accepted"] for case in cases) + return dict( + cell=arm.name, + planned_routes=30, + planned_episodes=1, + cases=cases, + route_correct=correct, + route_accuracy=correct / 30, + route_evidence_basis="recorded cell identity and individually verified HTTP/slot/accepted act/environment step", + accepted_processed=processed, + accepted_coverage=processed / 30, + complete_episode=complete, + strict_batch_success=complete and correct == 30, + result_type=result_type, + normal_result_type_basis=NORMAL_BASIS, + quality_status="PROVISIONAL" if complete else "INCOMPLETE", + required_input="independent review of actual loaded framework result_type source", + framework_sources=framework, + identity=identity, + identity_basis="exact recorded assets; source-to-build ancestry unverified", + process_identity_state=process_identity_state, + original_job=job_source, + errors=errors, + ) + + +def framework_sources(arm, remote_arm): + path = arm / "framework-result-type-source.json" + if not path.exists(): + return {"state": "missing", "candidates": []} + record = read_json(path) + loaded = read_json(arm / "python-loaded-after.json") + for row in record["candidates"]: + if "capture_error" in row: + continue + raw = relocated(row["copy"], remote_arm, arm).read_bytes() + require( + len(raw) == row["bytes"] and digest(raw) == row["sha256"] == row["loaded_module_record_sha256"], + "Actual framework source copy hash mismatch", + ) + require( + loaded.get(row["module"]) == {"path": row["path"], "sha256": row["sha256"]}, + "Framework source capture differs from actually loaded module", + ) + record["state"] = ( + "verified" + if record["candidates"] and not any("capture_error" in row for row in record["candidates"]) + else "incomplete" + ) + return record # Exact files are supplied for independent enum review, not interpreted as normal here. + + +def aggregate(root, preparation): + try: + from s1a.agents.ticket_router import QUEUES, RULES, TicketRouterEnv, load_tickets + from s1a.decision_models.validation import choice_faults + except ModuleNotFoundError as error: + raise RuntimeError( + "Ticket scoring needs the existing System1-Agents environment (including openjiuwen); " + "public231 scoring is independent of that dependency" + ) from error + prep, binding, producer = preparation + current = Path(__file__).resolve().parents[1] + require( + Path(TicketRouterEnv.__init__.__code__.co_filename).resolve() == current / "s1a/agents/ticket_router.py" + and Path(choice_faults.__code__.co_filename).resolve() == current / "s1a/decision_models/validation.py", + "Offline scorer imported a different Agents contract root", + ) + fixture = load_tickets(prep / "runtime/s1a/agents/_data/ticket_router_eval.jsonl") + require(len(fixture) == 30 and len({row["id"] for row in fixture}) == 30, "Fixture must contain 30 unique tickets") + frozen_plan = [decode(line) for line in (prep / "planned-cases.jsonl").read_text(encoding="utf-8").splitlines()] + cells, root_errors, items = [], [], {} + try: + status = read_json(root / "status.json") + require( + status["planned_models"] == list(CELLS) and status["planned_tickets_per_model"] == 30, + "Measurement full planned cells/denominator mismatch", + ) + require( + len(status["models"]) == 3 and {item["name"] for item in status["models"]} == set(CELLS), + "Measurement duplicate or unknown cell", + ) + require( + (root / "binding.json").read_bytes() == (prep / "binding.json").read_bytes(), + "Measurement binding differs from frozen producer", + ) + source = read_json(root / "source.json") + require( + source + == { + "binding_sha256": BINDING_SHA256, + "files": binding["source_sha256"], + "runtime_revision": binding["agents_revision"], + }, + "Measurement source identity mismatch", + ) + items = {item["name"]: item for item in status["models"]} + except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: + root_errors.append(str(error)) + for name in CELLS: + try: + require(not root_errors and name in items, "Missing or invalid measurement root/cell identity") + cell = asyncio.run( + score_cell( + root, + root / name, + items[name], + binding, + producer, + fixture, + TicketRouterEnv, + QUEUES, + RULES, + choice_faults, + ) + ) + except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: + cases = [ + dict( + ticket_id=case["ticket_id"], + planned_sequence=case["sequence"], + attempted=False, + accepted=False, + correct=False, + predicted=None, + errors=[str(error)], + ) + for case in frozen_plan + ] + cell = dict( + cell=name, + planned_routes=30, + planned_episodes=1, + cases=cases, + route_correct=0, + route_accuracy=0.0, + accepted_processed=0, + accepted_coverage=0.0, + complete_episode=False, + strict_batch_success=False, + quality_status="MISSING_OR_INVALID", + errors=[str(error)], + identity=None, + process_identity_state="unavailable", + original_job=None, + normal_result_type_basis=NORMAL_BASIS, + ) + cells.append(cell) + return dict( + schema="ticket-raw-offline-quality-v1", + run=str(root.resolve()), + planned_routes=90, + planned_episodes=3, + cells=cells, + root_errors=root_errors, + latency_reported=False, + route_correct=sum(cell["route_correct"] for cell in cells), + accepted_processed=sum(cell["accepted_processed"] for cell in cells), + complete_episodes=sum(cell["complete_episode"] for cell in cells), + strict_batch_successes=sum(cell["strict_batch_success"] for cell in cells), + producer_binding_sha256=BINDING_SHA256, + normal_result_type_basis=NORMAL_BASIS, + ) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--run", type=Path, required=True, help="whole measurement root, including missing cells") + parser.add_argument("--preparation", type=Path, required=True, help="frozen ticket producer r3 directory") + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + result = aggregate(args.run, load_preparation(args.preparation)) + output = outside_repo(args.out) + output.mkdir(parents=True, exist_ok=False) + write_json(output / "summary.json", result) + print(dump(result)) + + +if __name__ == "__main__": + main() diff --git a/tests/benchmarks/test_quality_public231.py b/tests/benchmarks/test_quality_public231.py new file mode 100644 index 0000000..79359ff --- /dev/null +++ b/tests/benchmarks/test_quality_public231.py @@ -0,0 +1,232 @@ +"""Deterministic saved Omni responses, scored by the pinned public231 oracle.""" + +import copy +import json +import os +from pathlib import Path + +import pytest + +from benchmarks.evidence import digest, dump, load_reference +from benchmarks.quality_public231 import adapt_record, aggregate, load_collector + + +@pytest.fixture +def reference(): + if not os.environ.get("JEVBENCH_ROOT"): + pytest.skip("Set JEVBENCH_ROOT to the pinned reference") + return load_reference() + + +@pytest.fixture +def collector(): + source = os.environ.get("OMNI_BENCH_SOURCE") + if not source: + pytest.skip("Set OMNI_BENCH_SOURCE to the frozen raw collector") + return load_collector(Path(source)) + + +def response_record(answer, *, kind="invalid_response", status=200): + text = dump({"answers": {"decision": answer}, "model": "configured"}) + return dict( + id="unused", + state="completed", + status=status, + response=text, + response_sha256=digest(text.encode()), + answers={}, + error_kind=kind, + error="wire schema rejected" if kind else None, + latency_ms=12.0, + ) + + +def test_wire_rejection_does_not_discard_pinned_distribution(reference): + upstream, items, _ = reference + identity, task = next(row for row in items if row[1].question["type"] == "noul") + raw = response_record({"type": "noul", "probabilities": {"no": 0.5, "yes": 0.505}}) + row = adapt_record(upstream, identity, task, raw, "configured", 1, "fixture") + assert row["valid"] and row["renormalized"] and not row["strict_valid"] + assert row["wire_valid"] is False and row["ok"] is True + assert row["predicted"] == "yes" + assert row["latency_s"] is None and row["latency_basis"] is None and row["ts"] is None + + +def test_scalar_tie_and_label_only(reference): + upstream, items, _ = reference + for value, expected in ((0.5, "no"), (0.8, "yes")): + identity, task = next(row for row in items if row[1].question["type"] == "noul") + row = adapt_record( + upstream, identity, task, response_record({"type": "noul", "noul": value}), "configured", 1, "fixture" + ) + assert row["predicted"] == expected and row["probs_source"] == "native_noul_scalar" + identity, task = next(row for row in items if row[1].question["type"] == "choice") + row = adapt_record( + upstream, + identity, + task, + response_record({"type": "choice", "choice": task.expected}), + "configured", + 1, + "fixture", + ) + assert not row["valid"] and row["probs"] is None + + +def test_ordinal_accuracy_uses_argmax(reference): + upstream, items, _ = reference + identity, original = next(row for row in items if row[1].question["type"] == "score") + task = copy.copy(original) + task.labels, task.expected = ["0", "1", "2"], 0 + probabilities = {"0": 0.4, "1": 0.35, "2": 0.25} + row = adapt_record( + upstream, + identity, + task, + response_record({"type": "score", "score": 0.85, "probabilities": probabilities}), + "configured", + 1, + "fixture", + ) + assert row["predicted"] == "0" and row["correct"] is (task.expected == 0) + + +@pytest.mark.parametrize("kind,status", [("http_422", 422), ("timeout", None)]) +def test_failures_are_scored_as_invalid(reference, kind, status): + upstream, items, _ = reference + identity, task = items[0] + raw = response_record({"type": "choice", "probabilities": {}}, kind=kind, status=status) + if status is None: + raw.update(response=None, response_sha256=None) + row = adapt_record(upstream, identity, task, raw, "configured", 1, "fixture") + assert not row["valid"] and not row["correct"] and row["attempted"] + + +def saved_run(tmp_path, collector, input_path, *, count=231): + cases = collector.load_cases(input_path) + run = tmp_path / "round-1" + run.mkdir() + (run / "requests.jsonl").write_bytes(input_path.read_bytes()) + config = dict( + manifest_sha256=digest(input_path.read_bytes()), + runner_sha256=collector.SOURCE_SHA256, + model="configured", + metadata={"model_revision": "fixture"}, + phase="measured", + warmup_manifest_sha256=None, + latency_basis="post_to_body_json_and_schema_validation", + ) + (run / "config.json").write_text(dump(config)) + records = [] + for case in cases[:count]: + q = case["request"]["questions"]["decision"] + answer = {"type": q["type"]} + if q["type"] == "noul": + answer["noul"] = float(case["expected"]["decision"]) + else: + labels = list(q["criteria"]) if q["type"] == "choice" else list(map(str, range(len(q["criteria"])))) + answer["probabilities"] = {key: float(key == str(case["expected"]["decision"])) for key in labels} + answer[q["type"]] = case["expected"]["decision"] + raw = response_record(answer, kind=None) + raw.update(id=case["id"], answers=collector.read_answers(case, json.loads(raw["response"]))) + records.append(raw) + (run / "responses.jsonl").write_text("".join(dump(row) + "\n" for row in records)) + state = dict( + manifest_sha256=config["manifest_sha256"], + config_sha256=digest((run / "config.json").read_bytes()), + responses_sha256=digest((run / "responses.jsonl").read_bytes()), + attempted_ids=[r["id"] for r in records], + completed_ids=[r["id"] for r in records], + active_ids=[], + started_at=1.0, + finished_at=2.0, + wall_seconds=1.0, + stop_reason="completed" if count == 231 else "interrupted", + ) + (run / "completion.json").write_text(dump(state)) + return run + + +def test_full_plan_missing_round_and_raw_integrity(tmp_path, collector, reference): + manifest = os.environ.get("PUBLIC231_INPUT_MANIFEST") + if not manifest: + pytest.skip("Set PUBLIC231_INPUT_MANIFEST to the frozen input manifest") + manifest = Path(manifest) + inputs = manifest.with_name("public231.jsonl") + run = saved_run(tmp_path, collector, inputs, count=230) + result, records = aggregate( + [run, tmp_path / "missing", tmp_path / "missing-3"], manifest, collector, "fixture-config" + ) + assert result["distinct_tasks"] == 231 and result["pooled"]["planned"] == 693 + assert result["rounds"][0]["accuracy"] == 230 / 231 + assert result["rounds"][1]["accuracy"] == 0 and result["rounds"][1]["coverage"] == 0 + assert len(records) == 693 and sum(row["attempted"] for row in records) == 230 + (run / "responses.jsonl").write_text("{}\n") + with pytest.raises(ValueError, match="checksum"): + aggregate([run], manifest, collector, "fixture-config") + + +def test_source_hash_is_required(tmp_path): + source = tmp_path / "bench.py" + source.write_text("raise AssertionError('must not execute')") + with pytest.raises(ValueError, match="collector hash"): + load_collector(source) + + +def test_recorded_round_identity_is_separate_from_stable_model_config(tmp_path, collector, reference): + import shutil + + manifest = os.environ.get("PUBLIC231_INPUT_MANIFEST") + if not manifest: + pytest.skip("Set PUBLIC231_INPUT_MANIFEST to the frozen input manifest") + manifest = Path(manifest) + first = saved_run(tmp_path, collector, manifest.with_name("public231.jsonl")) + runs = [first, tmp_path / "round-2", tmp_path / "round-3"] + for path in runs[1:]: + shutil.copytree(first, path) + + def set_metadata(path, **values): + config = json.loads((path / "config.json").read_text()) + config["metadata"].update(values) + (path / "config.json").write_text(dump(config)) + completion = json.loads((path / "completion.json").read_text()) + completion["config_sha256"] = digest((path / "config.json").read_bytes()) + (path / "completion.json").write_text(dump(completion)) + + for number, run in enumerate(runs, 1): + set_metadata(run, round=number) + result, records = aggregate(runs, manifest, collector, "fixture-config") + assert result["distinct_tasks"] == 231 and result["pooled"]["planned"] == 693 + assert len(records) == 693 and result["evidence_complete"] + assert [row["source"]["recorded_round"] for row in result["rounds"]] == [1, 2, 3] + with pytest.raises(ValueError, match="Recorded round"): + aggregate([runs[1], runs[0], runs[2]], manifest, collector, "fixture-config") + set_metadata(runs[1], round=True) + with pytest.raises(ValueError, match="Recorded round"): + aggregate(runs, manifest, collector, "fixture-config") + set_metadata(runs[1], round=2, model_revision="other") + with pytest.raises(ValueError, match="Model configuration"): + aggregate(runs, manifest, collector, "fixture-config") + + +def test_started_without_terminal_keeps_attempt_and_full_denominator(tmp_path, collector, reference): + manifest = os.environ.get("PUBLIC231_INPUT_MANIFEST") + if not manifest: + pytest.skip("Set PUBLIC231_INPUT_MANIFEST to the frozen input manifest") + manifest = Path(manifest) + run = saved_run(tmp_path, collector, manifest.with_name("public231.jsonl"), count=0) + _, items, _ = reference + state = json.loads((run / "completion.json").read_text()) + state.update( + attempted_ids=[items[0][1].id], + active_ids=[items[0][1].id], + finished_at=None, + wall_seconds=None, + stop_reason=None, + ) + (run / "completion.json").write_text(dump(state)) + result, rows = aggregate([run], manifest, collector, "fixture-config") + assert rows[0]["attempted"] and rows[0]["status"] == "started_without_saved_terminal" + assert result["rounds"][0]["attempted"] == 1 and result["rounds"][0]["planned"] == 231 + assert result["rounds"][0]["upstream_attempted_metrics"]["brier_mean"] is None + assert not result["evidence_complete"] diff --git a/tests/benchmarks/test_quality_ticket.py b/tests/benchmarks/test_quality_ticket.py new file mode 100644 index 0000000..2681e9d --- /dev/null +++ b/tests/benchmarks/test_quality_ticket.py @@ -0,0 +1,913 @@ +"""Producer-shaped CPU evidence; framework stubs are confined to these tests.""" + +import base64 +import importlib.util +import json +import os +import sys +from pathlib import Path +from types import ModuleType + +import pytest + +from benchmarks.evidence import digest, dump +from benchmarks.quality_ticket import aggregate, load_preparation + + +@pytest.fixture +def preparation(monkeypatch, tmp_path): + path = os.environ.get("TICKET_PREPARATION_ROOT") + if not path: + pytest.skip("Set TICKET_PREPARATION_ROOT to the frozen producer preparation") + # Execute the actual validator and environment, without installing or initializing the framework. + if importlib.util.find_spec("openjiuwen") is None: + for name in ("openjiuwen.core.common.exception.codes", "openjiuwen.core.common.exception.errors"): + module = ModuleType(name) + module.StatusCode = object() + module.BaseError = ValueError + module.build_error = lambda *a, **k: ValueError("framework stub") + monkeypatch.setitem(sys.modules, name, module) + package = ModuleType("s1a.decision_models") + package.__path__ = [str(Path(__file__).resolve().parents[2] / "s1a/decision_models")] + monkeypatch.setitem(sys.modules, "s1a.decision_models", package) + config = ModuleType("s1a.config") + config.HOME = tmp_path + monkeypatch.setitem(sys.modules, "s1a.config", config) + rail = ModuleType("openjiuwen.core.single_agent.rail.base") + rail.AgentCallbackContext = type("AgentCallbackContext", (), {}) + rail.AgentCallbackEvent = type( + "AgentCallbackEvent", (), {"BEFORE_TOOL_CALL": "before", "AFTER_TOOL_CALL": "after"} + ) + monkeypatch.setitem(sys.modules, rail.__name__, rail) + logging = ModuleType("openjiuwen.core.common.logging") + logging.logger = None + monkeypatch.setitem(sys.modules, logging.__name__, logging) + return load_preparation(Path(path)) + + +def body(value): + raw = dump(value).encode() + return dict(bytes=len(raw), sha256=digest(raw), base64=base64.b64encode(raw).decode(), utf8=raw.decode()) + + +@pytest.fixture +def measurement(tmp_path, preparation): + prep, binding, producer = preparation + root = tmp_path / "cpu-ticket-fixture" + root.mkdir() + (root / "binding.json").write_bytes((prep / "binding.json").read_bytes()) + (root / "source.json").write_text( + dump( + dict( + binding_sha256=digest((prep / "binding.json").read_bytes()), + files=binding["source_sha256"], + runtime_revision=binding["agents_revision"], + ) + ) + ) + models = [ + dict(name=name, arm=f"/remote/{root.name}/{name}", collector_exit_code=0) for name in binding["service_order"] + ] + (root / "status.json").write_text( + dump(dict(job_id="123", planned_models=binding["service_order"], planned_tickets_per_model=30, models=models)) + ) + from s1a.agents.ticket_router import PUBLIC_FIELDS, QUEUES, RULES, load_tickets + import random + + rows = load_tickets(prep / "runtime/s1a/agents/_data/ticket_router_eval.jsonl") + rows = [{key: row[key] for key in (*PUBLIC_FIELDS, "label") if key in row} for row in rows] + random.Random(0).shuffle(rows) + name = "open-9b" + arm = root / name + arm.mkdir() + remote = f"/remote/{root.name}/{name}" + assets = { + "binary_sha256": {"/remote/worker": "a" * 64, "/remote/frontend": "c" * 64}, + "export": {"model_id": "fixture"}, + "verified_export_files": {"weights": {"sha256": "b" * 64}}, + } + identity = dict( + benchmark_run_id=root.name, + planned_model=name, + requested_model="open-jev-9b", + job_id="123", + source_binding_sha256=digest((prep / "binding.json").read_bytes()), + actual_assets=assets, + identity_state="current_exact_assets_recorded", + gpu={"name": "CPU mock; no GPU"}, + model_identity_sha256=digest(json.dumps(assets, ensure_ascii=False, sort_keys=True).encode()), + ) + (arm / "identity.json").write_text(dump(identity)) + (arm / "launch.json").write_text(dump(dict(worker="/remote/worker", frontend="/remote/frontend"))) + processes = {} + for role, sha in (("worker", "a" * 64), ("frontend", "c" * 64)): + maps = f"1000-2000 r-xp 00000000 00:00 1 /remote/{role}\n".encode() + processes[role] = dict( + pid=1 if role == "worker" else 2, + exe=f"/remote/{role}", + argv=[f"/remote/{role}"], + exe_sha256=sha, + maps_sha256=digest(maps), + loaded_libraries=[], + ) + for phase in ("ready", "after"): + (arm / f"{role}-{phase}.maps").write_bytes(maps) + for phase in ("ready", "after"): + (arm / f"process-{phase}.json").write_text(dump(processes)) + planned = [json.loads(line) for line in (prep / "planned-cases.jsonl").read_text().splitlines()] + (arm / "planned-initial.json").write_text( + dump( + dict( + planned_denominator=30, + cases=planned, + identity={ + "benchmark_run_id": root.name, + "planned_model": "open-9b", + "identity_state": "planned_contract_only", + }, + ) + ) + ) + plan_identity = {key: identity[key] for key in ("benchmark_run_id", "model_identity_sha256")} + plan_identity["record"] = "identity.json" + (arm / "planned-cases.json").write_text( + dump( + dict( + identity=plan_identity, + planned_denominator=30, + cases=[ + { + **case, + "logical_key": [ + root.name, + identity["model_identity_sha256"], + "ticket_router", + 0, + case["ticket_id"], + ], + } + for case in planned + ], + ) + ) + ) + modules = { + key[8:-3].replace("/", "."): {"path": "/remote/source/" + key[8:], "sha256": value} + for key, value in binding["source_sha256"].items() + if key.startswith("runtime/s1a/") and key.endswith(".py") + } + for key in list(modules): + if key.endswith(".__init__"): + modules[key[:-9]] = modules.pop(key) + (arm / "python-identity.json").write_text( + dump( + dict( + executable="/remote/python", + executable_sha256="c" * 64, + version="fixture", + modules=modules, + expected_s1a_root="/remote/source", + actual_s1a_root="/remote/source/s1a", + environment={ + "MODEL_NAME": "", + "S1A_HOME": remote + "/s1a-home", + "S1A_DECISION_TIMEOUT_S": "120", + "LAYA_SERVED_TIMEOUT_S": "120", + "TYPESAFE_MODEL": "open-jev-9b", + }, + ) + ) + ) + framework = b"# synthetic source capture containing result_type\n" + (arm / "framework.py").write_bytes(framework) + framework_sha = digest(framework) + modules["openjiuwen.fixture"] = {"path": "/remote/framework.py", "sha256": framework_sha} + (arm / "python-loaded-after.json").write_text(dump(modules)) + (arm / "framework-result-type-source.json").write_text( + dump( + dict( + normal_result_type_allowlist=["answer"], + gate_basis="conservative_provisional_pending_actual_framework_source_verification", + candidates=[ + dict( + module="openjiuwen.fixture", + path="/remote/framework.py", + copy=remote + "/framework.py", + bytes=len(framework), + sha256=framework_sha, + loaded_module_record_sha256=framework_sha, + ) + ], + ) + ) + ) + argv = [ + "s1a", + "run", + "ticket_router", + "--model", + "jev", + "--rethink", + "off", + "--episodes", + "1", + "--seed", + "0", + "--max-steps", + "30", + "--timeout", + "300", + "--dataset", + "/remote/source/s1a/agents/_data/ticket_router_eval.jsonl", + "--batch-size", + "30", + "--log", + ] + events = [] + compact = {key: identity[key] for key in ("benchmark_run_id", "model_identity_sha256")} + compact["record"] = "identity.json" + + def add(kind, **fields): + events.append( + dict( + event_index=len(events) + 1, + utc="fixture", + monotonic_ns=len(events) + 1, + kind=kind, + identity=compact, + **fields, + ) + ) + return len(events) + + def pair(kind, start, terminal): + index = add(kind + "_start", **start) + return add(kind + "_terminal", start_event=index, status="returned", **terminal) + + def observation(index): + if index == 30: + return {"ticket": None, "progress": {"ticket_id": None}} + return { + "ticket": {key: rows[index][key] for key in PUBLIC_FIELDS if key in rows[index]}, + "progress": {"ticket_id": rows[index]["id"]}, + } + + def view(index): + return dict( + state=observation(index), candidates={} if index == 30 else QUEUES, done=index == 30, score=float(index) + ) + + add("cli_start", argv=argv) + for stage in ("Runner.start", "build_model", "make_series", "DecisionModel.warm", "TicketRouterEnv.__init__"): + pair("lifecycle", {"stage": stage}, {"stage": stage}) + episode_start = add("lifecycle_start", stage="run_episode") + pair("env_reset", {}, dict(view=observation(0), candidates=QUEUES)) + pair("lifecycle", {"stage": "DeepAgent.ensure_initialized"}, {"stage": "DeepAgent.ensure_initialized"}) + runner_start = add("lifecycle_start", stage="Runner.run_agent") + ticks, routes = [], [] + for index, row in enumerate(rows): + context = dict( + decision_index=index + 1, + case_id=row["id"], + planned_sequence=index + 1, + suite="ticket_router", + episode_seed=0, + env_instance=123, + env_index=index, + logical_key=[root.name, identity["model_identity_sha256"], "ticket_router", 0, row["id"]], + ) + key = row["label"] + probabilities = {queue: float(queue == key) for queue in QUEUES} + question = dict(type="choice", criteria=QUEUES, instructions={"rules": RULES}) + request = dict(model="open-jev-9b", state=observation(index), questions={"pick": question}) + reply = dict( + model="fixture", + answers={"pick": dict(type="choice", choice=key, probabilities=probabilities, confidence=1.0)}, + ) + start_slot = add("slot_start", context=context, observation=observation(index), candidates=QUEUES) + start_validation = add( + "decision_validation_start", + context=context, + observation={"state": observation(index), "images": []}, + questions={"pick": dict(options=QUEUES, rules=[RULES], goal="", operation="")}, + kwargs={}, + ) + http = add( + "http_start", + context=context, + method="POST", + url="http://127.0.0.1:18188/v1/systemone", + category="decision_post", + post_attempt_index=1, + request_body=body(request), + ) + add( + "http_terminal", + context=context, + start_event=http, + status="response", + post_attempt_index=1, + http_status=200, + response_body=body(reply), + ) + add( + "decision_validation_terminal", + context=context, + start_event=start_validation, + status="returned", + result=dict( + answers={"pick": {"key": key, "probabilities": probabilities, "confidence": 1.0}}, + model="fixture", + raw=reply, + ), + ) + tick = dict(step=index + 1, key=key, confidence=1.0, probabilities=probabilities, source="jev", model="fixture") + ticks.append(tick) + add( + "slot_terminal", + context=context, + start_event=start_slot, + status="returned", + new_decisions=[tick], + state_error=None, + ) + act_context = dict(context, selected_key=key) + act = add("act_start", context=act_context, inputs={"key": key}, association_matches=True, before=view(index)) + pair( + "env_step", + dict( + context=act_context, + key=key, + before=dict(index=index, done=False, observation=observation(index), candidates=QUEUES), + ), + dict( + context=act_context, + after=dict( + index=index + 1, + done=index == 29, + observation=observation(index + 1), + candidates={} if index == 29 else QUEUES, + ), + ), + ) + add( + "act_terminal", + context=act_context, + start_event=act, + status="returned", + act_calls=[{"key": key, "accepted": True}], + after=view(index + 1), + raw_tool_result=dump(view(index + 1)), + ) + routes.append(dict(id=row["id"], predicted=key, expected=key, correct=True)) + add( + "lifecycle_terminal", + start_event=runner_start, + stage="Runner.run_agent", + status="returned", + result_type="answer", + ) + for stage in ("DeepAgent.cleanup_task_resources", "ability_manager.teardown_tools", "Runner.release"): + pair("lifecycle", {"stage": stage}, {"stage": stage}) + add("lifecycle_terminal", start_event=episode_start, stage="run_episode", status="returned") + for stage in ("annotate", "DecisionModel.close"): + pair("lifecycle", {"stage": stage}, {"stage": stage}) + job = arm / "s1a-home/evals/results/ticket_router/fixture__jev" + trial = job / "ticket_router--0__fixture" + (trial / "agent").mkdir(parents=True) + report = dict( + batch_sha256=binding["batch_sha256"], + seed=0, + total=30, + processed=30, + correct=30, + coverage=1, + ticket_ids=[row["id"] for row in rows], + unprocessed_ids=[], + routes=routes, + ) + episode = dict( + decisions=ticks, + final_state=observation(30), + views=[view(i) for i in range(31)], + extra={"ticket_router": report, "rethink": False, "rethinks": [], "result_type": "answer"}, + ) + (trial / "agent/episode.json").write_text(dump(episode)) + (trial / "result.json").write_text( + dump( + dict( + task_name="ticket_router/0", + source="ticket_router", + exception_info=None, + agent_result={"metadata": {"steps": 30, "decisions": 30, "chat_calls": 0, "invalid_keys": 0}}, + ) + ) + ) + (job / "config.json").write_text( + dump( + dict( + agents=[{"name": "s1a-evals/jev", "model_name": "jev"}], + datasets=[{"name": "ticket_router", "n_tasks": 1}], + ) + ) + ) + for name in ("result.json", "summary.json"): + (job / name).write_text("{}") + pair( + "lifecycle", + {"stage": "write_job"}, + {"stage": "write_job", "result": remote + "/s1a-home/evals/results/ticket_router/fixture__jev"}, + ) + pair("lifecycle", {"stage": "Runner.stop"}, {"stage": "Runner.stop"}) + add("process_terminal", state="returned", exit_code=0, stage="original_cli_entry") + + def save_events(): + (arm / "journal.jsonl").write_text("".join(dump(event) + "\n" for event in events)) + + save_events() + return root, arm, events, save_events, preparation + + +def test_complete_routes_are_provisional_and_missing_cells_remain(measurement): + root, _, _, _, preparation = measurement + result = aggregate(root, preparation) + cell = result["cells"][0] + assert cell["route_correct"] == cell["accepted_processed"] == 30 + assert cell["complete_episode"] and cell["strict_batch_success"] + assert cell["quality_status"] == "PROVISIONAL" + assert cell["process_identity_state"] == "verified" + assert result["planned_routes"] == 90 and result["planned_episodes"] == 3 + assert [row["accepted_processed"] for row in result["cells"]] == [30, 0, 0] + + +@pytest.mark.parametrize( + "mutation", ["expected", "probability", "request_label", "selected_key", "after_ticket", "chat", "timeout"] +) +def test_corrupt_or_abnormal_evidence_cannot_pass(measurement, mutation): + root, arm, events, save_events, preparation = measurement + episode_path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) + episode = json.loads(episode_path.read_text()) + if mutation == "expected": + episode["extra"]["ticket_router"]["routes"][0]["expected"] = "tampered" + elif mutation == "probability": + next(e for e in events if e["kind"] == "slot_terminal")["new_decisions"][0]["probabilities"] = {"human": 1} + elif mutation == "request_label": + event = next(e for e in events if e["kind"] == "http_start") + request = json.loads(event["request_body"]["utf8"]) + request["state"]["ticket"]["label"] = "human" + event["request_body"] = body(request) + elif mutation == "selected_key": + next(e for e in events if e["kind"] == "act_start")["inputs"]["key"] = "tampered" + elif mutation == "after_ticket": + next(e for e in events if e["kind"] == "env_step_terminal")["after"]["observation"]["ticket"]["id"] = "wrong" + elif mutation == "chat": + path = episode_path.parents[1] / "result.json" + trial = json.loads(path.read_text()) + trial["agent_result"]["metadata"]["chat_calls"] = 1 + path.write_text(dump(trial)) + elif mutation == "timeout": + episode["extra"]["result_type"] = "timeout" + next(e for e in events if e.get("stage") == "Runner.run_agent" and e["kind"] == "lifecycle_terminal")[ + "result_type" + ] = "timeout" + episode_path.write_text(dump(episode)) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert not cell["strict_batch_success"] and not cell["complete_episode"] + assert cell["planned_routes"] == 30 and cell["errors"] + if mutation == "timeout": + assert cell["route_correct"] == 30 + + +def test_journal_gap_and_body_hash_fail_closed(measurement): + root, _, events, save_events, preparation = measurement + next(e for e in events if e["kind"] == "http_start")["request_body"]["sha256"] = "0" * 64 + events[-1]["event_index"] += 1 + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert not cell["strict_batch_success"] and cell["errors"] + + +@pytest.mark.parametrize("value", ["off", True, 0, None]) +def test_rethink_requires_original_false_boolean(measurement, value): + root, arm, _, _, preparation = measurement + path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) + episode = json.loads(path.read_text()) + episode["extra"]["rethink"] = value + path.write_text(dump(episode)) + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == cell["route_correct"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] + + +@pytest.mark.parametrize("damage", ["final_newline", "stop_terminal", "process_terminal"]) +def test_tail_damage_retains_closed_routes(measurement, damage): + root, arm, events, save_events, preparation = measurement + if damage == "final_newline": + path = arm / "journal.jsonl" + path.write_bytes(path.read_bytes().rstrip(b"\n")) + else: + events[:] = [ + row + for row in events + if not ( + row["kind"] == "process_terminal" + if damage == "process_terminal" + else row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.stop" + ) + ] + renumber(events) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == cell["route_correct"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] and cell["errors"] + + +def test_broken_ticket_pair_only_loses_affected_ticket(measurement): + root, _, events, save_events, preparation = measurement + first = next(row for row in events if row["kind"] == "act_terminal") + events.remove(first) + renumber(events) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert not cell["cases"][0]["accepted"] + assert cell["accepted_processed"] == cell["route_correct"] == 29 + assert not cell["complete_episode"] + + +def test_corrupt_health_body_blocks_episode_without_losing_routes(measurement): + root, _, events, save_events, preparation = measurement + compact = events[-1]["identity"] + events[-1:-1] = [ + dict( + event_index=-1, + kind="http_start", + identity=compact, + context=None, + method="GET", + url="http://127.0.0.1:18188/health", + category="health", + request_body=body({}), + ), + dict( + event_index=-2, + kind="http_terminal", + identity=compact, + context=None, + start_event=-1, + status="response", + http_status=200, + response_body=body({"status": "ready"}), + ), + ] + renumber(events) + save_events() + assert aggregate(root, preparation)["cells"][0]["complete_episode"] + events[-2]["response_body"]["sha256"] = "0" * 64 + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == cell["route_correct"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] + assert any("body length/hash" in error for error in cell["errors"]) + + +@pytest.mark.parametrize("damage", ["ready_hash", "after_hash", "launch", "maps", "after_pid"]) +def test_process_asset_binding_blocks_episode_preserving_routes(measurement, damage): + root, arm, _, _, preparation = measurement + assert aggregate(root, preparation)["cells"][0]["complete_episode"] + path = arm / ("process-after.json" if damage in ("after_hash", "after_pid") else "process-ready.json") + capture = json.loads(path.read_text()) + if damage.endswith("hash"): + capture["worker"]["exe_sha256"] = "0" * 64 + path.write_text(dump(capture)) + elif damage == "after_pid": + capture["worker"]["pid"] += 1 + path.write_text(dump(capture)) + elif damage == "launch": + launch = json.loads((arm / "launch.json").read_text()) + launch["worker"] = "/remote/other-worker" + (arm / "launch.json").write_text(dump(launch)) + else: + (arm / "worker-ready.maps").write_text("changed") + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == cell["route_correct"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] and cell["errors"] + assert cell["process_identity_state"] == "invalid" + + +def test_framework_copy_corruption_preserves_verified_routes(measurement): + root, arm, _, _, preparation = measurement + remote = f"/remote/{root.name}/open-9b" + copy = arm / "framework.py" + copy.write_text("# captured framework\n") + sha = digest(copy.read_bytes()) + row = dict( + module="openjiuwen.normal", + path="/remote/framework.py", + copy=remote + "/framework.py", + bytes=copy.stat().st_size, + sha256=sha, + loaded_module_record_sha256=sha, + ) + path = arm / "python-loaded-after.json" + modules = json.loads(path.read_text()) + modules[row["module"]] = {"path": row["path"], "sha256": sha} + path.write_text(dump(modules)) + (arm / "framework-result-type-source.json").write_text(dump(dict(candidates=[row]))) + assert aggregate(root, preparation)["cells"][0]["complete_episode"] + copy.write_text("changed") + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == cell["route_correct"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] + assert cell["framework_sources"]["state"] == "invalid" + assert any("framework source copy hash" in error for error in cell["errors"]) + + +@pytest.mark.parametrize("damage", ["missing", "empty", "capture_error", "partial_capture_error"]) +def test_missing_or_incomplete_framework_capture_cannot_complete(measurement, damage): + root, arm, _, _, preparation = measurement + assert aggregate(root, preparation)["cells"][0]["complete_episode"] + path = arm / "framework-result-type-source.json" + capture = json.loads(path.read_text()) + failure = dict(module="openjiuwen.failed", path="/remote/failed.py", capture_error="read denied") + if damage == "missing": + path.unlink() + else: + capture["candidates"] = ( + [] if damage == "empty" else ([failure] if damage == "capture_error" else capture["candidates"] + [failure]) + ) + path.write_text(dump(capture)) + cell = aggregate(root, preparation)["cells"][0] + assert cell["route_correct"] == cell["accepted_processed"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] and cell["errors"] + assert cell["framework_sources"]["state"] == ("missing" if damage == "missing" else "incomplete") + + +@pytest.mark.parametrize( + "damage", ["early_runner_return", "early_release", "early_episode_return", "late_reset", "late_init"] +) +def test_original_runner_and_cleanup_order_is_required(measurement, damage): + root, _, events, save_events, preparation = measurement + assert aggregate(root, preparation)["cells"][0]["complete_episode"] + if damage == "early_runner_return": + terminal = next( + row for row in events if row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.run_agent" + ) + events.remove(terminal) + events.insert(next(i for i, row in enumerate(events) if row.get("stage") == "Runner.run_agent") + 1, terminal) + elif damage in ("late_reset", "late_init"): + moved = [ + row + for row in events + if ( + row["kind"] in ("env_reset_start", "env_reset_terminal") + if damage == "late_reset" + else row.get("stage") == "Runner.start" + ) + ] + events[:] = [row for row in events if row not in moved] + at = next( + i + for i, row in enumerate(events) + if row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.run_agent" + ) + if damage == "late_init": + at += 1 + events[at:at] = moved + else: + stage = "Runner.release" if damage == "early_release" else "run_episode" + moved = [ + row + for row in events + if row.get("stage") == stage and (stage == "Runner.release" or row["kind"] == "lifecycle_terminal") + ] + events[:] = [row for row in events if row not in moved] + at = next( + i + for i, row in enumerate(events) + if row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.run_agent" + ) + events[at:at] = moved + renumber(events) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert cell["route_correct"] == cell["accepted_processed"] == 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] + assert any("lifecycle order" in error or "outside original Runner" in error for error in cell["errors"]) + + +@pytest.mark.parametrize("provenance_source", ["response", "routing", "health"]) +def test_served_laya_original_raw_provenance_transform(measurement, provenance_source): + import asyncio + import copy + + from benchmarks.quality_ticket import verify_decision + from s1a.agents.ticket_router import QUEUES, RULES + from s1a.decision_models.served import ServedLayaModel + from s1a.decision_models.types import ChoiceQuestion, Observation + from s1a.decision_models.validation import choice_faults, validate_answers + + _, _, events, _, preparation = measurement + context = next(row["context"] for row in events if row["kind"] == "slot_start") + related = copy.deepcopy([row for row in events if (row.get("context") or {}).get("case_id") == context["case_id"]]) + slot = next(row for row in related if row["kind"] == "slot_start") + terminal = next(row for row in related if row["kind"] == "http_terminal") + payload = json.loads(terminal["response_body"]["utf8"]) + payload.update( + model="english", + served_by={"checkpoint": "laya-english", "revision": "r1", "source": "wire"}, + diagnostic="original wire field", + ) + health, read_at = {}, None + if provenance_source != "response": + payload["served_by"] = None + if provenance_source == "health": + payload["routing"] = {"model": "english", "repo": "laya-english", "reason": "explicit model english"} + health = {"models": {"english": {"checkpoint": "laya-english", "revision": "r1", "device": "cuda"}}} + read_at = "2026-10-10T09:00:00+00:00" + + class OfflineClient: + url = "http://127.0.0.1:18188" + + def deadline(self): + return 1.0 + + async def decide(self, request, request_id, deadline): + assert request == dict( + model="english", + state=slot["observation"], + questions={"pick": dict(type="choice", criteria=QUEUES, instructions={"rules": RULES})}, + ) + return copy.deepcopy(payload), {}, 1.0 + + model = ServedLayaModel(OfflineClient(), model="english", clock=lambda: 1.0) + model._health_tried = 1.0 + model._health, model._health_read_at = health, read_at + questions = {"pick": ChoiceQuestion(QUEUES, rules=RULES)} + reply = asyncio.run(model._decide(Observation(slot["observation"]), questions)) + validated = validate_answers(reply, questions) + captured = preparation[2].value(validated) + assert captured["raw"]["served_by"]["source"] == provenance_source + journal = related + if health: + start = dict( + kind="http_start", event_index=slot["event_index"] - 2, method="GET", url="http://127.0.0.1:18188/health" + ) + end = dict( + kind="http_terminal", + event_index=slot["event_index"] - 1, + start_event=start["event_index"], + status="response", + http_status=200, + response_body=body(health), + ) + journal = [start, end, *related] + for row in related: + if row["kind"] == "http_start": + request = json.loads(row["request_body"]["utf8"]) + request["model"] = "english" + row["request_body"] = body(request) + elif row["kind"] == "http_terminal": + row["response_body"] = body(payload) + elif row["kind"] == "decision_validation_terminal": + row["result"] = captured + elif row["kind"] == "slot_terminal": + row["new_decisions"][0].update(model=validated.model, source="laya-served") + identity = {"requested_model": "english"} + assert verify_decision(related, slot["observation"], QUEUES, RULES, identity, choice_faults, journal=journal) + raw = captured["raw"] + for target, field, wrong in ( + (raw["served_by"], "source", "wrong"), + (raw["served_by"], "revision", "wrong"), + (raw, "diagnostic", "wrong"), + (raw["answers"]["pick"], "confidence", 0.5), + ): + original = target[field] + target[field] = wrong + with pytest.raises(ValueError, match="Validated raw differs"): + verify_decision(related, slot["observation"], QUEUES, RULES, identity, choice_faults, journal=journal) + target[field] = original + + +def test_unknown_result_type_requests_actual_source(measurement): + root, arm, events, save_events, preparation = measurement + path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) + episode = json.loads(path.read_text()) + episode["extra"]["result_type"] = "new-framework-type" + path.write_text(dump(episode)) + next(e for e in events if e.get("stage") == "Runner.run_agent" and e["kind"] == "lifecycle_terminal")[ + "result_type" + ] = "new-framework-type" + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert cell["route_correct"] == 30 and not cell["strict_batch_success"] + assert cell["required_input"] == "independent review of actual loaded framework result_type source" + + +def renumber(events): + indices = {row["event_index"]: number for number, row in enumerate(events, 1)} + for row in events: + old = row["event_index"] + if "start_event" in row: + row["start_event"] = indices[row["start_event"]] + row["event_index"] = indices[old] + + +def test_retry_is_one_ticket_and_each_request_is_checked(measurement): + root, _, events, save_events, preparation = measurement + success = next(row for row in events if row["kind"] == "http_start") + terminal = next(row for row in events if row.get("start_event") == success["event_index"]) + import copy + + failed_start, failed_end = copy.deepcopy(success), copy.deepcopy(terminal) + failed_start["event_index"], failed_end["event_index"] = 10000, 10001 + failed_end.update(start_event=10000, http_status=503, response_body=body({"error": "retry fixture"})) + success["post_attempt_index"] = terminal["post_attempt_index"] = 2 + position = events.index(success) + events[position:position] = [failed_start, failed_end] + renumber(events) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == 30 and cell["cases"][0]["post_attempts"] == 2 + request = json.loads(failed_start["request_body"]["utf8"]) + request["state"]["ticket"]["id"] = "another-ticket" + failed_start["request_body"] = body(request) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert not cell["strict_batch_success"] and cell["errors"] + + +def test_29_routes_and_missing_job_preserve_denominator(measurement): + root, arm, events, save_events, preparation = measurement + last_id = next(row["context"]["case_id"] for row in reversed(events) if row.get("context")) + events[:] = [row for row in events if (row.get("context") or {}).get("case_id") != last_id] + renumber(events) + save_events() + path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) + episode = json.loads(path.read_text()) + report = episode["extra"]["ticket_router"] + report.update(processed=29, correct=29, coverage=29 / 30, unprocessed_ids=[last_id], routes=report["routes"][:29]) + episode.update( + decisions=episode["decisions"][:29], views=episode["views"][:30], final_state=episode["views"][29]["state"] + ) + path.write_text(dump(episode)) + trial_path = path.parents[1] / "result.json" + trial = json.loads(trial_path.read_text()) + trial["agent_result"]["metadata"].update(steps=29, decisions=29) + trial_path.write_text(dump(trial)) + cell = aggregate(root, preparation)["cells"][0] + assert cell["route_correct"] == cell["accepted_processed"] == 29 and cell["route_accuracy"] == 29 / 30 + assert not cell["complete_episode"] and not cell["strict_batch_success"] + path.unlink() + cell = aggregate(root, preparation)["cells"][0] + assert cell["route_correct"] == 29 and not cell["complete_episode"] + + +def test_all_wrong_complete_episode_is_not_strict_success(measurement): + root, arm, events, save_events, preparation = measurement + from s1a.agents.ticket_router import QUEUES + + slots = [row for row in events if row["kind"] == "slot_terminal"] + choices = { + row["context"]["decision_index"]: next(key for key in QUEUES if key != row["new_decisions"][0]["key"]) + for row in slots + } + for row in events: + context = row.get("context") + if not context: + continue + key = choices[context["decision_index"]] + probabilities = {queue: float(queue == key) for queue in QUEUES} + if "selected_key" in context: + context["selected_key"] = key + if row["kind"] == "http_terminal" and row["status"] == "response": + reply = json.loads(row["response_body"]["utf8"]) + reply["answers"]["pick"].update(choice=key, probabilities=probabilities) + row["response_body"] = body(reply) + elif row["kind"] == "decision_validation_terminal": + row["result"]["answers"]["pick"].update(key=key, probabilities=probabilities) + row["result"]["raw"]["answers"]["pick"].update(choice=key, probabilities=probabilities) + elif row["kind"] == "slot_terminal": + row["new_decisions"][0].update(key=key, probabilities=probabilities) + elif row["kind"] == "act_start": + row["inputs"]["key"] = key + row["before"]["score"] = 0.0 + elif row["kind"] == "env_step_start": + row["key"] = key + elif row["kind"] == "act_terminal": + row["act_calls"][0]["key"] = key + row["after"]["score"] = 0.0 + row["raw_tool_result"] = dump(row["after"]) + path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) + episode = json.loads(path.read_text()) + episode["decisions"] = [row["new_decisions"][0] for row in slots] + for route, key in zip(episode["extra"]["ticket_router"]["routes"], choices.values(), strict=True): + route.update(predicted=key, correct=False) + episode["extra"]["ticket_router"]["correct"] = 0 + for view in episode["views"]: + view["score"] = 0.0 + path.write_text(dump(episode)) + save_events() + cell = aggregate(root, preparation)["cells"][0] + assert cell["accepted_processed"] == 30 and cell["route_correct"] == 0 + assert cell["complete_episode"] and not cell["strict_batch_success"] From 3400a60114d38b57a9048b0f6bdeaa4fb625bb03 Mon Sep 17 00:00:00 2001 From: linear3735 Date: Sat, 10 Oct 2026 23:25:25 +0800 Subject: [PATCH 4/5] bench: focus saved-decision scoring on public231 --- .github/workflows/test.yml | 37 + CHANGELOG.md | 4 +- benchmarks/README.md | 68 +- benchmarks/accuracy-leaderboard.md | 71 ++ benchmarks/quality_public231.py | 7 +- benchmarks/quality_ticket.py | 887 -------------------- tests/benchmarks/test_quality_public231.py | 55 +- tests/benchmarks/test_quality_ticket.py | 913 --------------------- 8 files changed, 170 insertions(+), 1872 deletions(-) create mode 100644 benchmarks/accuracy-leaderboard.md delete mode 100644 benchmarks/quality_ticket.py delete mode 100644 tests/benchmarks/test_quality_ticket.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 521305e..67f8867 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -87,3 +87,40 @@ jobs: # The Chromium revision that playwright-core inside the runtime's pinned @playwright/mcp expects. - run: npx -y -p "$(uv run --no-sync python -c 'import s1a.entry; from s1a.config import DEFAULT_PLAYWRIGHT_MCP_ARGS as a; print(a.split()[-1])')" playwright-core install --with-deps chromium - run: S1A_BROWSER_TESTS=1 uv run pytest -q tests/system + + benchmarks: + runs-on: ubuntu-latest + timeout-minutes: 15 + env: + JEVBENCH_ROOT: ${{ github.workspace }}/_reference/jevbench + OMNI_BENCH_SOURCE: ${{ github.workspace }}/_reference/omni/benchmarks/bench.py + steps: + - uses: actions/checkout@v7 + - uses: actions/checkout@v7 + with: + repository: fstandhartinger/jevbench + ref: 7ce310c7262ed49cc85853339a8a42459298e3f3 + path: _reference/jevbench + persist-credentials: false + - uses: actions/checkout@v7 + with: + repository: ThinkFlowLab/system1-omni + ref: 4b9cccbde841272ff5ace6aee3cf51610fe1324f + path: _reference/omni + persist-credentials: false + - uses: astral-sh/setup-uv@v10.2.0 + with: + python-version: "3.11" + enable-cache: true + - run: uv sync --extra dev + - name: Required benchmark tests + run: | + uv run pytest tests/benchmarks -q --junitxml="${RUNNER_TEMP}/benchmark-tests.xml" + uv run python - <<'PY' + import os + import xml.etree.ElementTree as ET + from pathlib import Path + + report = Path(os.environ["RUNNER_TEMP"]) / "benchmark-tests.xml" + raise SystemExit(bool(ET.parse(report).findall(".//skipped"))) + PY diff --git a/CHANGELOG.md b/CHANGELOG.md index 786ebe0..39e1da6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver ### Added -- Offline quality consumers for shared Omni public231 raw and original ticket-router - evidence, retaining planned denominators, original scoring and provisional completion gates. +- Offline quality consumer for shared Omni public231 raw, retaining planned + denominators and original scoring. - Public231 decision-model benchmark with raw HTTP evidence, pinned upstream scoring, complete-denominator quality metrics and per-round service latency. diff --git a/benchmarks/README.md b/benchmarks/README.md index 93a2d44..2f392b6 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -7,6 +7,10 @@ command re-scores them with the pinned upstream scorer. [H800 results](results/public231-h800.md) compare five open checkpoints. This is the public231 set, not the article's 248-question or 23-model leaderboard. +The [accuracy leaderboard](accuracy-leaderboard.md) ranks that frozen run and +defines how to add comparable results. Request latency has a separate +[System1-Omni leaderboard](https://github.com/ThinkFlowLab/system1-omni/pull/131). + ## Run Use the repository's development environment and a running service. Clone @@ -84,8 +88,8 @@ check collection and scoring behavior; model quality and latency come from the l ## Offline quality from shared raw -These consumers re-score saved Omni requests and original ticket episodes without -another model run. The existing `predict` and `aggregate` commands and their +This consumer re-scores saved Omni requests without another model run. +The existing `predict` and `aggregate` commands and their default timing basis are unchanged. For public231, provide the frozen input manifest, the independently frozen Omni @@ -123,56 +127,18 @@ pointers and hashes. Quality output uses null timestamps/latencies; it does not report speed. The raw clock is POST through body decoding, JSON and schema validation, and the saved response is decoded HTTPX text, not compressed wire bytes. -Ticket quality needs the existing System1-Agents environment, including openJiuwen. -Public231 does not need that framework. Use the frozen ticket producer r3 preparation -and the **whole** measurement root, including unsuccessful or missing cells: +Use a fresh output directory outside the repository. The tool checks archive +consistency; it does not authenticate a coordinated rewrite of unsigned evidence. +CPU fixtures exercise the reader and scorer, not model quality or GPU execution. + +The required benchmark CI job fetches the pinned JevBench reference and Omni +collector, generates the public231 fixture, and rejects skipped tests. To run the +same suite locally, set `JEVBENCH_ROOT` and `OMNI_BENCH_SOURCE` to those checkouts: ```bash -python -m benchmarks.quality_ticket \ - --preparation /absolute/path/to/ticket-gpu-prepare/r3 \ - --run /absolute/path/to/measurement-root \ - --out /absolute/path/to/new-ticket-quality-output +python -m pytest tests/benchmarks -q --confcutdir=tests/benchmarks ``` -This version accepts the frozen `open-9b`, `open-27b` and `laya-english` cells, -seed0, one episode, batch30, maxsteps30, timeout300 and decision deadline120. -It reuses the original fixture/environment and choice validator, checks every -POST/retry against the current public ticket and queues, and follows original -validated slot → accepted act → committed environment step → returned job. -Decisions and actions must occur inside the original `Runner.run_agent` call; -cleanup/release, episode return, annotate/close, write_job and stop must follow -the original caller's order. Laya-served response provenance is checked with the -original model's response augmentation or captured health/routing fallback; -other wire fields remain exact comparisons. -Labels come only from the fixed fixture. Requests, probabilities/confidence, -before/after observations, original job ticks/views, loaded source and cell/asset -identity are checked; archived report scores are diagnostic comparisons. Non-decision -HTTP bodies are also checked for exact saved bytes, hash and viewing text. Captured -worker/frontend executables must match launch paths and recorded asset hashes, and -the ready/after process IDs and maps must remain consistent. -Remote artifact paths are relocated only inside that cell's recorded arm prefix. - -Each cell reports fixture-derived `route_correct / 30`, `accepted_processed / 30`, -`complete_episode`, and `strict_batch_success` (normal complete episode and all -30 routes correct). Timeout/error/missing evidence retain verified prefixes and -the full 30 denominator. `MODEL_NAME` must be empty, rethink off, and original -episode `rethink` must be boolean `false`, with `chat_calls` zero. Route counts -describe individually verified calls under the recorded cell identity. Damaged -cleanup, process/asset cross-checks or framework copies block complete/strict -results without removing those closed routes; a broken ticket association blocks -that ticket. `process_identity_state` reports that separate process/asset check. -`answer` is an explicitly **provisional** normal-type gate; -missing, empty or incomplete framework source captures cannot be complete/strict. -all results require independent review of the actual captured framework source. -Other/unknown result types cannot be complete or strict successes. Exact recorded -asset/source consistency does not establish historical binary build ancestry. - -Both tools require fresh output outside the repository. They detect inconsistent -unsigned archives, not coordinated rewriting of all evidence. No real model quality, -GPU execution or framework normal-type acceptance is established by their CPU mocks. - -The new tests additionally use `OMNI_BENCH_SOURCE`, `PUBLIC231_INPUT_MANIFEST`, and -`TICKET_PREPARATION_ROOT` pointing to those frozen external inputs. When the CPU -environment lacks openJiuwen, ticket tests stub only framework imports and execute -the repository's actual environment/validator; that is separate from real framework -integration. Run the benchmark tests with the same pytest command above. +The reader accepts the original collector (`5aa20a2c…`). Each archive must match +the exact collector supplied with `--collector`; the probability rules, input +bytes and timing basis are unchanged. diff --git a/benchmarks/accuracy-leaderboard.md b/benchmarks/accuracy-leaderboard.md new file mode 100644 index 0000000..dba281b --- /dev/null +++ b/benchmarks/accuracy-leaderboard.md @@ -0,0 +1,71 @@ +# Public231 accuracy leaderboard + +This board ranks five deployed decision models on 231 public JevBench tasks: +easy48, original72 and hard111. It is a fixed evaluation group, not the article's +248-question leaderboard or a claim about general model quality. + +## H800 group: Slurm 415076, 2026-10-07 UTC + +Rank uses correct / 231 in the first measured round. All three rounds +produced identical predictions and probability distributions for each deployment. +The repeated rounds add timing samples, not unique quality tasks. Equal correct +counts share a rank; calibration does not break ties. + +| Rank | Deployment | Correct / planned | Accuracy | Validity | Brier | ECE | +|---:|---|---:|---:|---:|---:|---:| +| 1 | Intern-Decision-4B, official HF | 200 / 231 | 86.5801% | 100% | 0.189601 | 0.067214 | +| 2 | Open-Jev-27B-v1.1, Omni native | 198 / 231 | 85.7143% | 100% | 0.240493 | 0.130333 | +| 3 | Intern-Decision-2B, official HF | 181 / 231 | 78.3550% | 100% | 0.306049 | 0.115420 | +| 4 | Open-Jev-9B, Omni native | 179 / 231 | 77.4892% | 100% | 0.320752 | 0.086786 | +| 5 | Intern-Decision-0.8B, official HF | 164 / 231 | 70.9957% | 100% | 0.369906 | 0.111077 | + +Each deployment completed 693 / 693 measured requests without failed or invalid +responses. Validity, Brier and 10-bin top-label ECE use those 693 distributions; +their repetition does not increase the independent sample size. Brier and ECE +are secondary calibration measures, with lower values better. + +## Source and controls + +- Dataset and scorer: [JevBench 7ce310c7](https://github.com/fstandhartinger/jevbench/tree/7ce310c7262ed49cc85853339a8a42459298e3f3). + The collector checks the pinned dataset and imported scorer file hashes. +- Canonical dataset SHA-256 (`dataset_hash` of the pinned tasks, also recorded + as `canonical_dataset_sha256` in the numeric attachment's source manifest): + `dc3995d8ae1e2fc8e81ce38431add509eb8bb39b85aadfd0c7c32079382dde51`. +- One H800, BF16, concurrency 1, thinking off, no retries, fixed task and option + order; five client warmups per deployment followed by three measured traversals. +- Intern used its official HF service at T=1, with optional fast paths unavailable. + Open-Jev used native Rust/CUDA, merged LoRA/head and its checkpoint temperature; + graph and prefix cache were disabled. These are different service configurations. +- The [immutable run report](https://github.com/linear3735/system1-agents/blob/40bf742e2b84ad6157bf2990dd5e4bab73571702/benchmarks/results/public231-h800.md) + records exact checkpoint, backend and dependency revisions, all round results + and limitations. The companion [latency contribution](https://github.com/ThinkFlowLab/system1-omni/pull/131) + keeps request timing separate from quality. + +The published report is available now. The numeric audit attachment for this run +is prepared but not yet published; complete original responses are not included +in the repository. The attachment removes question/option text and retains +probability vectors, label indices, timings and original record hashes. It supports +numeric recomputation; full response replay additionally requires the original raw. + +## Add a result + +Use the [collection and aggregation commands](README.md#run) against an existing +decision endpoint. Keep the dataset/scorer version, task order, precision and +sampling policy fixed; record exact model/backend revisions and any required +configuration differences. Labels and provenance must never enter model requests. + +Publish planned, attempted, valid, failed, incomplete and unattempted counts. +Failures and missing tasks stay in the planned accuracy denominator. Score the +original probabilities with the pinned scorer; do not replace them with one-hot +labels. Missing calibration inputs remain missing, rather than becoming zero error. + +Predeclare the primary round and repeat plan. This group uses the first measured +round for ranking; publish every round and any prediction disagreement. Never +choose the best round after seeing results. A partial run remains an unranked +observation with its coverage shown. + +Add a dated campaign with a linked report and auditable numeric evidence; preserve +prior campaigns. Changed datasets, scorer rules or sampling policies need a separate +group. Compare new deployments with a reference in the same campaign. Ticket-routing +completion, screenshot/VL tasks and other suites also need separate groups and +their own fixed inputs; they cannot inherit this text board's rank. diff --git a/benchmarks/quality_public231.py b/benchmarks/quality_public231.py index df18d15..3a19387 100644 --- a/benchmarks/quality_public231.py +++ b/benchmarks/quality_public231.py @@ -23,12 +23,13 @@ def load_collector(path): # Only this reviewed interface is executable; never import a file selected by raw metadata. - if digest(path.read_bytes()) != COLLECTOR_SHA256: + source_hash = digest(path.read_bytes()) + if source_hash != COLLECTOR_SHA256: raise ValueError("Frozen Omni collector hash mismatch") spec = importlib.util.spec_from_file_location("public231_omni_raw", path) collector = importlib.util.module_from_spec(spec) spec.loader.exec_module(collector) - collector.SOURCE_SHA256 = COLLECTOR_SHA256 + collector.SOURCE_SHA256 = source_hash return collector @@ -109,7 +110,7 @@ def aggregate(runs, input_manifest, collector, model_config_id): if (run / "requests.jsonl").read_bytes() != frozen: raise ValueError("Omni requests differ from the complete frozen 231 plan") config = decode((run / "config.json").read_text(encoding="utf-8")) - if config["runner_sha256"] != COLLECTOR_SHA256 or config["latency_basis"] != RAW_CLOCK: + if config["runner_sha256"] != collector.SOURCE_SHA256 or config["latency_basis"] != RAW_CLOCK: raise ValueError("Omni runner or clock differs from the frozen interface") identity = {key: value for key, value in config.items() if key not in ("python", "httpx")} metadata = dict(config.get("metadata", {})) diff --git a/benchmarks/quality_ticket.py b/benchmarks/quality_ticket.py deleted file mode 100644 index 0e56d45..0000000 --- a/benchmarks/quality_ticket.py +++ /dev/null @@ -1,887 +0,0 @@ -"""Independently verify frozen ticket-router raw evidence offline, with all 3 x 30 cases.""" - -import argparse -import asyncio -import base64 -import importlib.util -import json -from pathlib import Path -from urllib.parse import urlsplit - -from benchmarks.client import decode -from benchmarks.evidence import digest, dump, outside_repo, write_json - -BINDING_SHA256 = "8bd28a3624842cdaf7fcd35dac187dc0378f029d559daa2f870ed3ca4d7a573a" -SCHEMA_SHA256 = "34b2014a5f62db9801a52eab7831de6ffa4ef6af0cd3ce3bde8cb5b75b7499af" -CELLS = { - "open-9b": ("jev", "open-jev-9b"), - "open-27b": ("jev", "open-jev-27b-v1.1"), - "laya-english": ("laya-served", "english"), -} -NORMAL_BASIS = "answer_only_provisional_pending_actual_framework_source_verification" - - -def read_json(path): - return decode(path.read_text(encoding="utf-8")) - - -def require(condition, message): - if not condition: - raise ValueError(message) - - -def load_preparation(path): - require(digest((path / "binding.json").read_bytes()) == BINDING_SHA256, "Frozen producer binding hash mismatch") - require(digest((path / "schema-r1.md").read_bytes()) == SCHEMA_SHA256, "Frozen producer schema hash mismatch") - binding = read_json(path / "binding.json") - source = path / "evidence.py" - require( - digest(source.read_bytes()) == binding["source_sha256"]["evidence.py"], "Frozen journal parser hash mismatch" - ) - spec = importlib.util.spec_from_file_location("ticket_raw_journal", source) - producer = importlib.util.module_from_spec(spec) - spec.loader.exec_module(producer) - # Imports below use the current checkout, whose relevant contracts must match this producer. - current = Path(__file__).resolve().parents[1] - for name in ( - "s1a/agents/ticket_router.py", - "s1a/decision_models/validation.py", - "s1a/decision_models/types.py", - "s1a/decision_models/jev.py", - "s1a/decision_models/laya.py", - "s1a/decision_models/served.py", - "s1a/tool/models.py", - "s1a/tool/loop.py", - "s1a/jobs.py", - ): - require( - digest((current / name).read_bytes()) == binding["source_sha256"]["runtime/" + name], - f"Current contract differs from frozen producer: {name}", - ) - fixture = path / "runtime/s1a/agents/_data/ticket_router_eval.jsonl" - require(digest(fixture.read_bytes()) == binding["fixture_sha256"], "Frozen ticket fixture hash mismatch") - require( - digest((path / "planned-cases.jsonl").read_bytes()) == binding["planned_cases_sha256"], - "Frozen ticket plan hash mismatch", - ) - return path, binding, producer - - -def body_bytes(record): - raw = base64.b64decode(record["base64"], validate=True) - require(len(raw) == record["bytes"] and digest(raw) == record["sha256"], "HTTP body length/hash mismatch") - require(raw.decode("utf-8", errors="replace") == record["utf8"], "HTTP body viewing text mismatch") - return raw - - -def bytes_json(record): - return decode(body_bytes(record)) - - -def relocated(path, remote_arm, arm): - suffix = Path(path).relative_to(Path(remote_arm)) - local = (arm / suffix).resolve() - require(local.is_relative_to(arm.resolve()), "Original artifact path escapes current cell") - return local - - -def verify_identity(root, arm, item, binding): - identity = read_json(arm / "identity.json") - require( - identity["benchmark_run_id"] == root.name and identity["planned_model"] == arm.name, - "Current run/cell identity mismatch", - ) - require( - identity["requested_model"] == CELLS[arm.name][1] - and identity["identity_state"] == "current_exact_assets_recorded", - "Actual model identity unavailable", - ) - assets_hash = digest(json.dumps(identity["actual_assets"], sort_keys=True, ensure_ascii=False).encode("utf-8")) - require(assets_hash == identity["model_identity_sha256"], "Actual asset identity hash mismatch") - require(identity["source_binding_sha256"] == BINDING_SHA256, "Cell source binding mismatch") - require(identity["job_id"] == read_json(root / "status.json")["job_id"], "Job identity mismatch") - python = read_json(arm / "python-identity.json") - loaded = read_json(arm / "python-loaded-after.json") - require( - python["actual_s1a_root"] == str(Path(python["expected_s1a_root"]) / "s1a"), - "Actual loaded Agents root differs from requested source", - ) - for modules in (python["modules"], loaded): - for name, row in modules.items(): - if name == "s1a" or name.startswith("s1a."): - suffix = Path(row["path"]).relative_to(python["expected_s1a_root"]) - require( - row["sha256"] == binding["source_sha256"].get("runtime/" + suffix.as_posix()), - f"Loaded source hash mismatch: {name}", - ) - for name in ("s1a.agents.ticket_router", "s1a.decision_models.validation", "s1a.tool.models", "s1a.tool.loop"): - require(name in modules, f"Missing actual loaded source: {name}") - env = python["environment"] - require( - env["MODEL_NAME"] == "" and env["S1A_HOME"] == str(Path(item["arm"]) / "s1a-home"), - "Chat model must be empty and S1A_HOME must belong to this cell", - ) - require( - env["S1A_DECISION_TIMEOUT_S"] == "120" and env["LAYA_SERVED_TIMEOUT_S"] == "120", - "Actual decision deadline differs from 120s", - ) - model_field = "LAYA_SERVED_MODEL" if arm.name == "laya-english" else "TYPESAFE_MODEL" - require(env[model_field] == identity["requested_model"], "Actual request model environment mismatch") - return identity, python - - -def verify_processes(arm, identity): - launch = read_json(arm / "launch.json") - assets = identity["actual_assets"] - binaries = assets["binaries"] if arm.name == "laya-english" else assets["binary_sha256"] - ready = read_json(arm / "process-ready.json") - after = read_json(arm / "process-after.json") - for phase, capture in (("ready", ready), ("after", after)): - for role in ("worker", "frontend"): - process = capture[role] - require( - process["exe"] == launch[role] - and process["argv"] == [launch[role]] - and process["exe_sha256"] == binaries[launch[role]], - f"Actual {phase} {role} executable differs from launch/assets", - ) - require( - type(process["pid"]) is int and process["pid"] > 0 and process["pid"] == ready[role]["pid"], - f"Actual {role} process changed during episode", - ) - maps = (arm / f"{role}-{phase}.maps").read_bytes() - require(digest(maps) == process["maps_sha256"], f"Actual {phase} {role} maps hash mismatch") - mapped = { - line.split(maxsplit=5)[5] for line in maps.decode().splitlines() if len(line.split(maxsplit=5)) == 6 - } - require( - process["exe"] in mapped and set(process["loaded_libraries"]).issubset(mapped), - f"Actual {phase} {role} executable/libraries absent from captured maps", - ) - - -def verify_decision(events, observation, queues, rules, identity, choice_faults, *, journal=None): - slots = [row for row in events if row["kind"] == "slot_start"] - require(len(slots) == 1, "Decision must have exactly one original slot start") - slot = slots[0] - context = slot["context"] - require( - all( - {key: value for key, value in row["context"].items() if key != "selected_key"} == context for row in events - ), - "Decision context/environment instance differs across original calls", - ) - require( - slot["observation"] == observation and slot["candidates"] == queues, "Slot current ticket/candidates mismatch" - ) - terminals = {row.get("start_event"): row for row in events if row["kind"].endswith("_terminal")} - posts = [row for row in events if row["kind"] == "http_start" and row["method"] == "POST"] - require( - all(type(row["post_attempt_index"]) is int for row in posts) - and [row["post_attempt_index"] for row in posts] == list(range(1, len(posts) + 1)), - "POST retry sequence mismatch", - ) - expected_request = dict( - model=identity["requested_model"], - state=observation, - questions={"pick": {"type": "choice", "criteria": queues, "instructions": {"rules": rules}}}, - ) - replies = [] - for post in posts: - url = urlsplit(post["url"]) - require( - post["category"] == "decision_post" - and url.hostname in ("127.0.0.1", "localhost") - and url.port == 18188 - and url.path == "/v1/systemone", - "Unexpected decision POST destination", - ) - require( - dump(bytes_json(post["request_body"])) == dump(expected_request), - "Actual POST differs from current public ticket/queues/rules/model", - ) - terminal = terminals[post["event_index"]] - if terminal["status"] == "response": - # HTTP errors are still decoded/hash checked; non-JSON error bodies remain diagnostic. - try: - payload = bytes_json(terminal["response_body"]) - except (UnicodeError, json.JSONDecodeError): - require(not 200 <= terminal["http_status"] < 300, "Successful response is not JSON") - continue - if 200 <= terminal["http_status"] < 300: - replies.append((post, terminal, payload)) - slot_result = terminals[slot["event_index"]] - if slot_result["status"] != "returned" or not slot_result.get("new_decisions"): - return None - require( - slot_result.get("state_error") is None and len(slot_result["new_decisions"]) == 1, - "Returned slot is blocked or has multiple decisions", - ) - tick = slot_result["new_decisions"][0] - validations = [row for row in events if row["kind"] == "decision_validation_start"] - require(len(validations) == 1, "Missing or duplicate original validation start") - validation = validations[0] - require(validation["observation"] == {"state": observation, "images": []}, "Validator observation mismatch") - require( - validation["questions"] == {"pick": {"options": queues, "rules": [rules], "goal": "", "operation": ""}}, - "Validator offered question mismatch", - ) - result = terminals[validation["event_index"]] - require(result["status"] == "returned" and replies, "No successful POST for original validated decision") - validated = result["result"] - answer = validated["answers"]["pick"] - faults = choice_faults( - {"choice": answer["key"], "probabilities": answer["probabilities"], "confidence": answer["confidence"]}, - list(queues), - ) - require(not faults, "Invalid original validated probabilities/confidence: " + "; ".join(faults)) - payload = replies[-1][2] - wire = payload["answers"]["pick"] - require( - wire.get("type") == "choice" and not choice_faults(wire, list(queues)), - "Invalid wire choice probabilities/confidence", - ) - require( - (wire["choice"], wire["probabilities"], wire["confidence"]) - == (answer["key"], answer["probabilities"], answer["confidence"]), - "Wire and original validation result differ", - ) - expected_raw = payload - if identity["requested_model"] == "english": - from s1a.decision_models.served import served_by_from_health - - sent = payload.get("served_by") - if isinstance(sent, dict): - served_by = {**sent, "source": "response"} - else: - history = events if journal is None else journal - starts = {row["event_index"]: row for row in history if row["kind"] == "http_start"} - health = {} - for row in history: - if row["event_index"] >= replies[-1][0]["event_index"]: - break - start = starts.get(row.get("start_event"), {}) - url = urlsplit(start.get("url", "")) - if ( - row["kind"] == "http_terminal" - and row.get("status") == "response" - and row.get("http_status") == 200 - and start.get("method") == "GET" - and url.hostname in ("127.0.0.1", "localhost") - and url.port == 18188 - and url.path == "/health" - ): - try: - reading = bytes_json(row["response_body"]) - except (UnicodeError, json.JSONDecodeError): - continue - if isinstance(reading, dict) and reading: - health = reading - provenance = validated["raw"]["served_by"] - read_at = provenance.get("read_at") - require(read_at is None or isinstance(read_at, str), "Invalid original health reading timestamp") - served_by = served_by_from_health(health, payload.get("routing"), read_at) - expected_raw = {**payload, "served_by": served_by} - require( - all(validated["raw"].get(key) == value for key, value in expected_raw.items()), - "Validated raw differs from actual HTTP reply", - ) - require( - tick["key"] == answer["key"] - and tick["probabilities"] == answer["probabilities"] - and tick["confidence"] == round(answer["confidence"], 3) - and tick["model"] == validated["model"], - "Slot tick differs from original validation result", - ) - require( - slot["event_index"] < validation["event_index"] < posts[0]["event_index"] - and replies[-1][1]["event_index"] < result["event_index"] < slot_result["event_index"], - "Decision/POST/validation/slot event order mismatch", - ) - return tick, context, slot_result - - -async def score_cell(root, arm, item, binding, producer, fixture, TicketRouterEnv, queues, rules, choice_faults): - identity, python = verify_identity(root, arm, item, binding) - events, integrity = producer.read_journal(arm / "journal.jsonl", identity) - errors = [f"journal: {row['error']}" for row in integrity] - # Pair/identity faults affect their events; a broken cleanup cannot erase closed ticket evidence. - damaged_events = {row[key] for row in integrity for key in ("event_index", "start_event") if key in row} - malformed_lines = [row["line"] for row in integrity if row["code"] == "invalid_json_event"] - if events and any(row["code"] == "missing_final_newline" for row in integrity): - damaged_events.add(events[-1]["event_index"]) - terminals = {row.get("start_event"): row for row in events if row["kind"].endswith("_terminal")} - env = TicketRouterEnv(0, tickets=fixture, batch_size=30) - await env.reset() - planned_ids = env.report()["ticket_ids"] - labels = {row["id"]: row["label"] for row in fixture} - observations = [] - for ticket_id in planned_ids: - observations.append(await env.observe()) - await env.step(labels[ticket_id]) - observations.append(await env.observe()) - planned = [ - dict( - suite="ticket_router", - episode_seed=0, - sequence=index + 1, - ticket_id=ticket_id, - public_ticket=observations[index]["ticket"], - public_ticket_sha256=digest(dump(observations[index]["ticket"]).encode("utf-8")), - expected_next_ticket_id=planned_ids[index + 1] if index < 29 else None, - planned_denominator=30, - ) - for index, ticket_id in enumerate(planned_ids) - ] - initial = read_json(arm / "planned-initial.json") - actual_plan = read_json(arm / "planned-cases.json") - require( - initial["planned_denominator"] == actual_plan["planned_denominator"] == 30 and initial["cases"] == planned, - "Prebuilt initial plan differs from fixture-derived seed0 order/public fields", - ) - require( - initial["identity"]["benchmark_run_id"] == root.name - and initial["identity"]["planned_model"] == arm.name - and initial["identity"]["identity_state"] == "planned_contract_only", - "Prebuilt plan identity differs from this run/cell", - ) - require( - actual_plan["identity"] - == {key: identity[key] for key in ("benchmark_run_id", "model_identity_sha256")} | {"record": "identity.json"}, - "Current actual plan identity mismatch", - ) - require( - actual_plan["cases"] - == [ - { - **case, - "logical_key": [root.name, identity["model_identity_sha256"], "ticket_router", 0, case["ticket_id"]], - } - for case in planned - ], - "Current actual plan/logical keys mismatch", - ) - cases = [ - dict( - ticket_id=ticket_id, - planned_sequence=index + 1, - expected=labels[ticket_id], - attempted=False, - post_attempts=0, - accepted=False, - correct=False, - predicted=None, - errors=[], - ) - for index, ticket_id in enumerate(planned_ids) - ] - decisions = {} - for row in events: - context = row.get("context") - if context is None: - if row["kind"] in ("http_start", "http_terminal"): - try: - for field in ("request_body", "response_body"): - if field in row: - body_bytes(row[field]) - except (KeyError, TypeError, ValueError) as error: - errors.append(f"Non-decision HTTP evidence: {error}") - if row["kind"] == "http_start" and row["method"] == "POST": - errors.append("POST outside original decision context; zero-chat provenance insufficient") - continue - ticket_id = context.get("case_id") - if ticket_id not in planned_ids: - errors.append("Unknown current ticket in journal context") - continue - index = planned_ids.index(ticket_id) - expected_key = [root.name, identity["model_identity_sha256"], "ticket_router", 0, ticket_id] - if ( - context.get("logical_key") != expected_key - or any( - type(context.get(key)) is not int - for key in ("decision_index", "planned_sequence", "env_index", "episode_seed") - ) - or context.get("decision_index", 0) <= 0 - or context.get("planned_sequence") != index + 1 - or context.get("env_index") != index - or context.get("suite") != "ticket_router" - or context.get("episode_seed") != 0 - ): - cases[index]["errors"].append("Current case/seed/logical identity mismatch") - decisions.setdefault(context.get("decision_index"), []).append(row) - verified_ticks, verified_acts = [], [] - for related in decisions.values(): - context = related[0]["context"] - ticket_id = context.get("case_id") - index = planned_ids.index(ticket_id) - case = cases[index] - case["attempted"] = True - case["post_attempts"] += sum(row["kind"] == "http_start" and row["method"] == "POST" for row in related) - try: - require( - not case["errors"] - and not any(row["event_index"] in damaged_events for row in related) - and not any(line <= max(row["event_index"] for row in related) for line in malformed_lines), - "Journal/current case integrity unavailable", - ) - verified = verify_decision( - related, observations[index], queues, rules, identity, choice_faults, journal=events - ) - acts = [row for row in related if row["kind"] == "act_start"] - steps = [row for row in related if row["kind"] == "env_step_start"] - if verified is None: - require(not acts and not steps, "No valid slot decision for executed act/step") - continue - tick, slot_context, slot_result = verified - verified_ticks.append(tick) - if not acts: - require(not steps, "Environment step without original act") - continue - require(len(acts) == len(steps) == 1 and not case["accepted"], "Duplicate act/step for planned ticket") - act, step = acts[0], steps[0] - act_result, step_result = terminals[act["event_index"]], terminals[step["event_index"]] - key = tick["key"] - act_context = {**slot_context, "selected_key": key} - require( - act["context"] == step["context"] == act_context - and act["inputs"] == {"key": key} - and act["association_matches"] is True - and step["key"] == key, - "Selected key/accepted act association mismatch", - ) - require( - act_result["status"] == step_result["status"] == "returned" - and act_result["act_calls"] == [{"key": key, "accepted": True}], - "Original act/step was not accepted", - ) - require( - slot_result["event_index"] - < act["event_index"] - < step["event_index"] - < step_result["event_index"] - < act_result["event_index"], - "Original act/step event order mismatch", - ) - require( - step["before"] == dict(index=index, done=False, observation=observations[index], candidates=queues) - and step_result["after"] - == dict( - index=index + 1, - done=index == 29, - observation=observations[index + 1], - candidates={} if index == 29 else queues, - ), - "Original environment step before/after differs from seeded fixture", - ) - for view, expected_index in ((act["before"], index), (act_result["after"], index + 1)): - require( - view["state"] == observations[expected_index] - and view["done"] == (expected_index == 30) - and view["candidates"] == ({} if expected_index == 30 else queues), - "Original act view differs from seeded fixture", - ) - require(decode(act_result["raw_tool_result"]) == act_result["after"], "Original act output/view mismatch") - case.update( - accepted=True, - correct=key == labels[ticket_id], - predicted=key, - accepted_event=act_result["event_index"], - step_event=step_result["event_index"], - ) - verified_acts.append((case, act, act_result)) - except (AttributeError, KeyError, TypeError, ValueError, OverflowError) as error: - case["errors"].append(str(error)) - for case in cases: - errors.extend(f"{case['ticket_id']}: {error}" for error in case["errors"]) - process_identity_state = "verified" - try: - verify_processes(arm, identity) - except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: - process_identity_state = "invalid" - errors.append(str(error)) - # Partial/timeout episodes retain independently verified accepted prefixes. - complete, result_type, job_source = False, None, None - try: - writes = [ - row - for row in events - if row["kind"] == "lifecycle_terminal" - and row.get("stage") == "write_job" - and row.get("status") == "returned" - ] - require(len(writes) == 1, "Missing unique actual write_job result") - job = relocated(writes[0]["result"], item["arm"], arm) - require( - job.is_relative_to((arm / "s1a-home/evals/results/ticket_router").resolve()), - "Original job outside current cell", - ) - paths = list(job.glob("*/agent/episode.json")) - require(len(paths) == 1, "Missing unique original episode in actual returned job") - path = paths[0] - episode, trial, config = ( - read_json(path), - read_json(path.parents[1] / "result.json"), - read_json(job / "config.json"), - ) - for name in ("result.json", "summary.json"): - read_json(job / name) # Existence/JSON only: score and DONE are not an oracle. - job_source = dict( - job=str(job), - original_path=writes[0]["result"], - episode_sha256=digest(path.read_bytes()), - trial_sha256=digest((path.parents[1] / "result.json").read_bytes()), - ) - require( - config["agents"] == [{"name": "s1a-evals/" + CELLS[arm.name][0], "model_name": CELLS[arm.name][0]}] - and config["datasets"] == [{"name": "ticket_router", "n_tasks": 1}], - "Original job policy/plan mismatch", - ) - require( - trial["task_name"] == "ticket_router/0" and trial["source"] == "ticket_router", - "Original trial seed/task mismatch", - ) - extra, metadata = episode["extra"], trial["agent_result"]["metadata"] - result_type = extra.get("result_type") - require( - extra["rethink"] is False - and not extra.get("rethinks") - and type(metadata["chat_calls"]) is int - and metadata["chat_calls"] == 0, - "Original episode must have rethink off and zero chat", - ) - require(episode["decisions"] == verified_ticks, "Original job decisions differ from verified slot ticks") - accepted = [case for case in cases if case["accepted"]] - report = extra["ticket_router"] - expected_routes = [ - { - "id": case["ticket_id"], - "expected": case["expected"], - "predicted": case["predicted"], - "correct": case["correct"], - } - for case in accepted - ] - require( - report["seed"] == 0 - and report["batch_sha256"] == binding["batch_sha256"] - and report["total"] == 30 - and report["processed"] == len(accepted) - and report["ticket_ids"] == planned_ids - and report["unprocessed_ids"] == planned_ids[len(accepted) :] - and report["routes"] == expected_routes - and report["correct"] == sum(case["correct"] for case in accepted), - "Original report differs from fixture-derived routes/labels/full denominator", - ) - require( - [case["planned_sequence"] for case in accepted] == list(range(1, len(accepted) + 1)), - "Accepted routes are not a prefix", - ) - views = episode["views"] - require( - len(views) == len(accepted) + 1 and episode["final_state"] == observations[len(accepted)], - "Original job views/final state mismatch", - ) - for index, view in enumerate(views): - require( - view["state"] == observations[index] - and view["candidates"] == ({} if index == 30 else queues) - and view["done"] == (index == 30) - and view["score"] == sum(case["correct"] for case in accepted[:index]), - "Original job view differs from independently replayed prefix", - ) - for index, (_, act, act_result) in enumerate(verified_acts): - require( - act["before"] == views[index] and act_result["after"] == views[index + 1], - "Original job/act views mismatch", - ) - require( - metadata["steps"] == len(accepted) - and metadata["decisions"] == len(verified_ticks) - and metadata["invalid_keys"] == 0, - "Original job action counts mismatch", - ) - lifecycle = [row for row in events if row["kind"] == "lifecycle_terminal"] - required = { - stage: [row for row in lifecycle if row.get("stage") == stage] for stage in producer.REQUIRED_LIFECYCLES - } - require( - all(len(rows) == 1 and rows[0]["status"] == "returned" for rows in required.values()), - "Original lifecycle incomplete or abnormal", - ) - require( - required["Runner.run_agent"][0].get("result_type") == result_type - and required["Runner.stop"][0]["event_index"] > writes[0]["event_index"], - "Original runner result/stop mismatch", - ) - runner = required["Runner.run_agent"][0] - require( - all( - runner["start_event"] < row["event_index"] < runner["event_index"] - for row in events - if row.get("context") is not None - ), - "Decision/act/environment evidence outside original Runner.run_agent", - ) - resets = [row for row in events if row["kind"] == "env_reset_terminal"] - require(len(resets) == 1, "Original episode must have exactly one environment reset") - cli = [row for row in events if row["kind"] == "cli_start"] - require(len(cli) == 1, "Missing unique original CLI start") - ordered = [cli[0]["event_index"]] - for stage in ("Runner.start", "build_model", "make_series", "DecisionModel.warm", "TicketRouterEnv.__init__"): - row = required[stage][0] - ordered.extend((row["start_event"], row["event_index"])) - ordered.extend((required["run_episode"][0]["start_event"], resets[0]["start_event"], resets[0]["event_index"])) - for stage in ( - "DeepAgent.ensure_initialized", - "Runner.run_agent", - "DeepAgent.cleanup_task_resources", - "ability_manager.teardown_tools", - "Runner.release", - ): - row = required[stage][0] - ordered.extend((row["start_event"], row["event_index"])) - ordered.append(required["run_episode"][0]["event_index"]) - for stage in ("annotate", "DecisionModel.close", "write_job", "Runner.stop"): - row = required[stage][0] - ordered.extend((row["start_event"], row["event_index"])) - require(all(left < right for left, right in zip(ordered, ordered[1:])), "Original lifecycle order mismatch") - processes = [row for row in events if row["kind"] == "process_terminal"] - expected_argv = [ - "s1a", - "run", - "ticket_router", - "--model", - CELLS[arm.name][0], - "--rethink", - "off", - "--episodes", - "1", - "--seed", - "0", - "--max-steps", - "30", - "--timeout", - "300", - "--dataset", - str(Path(python["expected_s1a_root"]) / "s1a/agents/_data/ticket_router_eval.jsonl"), - "--batch-size", - "30", - "--log", - ] - require(cli[0]["argv"] == expected_argv, "Actual CLI plan/budget differs from frozen contract") - require( - len(processes) == 1 - and events[-1] == processes[0] - and processes[0].get("state") == "returned" - and processes[0].get("exit_code") == 0 - and processes[0].get("stage") == "original_cli_entry" - and not processes[0].get("error") - and not processes[0].get("received_signal") - and item.get("collector_exit_code") == 0, - "Original process/collector did not complete normally", - ) - require( - not any( - row["kind"] == "process_signal" or (row["kind"] != "http_terminal" and row.get("status") == "exception") - for row in events - ), - "Abnormal non-HTTP termination", - ) - require( - trial.get("exception_info") is None and result_type == "answer", - "Non-normal or unknown original result_type", - ) - require(len(accepted) == 30 and not errors, "Incomplete accepted coverage or evidence errors") - complete = True - except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: - errors.append(str(error)) - try: - framework = framework_sources(arm, item["arm"]) - if framework["state"] != "verified": - errors.append("Actual framework source capture missing or incomplete") - complete = False - except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: - framework = {"state": "invalid", "record": str(arm / "framework-result-type-source.json")} - errors.append(str(error)) - complete = False - correct = sum(case["correct"] for case in cases if case["accepted"]) - processed = sum(case["accepted"] for case in cases) - return dict( - cell=arm.name, - planned_routes=30, - planned_episodes=1, - cases=cases, - route_correct=correct, - route_accuracy=correct / 30, - route_evidence_basis="recorded cell identity and individually verified HTTP/slot/accepted act/environment step", - accepted_processed=processed, - accepted_coverage=processed / 30, - complete_episode=complete, - strict_batch_success=complete and correct == 30, - result_type=result_type, - normal_result_type_basis=NORMAL_BASIS, - quality_status="PROVISIONAL" if complete else "INCOMPLETE", - required_input="independent review of actual loaded framework result_type source", - framework_sources=framework, - identity=identity, - identity_basis="exact recorded assets; source-to-build ancestry unverified", - process_identity_state=process_identity_state, - original_job=job_source, - errors=errors, - ) - - -def framework_sources(arm, remote_arm): - path = arm / "framework-result-type-source.json" - if not path.exists(): - return {"state": "missing", "candidates": []} - record = read_json(path) - loaded = read_json(arm / "python-loaded-after.json") - for row in record["candidates"]: - if "capture_error" in row: - continue - raw = relocated(row["copy"], remote_arm, arm).read_bytes() - require( - len(raw) == row["bytes"] and digest(raw) == row["sha256"] == row["loaded_module_record_sha256"], - "Actual framework source copy hash mismatch", - ) - require( - loaded.get(row["module"]) == {"path": row["path"], "sha256": row["sha256"]}, - "Framework source capture differs from actually loaded module", - ) - record["state"] = ( - "verified" - if record["candidates"] and not any("capture_error" in row for row in record["candidates"]) - else "incomplete" - ) - return record # Exact files are supplied for independent enum review, not interpreted as normal here. - - -def aggregate(root, preparation): - try: - from s1a.agents.ticket_router import QUEUES, RULES, TicketRouterEnv, load_tickets - from s1a.decision_models.validation import choice_faults - except ModuleNotFoundError as error: - raise RuntimeError( - "Ticket scoring needs the existing System1-Agents environment (including openjiuwen); " - "public231 scoring is independent of that dependency" - ) from error - prep, binding, producer = preparation - current = Path(__file__).resolve().parents[1] - require( - Path(TicketRouterEnv.__init__.__code__.co_filename).resolve() == current / "s1a/agents/ticket_router.py" - and Path(choice_faults.__code__.co_filename).resolve() == current / "s1a/decision_models/validation.py", - "Offline scorer imported a different Agents contract root", - ) - fixture = load_tickets(prep / "runtime/s1a/agents/_data/ticket_router_eval.jsonl") - require(len(fixture) == 30 and len({row["id"] for row in fixture}) == 30, "Fixture must contain 30 unique tickets") - frozen_plan = [decode(line) for line in (prep / "planned-cases.jsonl").read_text(encoding="utf-8").splitlines()] - cells, root_errors, items = [], [], {} - try: - status = read_json(root / "status.json") - require( - status["planned_models"] == list(CELLS) and status["planned_tickets_per_model"] == 30, - "Measurement full planned cells/denominator mismatch", - ) - require( - len(status["models"]) == 3 and {item["name"] for item in status["models"]} == set(CELLS), - "Measurement duplicate or unknown cell", - ) - require( - (root / "binding.json").read_bytes() == (prep / "binding.json").read_bytes(), - "Measurement binding differs from frozen producer", - ) - source = read_json(root / "source.json") - require( - source - == { - "binding_sha256": BINDING_SHA256, - "files": binding["source_sha256"], - "runtime_revision": binding["agents_revision"], - }, - "Measurement source identity mismatch", - ) - items = {item["name"]: item for item in status["models"]} - except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: - root_errors.append(str(error)) - for name in CELLS: - try: - require(not root_errors and name in items, "Missing or invalid measurement root/cell identity") - cell = asyncio.run( - score_cell( - root, - root / name, - items[name], - binding, - producer, - fixture, - TicketRouterEnv, - QUEUES, - RULES, - choice_faults, - ) - ) - except (AttributeError, OSError, KeyError, TypeError, ValueError) as error: - cases = [ - dict( - ticket_id=case["ticket_id"], - planned_sequence=case["sequence"], - attempted=False, - accepted=False, - correct=False, - predicted=None, - errors=[str(error)], - ) - for case in frozen_plan - ] - cell = dict( - cell=name, - planned_routes=30, - planned_episodes=1, - cases=cases, - route_correct=0, - route_accuracy=0.0, - accepted_processed=0, - accepted_coverage=0.0, - complete_episode=False, - strict_batch_success=False, - quality_status="MISSING_OR_INVALID", - errors=[str(error)], - identity=None, - process_identity_state="unavailable", - original_job=None, - normal_result_type_basis=NORMAL_BASIS, - ) - cells.append(cell) - return dict( - schema="ticket-raw-offline-quality-v1", - run=str(root.resolve()), - planned_routes=90, - planned_episodes=3, - cells=cells, - root_errors=root_errors, - latency_reported=False, - route_correct=sum(cell["route_correct"] for cell in cells), - accepted_processed=sum(cell["accepted_processed"] for cell in cells), - complete_episodes=sum(cell["complete_episode"] for cell in cells), - strict_batch_successes=sum(cell["strict_batch_success"] for cell in cells), - producer_binding_sha256=BINDING_SHA256, - normal_result_type_basis=NORMAL_BASIS, - ) - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--run", type=Path, required=True, help="whole measurement root, including missing cells") - parser.add_argument("--preparation", type=Path, required=True, help="frozen ticket producer r3 directory") - parser.add_argument("--out", type=Path, required=True) - args = parser.parse_args() - result = aggregate(args.run, load_preparation(args.preparation)) - output = outside_repo(args.out) - output.mkdir(parents=True, exist_ok=False) - write_json(output / "summary.json", result) - print(dump(result)) - - -if __name__ == "__main__": - main() diff --git a/tests/benchmarks/test_quality_public231.py b/tests/benchmarks/test_quality_public231.py index 79359ff..77fb4f9 100644 --- a/tests/benchmarks/test_quality_public231.py +++ b/tests/benchmarks/test_quality_public231.py @@ -7,7 +7,7 @@ import pytest -from benchmarks.evidence import digest, dump, load_reference +from benchmarks.evidence import body_for, digest, dump, load_reference from benchmarks.quality_public231 import adapt_record, aggregate, load_collector @@ -26,6 +26,38 @@ def collector(): return load_collector(Path(source)) +@pytest.fixture +def input_manifest(tmp_path, reference): + _, items, dataset = reference + rows = [ + dict( + id=task.id, + request=body_for(task, None), + expected={"decision": task.expected == "yes" if task.question["type"] == "noul" else task.expected}, + source={**identity, "jevbench_pin": dataset["jevbench_pin"]}, + labels=task.labels, + provenance=task.provenance, + group=task.group, + reference_expected=task.expected, + ) + for identity, task in items + ] + data = "".join(dump(row) + "\n" for row in rows).encode("utf-8") + assert digest(data) == "00d1f10ca1142499a598b5f1c6602d329cce1cef04c961c46fc0318daa2c67d3" + (tmp_path / "public231.jsonl").write_bytes(data) + manifest = tmp_path / "manifest.json" + manifest.write_text( + dump( + dict( + output={"path": "public231.jsonl", "sha256": digest(data)}, + sources={"jevbench_pin": dataset["jevbench_pin"], "reference_dataset_hash": dataset["dataset_hash"]}, + ) + ), + encoding="utf-8", + ) + return manifest + + def response_record(answer, *, kind="invalid_response", status=200): text = dump({"answers": {"decision": answer}, "model": "configured"}) return dict( @@ -147,11 +179,8 @@ def saved_run(tmp_path, collector, input_path, *, count=231): return run -def test_full_plan_missing_round_and_raw_integrity(tmp_path, collector, reference): - manifest = os.environ.get("PUBLIC231_INPUT_MANIFEST") - if not manifest: - pytest.skip("Set PUBLIC231_INPUT_MANIFEST to the frozen input manifest") - manifest = Path(manifest) +def test_full_plan_missing_round_and_raw_integrity(tmp_path, collector, input_manifest): + manifest = input_manifest inputs = manifest.with_name("public231.jsonl") run = saved_run(tmp_path, collector, inputs, count=230) result, records = aggregate( @@ -173,13 +202,10 @@ def test_source_hash_is_required(tmp_path): load_collector(source) -def test_recorded_round_identity_is_separate_from_stable_model_config(tmp_path, collector, reference): +def test_recorded_round_identity_is_separate_from_stable_model_config(tmp_path, collector, input_manifest): import shutil - manifest = os.environ.get("PUBLIC231_INPUT_MANIFEST") - if not manifest: - pytest.skip("Set PUBLIC231_INPUT_MANIFEST to the frozen input manifest") - manifest = Path(manifest) + manifest = input_manifest first = saved_run(tmp_path, collector, manifest.with_name("public231.jsonl")) runs = [first, tmp_path / "round-2", tmp_path / "round-3"] for path in runs[1:]: @@ -209,11 +235,8 @@ def set_metadata(path, **values): aggregate(runs, manifest, collector, "fixture-config") -def test_started_without_terminal_keeps_attempt_and_full_denominator(tmp_path, collector, reference): - manifest = os.environ.get("PUBLIC231_INPUT_MANIFEST") - if not manifest: - pytest.skip("Set PUBLIC231_INPUT_MANIFEST to the frozen input manifest") - manifest = Path(manifest) +def test_started_without_terminal_keeps_attempt_and_full_denominator(tmp_path, collector, reference, input_manifest): + manifest = input_manifest run = saved_run(tmp_path, collector, manifest.with_name("public231.jsonl"), count=0) _, items, _ = reference state = json.loads((run / "completion.json").read_text()) diff --git a/tests/benchmarks/test_quality_ticket.py b/tests/benchmarks/test_quality_ticket.py deleted file mode 100644 index 2681e9d..0000000 --- a/tests/benchmarks/test_quality_ticket.py +++ /dev/null @@ -1,913 +0,0 @@ -"""Producer-shaped CPU evidence; framework stubs are confined to these tests.""" - -import base64 -import importlib.util -import json -import os -import sys -from pathlib import Path -from types import ModuleType - -import pytest - -from benchmarks.evidence import digest, dump -from benchmarks.quality_ticket import aggregate, load_preparation - - -@pytest.fixture -def preparation(monkeypatch, tmp_path): - path = os.environ.get("TICKET_PREPARATION_ROOT") - if not path: - pytest.skip("Set TICKET_PREPARATION_ROOT to the frozen producer preparation") - # Execute the actual validator and environment, without installing or initializing the framework. - if importlib.util.find_spec("openjiuwen") is None: - for name in ("openjiuwen.core.common.exception.codes", "openjiuwen.core.common.exception.errors"): - module = ModuleType(name) - module.StatusCode = object() - module.BaseError = ValueError - module.build_error = lambda *a, **k: ValueError("framework stub") - monkeypatch.setitem(sys.modules, name, module) - package = ModuleType("s1a.decision_models") - package.__path__ = [str(Path(__file__).resolve().parents[2] / "s1a/decision_models")] - monkeypatch.setitem(sys.modules, "s1a.decision_models", package) - config = ModuleType("s1a.config") - config.HOME = tmp_path - monkeypatch.setitem(sys.modules, "s1a.config", config) - rail = ModuleType("openjiuwen.core.single_agent.rail.base") - rail.AgentCallbackContext = type("AgentCallbackContext", (), {}) - rail.AgentCallbackEvent = type( - "AgentCallbackEvent", (), {"BEFORE_TOOL_CALL": "before", "AFTER_TOOL_CALL": "after"} - ) - monkeypatch.setitem(sys.modules, rail.__name__, rail) - logging = ModuleType("openjiuwen.core.common.logging") - logging.logger = None - monkeypatch.setitem(sys.modules, logging.__name__, logging) - return load_preparation(Path(path)) - - -def body(value): - raw = dump(value).encode() - return dict(bytes=len(raw), sha256=digest(raw), base64=base64.b64encode(raw).decode(), utf8=raw.decode()) - - -@pytest.fixture -def measurement(tmp_path, preparation): - prep, binding, producer = preparation - root = tmp_path / "cpu-ticket-fixture" - root.mkdir() - (root / "binding.json").write_bytes((prep / "binding.json").read_bytes()) - (root / "source.json").write_text( - dump( - dict( - binding_sha256=digest((prep / "binding.json").read_bytes()), - files=binding["source_sha256"], - runtime_revision=binding["agents_revision"], - ) - ) - ) - models = [ - dict(name=name, arm=f"/remote/{root.name}/{name}", collector_exit_code=0) for name in binding["service_order"] - ] - (root / "status.json").write_text( - dump(dict(job_id="123", planned_models=binding["service_order"], planned_tickets_per_model=30, models=models)) - ) - from s1a.agents.ticket_router import PUBLIC_FIELDS, QUEUES, RULES, load_tickets - import random - - rows = load_tickets(prep / "runtime/s1a/agents/_data/ticket_router_eval.jsonl") - rows = [{key: row[key] for key in (*PUBLIC_FIELDS, "label") if key in row} for row in rows] - random.Random(0).shuffle(rows) - name = "open-9b" - arm = root / name - arm.mkdir() - remote = f"/remote/{root.name}/{name}" - assets = { - "binary_sha256": {"/remote/worker": "a" * 64, "/remote/frontend": "c" * 64}, - "export": {"model_id": "fixture"}, - "verified_export_files": {"weights": {"sha256": "b" * 64}}, - } - identity = dict( - benchmark_run_id=root.name, - planned_model=name, - requested_model="open-jev-9b", - job_id="123", - source_binding_sha256=digest((prep / "binding.json").read_bytes()), - actual_assets=assets, - identity_state="current_exact_assets_recorded", - gpu={"name": "CPU mock; no GPU"}, - model_identity_sha256=digest(json.dumps(assets, ensure_ascii=False, sort_keys=True).encode()), - ) - (arm / "identity.json").write_text(dump(identity)) - (arm / "launch.json").write_text(dump(dict(worker="/remote/worker", frontend="/remote/frontend"))) - processes = {} - for role, sha in (("worker", "a" * 64), ("frontend", "c" * 64)): - maps = f"1000-2000 r-xp 00000000 00:00 1 /remote/{role}\n".encode() - processes[role] = dict( - pid=1 if role == "worker" else 2, - exe=f"/remote/{role}", - argv=[f"/remote/{role}"], - exe_sha256=sha, - maps_sha256=digest(maps), - loaded_libraries=[], - ) - for phase in ("ready", "after"): - (arm / f"{role}-{phase}.maps").write_bytes(maps) - for phase in ("ready", "after"): - (arm / f"process-{phase}.json").write_text(dump(processes)) - planned = [json.loads(line) for line in (prep / "planned-cases.jsonl").read_text().splitlines()] - (arm / "planned-initial.json").write_text( - dump( - dict( - planned_denominator=30, - cases=planned, - identity={ - "benchmark_run_id": root.name, - "planned_model": "open-9b", - "identity_state": "planned_contract_only", - }, - ) - ) - ) - plan_identity = {key: identity[key] for key in ("benchmark_run_id", "model_identity_sha256")} - plan_identity["record"] = "identity.json" - (arm / "planned-cases.json").write_text( - dump( - dict( - identity=plan_identity, - planned_denominator=30, - cases=[ - { - **case, - "logical_key": [ - root.name, - identity["model_identity_sha256"], - "ticket_router", - 0, - case["ticket_id"], - ], - } - for case in planned - ], - ) - ) - ) - modules = { - key[8:-3].replace("/", "."): {"path": "/remote/source/" + key[8:], "sha256": value} - for key, value in binding["source_sha256"].items() - if key.startswith("runtime/s1a/") and key.endswith(".py") - } - for key in list(modules): - if key.endswith(".__init__"): - modules[key[:-9]] = modules.pop(key) - (arm / "python-identity.json").write_text( - dump( - dict( - executable="/remote/python", - executable_sha256="c" * 64, - version="fixture", - modules=modules, - expected_s1a_root="/remote/source", - actual_s1a_root="/remote/source/s1a", - environment={ - "MODEL_NAME": "", - "S1A_HOME": remote + "/s1a-home", - "S1A_DECISION_TIMEOUT_S": "120", - "LAYA_SERVED_TIMEOUT_S": "120", - "TYPESAFE_MODEL": "open-jev-9b", - }, - ) - ) - ) - framework = b"# synthetic source capture containing result_type\n" - (arm / "framework.py").write_bytes(framework) - framework_sha = digest(framework) - modules["openjiuwen.fixture"] = {"path": "/remote/framework.py", "sha256": framework_sha} - (arm / "python-loaded-after.json").write_text(dump(modules)) - (arm / "framework-result-type-source.json").write_text( - dump( - dict( - normal_result_type_allowlist=["answer"], - gate_basis="conservative_provisional_pending_actual_framework_source_verification", - candidates=[ - dict( - module="openjiuwen.fixture", - path="/remote/framework.py", - copy=remote + "/framework.py", - bytes=len(framework), - sha256=framework_sha, - loaded_module_record_sha256=framework_sha, - ) - ], - ) - ) - ) - argv = [ - "s1a", - "run", - "ticket_router", - "--model", - "jev", - "--rethink", - "off", - "--episodes", - "1", - "--seed", - "0", - "--max-steps", - "30", - "--timeout", - "300", - "--dataset", - "/remote/source/s1a/agents/_data/ticket_router_eval.jsonl", - "--batch-size", - "30", - "--log", - ] - events = [] - compact = {key: identity[key] for key in ("benchmark_run_id", "model_identity_sha256")} - compact["record"] = "identity.json" - - def add(kind, **fields): - events.append( - dict( - event_index=len(events) + 1, - utc="fixture", - monotonic_ns=len(events) + 1, - kind=kind, - identity=compact, - **fields, - ) - ) - return len(events) - - def pair(kind, start, terminal): - index = add(kind + "_start", **start) - return add(kind + "_terminal", start_event=index, status="returned", **terminal) - - def observation(index): - if index == 30: - return {"ticket": None, "progress": {"ticket_id": None}} - return { - "ticket": {key: rows[index][key] for key in PUBLIC_FIELDS if key in rows[index]}, - "progress": {"ticket_id": rows[index]["id"]}, - } - - def view(index): - return dict( - state=observation(index), candidates={} if index == 30 else QUEUES, done=index == 30, score=float(index) - ) - - add("cli_start", argv=argv) - for stage in ("Runner.start", "build_model", "make_series", "DecisionModel.warm", "TicketRouterEnv.__init__"): - pair("lifecycle", {"stage": stage}, {"stage": stage}) - episode_start = add("lifecycle_start", stage="run_episode") - pair("env_reset", {}, dict(view=observation(0), candidates=QUEUES)) - pair("lifecycle", {"stage": "DeepAgent.ensure_initialized"}, {"stage": "DeepAgent.ensure_initialized"}) - runner_start = add("lifecycle_start", stage="Runner.run_agent") - ticks, routes = [], [] - for index, row in enumerate(rows): - context = dict( - decision_index=index + 1, - case_id=row["id"], - planned_sequence=index + 1, - suite="ticket_router", - episode_seed=0, - env_instance=123, - env_index=index, - logical_key=[root.name, identity["model_identity_sha256"], "ticket_router", 0, row["id"]], - ) - key = row["label"] - probabilities = {queue: float(queue == key) for queue in QUEUES} - question = dict(type="choice", criteria=QUEUES, instructions={"rules": RULES}) - request = dict(model="open-jev-9b", state=observation(index), questions={"pick": question}) - reply = dict( - model="fixture", - answers={"pick": dict(type="choice", choice=key, probabilities=probabilities, confidence=1.0)}, - ) - start_slot = add("slot_start", context=context, observation=observation(index), candidates=QUEUES) - start_validation = add( - "decision_validation_start", - context=context, - observation={"state": observation(index), "images": []}, - questions={"pick": dict(options=QUEUES, rules=[RULES], goal="", operation="")}, - kwargs={}, - ) - http = add( - "http_start", - context=context, - method="POST", - url="http://127.0.0.1:18188/v1/systemone", - category="decision_post", - post_attempt_index=1, - request_body=body(request), - ) - add( - "http_terminal", - context=context, - start_event=http, - status="response", - post_attempt_index=1, - http_status=200, - response_body=body(reply), - ) - add( - "decision_validation_terminal", - context=context, - start_event=start_validation, - status="returned", - result=dict( - answers={"pick": {"key": key, "probabilities": probabilities, "confidence": 1.0}}, - model="fixture", - raw=reply, - ), - ) - tick = dict(step=index + 1, key=key, confidence=1.0, probabilities=probabilities, source="jev", model="fixture") - ticks.append(tick) - add( - "slot_terminal", - context=context, - start_event=start_slot, - status="returned", - new_decisions=[tick], - state_error=None, - ) - act_context = dict(context, selected_key=key) - act = add("act_start", context=act_context, inputs={"key": key}, association_matches=True, before=view(index)) - pair( - "env_step", - dict( - context=act_context, - key=key, - before=dict(index=index, done=False, observation=observation(index), candidates=QUEUES), - ), - dict( - context=act_context, - after=dict( - index=index + 1, - done=index == 29, - observation=observation(index + 1), - candidates={} if index == 29 else QUEUES, - ), - ), - ) - add( - "act_terminal", - context=act_context, - start_event=act, - status="returned", - act_calls=[{"key": key, "accepted": True}], - after=view(index + 1), - raw_tool_result=dump(view(index + 1)), - ) - routes.append(dict(id=row["id"], predicted=key, expected=key, correct=True)) - add( - "lifecycle_terminal", - start_event=runner_start, - stage="Runner.run_agent", - status="returned", - result_type="answer", - ) - for stage in ("DeepAgent.cleanup_task_resources", "ability_manager.teardown_tools", "Runner.release"): - pair("lifecycle", {"stage": stage}, {"stage": stage}) - add("lifecycle_terminal", start_event=episode_start, stage="run_episode", status="returned") - for stage in ("annotate", "DecisionModel.close"): - pair("lifecycle", {"stage": stage}, {"stage": stage}) - job = arm / "s1a-home/evals/results/ticket_router/fixture__jev" - trial = job / "ticket_router--0__fixture" - (trial / "agent").mkdir(parents=True) - report = dict( - batch_sha256=binding["batch_sha256"], - seed=0, - total=30, - processed=30, - correct=30, - coverage=1, - ticket_ids=[row["id"] for row in rows], - unprocessed_ids=[], - routes=routes, - ) - episode = dict( - decisions=ticks, - final_state=observation(30), - views=[view(i) for i in range(31)], - extra={"ticket_router": report, "rethink": False, "rethinks": [], "result_type": "answer"}, - ) - (trial / "agent/episode.json").write_text(dump(episode)) - (trial / "result.json").write_text( - dump( - dict( - task_name="ticket_router/0", - source="ticket_router", - exception_info=None, - agent_result={"metadata": {"steps": 30, "decisions": 30, "chat_calls": 0, "invalid_keys": 0}}, - ) - ) - ) - (job / "config.json").write_text( - dump( - dict( - agents=[{"name": "s1a-evals/jev", "model_name": "jev"}], - datasets=[{"name": "ticket_router", "n_tasks": 1}], - ) - ) - ) - for name in ("result.json", "summary.json"): - (job / name).write_text("{}") - pair( - "lifecycle", - {"stage": "write_job"}, - {"stage": "write_job", "result": remote + "/s1a-home/evals/results/ticket_router/fixture__jev"}, - ) - pair("lifecycle", {"stage": "Runner.stop"}, {"stage": "Runner.stop"}) - add("process_terminal", state="returned", exit_code=0, stage="original_cli_entry") - - def save_events(): - (arm / "journal.jsonl").write_text("".join(dump(event) + "\n" for event in events)) - - save_events() - return root, arm, events, save_events, preparation - - -def test_complete_routes_are_provisional_and_missing_cells_remain(measurement): - root, _, _, _, preparation = measurement - result = aggregate(root, preparation) - cell = result["cells"][0] - assert cell["route_correct"] == cell["accepted_processed"] == 30 - assert cell["complete_episode"] and cell["strict_batch_success"] - assert cell["quality_status"] == "PROVISIONAL" - assert cell["process_identity_state"] == "verified" - assert result["planned_routes"] == 90 and result["planned_episodes"] == 3 - assert [row["accepted_processed"] for row in result["cells"]] == [30, 0, 0] - - -@pytest.mark.parametrize( - "mutation", ["expected", "probability", "request_label", "selected_key", "after_ticket", "chat", "timeout"] -) -def test_corrupt_or_abnormal_evidence_cannot_pass(measurement, mutation): - root, arm, events, save_events, preparation = measurement - episode_path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) - episode = json.loads(episode_path.read_text()) - if mutation == "expected": - episode["extra"]["ticket_router"]["routes"][0]["expected"] = "tampered" - elif mutation == "probability": - next(e for e in events if e["kind"] == "slot_terminal")["new_decisions"][0]["probabilities"] = {"human": 1} - elif mutation == "request_label": - event = next(e for e in events if e["kind"] == "http_start") - request = json.loads(event["request_body"]["utf8"]) - request["state"]["ticket"]["label"] = "human" - event["request_body"] = body(request) - elif mutation == "selected_key": - next(e for e in events if e["kind"] == "act_start")["inputs"]["key"] = "tampered" - elif mutation == "after_ticket": - next(e for e in events if e["kind"] == "env_step_terminal")["after"]["observation"]["ticket"]["id"] = "wrong" - elif mutation == "chat": - path = episode_path.parents[1] / "result.json" - trial = json.loads(path.read_text()) - trial["agent_result"]["metadata"]["chat_calls"] = 1 - path.write_text(dump(trial)) - elif mutation == "timeout": - episode["extra"]["result_type"] = "timeout" - next(e for e in events if e.get("stage") == "Runner.run_agent" and e["kind"] == "lifecycle_terminal")[ - "result_type" - ] = "timeout" - episode_path.write_text(dump(episode)) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert not cell["strict_batch_success"] and not cell["complete_episode"] - assert cell["planned_routes"] == 30 and cell["errors"] - if mutation == "timeout": - assert cell["route_correct"] == 30 - - -def test_journal_gap_and_body_hash_fail_closed(measurement): - root, _, events, save_events, preparation = measurement - next(e for e in events if e["kind"] == "http_start")["request_body"]["sha256"] = "0" * 64 - events[-1]["event_index"] += 1 - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert not cell["strict_batch_success"] and cell["errors"] - - -@pytest.mark.parametrize("value", ["off", True, 0, None]) -def test_rethink_requires_original_false_boolean(measurement, value): - root, arm, _, _, preparation = measurement - path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) - episode = json.loads(path.read_text()) - episode["extra"]["rethink"] = value - path.write_text(dump(episode)) - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == cell["route_correct"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] - - -@pytest.mark.parametrize("damage", ["final_newline", "stop_terminal", "process_terminal"]) -def test_tail_damage_retains_closed_routes(measurement, damage): - root, arm, events, save_events, preparation = measurement - if damage == "final_newline": - path = arm / "journal.jsonl" - path.write_bytes(path.read_bytes().rstrip(b"\n")) - else: - events[:] = [ - row - for row in events - if not ( - row["kind"] == "process_terminal" - if damage == "process_terminal" - else row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.stop" - ) - ] - renumber(events) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == cell["route_correct"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] and cell["errors"] - - -def test_broken_ticket_pair_only_loses_affected_ticket(measurement): - root, _, events, save_events, preparation = measurement - first = next(row for row in events if row["kind"] == "act_terminal") - events.remove(first) - renumber(events) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert not cell["cases"][0]["accepted"] - assert cell["accepted_processed"] == cell["route_correct"] == 29 - assert not cell["complete_episode"] - - -def test_corrupt_health_body_blocks_episode_without_losing_routes(measurement): - root, _, events, save_events, preparation = measurement - compact = events[-1]["identity"] - events[-1:-1] = [ - dict( - event_index=-1, - kind="http_start", - identity=compact, - context=None, - method="GET", - url="http://127.0.0.1:18188/health", - category="health", - request_body=body({}), - ), - dict( - event_index=-2, - kind="http_terminal", - identity=compact, - context=None, - start_event=-1, - status="response", - http_status=200, - response_body=body({"status": "ready"}), - ), - ] - renumber(events) - save_events() - assert aggregate(root, preparation)["cells"][0]["complete_episode"] - events[-2]["response_body"]["sha256"] = "0" * 64 - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == cell["route_correct"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] - assert any("body length/hash" in error for error in cell["errors"]) - - -@pytest.mark.parametrize("damage", ["ready_hash", "after_hash", "launch", "maps", "after_pid"]) -def test_process_asset_binding_blocks_episode_preserving_routes(measurement, damage): - root, arm, _, _, preparation = measurement - assert aggregate(root, preparation)["cells"][0]["complete_episode"] - path = arm / ("process-after.json" if damage in ("after_hash", "after_pid") else "process-ready.json") - capture = json.loads(path.read_text()) - if damage.endswith("hash"): - capture["worker"]["exe_sha256"] = "0" * 64 - path.write_text(dump(capture)) - elif damage == "after_pid": - capture["worker"]["pid"] += 1 - path.write_text(dump(capture)) - elif damage == "launch": - launch = json.loads((arm / "launch.json").read_text()) - launch["worker"] = "/remote/other-worker" - (arm / "launch.json").write_text(dump(launch)) - else: - (arm / "worker-ready.maps").write_text("changed") - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == cell["route_correct"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] and cell["errors"] - assert cell["process_identity_state"] == "invalid" - - -def test_framework_copy_corruption_preserves_verified_routes(measurement): - root, arm, _, _, preparation = measurement - remote = f"/remote/{root.name}/open-9b" - copy = arm / "framework.py" - copy.write_text("# captured framework\n") - sha = digest(copy.read_bytes()) - row = dict( - module="openjiuwen.normal", - path="/remote/framework.py", - copy=remote + "/framework.py", - bytes=copy.stat().st_size, - sha256=sha, - loaded_module_record_sha256=sha, - ) - path = arm / "python-loaded-after.json" - modules = json.loads(path.read_text()) - modules[row["module"]] = {"path": row["path"], "sha256": sha} - path.write_text(dump(modules)) - (arm / "framework-result-type-source.json").write_text(dump(dict(candidates=[row]))) - assert aggregate(root, preparation)["cells"][0]["complete_episode"] - copy.write_text("changed") - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == cell["route_correct"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] - assert cell["framework_sources"]["state"] == "invalid" - assert any("framework source copy hash" in error for error in cell["errors"]) - - -@pytest.mark.parametrize("damage", ["missing", "empty", "capture_error", "partial_capture_error"]) -def test_missing_or_incomplete_framework_capture_cannot_complete(measurement, damage): - root, arm, _, _, preparation = measurement - assert aggregate(root, preparation)["cells"][0]["complete_episode"] - path = arm / "framework-result-type-source.json" - capture = json.loads(path.read_text()) - failure = dict(module="openjiuwen.failed", path="/remote/failed.py", capture_error="read denied") - if damage == "missing": - path.unlink() - else: - capture["candidates"] = ( - [] if damage == "empty" else ([failure] if damage == "capture_error" else capture["candidates"] + [failure]) - ) - path.write_text(dump(capture)) - cell = aggregate(root, preparation)["cells"][0] - assert cell["route_correct"] == cell["accepted_processed"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] and cell["errors"] - assert cell["framework_sources"]["state"] == ("missing" if damage == "missing" else "incomplete") - - -@pytest.mark.parametrize( - "damage", ["early_runner_return", "early_release", "early_episode_return", "late_reset", "late_init"] -) -def test_original_runner_and_cleanup_order_is_required(measurement, damage): - root, _, events, save_events, preparation = measurement - assert aggregate(root, preparation)["cells"][0]["complete_episode"] - if damage == "early_runner_return": - terminal = next( - row for row in events if row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.run_agent" - ) - events.remove(terminal) - events.insert(next(i for i, row in enumerate(events) if row.get("stage") == "Runner.run_agent") + 1, terminal) - elif damage in ("late_reset", "late_init"): - moved = [ - row - for row in events - if ( - row["kind"] in ("env_reset_start", "env_reset_terminal") - if damage == "late_reset" - else row.get("stage") == "Runner.start" - ) - ] - events[:] = [row for row in events if row not in moved] - at = next( - i - for i, row in enumerate(events) - if row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.run_agent" - ) - if damage == "late_init": - at += 1 - events[at:at] = moved - else: - stage = "Runner.release" if damage == "early_release" else "run_episode" - moved = [ - row - for row in events - if row.get("stage") == stage and (stage == "Runner.release" or row["kind"] == "lifecycle_terminal") - ] - events[:] = [row for row in events if row not in moved] - at = next( - i - for i, row in enumerate(events) - if row["kind"] == "lifecycle_terminal" and row.get("stage") == "Runner.run_agent" - ) - events[at:at] = moved - renumber(events) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert cell["route_correct"] == cell["accepted_processed"] == 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] - assert any("lifecycle order" in error or "outside original Runner" in error for error in cell["errors"]) - - -@pytest.mark.parametrize("provenance_source", ["response", "routing", "health"]) -def test_served_laya_original_raw_provenance_transform(measurement, provenance_source): - import asyncio - import copy - - from benchmarks.quality_ticket import verify_decision - from s1a.agents.ticket_router import QUEUES, RULES - from s1a.decision_models.served import ServedLayaModel - from s1a.decision_models.types import ChoiceQuestion, Observation - from s1a.decision_models.validation import choice_faults, validate_answers - - _, _, events, _, preparation = measurement - context = next(row["context"] for row in events if row["kind"] == "slot_start") - related = copy.deepcopy([row for row in events if (row.get("context") or {}).get("case_id") == context["case_id"]]) - slot = next(row for row in related if row["kind"] == "slot_start") - terminal = next(row for row in related if row["kind"] == "http_terminal") - payload = json.loads(terminal["response_body"]["utf8"]) - payload.update( - model="english", - served_by={"checkpoint": "laya-english", "revision": "r1", "source": "wire"}, - diagnostic="original wire field", - ) - health, read_at = {}, None - if provenance_source != "response": - payload["served_by"] = None - if provenance_source == "health": - payload["routing"] = {"model": "english", "repo": "laya-english", "reason": "explicit model english"} - health = {"models": {"english": {"checkpoint": "laya-english", "revision": "r1", "device": "cuda"}}} - read_at = "2026-10-10T09:00:00+00:00" - - class OfflineClient: - url = "http://127.0.0.1:18188" - - def deadline(self): - return 1.0 - - async def decide(self, request, request_id, deadline): - assert request == dict( - model="english", - state=slot["observation"], - questions={"pick": dict(type="choice", criteria=QUEUES, instructions={"rules": RULES})}, - ) - return copy.deepcopy(payload), {}, 1.0 - - model = ServedLayaModel(OfflineClient(), model="english", clock=lambda: 1.0) - model._health_tried = 1.0 - model._health, model._health_read_at = health, read_at - questions = {"pick": ChoiceQuestion(QUEUES, rules=RULES)} - reply = asyncio.run(model._decide(Observation(slot["observation"]), questions)) - validated = validate_answers(reply, questions) - captured = preparation[2].value(validated) - assert captured["raw"]["served_by"]["source"] == provenance_source - journal = related - if health: - start = dict( - kind="http_start", event_index=slot["event_index"] - 2, method="GET", url="http://127.0.0.1:18188/health" - ) - end = dict( - kind="http_terminal", - event_index=slot["event_index"] - 1, - start_event=start["event_index"], - status="response", - http_status=200, - response_body=body(health), - ) - journal = [start, end, *related] - for row in related: - if row["kind"] == "http_start": - request = json.loads(row["request_body"]["utf8"]) - request["model"] = "english" - row["request_body"] = body(request) - elif row["kind"] == "http_terminal": - row["response_body"] = body(payload) - elif row["kind"] == "decision_validation_terminal": - row["result"] = captured - elif row["kind"] == "slot_terminal": - row["new_decisions"][0].update(model=validated.model, source="laya-served") - identity = {"requested_model": "english"} - assert verify_decision(related, slot["observation"], QUEUES, RULES, identity, choice_faults, journal=journal) - raw = captured["raw"] - for target, field, wrong in ( - (raw["served_by"], "source", "wrong"), - (raw["served_by"], "revision", "wrong"), - (raw, "diagnostic", "wrong"), - (raw["answers"]["pick"], "confidence", 0.5), - ): - original = target[field] - target[field] = wrong - with pytest.raises(ValueError, match="Validated raw differs"): - verify_decision(related, slot["observation"], QUEUES, RULES, identity, choice_faults, journal=journal) - target[field] = original - - -def test_unknown_result_type_requests_actual_source(measurement): - root, arm, events, save_events, preparation = measurement - path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) - episode = json.loads(path.read_text()) - episode["extra"]["result_type"] = "new-framework-type" - path.write_text(dump(episode)) - next(e for e in events if e.get("stage") == "Runner.run_agent" and e["kind"] == "lifecycle_terminal")[ - "result_type" - ] = "new-framework-type" - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert cell["route_correct"] == 30 and not cell["strict_batch_success"] - assert cell["required_input"] == "independent review of actual loaded framework result_type source" - - -def renumber(events): - indices = {row["event_index"]: number for number, row in enumerate(events, 1)} - for row in events: - old = row["event_index"] - if "start_event" in row: - row["start_event"] = indices[row["start_event"]] - row["event_index"] = indices[old] - - -def test_retry_is_one_ticket_and_each_request_is_checked(measurement): - root, _, events, save_events, preparation = measurement - success = next(row for row in events if row["kind"] == "http_start") - terminal = next(row for row in events if row.get("start_event") == success["event_index"]) - import copy - - failed_start, failed_end = copy.deepcopy(success), copy.deepcopy(terminal) - failed_start["event_index"], failed_end["event_index"] = 10000, 10001 - failed_end.update(start_event=10000, http_status=503, response_body=body({"error": "retry fixture"})) - success["post_attempt_index"] = terminal["post_attempt_index"] = 2 - position = events.index(success) - events[position:position] = [failed_start, failed_end] - renumber(events) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == 30 and cell["cases"][0]["post_attempts"] == 2 - request = json.loads(failed_start["request_body"]["utf8"]) - request["state"]["ticket"]["id"] = "another-ticket" - failed_start["request_body"] = body(request) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert not cell["strict_batch_success"] and cell["errors"] - - -def test_29_routes_and_missing_job_preserve_denominator(measurement): - root, arm, events, save_events, preparation = measurement - last_id = next(row["context"]["case_id"] for row in reversed(events) if row.get("context")) - events[:] = [row for row in events if (row.get("context") or {}).get("case_id") != last_id] - renumber(events) - save_events() - path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) - episode = json.loads(path.read_text()) - report = episode["extra"]["ticket_router"] - report.update(processed=29, correct=29, coverage=29 / 30, unprocessed_ids=[last_id], routes=report["routes"][:29]) - episode.update( - decisions=episode["decisions"][:29], views=episode["views"][:30], final_state=episode["views"][29]["state"] - ) - path.write_text(dump(episode)) - trial_path = path.parents[1] / "result.json" - trial = json.loads(trial_path.read_text()) - trial["agent_result"]["metadata"].update(steps=29, decisions=29) - trial_path.write_text(dump(trial)) - cell = aggregate(root, preparation)["cells"][0] - assert cell["route_correct"] == cell["accepted_processed"] == 29 and cell["route_accuracy"] == 29 / 30 - assert not cell["complete_episode"] and not cell["strict_batch_success"] - path.unlink() - cell = aggregate(root, preparation)["cells"][0] - assert cell["route_correct"] == 29 and not cell["complete_episode"] - - -def test_all_wrong_complete_episode_is_not_strict_success(measurement): - root, arm, events, save_events, preparation = measurement - from s1a.agents.ticket_router import QUEUES - - slots = [row for row in events if row["kind"] == "slot_terminal"] - choices = { - row["context"]["decision_index"]: next(key for key in QUEUES if key != row["new_decisions"][0]["key"]) - for row in slots - } - for row in events: - context = row.get("context") - if not context: - continue - key = choices[context["decision_index"]] - probabilities = {queue: float(queue == key) for queue in QUEUES} - if "selected_key" in context: - context["selected_key"] = key - if row["kind"] == "http_terminal" and row["status"] == "response": - reply = json.loads(row["response_body"]["utf8"]) - reply["answers"]["pick"].update(choice=key, probabilities=probabilities) - row["response_body"] = body(reply) - elif row["kind"] == "decision_validation_terminal": - row["result"]["answers"]["pick"].update(key=key, probabilities=probabilities) - row["result"]["raw"]["answers"]["pick"].update(choice=key, probabilities=probabilities) - elif row["kind"] == "slot_terminal": - row["new_decisions"][0].update(key=key, probabilities=probabilities) - elif row["kind"] == "act_start": - row["inputs"]["key"] = key - row["before"]["score"] = 0.0 - elif row["kind"] == "env_step_start": - row["key"] = key - elif row["kind"] == "act_terminal": - row["act_calls"][0]["key"] = key - row["after"]["score"] = 0.0 - row["raw_tool_result"] = dump(row["after"]) - path = next((arm / "s1a-home").glob("evals/results/ticket_router/*/*/agent/episode.json")) - episode = json.loads(path.read_text()) - episode["decisions"] = [row["new_decisions"][0] for row in slots] - for route, key in zip(episode["extra"]["ticket_router"]["routes"], choices.values(), strict=True): - route.update(predicted=key, correct=False) - episode["extra"]["ticket_router"]["correct"] = 0 - for view in episode["views"]: - view["score"] = 0.0 - path.write_text(dump(episode)) - save_events() - cell = aggregate(root, preparation)["cells"][0] - assert cell["accepted_processed"] == 30 and cell["route_correct"] == 0 - assert cell["complete_episode"] and not cell["strict_batch_success"] From 9df13225532c7f3af380cf50878ca4f00ae639f7 Mon Sep 17 00:00:00 2001 From: linear3735 Date: Mon, 12 Oct 2026 10:29:53 +0800 Subject: [PATCH 5/5] docs: clarify public benchmark evidence scope --- benchmarks/accuracy-leaderboard.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/benchmarks/accuracy-leaderboard.md b/benchmarks/accuracy-leaderboard.md index dba281b..a473840 100644 --- a/benchmarks/accuracy-leaderboard.md +++ b/benchmarks/accuracy-leaderboard.md @@ -41,11 +41,13 @@ are secondary calibration measures, with lower values better. and limitations. The companion [latency contribution](https://github.com/ThinkFlowLab/system1-omni/pull/131) keeps request timing separate from quality. -The published report is available now. The numeric audit attachment for this run -is prepared but not yet published; complete original responses are not included -in the repository. The attachment removes question/option text and retains -probability vectors, label indices, timings and original record hashes. It supports -numeric recomputation; full response replay additionally requires the original raw. +The [public summary bundle](https://github.com/user-attachments/files/33304077/public231-benchmark-output.zip) +contains this campaign's five aggregate summaries, run metadata and collector replay +receipt. It omits per-question predictions, probabilities and raw responses, so it +does not support independent numeric recomputation. The separate per-question +numeric audit attachment remains unpublished; full response replay also requires +the original raw responses. The public bundle is collector evidence from PR #52, +not a replay of this PR's `quality_public231.py` consumer. ## Add a result