diff --git a/CHANGELOG.md b/CHANGELOG.md index c5f8a66..ebbf8bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,25 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added + +- **The universal router's benchmark is independently replicated.** harness-bench run 4 was + re-run from the pinned public data (SWE-bench Verified `78f471b`, SWE-bench/experiments + `40f164d`) on a second machine. Results: + - **Split:** the same 150/350 split. + - **Held-out test:** 217 of 218 metrics identical, with only wall-clock fit time differing. + The headline reproduces: 76.3% solved at $0.093 per task against 75.1% at $0.364. + - **Shipped prior:** refits bit for bit (176 of 176 values). + + `bench/universal-router/README.md` records the commands, including the sub-1 MB sparse fetch + of the experiments data. + +### Fixed + +- **`fit_prior.mjs` writes its default output on Windows.** The default `--out` used + `new URL(…).pathname`, which is `/C:/…` on Windows and not a usable path; it now uses + `fileURLToPath`. + ## [1.1.1] - 2026-09-22 ### Fixed diff --git a/bench/universal-router/README.md b/bench/universal-router/README.md index 0c9be56..5883e2a 100644 --- a/bench/universal-router/README.md +++ b/bench/universal-router/README.md @@ -10,12 +10,42 @@ results: To regenerate it, build the input JSON, then run: ```bash -# harness-bench writes the input from SWE-bench Verified + SWE-bench/experiments: +# harness-bench writes the input from SWE-bench Verified + SWE-bench/experiments. +# `build routing` makes the 150/350 split that `build universal` reads, so it runs first: +# python3 -m hbench.cli build routing --swe-exp # python3 -m hbench.cli build universal --swe-exp --forgekit # python3 -c "from hbench.tracks import universal as U; U.export_for_forgekit(Path('universal_all.json'))" node bench/universal-router/fit_prior.mjs universal_all.json --out data/router_prior.json ``` +The experiments checkout only needs the eleven runs' `per_instance_details.json` at `40f164d`. +A sparse, blob-less fetch of that commit brings down under 1 MB: + +```bash +git init swe-exp && git -C swe-exp remote add origin https://github.com/SWE-bench/experiments +git -C swe-exp sparse-checkout set --no-cone evaluation/verified/20260217_mini-v2.0.0_/per_instance_details.json # ×11 +git -C swe-exp fetch --depth 1 --filter=blob:none origin 40f164d5b8f1d249bf95a6df8b74b577fd8e519d +git -C swe-exp checkout FETCH_HEAD +``` + The fit chooses the latent dimension k and the prior scale by 3-fold cross-validation. That selection is recorded in `selection` inside the file. Raw per-task results are not redistributed here, only the fitted parameters and their provenance. + +## Independent replication (2026-09-22) + +The run-4 benchmark and this prior were re-run from scratch on a second machine (Windows, Node 24, Python 3.12). The inputs were fetched as above, and the code was forgekit at `c5227db` (router code unchanged since the fit). + +| What | Result | +|---|---| +| Data | 11 runs × 500 issues; 616 KB of `per_instance_details.json` | +| Split | dev 150 and held-out 350: the same issue ids as the original run | +| Held-out test (`hbench test universal`) | 217 of 218 metric values identical; the only difference is wall-clock `fitSeconds` (37.6 s vs 45.9 s) | +| Headline | router 76.3% solved at $0.093 per task; best single model chosen on dev 75.1% at $0.364; pre-registered endpoint met | +| Prior refit (`fit_prior.mjs`, all 500 issues) | all 176 fitted values identical to `data/router_prior.json` (k=1, scale 4); only `provenance.fittedAt` differs | + +Two portability fixes were needed on Windows: +- `fit_prior.mjs`'s default `--out` path (fixed here). +- harness-bench's `adapters/forgekit_universal.mjs`, which passes plain paths to `import()` and so needs `pathToFileURL(...).href`. That file lives in harness-bench, not in this repo. + +Neither fix changes a measured value. diff --git a/bench/universal-router/fit_prior.mjs b/bench/universal-router/fit_prior.mjs index 841bae0..1e8e522 100644 --- a/bench/universal-router/fit_prior.mjs +++ b/bench/universal-router/fit_prior.mjs @@ -6,12 +6,14 @@ // // node bench/universal-router/fit_prior.mjs [--out data/router_prior.json] [--only ] import { readFileSync, writeFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; import { buildPrior } from "../../src/router/prior.js"; const args = process.argv.slice(2); const opt = (n, d) => (args.includes(n) ? args[args.indexOf(n) + 1] : d); const input = JSON.parse(readFileSync(args[0], "utf8")); -const out = opt("--out", new URL("../../data/router_prior.json", import.meta.url).pathname); +// fileURLToPath, not `.pathname`: on Windows `.pathname` is `/C:/…`, which is not a path. +const out = opt("--out", fileURLToPath(new URL("../../data/router_prior.json", import.meta.url))); const only = opt("--only") ? new Set(JSON.parse(readFileSync(opt("--only"), "utf8"))) : null; const t0 = Date.now(); const prior = buildPrior(input, only); diff --git a/docs/UNIVERSAL_ROUTING.md b/docs/UNIVERSAL_ROUTING.md index 749c1b5..2af5941 100644 --- a/docs/UNIVERSAL_ROUTING.md +++ b/docs/UNIVERSAL_ROUTING.md @@ -91,6 +91,8 @@ harness-bench run 4 is pre-registered. It fits on 150 dev issues and scores 350 **Against the best single model:** non-inferior (+1.1 points, CI [−2.0, +4.3]) at 74% lower cost. In 5-fold cross-validation it is +3.2 points (CI [+0.4, +6.2]) at −$0.58 per task. +**Replicated.** An independent re-run from the pinned public data reproduced 217 of 218 test metrics exactly (only the wall-clock fit time differs). The shipped prior also refits bit for bit. See `bench/universal-router/README.md`. + **Limits (measured):** - **Where the gain comes from.** Most of it comes from choosing across providers. On the 150-issue fit the router does not beat a fixed cascade chosen on the same dev data; with 400 training issues its target modes are cheaper than the fixed equivalents. - **Targets are optimistic.** Predicted cascade success is optimistic by 4 to 6 points on the test split, so `target:p` lands below p. A cross-validated calibration map is the planned fix.