docs(router): independent replication of the universal-router benchmark; fit_prior Windows fix

CHANGELOG.md

@@ -6,6 +6,25 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+
+- **The universal router's benchmark is independently replicated.** harness-bench run 4 was
+  re-run from the pinned public data (SWE-bench Verified `78f471b`, SWE-bench/experiments
+  `40f164d`) on a second machine. Results:
+  - **Split:** the same 150/350 split.
+  - **Held-out test:** 217 of 218 metrics identical, with only wall-clock fit time differing.
+    The headline reproduces: 76.3% solved at $0.093 per task against 75.1% at $0.364.
+  - **Shipped prior:** refits bit for bit (176 of 176 values).
+
+  `bench/universal-router/README.md` records the commands, including the sub-1 MB sparse fetch
+  of the experiments data.
+
+### Fixed
+
+- **`fit_prior.mjs` writes its default output on Windows.** The default `--out` used
+  `new URL(…).pathname`, which is `/C:/…` on Windows and not a usable path; it now uses
+  `fileURLToPath`.
+
 ## [1.1.1] - 2026-09-22
 
 ### Fixed

bench/universal-router/README.md

@@ -10,12 +10,42 @@ results:
 To regenerate it, build the input JSON, then run:
 
 ```bash
-# harness-bench writes the input from SWE-bench Verified + SWE-bench/experiments:
+# harness-bench writes the input from SWE-bench Verified + SWE-bench/experiments.
+# `build routing` makes the 150/350 split that `build universal` reads, so it runs first:
+#   python3 -m hbench.cli build routing   --swe-exp <experiments checkout>
 #   python3 -m hbench.cli build universal --swe-exp <experiments checkout> --forgekit <this repo>
 #   python3 -c "from hbench.tracks import universal as U; U.export_for_forgekit(Path('universal_all.json'))"
 node bench/universal-router/fit_prior.mjs universal_all.json --out data/router_prior.json
 ```
 
+The experiments checkout only needs the eleven runs' `per_instance_details.json` at `40f164d`.
+A sparse, blob-less fetch of that commit brings down under 1 MB:
+
+```bash
+git init swe-exp && git -C swe-exp remote add origin https://github.com/SWE-bench/experiments
+git -C swe-exp sparse-checkout set --no-cone evaluation/verified/20260217_mini-v2.0.0_<run>/per_instance_details.json  # ×11
+git -C swe-exp fetch --depth 1 --filter=blob:none origin 40f164d5b8f1d249bf95a6df8b74b577fd8e519d
+git -C swe-exp checkout FETCH_HEAD
+```
+
 The fit chooses the latent dimension k and the prior scale by 3-fold cross-validation. That selection is recorded in `selection` inside the file.
 
 Raw per-task results are not redistributed here, only the fitted parameters and their provenance.
+
+## Independent replication (2026-09-22)
+
+The run-4 benchmark and this prior were re-run from scratch on a second machine (Windows, Node 24, Python 3.12). The inputs were fetched as above, and the code was forgekit at `c5227db` (router code unchanged since the fit).
+
+| What | Result |
+|---|---|
+| Data | 11 runs × 500 issues; 616 KB of `per_instance_details.json` |
+| Split | dev 150 and held-out 350: the same issue ids as the original run |
+| Held-out test (`hbench test universal`) | 217 of 218 metric values identical; the only difference is wall-clock `fitSeconds` (37.6 s vs 45.9 s) |
+| Headline | router 76.3% solved at $0.093 per task; best single model chosen on dev 75.1% at $0.364; pre-registered endpoint met |
+| Prior refit (`fit_prior.mjs`, all 500 issues) | all 176 fitted values identical to `data/router_prior.json` (k=1, scale 4); only `provenance.fittedAt` differs |
+
+Two portability fixes were needed on Windows:
+- `fit_prior.mjs`'s default `--out` path (fixed here).
+- harness-bench's `adapters/forgekit_universal.mjs`, which passes plain paths to `import()` and so needs `pathToFileURL(...).href`. That file lives in harness-bench, not in this repo.
+
+Neither fix changes a measured value.

bench/universal-router/fit_prior.mjs

@@ -6,12 +6,14 @@
 //
 //   node bench/universal-router/fit_prior.mjs <input.json> [--out data/router_prior.json] [--only <task ids json>]
 import { readFileSync, writeFileSync } from "node:fs";
+import { fileURLToPath } from "node:url";
 import { buildPrior } from "../../src/router/prior.js";
 
 const args = process.argv.slice(2);
 const opt = (n, d) => (args.includes(n) ? args[args.indexOf(n) + 1] : d);
 const input = JSON.parse(readFileSync(args[0], "utf8"));
-const out = opt("--out", new URL("../../data/router_prior.json", import.meta.url).pathname);
+// fileURLToPath, not `.pathname`: on Windows `.pathname` is `/C:/…`, which is not a path.
+const out = opt("--out", fileURLToPath(new URL("../../data/router_prior.json", import.meta.url)));
 const only = opt("--only") ? new Set(JSON.parse(readFileSync(opt("--only"), "utf8"))) : null;
 const t0 = Date.now();
 const prior = buildPrior(input, only);

docs/UNIVERSAL_ROUTING.md

@@ -91,6 +91,8 @@ harness-bench run 4 is pre-registered. It fits on 150 dev issues and scores 350
 
 **Against the best single model:** non-inferior (+1.1 points, CI [−2.0, +4.3]) at 74% lower cost. In 5-fold cross-validation it is +3.2 points (CI [+0.4, +6.2]) at −$0.58 per task.
 
+**Replicated.** An independent re-run from the pinned public data reproduced 217 of 218 test metrics exactly (only the wall-clock fit time differs). The shipped prior also refits bit for bit. See `bench/universal-router/README.md`.
+
 **Limits (measured):**
 - **Where the gain comes from.** Most of it comes from choosing across providers. On the 150-issue fit the router does not beat a fixed cascade chosen on the same dev data; with 400 training issues its target modes are cheaper than the fixed equivalents.
 - **Targets are optimistic.** Predicted cascade success is optimistic by 4 to 6 points on the test split, so `target:p` lands below p. A cross-validated calibration map is the planned fix.