Skip to content

docs(router): independent replication of the universal-router benchmark; fit_prior Windows fix - #154

Merged
CodeWithJuber merged 1 commit into
masterfrom
docs/router-replication
Sep 22, 2026
Merged

CodeWithJuber merged 1 commit into
masterfrom
docs/router-replication

Conversation

@CodeWithJuber

Copy link
Copy Markdown
Owner

Summary

An independent replication of the universal router's benchmark (harness-bench run 4) and of the shipped prior, run on a second machine from the pinned public data. Also a one-line Windows fix in bench/universal-router/fit_prior.mjs.

What Result
Data SWE-bench/experiments @ 40f164d, sparse blob-less fetch of the 11 runs' per_instance_details.json: 616 KB, under 1 MB in .git
Split dev 150 / held-out 350: the same issue ids as the original run
Held-out test 217 of 218 metric values identical; only wall-clock fitSeconds differs (37.6 s vs 45.9 s)
Headline router 76.3% at $0.093 per task vs best-single-on-dev 75.1% at $0.364; cost −$0.271 CI [−0.288, −0.252]; solve +1.1 pt CI [−2.0, +4.3]; pre-registered endpoint met
Prior refit (all 500) 176 of 176 values identical to data/router_prior.json (k=1, scale 4); only provenance.fittedAt differs

The author's stated limit also reproduces: a fixed cascade chosen on dev gets 77.1% at $0.088, level with the router on this fit.

Changes

  • bench/universal-router/README.md:
    • the replication table;
    • the missing build routing step (the universal build reads its split);
    • the sparse-fetch commands.
  • docs/UNIVERSAL_ROUTING.md: a one-line "Replicated" note under Measured.
  • fit_prior.mjs: the default --out used new URL(…).pathname, which is /C:/… on Windows. It now uses fileURLToPath; the path was checked before and after the change.
  • Not in this repo: harness-bench's adapters/forgekit_universal.mjs passes plain paths to import() and needed pathToFileURL(...).href on Windows. That file lives in harness-bench.

Verification

  • forge docs check passes.
  • npm run lint: 0 errors.
  • The test suite doesn't cover the bench scripts, and no src/ file changed.

🤖 Generated with Claude Code

…or on Windows

harness-bench run 4 and the shipped prior were re-run from the pinned public
data on a second machine (Windows, Node 24, Python 3.12):
- data: SWE-bench/experiments@40f164d, sparse blob-less fetch of the eleven
  runs' per_instance_details.json (616 KB; under 1 MB in .git);
- split: dev 150 / held-out 350, the same issue ids as the original run;
- held-out test: 217 of 218 metric values identical; only fitSeconds differs;
- prior refit on all 500 issues: 176 of 176 values identical (k=1, scale 4);
  only provenance.fittedAt differs.

The bench README now records the commands, including `build routing` before
`build universal` (the universal build reads the routing split).

fit_prior.mjs defaulted --out to `new URL(...).pathname`, which is `/C:/...`
on Windows; it now uses fileURLToPath.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@CodeWithJuber
CodeWithJuber merged commit 673de5c into master Sep 22, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant