docs(router): independent replication of the universal-router benchmark; fit_prior Windows fix - #154
Merged
Merged
Conversation
…or on Windows harness-bench run 4 and the shipped prior were re-run from the pinned public data on a second machine (Windows, Node 24, Python 3.12): - data: SWE-bench/experiments@40f164d, sparse blob-less fetch of the eleven runs' per_instance_details.json (616 KB; under 1 MB in .git); - split: dev 150 / held-out 350, the same issue ids as the original run; - held-out test: 217 of 218 metric values identical; only fitSeconds differs; - prior refit on all 500 issues: 176 of 176 values identical (k=1, scale 4); only provenance.fittedAt differs. The bench README now records the commands, including `build routing` before `build universal` (the universal build reads the routing split). fit_prior.mjs defaulted --out to `new URL(...).pathname`, which is `/C:/...` on Windows; it now uses fileURLToPath. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An independent replication of the universal router's benchmark (harness-bench run 4) and of the shipped prior, run on a second machine from the pinned public data. Also a one-line Windows fix in
bench/universal-router/fit_prior.mjs.40f164d, sparse blob-less fetch of the 11 runs'per_instance_details.json: 616 KB, under 1 MB in.gitfitSecondsdiffers (37.6 s vs 45.9 s)data/router_prior.json(k=1, scale 4); onlyprovenance.fittedAtdiffersThe author's stated limit also reproduces: a fixed cascade chosen on dev gets 77.1% at $0.088, level with the router on this fit.
Changes
bench/universal-router/README.md:build routingstep (the universal build reads its split);docs/UNIVERSAL_ROUTING.md: a one-line "Replicated" note under Measured.fit_prior.mjs: the default--outusednew URL(…).pathname, which is/C:/…on Windows. It now usesfileURLToPath; the path was checked before and after the change.adapters/forgekit_universal.mjspasses plain paths toimport()and neededpathToFileURL(...).hrefon Windows. That file lives in harness-bench.Verification
forge docs checkpasses.npm run lint: 0 errors.src/file changed.🤖 Generated with Claude Code