feat(router): universal cross-provider router + research-audit gap fixes

[CodeWithJuber]

Merged at f64c540. Four follow-up commits were pushed to this branch after the merge, so they are not in it. They are re-applied on top of v1.1.0 in #153:

• the session learner on macOS;
• the Windows home folder;
• the Sonnet 5 price;
• the release post-publish check.

Summary

Adds the opt-in universal router (forge route universal|outcome|fit|models) and closes seven gaps from a research-to-code audit. It is an 11-patch series applied with git am onto v1.0.0 (61385b3), plus two small commits.

• Universal router. It recommends a model, or a cascade of models, across any provider. Model data lives in data/models.json and .forge/models.json, not in code, and a test enforces that.
  • What it learns: a multidimensional IRT model estimates each model's ability and each task's difficulty. Correlated failures across models are integrated with Gauss–Hermite quadrature, and a log-cost model estimates what each attempt costs.
  • What it picks: the cheapest cascade for the chosen objective: match-best-single (the default), target:p, value:$ or budget:$.
  • How it updates: a Bayesian refit from recorded outcomes (forge route outcome then forge route fit).
  • The default is unchanged: the tiered forge route (haiku / sonnet / opus / fable) stays as it is.
• Audit gaps closed:
  • Commit gate on binary files: a binary file (git reports it binary and it contains a NUL byte) is scanned with the credential-format rules only. The fixed XMP packet id no longer counts as a secret, and a ghp_ token inside a binary is still refused.
  • B02: the "repeated gates multiply catch rates" claim is removed; the stated law is now (1−p)(1−c_max).
  • C09: the handoff writer and loader share one byte budget.
  • E03: the substrate pre-action check (and so the prompt hook) and the Stop gate's repair checklist also list sibling and forward files, each tagged with its relation.
  • verifyToken: the docs now show the 0.88 the code actually computes.
  • A13: learned-lesson consolidation is deterministic and driven by ledger evidence; the model path needs an explicit --llm.
• Cleanup commit: the series re-serialized package.json and turned the description's em dash into a JSON unicode escape. The literal character is restored.
• macOS fix (from this PR's first CI run): learn-consolidate.sh --llm wrapped the model call in GNU timeout, which stock macOS doesn't ship.
  • The failure was silent: the model was never called, and every Mac run ended in "response too short". The bug predates this branch; the new stub test exposed it, and Install smoke (macos-latest) failed.
  • The call now uses timeout, else gtimeout, else runs without a limit. fix: follow-ups to #152 — session learner on macOS, Sonnet 5 price, release npm check #153 replaces that last fallback with a watchdog.

What stays the same

• forge impact, predict_impact, imagine and context still walk reverse dependencies only by default, as decided in fix: resolve the deep review's findings across impact, guards, memory, routing, primitives and research #151.
• The FORGE_ENFORCE gate still counts only dependents (reverse and llm-verified) toward its 25-file block. Counting all three relations would push 79 of this repo's 98 source files over the limit.

Verification

Run locally on Windows (Node 24) before the macOS fix; that fix changed no test count. CI was green on all 12 checks at f64c540.

Check	Result
npm test	1,385 tests: 1,379 pass, 0 fail, 6 skipped
npm run lint	0 errors, 14 warnings (same as master)
npm run typecheck	0 errors
forge docs check	passes

Why 6 tests skip here:

• 4 are Windows-only guards: three verify polyglot tests, plus the learn-consolidate bash test.
• 2 need opt-in settings: RUN_INTEGRATION and FORGE_PLAYWRIGHT.
• The author's Linux run reported 1,382 pass and 3 skipped. It lists totals only, so its 3 skips are unnamed.

Consolidation script: run by hand in Git Bash, with a stub claude first on PATH and the real CLI kept off it.

• The default path made 0 calls to claude.
• --llm made exactly 1, to the stub.
• With neither timeout nor gtimeout on PATH (the stock macOS setup), the original script never called the stub. The fixed one called it once, exited 1 and printed "originals kept", which is what the test asserts.

Router smoke test: forge route universal "Add pagination to the users endpoint" suggests gpt-5-mini, then minimax-m2.5 if a check fails. That gives P(success) 0.86 at an expected $0.040, against the best single model, claude-opus-4.6, at 0.84 and $0.271. The command writes no files.

Known limits

• Benchmark not re-run here. Two results come from the author's pre-registered run 4 and were not reproduced for this PR:
  • 76.3% solved at $0.093 per task, against 75.1% at $0.364 for the best single model, on 350 held-out issues.
  • target:p is optimistic by 4–6 points.
• Binary relaxation gap (inferred from the code, not tested). A text file with a deliberately added NUL byte skips the entropy check. The gate targets accidental leaks, so this is accepted.
• Windows home-directory mismatch. learnedDir() uses node's homedir(), which reads USERPROFILE on Windows, while the bash learner writes under $HOME. Fixed in fix: follow-ups to #152 — session learner on macOS, Sonnet 5 price, release npm check #153.
• Model ids. claude-fable-5 and claude-opus-4-8 are valid ids: Anthropic lists both under "Legacy models (still available)".
  • The current models are claude-fable-5-1 and claude-opus-5, at the same prices.
  • Moving the tier table onto them is a product decision, so this PR doesn't do it.
• Disclosure from the author. While the A13 test was being written, it twice ran the real claude -p in the build environment. It sent the fixed consolidation prompt plus one heading line; no lesson text and no secrets. The test now puts a stub first on PATH.

The router design, benchmark and research audit are from the patch author.

🤖 Generated with Claude Code