feat(router): universal cross-provider router + research-audit gap fixes

ARCHITECTURE.md

@@ -115,14 +115,19 @@ anything. The analysis is **hazard-aware**: SCC-aware propagation (a change to a
 in a circular-dependency cluster impacts all co-members, via Tarjan from `forge rank`)
 and a data-driven threshold derived from PageRank centrality and ledger incident history
 (`effectiveThreshold = base / (1 + hazard)`). `--basic` reverts to the fixed-threshold
-mode.
+mode. `forge impact` walks reverse dependents; the pre-action check, the ambient prompt hook
+and the Stop gate's repair checklist also walk the empirical refutation's repaired sibling
+and forward relations (frozen parameters, `SIBLING`/`FORWARD` in `src/atlas.js`) and tag
+every file `reverse`, `sibling` or `forward`, because the reverse-only walk missed the
+sibling files that were 94.7% of the refutation's misses.
 
 The verdict is **advisory by default** — it reports, it does not block. Set
 `FORGE_ENFORCE=1` to turn the strongest signals into a hard block:
 
 - a **vacuous or underspecified** prompt (preflight finds no actionable intent),
 - **un-assemblable required context** (the completeness gate cannot cover the edit set),
-- a **blast radius over threshold** (default ~25 files).
+- a **blast radius over threshold** (default ~25 dependent files; sibling/forward
+  co-change candidates are named in the reason but not counted).
 
 Everything else stays a warning the human can override.
 
@@ -200,8 +205,12 @@ to decide whether an edit is safe to start. Surface: `forge reuse query | mint |
 Two failure modes this layer exists to kill: **partial work** (code changes without the
 artifacts that depend on it) and **session amnesia** (the next session re-assumes what
 this one knew). Instructions raise the _probability_ of correct behavior; deterministic
-hooks guarantee a _floor_ — with per-task miss rate `1−p` and gate catch rate `c`,
-silent misses fall to `(1−p)(1−c)`, and every layer here is one more `c`.
+hooks guarantee a _floor_ — with per-task miss rate `1−p`, silent misses fall to
+`(1−p)·P(no check fires | miss)`: `(1−p)(1−c)` for one check with catch rate `c`. A second
+check lowers that only where it catches what the first cannot; the product `∏(1−cⱼ)` holds
+only if the checks fire independently. The same check repeated at another point (Stop,
+pre-commit, CI on the same diff) is nested, so the residual is `(1−p)(1−c_max)` (formal
+synthesis §5.3, corrected 2026-09-21).
 
 **The completion gate (Stop, `src/gate.js`).** The only Stop-path guard that may answer:
 `completion-gate.sh` runs synchronously (the lesson-mining `cortex.sh stop` stays
@@ -225,7 +234,10 @@ injects: learned lessons, the anchored goal, the handoff snapshot, recent commit
 uncommitted changes — a fresh session orients on evidence, not priors.
 
 **The state/decision stores (`src/handoff.js`, `src/decide.js`).** `state.md` is a
-bounded REWRITE (snapshot semantics — loader cost stays O(bound) forever);
+bounded REWRITE (snapshot semantics — loader cost stays O(bound) forever). Writer and
+loader share ONE budget in one unit (`STATE_BUDGET_BYTES`, 8 KB): the writer keeps rows in
+priority order (goal, next, decisions, gotchas, in-progress, done) until the body fits, so
+the SessionStart loader never cuts what the handoff wrote;
 `decisions.md` is append-only ADR-lite with a machine-readable `decision` ledger twin
 (log semantics — supersede, never edit). Both refuse secrets at write.
 
@@ -318,8 +330,11 @@ completeness score `s(x)` is a **logistic** over its features (concreteness, nam
 vagueness, a smooth `tanh` length term) instead of an additive rubric with magic coefficients and
 discontinuous word-count steps — the `sigmoid` bounds it to (0,1) with no clamp, every feature's
 pull stays attributable, and a labeled bank could refine the weights via `predictor.js`'s
-`trainLogistic`. The calibrated prior still lands the paper's own examples where they were
-(a bare "make the auth better" ≈ 0.23 → ask; a concrete verifyToken edit ≈ 0.63 → proceed).
+`trainLogistic`. The hand-set prior (not fit to data) puts the paper's own examples on the
+right side of the 0.6 threshold: a bare "make the auth better" ≈ 0.23 → ask; the concrete
+verifyToken edit ≈ 0.88 → proceed. That edit scored ≈ 0.63 when the weights were set, with one
+concrete anchor (the filename); since a named code identifier became a second anchor it scores
+≈ 0.88, and the weights were not re-fit.
 
 **The evidence trail (preflight).** Once a goal is anchored, every prompt appends its
 graded `driftScore` to the session log; `cusum` (until now test-only math) accumulates
@@ -332,8 +347,11 @@ the gate lattice (turn ⊂ commit ⊂ PR): the Stop hook gates the turn and CI's
 gates the PR, so this runs the SAME registry-derived completeness classifier
 (`classifyPath` from `gate.js`) plus `hasSecret` over staged added lines at the commit
 boundary — code staged without its doc/state artifact, or a staged secret, is caught
-while the fix is still one `git add` away. Each rung is an independent catch layer, so
-the silent-miss probability falls multiplicatively.
+while the fix is still one `git add` away. The rungs are **not** independent catch
+layers: on the same diff the copies fire together, so they do not multiply the catch rate
+and the residual stays `(1−p)(1−c_max)`. This rung adds catches only where it sees what the
+Stop hook could not — edits made after the turn ended, a host or session where the Stop
+hook never ran, or a session whose one Stop block was already spent.
 
 **Deep verification (`src/consensus.js`, `forge verify --deep`).** Where plain `verify`
 asks one oracle (the tests) plus one heuristic, this runs a table of independent lenses
@@ -526,6 +544,7 @@ forgekit/
     ledger_store.js       # git-native on-disk ledger (.forge/ledger/): sharded claims, append-only evidence/tombstone logs, normal-form verify
     ledger_bridge.js      # legacy-store bridge, dormant by default (ledger-only); `FORGE_LEDGER_ONLY=0` re-enables cortex/recall/brain shadow-writes + idempotent `ledger import`
     ledger_read.js        # ledger-only read path by default (`FORGE_LEDGER_ONLY=0` merges legacy∪ledger instead): cortex lesson/fact injection, `recall list`, brain's AGENTS.md index all see teammate knowledge from `ledger merge`
+    learn_consolidate.js  # bin/learn-consolidate.sh: deterministic consolidation of ~/.claude/skills/learned — merge duplicates, drop only ledger-refuted (dormant/retracted/attic) lessons; no model call
     reuse.js              # proof-carrying artifact cache: fingerprint (MinHash+LSH), exact→near→adapt→miss ladder, atlas revalidation
     embed.js              # optional embeddings tier (ADR-0005): FORGE_EMBED=cmd:<cmd>|http:<url>, swaps MinHash/Jaccard for cosine in `reuse query`/`ledger query`, disk-cached at .forge/embed-cache.jsonl, silent fallback to MinHash
     context.js            # budgeted context assembly + completeness gate: R(edit) set cover, compression ladder, computed missing-set
@@ -601,17 +620,17 @@ from the tree it describes.
 ```mermaid
 %%{init: {'theme':'base','themeVariables':{'primaryColor':'#201a15','primaryTextColor':'#f2ede7','primaryBorderColor':'#372c22','lineColor':'#f26430','secondaryColor':'#272019','tertiaryColor':'#171310','edgeLabelBackground':'#201a15','clusterBkg':'#171310','clusterBorder':'#4a3b2e','fontFamily':'ui-sans-serif, system-ui, sans-serif','fontSize':'14px'},'flowchart':{'curve':'basis','padding':10,'nodeSpacing':36,'rankSpacing':44}}}%%
 flowchart LR
-  test["test<br/>113 files"]
-  src["src<br/>98 files"]
+  test["test<br/>117 files"]
+  src["src<br/>109 files"]
   landing["landing<br/>61 files"]
   research["research<br/>37 files"]
   global["global<br/>5 files"]
-  bench["bench<br/>2 files"]
+  bench["bench<br/>3 files"]
   scripts["scripts<br/>2 files"]
   docs["docs<br/>1 file"]
   examples["examples<br/>1 file"]
-  test -- 227 --> src
-  bench -- 7 --> src
+  test -- 240 --> src
+  bench -- 8 --> src
   examples -- 4 --> src
   test -- 2 --> bench
   test -- 2 --> global

CHANGELOG.md

@@ -6,6 +6,91 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Added
+
+- **Universal router** (`src/router`, `forge route universal|outcome|fit|models`). It recommends a model or a cascade across any provider's models, with no vendor, tier or threshold in code.
+  - **Model:** multidimensional IRT for who solves what, with correlated failures integrated by Gauss–Hermite quadrature; a log-linear cost model; and a cascade policy with a parameter-free default objective (`match-best-single`) plus `target`, `value` and `budget`.
+  - **Data:** the models live in `data/models.json` and `.forge/models.json`. A shipped prior is fitted on public SWE-bench Verified runs of 11 models from 7 providers. `route outcome` and `route fit` perform a Bayesian update on the project's own outcomes.
+  - **Measured** (harness-bench run 4, pre-registered, 350 held-out issues): 76.3% solved at $0.093 per task, against 75.1% at $0.364 for the best single model chosen on dev (non-inferior, 74% cheaper). See docs/UNIVERSAL_ROUTING.md for the limits.
+
+### Fixed
+
+- **Binary files no longer trip the commit gate's secret scan.** The staged scan reads every
+  file with `git diff --text`, and the entropy leg flagged the XMP packet id
+  (`W5M0MpCehiHzreSzNTczkc9d`, a constant fixed by Adobe's XMP spec) that the XMP packet
+  wrapper carries inside PDFs, JPEGs and PNGs, so ordinary binary commits were refused. A staged file that git
+  reports as binary (`--numstat` prints `-`/`-`) and that contains a NUL byte now gets the
+  credential-format grammars only; a `binary` attribute on a text file does not qualify, so
+  `.gitattributes` cannot switch the entropy leg off. The XMP packet id is also exempt from the
+  entropy leg everywhere, like lockfile integrity digests. A `ghp_…` token inside a binary is
+  still refused, and an unreadable diff still fails closed.
+
+- **A handoff snapshot is read back whole at session start.** `forge handoff` wrote up to
+  150 lines to `.forge/state.md` but the SessionStart loader injected only the first 80, so
+  rows 81–150 of a valid handoff were silently dropped (the budget mismatch the formal
+  synthesis's T4 correction names). Writer and loader now share one budget in one unit,
+  `STATE_BUDGET_BYTES` (8 KB of snapshot body), and the writer keeps rows in priority order
+  (goal and acceptance criteria, next steps, decisions, gotchas and open assumptions,
+  in-progress files, then done) until the body fits. A section that lost rows ends with
+  "(+N more not kept …)". Sections are now written in that priority order. Only a hand-edited
+  or pre-budget file can still overflow the loader, and then the cut names the file.
+
+- **The verifyToken example's completeness score matches the code again.** The docs
+  (ARCHITECTURE.md, GUIDE, the cognitive-substrate README) and the `src/preflight.js` comment
+  said "Change verifyToken in src/auth.js to require length > 20; update tests" scores ≈ 0.63
+  (medium risk), but the code gives 0.878 (low risk). The prior was hand-set when that task
+  had one concrete anchor (the filename, 0.63); since 2026-09-21 a named code identifier is a
+  second anchor. The docs now show 0.88 and say why; the weights are unchanged. A test pins the
+  value and checks that the two example outputs print it.
+
+### Changed
+
+- **The everyday blast-radius checks walk sibling and forward relations, tagged.** The
+  substrate pre-action check (so also the ambient prompt hook and the `FORGE_ENFORCE` gate)
+  and the Stop gate's repair checklist ran the reverse-only walk that the empirical
+  refutation measured at recall 0.022, where 94.7% of the misses were sibling files. They now
+  walk reverse + the paper's repaired sibling and forward relations at the frozen parameters
+  already in `src/atlas.js`, and every file is tagged with the relation that reached it:
+  `forge substrate` and the ambient advisory print `path (reverse|sibling|forward)` with a
+  per-relation count, `--json` adds `impact.fileRelations` and `impact.relationCounts`, and the
+  Stop gate's block reason lists the untouched co-change candidates. The enforce gate still
+  counts only dependents toward its 25-file block (the wide walk would put 79 of this repo's
+  98 source files over it, against 35 today, at precision about 0.09) and names the other
+  candidates in its reason; `blastRelations` changes what it counts. A wide walk never relabels
+  a reverse dependent, so its reverse-tagged set equals the reverse-only answer. Scope
+  decomposition and lesson matching keep using dependents only. `relations: ["reverse"]`
+  (`substrateCheck`, `repairReason`) is the explicit reverse-only option; `forge impact` and
+  `predict_impact` are unchanged (reverse-only unless `--all-relations`). The
+  `source/substrate.json` impact faculties move from `operational-v1` to
+  `operational-v2-recall`, with a guarantee that says the frozen parameters were tuned on a
+  different graph builder and are not held-out validated here.
+
+- **`bin/learn-consolidate.sh` no longer lets a model prune memory.** It sent every learned
+  lesson to Haiku with "DROP anything … contradicted" and rewrote `~/.claude/skills/learned`
+  from the answer, which is pruning by the model's own judgment (the research requires pruning
+  by ground truth). Consolidation is now deterministic (`src/learn_consolidate.js`): exact and
+  near-duplicate lessons within a project merge (MinHash Jaccard ≥ 0.7, the ledger's own
+  consolidation threshold), and a lesson is dropped only when its matching ledger claim in
+  that project is dormant, retracted or pruned to the attic; a lesson the ledger knows nothing
+  about is kept. `--repo <root>` names the ledger (default: the current directory), and
+  `--dry-run` / `--json` report without writing. Originals are archived first, as before. The
+  model rewrite remains behind an explicit `--llm` first argument, with "contradicted" removed
+  from its prompt.
+
+- **`learn-consolidate.sh --llm` works on macOS.** It wrapped the model call in GNU `timeout`,
+  which stock macOS does not ship; with stderr discarded the missing command failed silently,
+  the model was never called, and every run ended in "response too short". The call now uses
+  `timeout`, else Homebrew's `gtimeout`, else runs unwrapped.
+
+- **The gate docs no longer claim that repeated gates multiply their catch rates.** The
+  headers of `src/commit_gate.js` and `src/gate.js`, ARCHITECTURE.md §5 and the Mintlify
+  verification-gates page said each rung (Stop, pre-commit, CI) was an independent catch
+  layer, so the silent-miss probability fell multiplicatively. The formal synthesis withdrew
+  that (§5.3, corrected 2026-09-21): the same classifier run on the same diff fires together,
+  so the residual is `(1−p)(1−c_max)`, and a later rung adds catches only where it sees what
+  the earlier one could not (edits after the turn, a host where the Stop hook never ran).
+  Comments and docs only; no behaviour change.
+
 ## [1.0.0] - 2026-09-22
 
 ### Added

README.md

@@ -232,9 +232,12 @@ from a fresh repository graph.
 - **Git-native team merge.** Claims and append-only logs merge by set union. The join is
   property-tested for commutativity, associativity, and idempotence.
 - **Heuristic impact prediction.** Forgekit builds a regex-derived code graph and walks
-  reverse dependencies to estimate affected files and tests. It is not conservative: it can
-  miss affected files (including constructs its parser does not recognize) as well as produce
-  false positives.
+  reverse dependencies to estimate affected files and tests; the pre-action check and the
+  Stop gate's repair checklist also walk the empirical refutation's sibling and forward
+  relations and tag each file with the relation that reached it. It is not conservative: it
+  can miss affected files (including constructs its parser does not recognize) as well as
+  produce false positives, and the sibling/forward files are lower-precision co-change
+  candidates.
 - **Budgeted context assembly.** Definitions, direct dependants, sibling tests, and trusted
   lessons are selected under a token budget. Missing required context becomes a question
   rather than invented context.
@@ -407,7 +410,7 @@ and output live in [`docs/GUIDE.md`](docs/GUIDE.md).
 |                         | `forge impact`       | hazard-aware blast radius — SCC-aware propagation + data-driven threshold from PageRank centrality and ledger incident history                                                                                              |
 |                         | `forge scope`        | decompose files into independent clusters (+ coupled files you didn't name)                                                                                                                                                 |
 |                         | `forge context`      | budgeted context assembly + completeness gate — what an edit NEEDS known                                                                                                                                                    |
-|                         | `forge route`        | recommend the cheapest capable model for a task (+ gateway config)                                                                                                                                                          |
+|                         | `forge route`        | recommend the cheapest capable model for a task (+ gateway config); `route universal`: any provider's models, lowest expected cost for the success asked for, learned from outcomes                                         |
 |                         | `forge verify`       | independent verification gate — tests + hallucinated-symbol + provenance (--deep: multi-lens consensus)                                                                                                                     |
 |                         | `forge precommit`    | commit-level gate — staged code w/o docs + secret scan (FORGE_COMMIT_GATE=block|warn|0)                                                                                                                                     |
 | **Memory**              | `forge cortex`       | self-correcting project memory — status / why <symbol>                                                                                                                                                                      |

bench/impact_cases.mjs

@@ -47,7 +47,7 @@
 //   - test/ledger.test.js imports { mergeStates } (:14) and calls it
 //   (src/ledger_sync.js:3 also names it in the module header — same file, already labeled.)
 //
-// claimText (src/ledger.js) — 8 files
+// claimText (src/ledger.js) — 9 files
 //   - src/ledger.js       defines it (:610); sketchOf() (:636), termsOf() (:637) and :880 call it
 //   - src/context.js      imports { claimText } (:13) and calls it (:185)
 //   - src/dash.js         imports { claimText } (:16) and calls it (:58, :389, :400)
@@ -56,6 +56,7 @@
 //   - src/cli.js          dynamic-imports { claimText } (:874, :1644) and calls it
 //   - src/cortex_mcp.js   dynamic-imports { claimText } (:91) and calls it (:96, :106)
 //   - test/ledger.test.js imports { claimText } (:8) and calls it
+//   - src/learn_consolidate.js imports { claimText } (:32) and calls it (:110)
 //   (test/dash.test.js:69 mentions the name only inside an assertion message — a string,
 //    not a reference — so it is NOT labeled as a dependent.)
 //
@@ -114,6 +115,7 @@ export const IMPACT_CASES = [
       "src/context.js",
       "src/dash.js",
       "src/deja.js",
+      "src/learn_consolidate.js",
       "src/ledger_store.js",
       "src/cli.js",
       "src/cortex_mcp.js",